A Transformer-based method for disparity estimation of light field epipolar geometric images

Abstract

The present invention discloses a method for disparity estimation of light field epipolar geometric images based on Transformer. The method designs an EpiFormer network structure with spatial consistency information preservation; the method extracts EPI image blocks from four directions of 0°, 45°, 90°, and 135°, and extracts three adjacent EPI image blocks in each direction, with a total of 12 EPI image blocks as input to the EpiFormer network structure, and outputs the disparity value of the center pixel. By splicing the disparity values ​​estimated for all pixels in the central sub-aperture of the light field, the corresponding light field disparity map is obtained. The present invention is based on the Transformer method, which effectively models the association between pixels in the EPI image block, while adding spatial information, alleviating the impact of occlusion, and also obtaining more accurate pixel matching.

Description

Technical Field

[0001] The present invention relates to light field imaging and computational applications in the field of computer vision, and in particular to a light field epipolar geometric image disparity estimation method based on a Transformer deep learning network. Background Art

[0002] Light field imaging and computation is an emerging application area in computer vision. It involves adding a microlens array between the camera's main lens and the image sensor to record the angle and radiation of light rays converging on the image sensor plane, forming a 4D light field image (two-dimensional spatial information and two-dimensional angular information). Because 4D light field images provide rich scene lighting information and depth cues, using light field images for scene depth estimation has garnered increasing attention. A light field epipolar image (EPI) is a two-dimensional slice of a 4D light field image, composed of a set of epipolar lines. By calculating the slopes of these lines, the disparity value of the current pixel can be obtained. After calculating the disparity values ​​of all pixels, a light field disparity map can be obtained. Although many EPI disparity estimation methods have been proposed, existing methods still face numerous challenges, such as learning and extracting light field geometric features, detecting and estimating disparity in occluded and textureless areas, and noise interference.

[0003] Deep learning is one of the most popular research directions in the field of artificial intelligence in the past two decades, and has achieved great success in computer vision, natural language processing, data mining, multimedia learning and other fields. It uses the powerful feature extraction and representation capabilities of convolutional neural networks to learn the inherent laws and representations of data from training samples. Introducing deep learning into the task of light field disparity estimation helps to solve various challenges in light field disparity estimation and effectively improves the estimation performance. At present, a variety of EPI disparity estimation methods based on convolutional neural networks have been proposed, but these methods are all based on convolutional neural networks and it is difficult to extract global geometric features in EPI. In addition, since EPI can only provide disparity clues for a single row or column of pixels, it is difficult for existing methods to ensure the spatial-angular consistency constraints of the light field during the disparity estimation process, which leads to the degradation of estimation performance.

[0004] With the introduction of Transformer technology in 2017, it has become a new paradigm in deep learning. A growing number of studies are applying the powerful modeling capabilities of Transformers to various computer vision tasks, achieving impressive performance. Therefore, this paper designs a Transformer network structure that preserves spatial consistency information to achieve EPI disparity estimation. Summary of the Invention

[0005] The purpose of the present invention is to address the shortcomings of the existing technology and provide a Transformer-based light field epipolar geometric image disparity estimation method. This method designs a Transformer deep learning network (EpiFormer) that maintains spatial consistency information. It extracts EPI image blocks from four directions: 0°, 45°, 90°, and 135°. Three adjacent EPI image blocks are extracted from each direction, for a total of 12 EPI image blocks. The input of EpiFormer is the disparity value of the central pixel. By splicing the disparity values ​​estimated for all pixels in the central sub-aperture of the light field, the corresponding light field disparity map can be obtained.

[0006] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0007] Step 1. Prepare the light field dataset and create training and test sets;

[0008] Step 2. Design and build the EpiFormer network structure;

[0009] Step 3. Train the EpiFormer model using the training set;

[0010] Step 4. Use the trained EpiFormer model to test on the test set;

[0011] Step 1 specifically includes the following steps:

[0012] Step 1-1: Use the 4D light field dataset provided by the Heidelberg Image Processing Laboratory as the experimental dataset (K. Honauer, O. Johannsen, D. Kondermann and B. Goldluecke, “A dataset and evaluation methodology for depth estimation on 4D light fields,” in Asian Conference on Computer Vision, 2016). This dataset contains 28 light field images and provides high-precision disparity maps and performance evaluation metrics. Each light field image consists of 81 512×512 sub-aperture images, arranged in 9 rows and 9 columns. The sub-aperture image in the 5th row and 5th column is the center sub-aperture image. Of the 28 light field images, 16 are training images and the remaining 12 are test images.

[0013] Step 1-2: Use the EPI image patch extraction method in the SOA-EPN method (W.Zhou, L.Liang, H.Zhang, A.Lumsdaine and L.Lin, "Scale and Orientation Aware EPI-Patch Learning for Light Field Depth Estimation," 24th International Conference on Pattern Recognition (ICPR), Beijing, China, 2018) to extract EPI image patches of size 9×21 for each pixel in the central sub-aperture image at four directions: 0°, 45°, 90°, and 135°.

[0014] The light field image is defined as LF(s,t,x,y), where (x,y) is the spatial coordinate of the light field image and (s,t) is the angular coordinate of the light field image. For the pixel point P(x,y) on the central sub-aperture image, the EPI image blocks of the pixel point P at 0°, 45°, 90°, and 135° are respectively recorded as EPI0(x,y), EPI 45 (x,y),EPI 90 (x,y),EPI 135 (x,y).

[0015] Step 1-3: The EPI image block of each pixel in the central sub-aperture image and the EPI image blocks of its neighboring pixels form an EPIS sample. All EPIS samples from the training light field images constitute the training set, and all EPIS samples from the test light field images constitute the test set.

[0016] For the pixel point P(x,y) on the central sub-aperture image of the light field image LF(s,t,x,y), the three EPI image blocks EPI0(x,y), EPI0(x,y+1) and EPI0(x,y-1) are combined into the horizontal sample EPIS0(x,y)={EPI0(x,y),EPI0(x,y+1),EPI0(x,y-1)} of the pixel point P(x,y); 90 (x,y),EPI 90 (x+1,y) and EPI 90 (x-1,y) constitutes the vertical sample EPIS of the pixel point P(x,y). 90 (x,y)={EPI 90 (x,y),EPI 90 (x+1,y),EPI 90 (x-1,y)}; change EPI 45 (x,y),EPI45 (x+1,y+1) and EPI 45 (x-1, y-1) forms the 45-degree sample EPIS of the pixel point P(x, y) 45 (x,y)={EPI 45 (x,y),EPI 45 (x+1,y+1),EPI 45 (x-1,y-1)}; change EPI 135 (x,y),EPI 135 (x+1,y-1) and EPI 135 (x-1, y+1) forms the 135-degree sample EPIS of the pixel point P(x, y) 135 (x,y)={EPI 135 (x,y),EPI 135 (x+1,y-1),EPI 135 (x-1,y+1)}, by {EPIS0(x,y), EPIS 45 (x,y), EPIS 90 (x,y), EPIS 135 (x,y)} constitutes the EPIS sample of pixel point P(x,y).

[0017] The step 2 is specifically implemented as follows:

[0018] Step 2-1: Build the EpiFormer network structure.

[0019] The EpiFormer network structure is composed of four spatial matching modules (Spatial Matching Module abbreviated as: SMM) and one multi-channel disparity merging module. The four spatial matching modules are 0° spatial matching module, 45° spatial matching module, 90° spatial matching module and 135° spatial matching module. For the EPIS sample of pixel point P(x,y), the input of the 0° spatial matching module is EPIS0(x,y) and the output disparity prediction vector logit0; the input of the 90° spatial matching module is EPIS 90 (x,y), output disparity prediction vector logit1; the input of the 45° spatial matching module is EPIS 45 (x,y), the output is the disparity prediction vector logit2; the input of the 135° spatial matching module is EPIS 135 (x, y), the output is the disparity prediction vector logit3, the dimensions of the disparity prediction vectors output in four directions are all 1×229 (since the disparity range of the dataset is -4 to 4 (unit: pixel), we define disparity prediction as a classification task with a prediction accuracy of 0.035 pixels, and a total of 229 categories).

[0020] The multi-channel disparity merging module stacks the outputs logit0, logit1, logit2, and logit3 of the four spatial matching modules on the channel to form a 1×916 feature vector. The predicted disparity vectors are then fused through a fully connected layer with 229 neurons to obtain the final disparity vector logit4.

[0021] The spatial matching module is composed of a depth feature representation module, a spatial feature fusion module, a disparity prediction module, and a multi-channel disparity merging module, as follows:

[0022] (1) Deep feature representation module

[0023] The deep feature representation module contains 8 convolutional layers, namely convolutional layer 1 to convolutional layer 8. 8 convolutional layers with a convolution kernel of 2×2 and a stride of 1×1 are used to extract deep features of the input EPI image block. The output of each convolutional layer is batch normalized, and then the batch normalized output is activated using the ReLU activation function; the output feature map channels of each convolutional layer in the 8 convolutional layers are (16, 32, 64, 128, 256, 384, 512, 512) in sequence, and the tensor size of the output of the last convolutional layer is 3×512×1×13 (EPIS feature×feature channel×height×width), that is, 3 adjacent EPI deep features in space, each consisting of 512 feature maps of size 1×13.

[0024] (2) Spatial feature fusion module

[0025] The structure of the spatial feature fusion module is as follows: Figure 1 As shown, the spatial feature fusion module encodes the output of the deep feature representation module, flattens it, and then passes it into the Transformer module. The pixels are encoded as image blocks (equivalent to performing a 1×1 convolution). The resulting features are N×3×1×13, where N is the encoded feature channel (default setting is 768), 3 is the number of adjacent EPI features, and 1×13 is the feature resolution after convolution. Similar to the traditional Transformer approach, a class token is set as the pixel depth at the center of the final EPI image to be predicted. The features are first reconstructed to N×39, then the class token is added to obtain an N×40 feature dimension. The positional encoding (learnable randomly initialized encoding information) is then added to obtain the Transformer input. The feature portion corresponding to the class token ([N, 0]) is denoted as Q, and the feature portion after the class token ([N, 1:]) is denoted as K. To calculate the similarity between pixels in the EPI image block, the features first pass through the self-attention module and the cross-attention module, as shown in the following formula:

[0026]

[0027] in is the root of the dimensions of the Q and K eigenvectors.

[0028] The feedforward module consists of two fully connected components, and the activation function is the GeLU activation function. The formula is as follows:

[0029] FFN(Z)=GeLU(GeLU(ZW1+b1)W2+b2)

[0030] Figure 3 The gray box in the figure represents a submodule, which includes the self-attention module, regularization, cross-attention module, regularization, feedforward module, and regularization. The entire spatial feature fusion module is composed of several submodules. By default, 8 submodules are used.

[0031] (3) Disparity Prediction Module

[0032] We use the disparity classification method in SOA-EPN (W.Zhou, L.Liang, H.Zhang, A.Lumsdaine and L.Lin, "Scale and Orientation Aware EPI-Patch Learning for Light Field Depth Estimation," 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China, 2018) to perform disparity classification on the output of the spatial matching module and obtain the disparity estimation results corresponding to the spatial matching module in four directions.

[0033] (4) Multi-channel parallax merging module

[0034] The input of the multi-channel disparity merging module is the output of the disparity prediction module, that is, the output of the four spatial matching modules logit0, logit1, logit2, and logit3, all of which have a dimension of 1×229. They are stacked together in the channel dimension to form a fused feature with a dimension of 1×916. The predicted feature vectors are then fused through a fully connected layer with 229 neurons to obtain the final disparity estimation vector logit4.

[0035] Compared with the prior art, the main contributions of the present invention are:

[0036] (1) Light field epipolar geometric image block shared weight network. It shares parameters in the spatial matching module to achieve multi-directional feature fusion.

[0037] (2) Spatial feature fusion module, based on the Transformer method, can effectively model the relationship between pixels in the EPI image block, while adding spatial information, which can alleviate the impact of occlusion to a certain extent and obtain more accurate pixel matching.

[0038] (3) A light field epipolar geometry image learning model for depth estimation. To achieve directional and spatial perception, it contains the above two network structures and takes light field epipolar geometry image patches as input. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 :Multi-directional EPI patch interception effect diagram.

[0040] Figure 2 : EpiFormer’s network structure diagram.

[0041] Figure 3 : Spatial matching module SMM structure diagram. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0043] like Figure 1-3 As shown in the figure, a Transformer deep learning network (EpiFormer) with spatial consistency information is designed. It extracts EPI image patches from four directions: 0°, 45°, 90°, and 135°. In each direction, three adjacent EPI image patches are extracted, for a total of 12 EPI image patches. The input of EpiFormer is the disparity value of the central pixel. By splicing the disparity values ​​estimated for all pixels in the central sub-aperture of the light field, the corresponding light field disparity map can be obtained.

[0044] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0045] Step 1. Prepare the light field dataset and create training and test sets;

[0046] Step 2. Design and build the EpiFormer network structure;

[0047] Step 3. Train the EpiFormer model using the training set;

[0048] Step 4. Use the trained EpiFormer model to test on the test set;

[0049] Step 1 specifically includes the following steps:

[0050] Step 1-1: Use the 4D light field dataset provided by the Heidelberg Image Processing Laboratory as the experimental dataset (K. Honauer, O. Johannsen, D. Kondermann and B. Goldluecke, “A dataset and evaluation methodology for depth estimation on 4D light fields,” in Asian Conference on Computer Vision, 2016). This dataset contains 28 light field images and provides high-precision disparity maps and performance evaluation metrics. Each light field image consists of 81 512×512 sub-aperture images, arranged in 9 rows and 9 columns. The sub-aperture image in the 5th row and 5th column is the center sub-aperture image. Of the 28 light field images, 16 are training images and the remaining 12 are test images.

[0051] Step 1-2: Use the EPI image patch extraction method in the SOA-EPN method (W.Zhou, L.Liang, H.Zhang, A.Lumsdaine and L.Lin, "Scale and Orientation Aware EPI-Patch Learning for Light Field Depth Estimation," 24th International Conference on Pattern Recognition (ICPR), Beijing, China, 2018) to extract EPI image patches of size 9×21 for each pixel in the central sub-aperture image at four directions: 0°, 45°, 90°, and 135°.

[0052] The light field image is defined as LF(s,t,x,y), where (x,y) is the spatial coordinate of the light field image and (s,t) is the angular coordinate of the light field image. For the pixel point P(x,y) on the central sub-aperture image, the EPI image blocks of the pixel point P at 0°, 45°, 90°, and 135° are respectively recorded as EPI0(x,y), EPI 45 (x,y),EPI 90 (x,y),EPI 135 (x,y).

[0053] Step 1-3: The EPI image block of each pixel in the central sub-aperture image and the EPI image blocks of its neighboring pixels form an EPIS sample. All EPIS samples from the training light field images constitute the training set, and all EPIS samples from the test light field images constitute the test set.

[0054] For the pixel point P(x,y) on the central sub-aperture image of the light field image LF(s,t,x,y), the three EPI image blocks EPI0(x,y), EPI0(x,y+1) and EPI0(x,y-1) are combined into the horizontal sample EPIS0(x,y)={EPI0(x,y),EPI0(x,y+1),EPI0(x,y-1)} of the pixel point P(x,y); 90 (x,y),EPI 90 (x+1,y) and EPI 90 (x-1,y) constitutes the vertical sample EPIS of the pixel point P(x,y). 90 (x,y)={EPI 90 (x,y),EPI 90 (x+1,y),EPI 90 (x-1,y)}; change EPI 45 (x,y),EPI 45 (x+1,y+1) and EPI 45 (x-1, y-1) forms the 45-degree sample EPIS of the pixel point P(x, y) 45 (x,y)={EPI 45 (x,y),EPI 45 (x+1,y+1),EPI 45 (x-1,y-1)}; change EPI 135 (x,y),EPI 135 (x+1,y-1) and EPI 135 (x-1, y+1) forms the 135-degree sample EPIS of the pixel point P(x, y) 135 (x,y)={EPI 135 (x,y),EPI 135 (x+1,y-1),EPI 135 (x-1,y+1)}, by {EPIS0(x,y), EPIS 45 (x,y), EPIS 90 (x,y), EPIS 135 (x,y)} constitutes the EPIS sample of pixel point P(x,y).

[0055] The step 2 is specifically implemented as follows:

[0056] Step 2-1: Build the EpiFormer network structure.

[0057] like Figure 2As shown in the figure, the EpiFormer consists of four spatial matching modules (Spatial Matching Module abbreviated as: SMM) and one multi-channel disparity merging module. The four spatial matching modules are 0° spatial matching module, 45° spatial matching module, 90° spatial matching module and 135° spatial matching module. In order to reduce the number of parameters, these four spatial matching modules use weight sharing to perform forward and reverse calculations. For the EPIS sample of the pixel point P(x,y), the input of the 0° spatial matching module is EPIS0(x,y), and the output disparity prediction vector logit0; the input of the 90° spatial matching module is EPIS 90 (x,y), output disparity prediction vector logit1; the input of the 45° spatial matching module is EPIS 45 (x,y), the output is the disparity prediction vector logit2; the input of the 135° spatial matching module is EPIS 135 (x, y), the output is the disparity prediction vector logit3, the dimensions of the disparity prediction vectors output in four directions are all 1×229 (since the disparity range of the dataset is -4 to 4 (unit: pixel), we define disparity prediction as a classification task with a prediction accuracy of 0.035 pixels, and a total of 229 categories).

[0058] The multi-channel disparity merging module stacks the outputs of the four spatial matching modules on the channel to form a 1×916 feature vector, and then fuses the predicted disparity vectors through a fully connected layer with 229 neurons to obtain the final disparity vector logit4.

[0059] The spatial matching module is composed of a depth feature module, a spatial feature fusion module and a disparity prediction module. For details, see Table 1:

[0060]

[0061]

[0062] Table 1

[0063] The four modules are as follows:

[0064] (1) Deep feature representation module

[0065] The deep feature representation module contains 8 convolutional layers, namely convolutional layer 1 to convolutional layer 8, which use 8 convolutional layers with a convolution kernel of 2×2 and a stride of 1×1 to extract deep features of the input EPI image block, batch normalize the output of each convolutional layer, and then use the ReLU activation function to activate the batch normalized output; the output feature map channels of each convolutional layer in the 8 convolutional layers are (16, 32, 64, 128, 256, 384, 512, 512) in sequence, and the tensor output by the last convolutional layer is denoted as x0, whose size is 3×512×1×13 (EPIS feature×feature channel×height×width), that is, 3 adjacent EPI deep features in space, each consisting of 512 feature maps of size 1×13.

[0066] (2) Spatial feature fusion module

[0067] The structure of the spatial feature fusion module is as follows: Figure 1 As shown, the spatial feature fusion module encodes the output of the deep feature representation module, flattens it, and then passes it into the Transformer module. The pixels are encoded as image blocks (equivalent to performing a 1×1 convolution). The resulting features are N×3×1×13, where N is the encoded feature channel (default setting is 768), 3 is the number of adjacent EPI features, and 1×13 is the feature resolution after convolution. Similar to the traditional Transformer approach, a class token is set as the pixel depth at the center of the final EPI image to be predicted. The features are first reconstructed to N×39, then the class token is added to obtain an N×40 feature dimension. The positional encoding (learnable randomly initialized encoding information) is then added to obtain the Transformer input. The feature portion corresponding to the class token ([N, 0]) is denoted as Q, and the feature portion after the class token ([N, 1:]) is denoted as K. To calculate the similarity between pixels in the EPI image block, the features first pass through the self-attention module and the cross-attention module, as shown in the following formula:

[0068]

[0069] in is the root of the Q and K dimensions.

[0070] The feedforward module consists of two fully connected components, and the activation function is the GeLU activation function. The formula is as follows:

[0071] FFN(Z)=GeLU(GeLU(ZW1+b1)W2+b2)

[0072] Figure 3The gray box in the figure represents a submodule, which includes the self-attention module, regularization, cross-attention module, regularization, feedforward module, and regularization. The entire spatial feature fusion module is composed of several submodules. By default, 8 submodules are used.

[0073] (3) Disparity Prediction Module

[0074] We use the disparity classification method in SOA-EPN (W.Zhou, L.Liang, H.Zhang, B.Lumsdaineand L.Lin, "Scale and Orientation Aware EPI-Patch Learning for Light Field Depth Estimation," 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China, 2018)

[0075] Perform disparity classification on the output of the spatial matching module to obtain the disparity estimation results corresponding to the spatial matching module in four directions.

[0076] (4) Multi-channel parallax merging module

[0077] The input of the multi-channel disparity merging module is the output of the disparity prediction module, that is, the output of the four spatial matching modules logit0, logit1, logit2, logit3, the dimensions of which are all 1×229.

[0078] The channels are stacked together to form a fused feature with a dimension of 1×916. The predicted feature vectors are fused through a fully connected layer with 229 neurons to obtain the final disparity estimation vector logit4.

[0079] Step 4: Use the trained EpiFormer model to test on the test set. We use badpixel (0.07) as the evaluation indicator of the test data. The calculation formula of badpixel (0.07) is as follows:

[0080] abs(gt-pred)>0.07

[0081] Where gt is the true disparity value of each pixel in the sub-aperture image at the center of the light field, and pred is the estimated pixel disparity value. Pixels with an absolute difference greater than 0.07 are called bad pixels. Based on the above formula, the smaller the badpixel value, the better. We selected Dots light field data and Sideboard light field data, two scenes that are more difficult to predict based on the EPI method, for comparison. The resulting indicators are shown in Table 2:

[0082] Table 2

[0083]

[0084] In summary, EpiFormer significantly improves the badpixel rate (0.07) in both Dots and Sideboard scenarios. Compared to the SOA-EPN approach, the badpixel rate (0.07) in Dots is improved by 45%, and in Sideboard by 60%. Its advantages are even more pronounced compared to EPI_Shift, EPI_ORM, and EPI1_OCC.

Claims

1. A Transformer-based light field epipolar geometric image disparity estimation method, characterized by This method designs an EpiFormer network structure that maintains spatial consistency information. The method extracts EPI image blocks from four directions: 0°, 45°, 90°, and 135°. Three adjacent EPI image blocks are extracted from each direction, for a total of 12 EPI image blocks. The output is the disparity value of the central pixel. The corresponding light field disparity map is obtained by splicing the disparity values ​​estimated for all pixels in the central sub-aperture of the light field. The method includes the following steps: Step 1. Prepare the light field dataset and create training and test sets; Step 2. Design and build the EpiFormer network structure. The specific implementation is as follows: The EpiFormer network structure consists of four spatial matching modules and one multi-channel disparity merging module; the four spatial matching modules are 0° spatial matching module, 45° spatial matching module, 90° spatial matching module and 135° spatial matching module; for the EPIS sample of pixel point P(x,y), the input of the 0° spatial matching module is EPIS0(x,y), and the output disparity prediction vector logit0; the input of the 90° spatial matching module is EPIS 90 (x,y), output disparity prediction vector logit1; the input of the 45° spatial matching module is EPIS 45 (x,y), the output is the disparity prediction vector logit2; the input of the 135° spatial matching module is EPIS 135 (x, y), the output is the disparity prediction vector logit3, the dimensions of the disparity prediction vectors output in all four directions are 1×229; The multi-channel disparity merging module stacks the outputs logit0, logit1, logit2, and logit3 of the four spatial matching modules on the channel to form a 1×916 feature vector, and then fuses the predicted disparity vectors through a fully connected layer with 229 neurons to obtain the final disparity vector logit4; The spatial matching module is composed of a depth feature representation module, a spatial feature fusion module, a disparity prediction module and a multi-channel disparity merging module; Step 3. Train the EpiFormer model using the training set; Step 4. Use the trained EpiFormer model to test on the test set.

2. The method for estimating disparity of light field epipolar geometric images based on Transformer according to claim 1, characterized in that Step 1 specifically includes the following steps: Step 1-1: Use the 4D light field dataset provided by the Heidelberg Image Processing Laboratory as the experimental dataset; Step 1-2: Use the EPI image block extraction method in the SOA-EPN method to extract EPI image blocks of size 9×21 for each pixel in the central sub-aperture image at four directions: 0°, 45°, 90°, and 135°. The light field image is defined as LF(s,t,x,y), where (x,y) is the spatial coordinate of the light field image and (s,t) is the angular coordinate of the light field image. For the pixel point P(x,y) on the central sub-aperture image, the EPI image blocks of the pixel point P at 0°, 45°, 90° and 135° are respectively recorded as EPI0(x,y), EPI 45 (x,y),EPI 90 (x,y),EPI 135 (x,y); Step 1-3: The EPI image block of each pixel in the central sub-aperture image and the EPI image blocks of its neighboring pixels form an EPIS sample; all EPIS samples from the training light field images constitute the training set, and all EPIS samples from the test light field images constitute the test set; For the pixel point P(x,y) on the central sub-aperture image of the light field image LF(s,t,x,y), the three EPI image blocks EPI0(x,y), EPI0(x,y+1) and EPI0(x,y-1) are combined into the horizontal sample EPIS0(x,y)={EPI0(x,y),EPI0(x,y+1),EPI0(x,y-1)} of the pixel point P(x,y); 90 (x,y),EPI 90 (x+1,y) and EPI 90 (x-1,y) constitutes the vertical sample EPIS of the pixel point P(x,y). 90 (x,y)={EPI 90 (x,y),EPI 90 (x+1,y),EPI 90 (x-1,y)}; change EPI 45 (x,y),EPI 45 (x+1,y+1) and EPI 45 (x-1, y-1) forms the 45-degree sample EPIS of the pixel point P(x, y) 45 (x,y)={EPI 45 (x,y),EPI 45 (x+1,y+1),EPI 45 (x-1,y-1)}; change EPI 135 (x,y),EPI 135 (x+1,y-1) and EPI 135 (x-1, y+1) forms the 135-degree sample EPIS of the pixel point P(x, y) 135 (x,y)={EPI 135 (x,y),EPI 135 (x+1,y-1),EPI 135 (x-1,y+1)}, by {EPIS0(x,y), EPIS 45 (x,y), EPIS 90 (x,y), EPIS 135 (x,y)} constitutes the EPIS sample of pixel point P(x,y).

3. The method for estimating disparity of light field epipolar geometric images based on Transformer according to claim 2, characterized in that The depth feature characterization module specifically includes: The deep feature representation module contains 8 convolutional layers, namely convolutional layer 1 to convolutional layer 8. Eight convolutional layers with a convolution kernel of 2×2 and a stride of 1×1 are used to extract deep features of the input EPI image blocks. The output of each convolutional layer is batch normalized, and then the batch normalized output is activated using the ReLU activation function; the output feature map channels of each convolutional layer in the 8 convolutional layers are (16, 32, 64, 128, 256, 384, 512, 512) in sequence, and the tensor size of the output of the last convolutional layer is 3×512×1×13, that is, 3 adjacent EPI deep features in space, each consisting of 512 feature maps of size 1×13.

4. The method for estimating disparity of light field epipolar geometric images based on Transformer according to claim 3, characterized in that The spatial feature fusion module specifically includes: The spatial feature fusion module encodes the output of the deep feature representation module, flattens it, and then passes it into the Transformer module. The pixels are encoded as image blocks, and the obtained features are N×3×1×13, where N is the encoded feature channel, which is set to 768 by default, 3 is the number of adjacent EPI features, and 1×13 is the feature resolution after convolution. A class token is set as the pixel depth of the center of the EPI image to be predicted. The feature is first reconstructed to N×39, and then the class token is added to obtain an N×40 feature dimension. The position encoding is added to obtain the input of the Transformer. The feature part corresponding to the class token ([N,0]) is recorded as Q, and the feature part after the class token ([N,1:]) is recorded as K. Since the similarity between pixels in the EPI image block needs to be calculated, the feature must first pass through the self-attention module and the cross-attention module. The formula is as follows: in is the root of the dimension of the Q and K eigenvectors; The feedforward module consists of two fully connected components, and the activation function is the GeLU activation function. The formula is as follows: FFN(Z)=GeLU(GeLU(ZW1+b1)W2+b2) The entire spatial feature fusion module is composed of several sub-modules, each of which includes a self-attention module, regularization, a cross-attention module, regularization, a feedforward module and regularization.

5. The method for estimating disparity of light field epipolar geometric images based on Transformer according to claim 4, characterized in that The disparity prediction module specifically includes: The disparity classification method in SOA-EPN is used to perform disparity classification on the output of the spatial matching module, and the disparity estimation results corresponding to the spatial matching module in four directions are obtained.

6. The method for estimating disparity of light field epipolar geometric images based on Transformer according to claim 5, characterized in that The multi-channel parallax merging module specifically includes: The input of the multi-channel disparity merging module is the output of the disparity prediction module, that is, the 4 spatial matching The module's outputs, logit0, logit1, logit2, and logit3, all have dimensions of 1×229. The four outputs are stacked together in the channel dimension to form a fused feature with a dimension of 1×916. The predicted feature vectors are then fused through a fully connected layer with 229 neurons to obtain the final disparity estimate vector, logit4.

Images