Panoramic optical flow estimation method based on tangent plane processing and global features

Abstract

The invention discloses a panoramic optical flow estimation method based on section processing and global features, which solves the problem of two-pole distortion of a panoramic image by using a series of distortionless section images and improves the precision of optical flow estimation of the panoramic image. And learning undistorted features after image coding by utilizing the capability of capturing a long-distance dependency relationship between pixels of Transform, and capturing corresponding features with the highest similarity for matching to generate an optical flow field.

Description

Technical Field

[0001] The present invention relates to the technical field of panoramic image three-dimensional reconstruction, and in particular to a panoramic optical flow estimation method based on section processing and global features. Background Art

[0002] Optical flow estimation is a computer vision task used to track pixel motion in an image, provide 3D information about objects, identify and remove noise or distortion in the image, and improve the quality and stability of 3D reconstruction. Optical flow information from panoramic images is widely used in 3D reconstruction, virtual reality, autonomous driving, and augmented reality.

[0003] Traditional optical flow estimation methods usually rely on architectures based on convolutional neural networks (CNNs). These methods may have some limitations when dealing with complex motions in scenes, and methods for estimating optical flow on perspective images cannot adapt to the omnidirectional nature of panoramic images. In contrast, the present invention proposes a new panoramic optical flow estimation method that uses the attention mechanism of the Transformer model to more effectively capture the dependencies between long-distance pixels, solve the problem of large displacement, and thus improve the accuracy and robustness of optical flow estimation. However, panoramic images are different from ordinary images. There will be varying degrees of distortion and distortion in panoramic images, and their rich visual information is difficult to fully utilize during optical flow estimation. The present invention uses spherical center projection to obtain a series of distortion-free cross-sectional images and input them into the Transformer, alleviating the distortion problem of panoramic images.

[0004] The information disclosed in this background technology section is only intended to enhance understanding of the overall background of the invention and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the above technical defects and provide a panoramic optical flow estimation method based on section processing and global features. A series of undistorted section images are used to solve the problem of bipolar distortion of panoramic images and improve the accuracy of optical flow estimation of panoramic images. The Transformer's ability to capture long-distance dependencies between pixels is utilized to learn the undistorted features after image encoding, and the corresponding features with the highest similarity are captured for matching to generate an optical flow field.

[0006] In order to solve the above problems, the technical solution of the present invention is a panoramic optical flow estimation method based on section processing and global features, which includes the following steps:

[0007] Step 1: Load 5000 training images containing four weather conditions: sunny, cloudy, foggy, and rainy by calling the Dataloader function. Pass the panoramic image, loop through the data in the 'train' folder, and load the equirectangular projection dataset Flowscape into the neural network.

[0008] Step 2: After entering the neural network, the input tensor [B, C, H, W] is trained through the encoder;

[0009] Step 3. Place 1400 test images containing four weather conditions: sunny, cloudy, foggy, and rainy in a folder named 'test' and test the test data with the trained model.

[0010] Step 4: In the dimension of B, use torch.cat to concatenate the upper and lower frames to obtain a tensor of size [4, 3, 512, 1024]. After isometric transformation, it is the operation of the perspective image function.

[0011] Step 5: The difference between the positions of different image regions between the spherical image and the equirectangular image is re-projected back to the spherical image, and the position information of different regions is calculated using the spherical image to obtain a tensor of size [1, 64, 32, 32, 18].

[0012] Step 6. The first convolution operation is Ks (7, 7, 1), St (2, 2, 1) and Pd (3, 3, 0), which is the BatchNorm and ReLU activation function. After the first encoding layer, the sliced tensor [4, 3, 128, 128, 18] is encoded as [4, 64, 64, 64, 18].

[0013] Step 7: Use the BatchNorm function to perform the first batch normalization on the tensor obtained from the first convolution operation of the isometric projection image. This makes the input distribution of each neuron more stable, allowing for faster convergence during training. It also improves the generalization ability of the model and suppresses overfitting, thereby improving the accuracy and reliability of the deep learning model.

[0014] Step 8: Use the ReLU activation function to perform nonlinear fitting on the tensor obtained by the first convolution operation of the cross-section image;

[0015] Step 9. Perform the maximum pooling operation, Ks, St, and Pd are (3, 3, 1), (2, 2, 1), and (1, 1, 0) respectively, and the resulting tensor is [4, 64, 32, 32, 18]. The first encoding layer encodes the contextual relationship between the 18 patch blocks. The first encoding layer contains three blocks. The first block contains a Conv3d convolution operation, whose Ks, St, and Pd are (3, 3, 1), (1, 1, 1), and (1, 1, 0) respectively. After the BatchNorm and ReLU activation functions, it undergoes another convolution operation. The Ks, St, and Pd of this convolution operation are (3, 3, 1), (1, 1, 1), and (1, 1, 0) and BatchNorm operation respectively.

[0016] Step 10. Use the BatchNorm function to perform the first batch normalization on the tensor obtained by the first convolution operation of the equirectangular projection image, and use the ReLU activation function for nonlinear fitting; the second convolution block, the second block contains a Conv3d convolution operation, whose Ks, St and Pd are (3, 3, 1), (1, 1, 1) and (1, 1, 0) respectively. After the BatchNorm and ReLU activation functions, it undergoes another convolution operation. The Ks, St and Pd of this convolution operation are (3, 3, 1), (1, 1, 1) and (1, 1, 0) and BatchNorm operation respectively;

[0017] Step 11: Use the BatchNorm function to perform the first batch normalization on the tensor obtained by the first convolution operation of the equirectangular projection image, and use the ReLU activation function for nonlinear fitting.

[0018] Step 12, the third encoding block, the third block contains a Conv3d convolution operation, its Ks, St and Pd are (3, 3, 1), (1, 1, 1) and (1, 1, 0) respectively. After the BatchNorm and ReLU activation functions, it undergoes another convolution operation. The Ks, St and Pd of this convolution operation are (3, 3, 1), (1, 1, 1) and (1, 1, 0) and BatchNorm operation respectively.

[0019] Step 13: Use the BatchNorm function to perform the first batch normalization on the tensor obtained from the first convolution operation of the equirectangular projection image, and use the ReLU activation function for nonlinear fitting; after encoding through the first encoding layer, a tensor of size [4, 64, 32, 32, 18] is obtained. The position information of the stereo embedding module (SE) is embedded into the encoding layer result of the first layer to obtain a tensor of size [4, 64, 32, 32, 18];

[0020] Step 14: The second encoding layer. The second layer contains three blocks. Each block undergoes a convolution operation, a BatchNorm operation, and a ReLU activation function operation. After encoding in the second encoding layer, a tensor of size [4, 128, 16, 16, 18] is obtained.

[0021] Step 15. The third encoding layer contains 6 convolution blocks. Each block undergoes a convolution operation, a BatchNorm operation, and a ReLU activation function operation. After passing through the third encoding layer, a tensor of size [4, 256, 8, 8, 18] is obtained.

[0022] Step 16. After reshape deformation, the tensor is [4, 36, 64, 128]. After a convolution operation with Ks and St as (1, 1, 1) and (1, 1, 1), the tensor is [4, 256, 64, 128].

[0023] Step 17. Use torch.split to separate the tensor [4, 256, 64, 128] into feature_list1 and feature_list2, whose tensors are [2, 256, 64, 128] and [2, 256, 64, 128] respectively.

[0024] Step 18. After the encoding layer, the tensors of feature_list1 and feature_list2 are [2, 256, 64, 128]. Entering the transformer module, first expand the feature_list1 and feature_list2 tensors to [B, H*W, C], and concatenate feature_list1 and feature_list2 in the dimension of B to obtain [2B, H*W, C], that is, the tensor is [4, 8192, 256].

[0025] Step 19: The Q, K, and V matrices of self-attention are obtained by passing image1 through the Linear fully connected layer. The dimensions of Q, K, and V are all [4, 8192, 256]. Calculate the transpose multiplication of the Q matrix and the K matrix, and obtain the attention score through torch.softmax. The attention score is multiplied by the V matrix to obtain the self-attention enhanced tensor. The tensor obtained after self-attention is [4, 8192, 256].

[0026] Step 20: Use the output of self-attention as the input of cross-attention. Cross-attention has two inputs, one from self-attention and the other is the initial tensor obtained by concatenating feature_list1 and feature_list2 in the dimension of B. The Q matrix is calculated by the output of self-attention, and the K and V matrices are calculated using the tensor obtained by concatenating feature_list1 and feature_list2. The Q, K and V matrices are all calculated through the Linear layer. The transpose of the Q and K matrices is multiplied, and the attention score is obtained by torch.softmax and multiplied with the V matrix to obtain the cross-attention enhanced tensor.

[0027] Step 21: MLP layer, including Linear layer, ReLU and Linear, and finally a Layernorm normalization, which can effectively alleviate the problem of network convergence difficulties. After corss-attention, the tensor is obtained as [4,8192,256], and then the size is restored to the input size, that is, [2,256,64,128] of fmap1 and fmap2.

[0028] Step 22: In the cnet module, use the initial image1 to learn the upper and lower relationships of the image areas, with image1 as input, the tensor is [2, 3, 512, 1024], the input tensor [2, 3, 512, 1024], and the isometric conversion to perspective image function. The parameter settings are as follows: erp_img represents the isometric image, fov represents the field of view, nrows represents the number of rows, patch_size represents the size of the cross-section image, the fov field of view is 80°, nrows is 18, and the patch_size cross-section image size is 128×128. After this function, image1 and image2 and two batches of pictures are returned, namely, four tensors perspective image (pers), xyz coordinates (xyz) in the perspective image, uv coordinates (uv) in the spherical coordinate system, and center point coordinates (center_p);

[0029] Step 23: Cut the image into 18 blocks of [128, 128]. That is, the tensor after cutting is [2, 3, 128, 128, 18]. Then, send the tensor to the encoding layer.

[0030] Step 24: Calculate the difference between the positions of different image regions between the spherical image and the equirectangular image, reproject the equirectangular image back into the spherical image, and use the spherical image to calculate the position information of different regions to obtain a tensor of size [1, 64, 32, 32, 18].

[0031] Step 25: The first convolution operation is with Ks (7, 7, 1), St (2, 2, 1), and Pd (3, 3, 0). It uses BatchNorm and ReLU activation functions. After the first encoding layer, the tensor becomes [2, 64, 64, 64, 18]. The BatchNorm function is used to perform the first batch normalization on the tensor obtained by the first convolution operation of the isometric projection image. The ReLU activation function is used to perform the first nonlinear fitting on the tensor obtained by the first convolution operation of the cross-section image.

[0032] Step 26, the maximum pooling operation, Ks, St and Pd are (3, 3, 1), (2, 2, 1) and (1, 1, 0) respectively. The tensor obtained is [2, 64, 32, 32, 18]. The first encoding layer encodes the contextual relationship between 18 patch blocks. The first encoding layer contains three blocks. The first block contains a Conv3d convolution operation, whose Ks, St and Pd are (3, 3, 1), (1, 1, 1) and (1, 1, 0) respectively. After BatchNorm and ReLU activation functions, it undergoes another convolution operation. The Ks, St and Pd of this convolution operation are Pd is (3, 3, 1), (1, 1, 1) and (1, 1, 0) respectively and BatchNorm operation, the BatchNorm function is used to perform the first batch normalization on the tensor obtained by the first convolution operation of the isometric projection image, and the ReLU activation function is used to perform nonlinear fitting on the tensor obtained by the first convolution operation of the cross-section image; the second convolution block, the second block contains a Conv3d convolution operation, its Ks, St and Pd are (3, 3, 1), (1, 1, 1) and (1, 1, 0) respectively, after BatchNorm and ReLU activation function, and then After a convolution operation, the Ks, St and Pd of this convolution operation are (3, 3, 1), (1, 1, 1) and (1, 1, 0) respectively, and the BatchNorm operation is used. The BatchNorm function is used to perform the first batch normalization on the tensor obtained by the first convolution operation of the equirectangular projection image. The ReLU activation function is used to perform nonlinear fitting on the tensor obtained by the first convolution operation of the cross-section image. The third encoding block contains a Conv3d convolution operation, whose Ks, St and Pd are (3, 3, 1), (1, 1, 1) and (1, 1, 0) respectively. , after BatchNorm and ReLU activation functions, it undergoes another convolution operation. The Ks, St, and Pd of this convolution operation are (3, 3, 1), (1, 1, 1), and (1, 1, 0), respectively. The BatchNorm operation uses the BatchNorm function to perform the first batch normalization processing on the tensor obtained by the first convolution operation of the isometric projection image, making the input distribution of each neuron more stable, which can converge faster during the training process. At the same time, it can also improve the generalization ability of the model and suppress overfitting, thereby improving the accuracy and reliability of the deep learning model;

[0033] Step 27: Use the ReLU activation function to perform the first nonlinear fitting on the tensor obtained by the first convolution operation of the cross-section image. After encoding by the first coding layer, a tensor of size [2, 64, 32, 32, 18] is obtained. The position information of the stereo embedding module (SE) is embedded into the coding layer result of the first layer to obtain a tensor of size [2, 64, 32, 32, 18]. The second coding layer also contains three blocks. Each block undergoes a convolution operation three times, a BatchNorm operation and a ReLU activation function operation. After encoding by the second coding layer, a tensor of size [2, 128, 16, 16, 18] is obtained. The third coding layer contains 6 convolution blocks. Each block undergoes a convolution operation three times, a BatchNorm operation and a ReLU activation function operation. After the third coding layer, a tensor of size [2, 256, 8, 8, 18] is obtained.

[0034] Step 28. After deformation, the tensor [2, 36, 64, 128] is obtained. After a convolution operation with Ks and St of (1, 1, 1) and (1, 1, 1), the tensor cnet is obtained. The dimension of cnet is [2, 256, 64, 128]. The tensor [B, C, H, W] of image1 is operated with [B, H / / 8, W / / 8] to obtain coords0: [2, 2, 64, 128] and coords1: [2, 2, 64, 128] for subsequent optical flow calculation;

[0035] Step 29: Fuse fmap1 and fmap2 into a tensor of corr:[2,648,64,128]. Subtract coords1 from coords0 to obtain a simple flow of size [2,2,64,128]. Divide cnet into two tensors net and inp, which are [2,128,64,128] and [2,128,64,128] respectively.

[0036] Step 30: Fuse net, inp, corr and flow. First, fuse flow and corr. Through 5 convolution operations, the first Ks and St are (1, 1) and (1, 1). The second Ks, St and Pd are (3, 3), (1, 1) and (1, 1) respectively. The third Ks, St and Pd are (7, 7), (1, 1) and (3, 3) respectively. The fourth Ks, St and Pd are (3, 3), (1, 1) and (1, 1) respectively. The fifth Ks, St and Pd are (3, 3), (1, 1) and (1, 1) respectively. Finally, the motion_features tensor is [2, 128, 64, 128].

[0037] Step 31: Fuse motion_features and inp to obtain the inp tensor of [2,256,64,128]. Fuse net and inp through 6 convolution operations. The first Ks, St, and Pd are (1,5), (1,1), and (0,2), the second Ks, St, and Pd are (1,5), (1,1), and (0,2), the third Ks, St, and Pd are (1,5), (1,1), and (0,2), the fourth Ks, St, and Pd are (5,1), (1,1), and (2,0), the fifth Ks, St, and Pd are (5,1), (1,1), and (2,0), and the sixth Ks, St, and Pd are (5,1), (1,1), and (2,0). Finally, the net tensor is [2,128,64,128].

[0038] Step 32: Use net to calculate the optical flow and get the delta_flow tensor as [2,2,64,128]. At the same time, use net to perform convolution operations with Ks, St, and Pd as (3,3), (1,1), and (1,1), respectively. It is a ReLU activation function, and the second convolution operation with Ks and St as (1,1) and (1,1) yields a mask tensor of [2,576,64,128].

[0039] Step 33, add delta_flow to coords1 by bitwise addition to get [2,2,64,128]; take coords1-coords0 and mask as input for upsampling, name coords1-coords0 flow, adjust mask to the shape of [2,1,9,8,8,64,128], and call F.unfold method to transform flow into up_flow of [2,18,8192]. The function of F.unfold is to divide the input multi-dimensional tensor (usually an image tensor) into multiple small image blocks (again) in a sliding window manner. Each image block is expanded into a one-dimensional vector, and finally all the expanded vectors are arranged into a two-dimensional tensor according to the specified dimension; up_flow is deformed to [2,2,9,1,1,64,128]; the torch.sum function is used to operate mask and up_flow to obtain the up_flow tensor as [2,2,8,8,64,128], and the up_flow is swapped to [2,2,64,8,128,8], and after the deformation of [N,2,8*H,8*W], it is [2,2,512,1024]; the final output is [2,2,512,1024];

[0040] Step 34: Select the loss function L loss :

[0041]

[0042] Where γ = 0.8Ni-1 is the weight of the sequence loss, the weight increases exponentially, N is the number of optical flow predictions, and the goal of this loss function is to minimize the loss between the ground truth and the predicted optical flow;

[0043] Step 35. Select Adam with default parameters as the optimizer for the training process, and use a constant learning rate of 0.000125, an image size of 512*1024, and a default number of slices of N=18; perform 100K iterations on Flowscape with a batch size of 2; set the number of GRU iterations to 12 during training and validation; use Flyingchairs→Flyingthings for pre-training and conduct experiments on Flowscape; during training, set pre-training on Flyingchairs for 100K iterations with a batch size of 4, training on Flyingthings for 100K iterations with a batch size of 4; finally, set training on the Flowscape dataset for 100K iterations with a batch size of 2;

[0044] Step 36: In the algorithm, we plan to use an evaluation metric that has been widely used in similar tasks in the past to quantitatively evaluate the model algorithm. EPE (endpoint error) measures the Euclidean distance between the calculated optical flow vector and the true optical flow vector. The calculation formula of the above evaluation metric is as follows:

[0045]

[0046] Where (u, v) is the calculated optical flow vector, and (u', v') is the real optical flow vector.

[0047] Furthermore, in step 1, data is loaded by calling the Dataloader function, the panoramic image is passed, the data in the 'train' folder is looped through, and the index address of each file is saved for easy later search and use; several panoramic images are set to be loaded each time and these panoramic images are packaged to generate indexes and input into the network; the index addresses stored in the panoramic images and the indexes generated by the packaged files are used to merge the data, and the obtained data is converted into tensor form.

[0048] Furthermore, after the isometric projection image in step 2 enters the neural network, since the model inputs two images of the upper and lower frames; batch_size is set to 2; the image is an RGB image with 3 channels; the image size is 512*1024; so the size of the tensor of image1 obtained is [2, 3, 512, 1024], image2 is the same as image1, and the corresponding data is loaded into the neural network.

[0049] Furthermore, after entering the neural network in step 3, the input tensor size is first required to be [B, C, H, W]. The meaning of each parameter in the input tensor is: B represents the input batch size of 2, C represents the number of channels of the input tensor of 3, H represents the height of the input image of 512, and W represents the width of the input image of 1024. The image is downsampled using a convolution operation. The convolution operation Ks represents the size of the convolution kernel, St represents the sliding step size of the convolution operation, and Pa represents the padding of the original 3D grid space.

[0050] Furthermore, the perspective image function parameters in step 4 are set as follows: erp_img represents the equirectangular image, fov represents the field of view, nrows represents the number of rows, and patch_size represents the size of the slice image. The fov field of view is 80°, nrows is 18, and the patch_size slice image size is 128×128. This function returns image1 and image2, as well as two batches of images, namely four tensors: the perspective image (pers), the xyz coordinates (xyz) of the perspective image, the uv coordinates (uv) in the spherical coordinate system, and the center point coordinates (center_p). The image is cut into 18 [128, 128] blocks, resulting in a tensor of [4, 3, 128, 128, 18] after cutting. This tensor is then fed into the encoding layer.

[0051] Furthermore, the ReLU activation function in step 21 is a smooth activation function, whose derivative is differentiable over the entire real number range, avoiding the problems of gradient vanishing and gradient exploding, while improving the stability of training; the Linear layer is a fully connected layer, that is, each input feature is connected to each weight of the output feature. This fully connected structure can capture the global information in the input data and help the model better understand the data.

[0052] The advantages of the present invention compared with the existing technology are:

[0053] 1. This paper proposes a novel framework for learning panoramic optical flow. The framework learns panoramic optical flow estimation based on the method of section processing and global features. A large number of experiments have demonstrated the effectiveness of the framework.

[0054] 2. To address the significant distortion of equirectangular projection images, this paper utilizes a method based on cross-sectional images to address inherent distortion. A stereo embedding module is introduced to provide supplementary 3D information, embedding stereo features into planar features to address the global inconsistency of ERP.

[0055] 3. In order to solve the large displacement problem, the present invention proposes a transformer-based global feature learning (GFL) module to extract global information of the encoder features and enhance image features.

[0056] 4. Through the analysis of experimental results, the present invention shows that the final model has established the most advanced performance and has certain advantages in generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of the panoramic optical flow estimation method of the present invention.

[0058] Figure 2 These are example diagrams of spherical center projection of the present invention, (a) is the projection of the section image, and (b) is an example of spherical center projection.

[0059] Figure 3 It is the global feature learning (GFL) module based on Transformer of the present invention. DETAILED DESCRIPTION

[0060] In order to make the contents of the present invention more clearly understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0061] Reference Figure 1 Two consecutive frames of ERP images are input into the network structure, and a series of cross-sectional images are generated by spherical center projection. The stereo embedding (SE) network is introduced to generate stereo features to make up for the missing 3D information. The feature information of the image is extracted through the encoder, and the undistorted cross-sectional images are processed by the decoder network to generate a panoramic optical flow map.

[0062] Place the 5000 training images in a folder named 'train' and the 1400 test images in a folder named 'test'. After the equirectangular projection images enter the neural network, the model inputs two images, the upper and lower frames; the batch_size is set to 2; the images are RGB images with three channels; and the image size is 512*1024. Therefore, the resulting tensor size for image1 is [2, 3, 512, 1024]. Image2 is the same as image1. Load the corresponding data into the neural network.

[0063] Reference Figure 2,Two adjacent slice images projected from the center point in different projection directions will have overlapping areas, and the same object will appear in different perspective images. Point A on the sphere s =(ρ,θ) is projected onto point B = (ρ B ,θ B )Point A on the tangent plane t =(x t ,y t ).

[0064] First, in the dimension B, torch.cat concatenates the upper and lower frames to produce a tensor of size [4, 3, 512, 1024]. This is then converted to a perspective image using the following parameters: erp_img represents the isometric image, fov represents the field of view, nrows represents the number of rows, and patch_size represents the size of the slice image. The fov field of view is set to 80°, nrows is set to 18, and the slice image size is 128×128. This function returns image1 and image2, along with two batches of images: the perspective image (pers), the xyz coordinates of the perspective image (xyz), the spherical uv coordinates (uv), and the center point coordinates (center_p). The image is then sliced into 18 [128, 128] blocks, resulting in a tensor of [4, 3, 128, 128, 18]. This tensor is then fed into the encoding layer.

[0065] Considering the difference between the positions of different image regions between the spherical image and the equirectangular image, the equirectangular image is reprojected back to the spherical image, and the position information of different regions is calculated using the spherical image to obtain a tensor of size [1, 64, 32, 32, 18].

[0066] The first convolution operation is Ks (7, 7, 1), St (2, 2, 1), and Pd (3, 3, 0), followed by BatchNorm and ReLU activation functions. After the first encoding layer, the sliced tensor [4, 3, 128, 128, 18] is encoded as [4, 64, 64, 64, 18].

[0067] The BatchNorm function is used to perform the first batch normalization on the tensors obtained from the first convolution operation of the isometric projection image. This makes the input distribution of each neuron more stable, allowing for faster convergence during training. It also improves the generalization ability of the model and suppresses overfitting, thereby improving the accuracy and reliability of the deep learning model.

[0068] Then, the ReLU activation function is used to perform the first nonlinear fitting on the tensor obtained by the first convolution operation of the cross-section image;

[0069] Next is the maximum pooling operation, where Ks, St, and Pd are (3, 3, 1), (2, 2, 1), and (1, 1, 0), respectively, and the resulting tensor is [4, 64, 32, 32, 18].

[0070] Next comes the first encoding layer, which encodes the contextual relationships between the 18 patches. The first encoding layer consists of three blocks. The first block contains a Conv3d convolution operation, followed by BatchNorm and ReLU activation functions, and then another convolution operation and BatchNorm operation. This is followed by the second convolution block. The second block contains a Conv3d convolution operation, followed by BatchNorm and ReLU activation functions, and then another convolution operation and BatchNorm operation, followed by nonlinear fitting using the ReLU activation function. This is followed by the third encoding block. The third block contains a Conv3d convolution operation, followed by BatchNorm and ReLU activation functions, and then another convolution operation and BatchNorm operation.

[0071] After encoding by the first encoding layer, a tensor of size [4, 64, 32, 32, 18] is obtained. Then, the position information of the stereo embedding module (SE) is embedded into the encoding layer result of the first layer to obtain a tensor of size [4, 64, 32, 32, 18].

[0072] Next comes the second encoding layer, which also consists of three blocks. The first block consists of the first convolution operation, followed by BatchNorm and ReLU activation functions, the second convolution operation, BatchNorm and ReLU activation functions, and the third convolution operation and BatchNorm function. In the second convolution block, the first convolution operation is followed by BatchNorm and ReLU activation functions. In the third convolution block, the first convolution operation is followed by BatchNorm and ReLU activation functions. After encoding in the second encoding layer, the resulting tensor is [4, 128, 16, 16, 18].

[0073] The third encoding layer contains six convolutional blocks. In the first block, the first convolution operation is followed by BatchNorm and ReLU activation functions. The second convolution operation is followed by BatchNorm and ReLU activation functions. The third convolution operation is followed by BatchNorm. In the second convolution block, the first convolution operation is followed by BatchNorm and ReLU activation functions. The second convolution operation is followed by BatchNorm. In the third convolution block, the first convolution operation is followed by BatchNorm and ReLU activation functions. The second convolution operation is followed by BatchNorm. In the fourth convolution block, the first convolution operation is followed by BatchNorm and ReLU activation functions. The second convolution operation is followed by BatchNorm. In the fifth convolution block, the first convolution operation is followed by BatchNorm and ReLU activation functions. The second convolution operation is followed by BatchNorm. In the sixth convolution block, the first convolution operation is followed by BatchNorm and ReLU activation functions. The second convolution operation is followed by a BatchNorm function, which passes through the third encoding layer to obtain a tensor of size [4, 256, 8, 8, 18].

[0074] After that, after reshape deformation, the tensor is obtained [4, 36, 64, 128], and after a convolution operation with Ks and St as (1, 1, 1) and (1, 1, 1), the tensor is obtained [4, 256, 64, 128].

[0075] The tensor [4, 256, 64, 128] is separated into feature_list1 and feature_list2 by torch.split, and their tensors are [2, 256, 64, 128] and [2, 256, 64, 128] respectively.

[0076] After the encoding layer, the tensors of feature_list1 and feature_list2 are [2, 256, 64, 128]. Then feature_list1 and feature_list2 are sent to the transformer module. The transformer module includes self-attention and cross-attention. After entering the transformer module, the feature_list1 and feature_list2 tensors are first expanded to [B, H*W, C], and feature_list1 and feature_list2 are spliced on the dimension of B to obtain [2B, H*W, C], that is, the tensor is [4, 8192, 256].

[0077] The Q, K, and V matrices for self-attention are obtained by passing image1 through a fully connected Linear layer. The dimensions of Q, K, and V are all [4, 8192, 256]. First, the transpose of the Q matrix and the K matrix are multiplied together. Then, the attention score is obtained through torch.softmax. The attention score is multiplied by the V matrix to obtain the self-attention enhanced tensor. The resulting tensor after self-attention is [4, 8192, 256].

[0078] Then, the output of self-attention is used as the input of cross-attention. Corss-attention has two inputs, one from self-attention and the other is the initial tensor obtained by concatenating feature_list1 and feature_list2 in the dimension of B. The Q matrix is calculated by the output of self-attention, and the K and V matrices are calculated using the tensor obtained by concatenating feature_list1 and feature_list2. The Q, K and V matrices are all calculated through the Linear layer. Then, the transpose of the Q and K matrices is multiplied, and the attention score is obtained through torch.softmax and then multiplied with the V matrix to obtain the cross-attention enhanced tensor.

[0079] Then comes the MLP layer, which consists of a Linear layer, ReLU and Linear. Finally, there is a Layernorm normalization, which can effectively alleviate the problem of network convergence difficulties. After corss-attention, the tensor is obtained as [4,8192,256], and then the size is restored to the input size, that is, [2,256,64,128] of fmap1 and fmap2.

[0080] =Then, in the CNet module, the up-down relationship of image regions is learned using the initial target frame image1. Image1 is used as input, with a tensor of [2, 3, 512, 1024]. The input tensor [2, 3, 512, 1024] is converted from isometric to perspective using the following parameters: erp_img represents the isometric image, fov represents the field of view, nrows represents the number of rows, and patch_size represents the size of the slice image. The fov field of view is 80°, nrows is 18, and patch_size represents the size of the slice image. This function returns image1 and image2, along with two batches of images: four tensors: the perspective image (pers), the xyz coordinates of the perspective image (xyz), the spherical coordinates (uv) (uv), and the center point coordinates (center_p).

[0081] First, the image is cut into 18 blocks of [128, 128]. That is, the tensor after cutting is [2, 3, 128, 128, 18]. Then, this tensor is sent to the encoding layer.

[0082] Considering the difference between the positions of different image regions between the spherical image and the equirectangular image, the equirectangular image is reprojected back to the spherical image, and the position information of different regions is calculated using the spherical image to obtain a tensor of size [1, 64, 32, 32, 18].

[0083] The first convolution operation is with Ks (7, 7, 1), St (2, 2, 1), and Pd (3, 3, 0), followed by BatchNorm and ReLU activation functions. After the first encoding layer, the tensor becomes [2, 64, 64, 64, 18].

[0084] The BatchNorm function is used to perform the first batch normalization processing on the tensor obtained by the first convolution operation of the isometric projection image, so that the input distribution of each neuron is more stable and converges faster during the training process. At the same time, it can also improve the generalization ability of the model and suppress overfitting, thereby improving the accuracy and reliability of the deep learning model. Then, the ReLU activation function is used for nonlinear fitting.

[0085] Next is the maximum pooling operation, where Ks, St, and Pd are (3, 3, 1), (2, 2, 1), and (1, 1, 0), respectively, and the resulting tensor is [2, 64, 32, 32, 18].

[0086] Next is the first encoding layer to encode the contextual relationship between the 18 patch blocks. The first encoding layer contains three blocks. The first block contains a Conv3d convolution operation, whose Ks, St, and Pd are (3, 3, 1), (1, 1, 1), and (1, 1, 0), respectively. Then, after the BatchNorm and ReLU activation functions, it undergoes another convolution operation. The Ks, St, and Pd of this convolution operation are (3, 3, 1), (1, 1, 1), and (1, 1, 0), respectively, and the BatchNorm operation.

[0087] The BatchNorm function is used to perform the first batch normalization on the tensor obtained by the first convolution operation of the equirectangular projection image, and then the ReLU activation function is used for nonlinear fitting.

[0088] This is followed by the second convolution block, which contains a Conv3d convolution operation with Ks, St, and Pd of (3, 3, 1), (1, 1, 1), and (1, 1, 0), respectively. Then, after BatchNorm and ReLU activation functions, it undergoes another convolution operation with Ks, St, and Pd of (3, 3, 1), (1, 1, 1), and (1, 1, 0), respectively, and BatchNorm operation.

[0089] The BatchNorm function is used to perform the first batch normalization on the tensor obtained by the first convolution operation of the equirectangular projection image, and then the ReLU activation function is used for nonlinear fitting.

[0090] This is followed by the third encoding block, which contains a Conv3d convolution operation with Ks, St, and Pd of (3, 3, 1), (1, 1, 1), and (1, 1, 0), respectively. Then, after BatchNorm and ReLU activation functions, it undergoes another convolution operation with Ks, St, and Pd of (3, 3, 1), (1, 1, 1), and (1, 1, 0), respectively, and BatchNorm operations.

[0091] The BatchNorm function is used to perform the first batch normalization on the tensor obtained by the first convolution operation of the equirectangular projection image, and then the ReLU activation function is used for nonlinear fitting.

[0092] Step 30: After encoding by the first encoding layer, a tensor of size [2, 64, 32, 32, 18] is obtained. Then, the position information of the stereo embedding module (SE) is embedded into the encoding layer result of the first layer to obtain a tensor of size [2, 64, 32, 32, 18].

[0093] Next is the second encoding layer. The second layer also contains three blocks. Each block is a convolution operation followed by BatchNorm and ReLU activation functions.

[0094] After encoding by the second encoding layer, a tensor of size [2, 128, 16, 16, 18] is obtained.

[0095] The third encoding layer contains 6 convolution blocks. Each block undergoes a convolution operation followed by a BatchNorm and ReLU activation function. After the third encoding layer, a tensor of size [2, 256, 8, 8, 18] is obtained.

[0096] After that, after deformation, the tensor [2, 36, 64, 128] is obtained. After a convolution operation with Ks and St as (1, 1, 1) and (1, 1, 1), the tensor cnet is obtained, and the dimension of cnet is [2, 256, 64, 128].

[0097] Then, the tensor [B, C, H, W] of image1 is operated with [B, H / / 8, W / / 8] to obtain coords0: [2, 2, 64, 128] and coords1: [2, 2, 64, 128] for subsequent optical flow calculation.

[0098] Fuse fmap1 and fmap2 into a tensor of corr:[2,648,64,128].

[0099] Then, a simple flow is obtained by subtracting coords1 from coords0, and the size is [2, 2, 64, 128].

[0100] After that, cnet is divided into two tensors net and inp are [2,128,64,128] and [2,128,64,128] respectively.

[0101] After that, net, inp, corr and flow are fused. First, flow and corr are fused. Through 5 convolution operations, the first Ks and St are (1, 1) and (1, 1). The second Ks, St and Pd are (3, 3), (1, 1) and (1, 1) respectively. The third Ks, St and Pd are (7, 7), (1, 1) and (3, 3) respectively. The fourth Ks, St and Pd are (3, 3), (1, 1) and (1, 1) respectively. The fifth Ks, St and Pd are (3, 3), (1, 1) and (1, 1) respectively. Finally, the motion_features tensor is [2, 128, 64, 128].

[0102] Next, motion_features and inp are fused to obtain the inp tensor of [2, 256, 64, 128].

[0103] Then, the fused net and input are passed through 6 convolution operations. The first Ks, St, and Pd are (1, 5), (1, 1), and (0, 2) respectively. The second Ks, St, and Pd are (1, 5), (1, 1), and (0, 2) respectively. The third Ks, St, and Pd are (1, 5), (1, 1), and (0, 2) respectively. The fourth Ks, St, and Pd are (5, 1), (1, 1), and (2, 0) respectively. The fifth Ks, St, and Pd are (5, 1), (1, 1), and (2, 0) respectively. The sixth Ks, St, and Pd are (5, 1), (1, 1), and (2, 0) respectively. The final net tensor is [2, 128, 64, 128].

[0104] Then use the net to calculate the optical flow and get the delta_flow tensor of [2,2,64,128]. At the same time, use the net to perform convolution operations with Ks, St, and Pd as (3,3), (1,1), and (1,1), respectively, followed by a ReLU activation function, and then a second convolution operation with Ks and St as (1,1) and (1,1), to get the mask tensor of [2,576,64,128].

[0105] By adding delta_flow to coords1, we get [2, 2, 64, 128].

[0106] Then, coords1-coords0 and mask are used as input for upsampling. coords1-coords0 is named flow. First, mask is adjusted to the shape of [2, 1, 9, 8, 8, 64, 128]. Then, flow is transformed into up_flow [2, 18, 8192] by calling the F.unfold method. The function of F.unfold is to split the input multi-dimensional tensor (usually an image tensor) into multiple small image blocks (also called image blocks or local receptive fields) in a sliding window manner, then expand each image block into a one-dimensional vector, and finally arrange all the expanded vectors into a two-dimensional tensor according to the specified dimensions. Then up_flow is transformed into [2, 2, 9, 1, 1, 64, 128]. Then, the torch.sum function is used to calculate the mask and up_flow tensors to obtain the up_flow tensor [2,2,8,8,64,128]. The up_flow tensor is then transposed to [2,2,64,8,128,8]. After the transformation to [N,2,8*H,8*W], the tensor is [2,2,512,1024]. The final output is [2,2,512,1024].

[0107] Selected loss function L loss :

[0108]

[0109] Where γ = 0.8Ni-1 is the weight of the sequence loss, the weight increases exponentially, and N is the number of optical flow predictions. The goal of this loss function is to minimize the loss between the ground truth and the predicted optical flow.

[0110] Adam with default parameters was selected as the optimizer for the training process, with a constant learning rate of 0.000125, an image size of 512*1024, and a default number of slices of N=18. 100K iterations were performed on Flowscape with a batch size of 2. The number of GRU iterations was set to 12 during training and validation. For comparative experiments, pre-training was performed on Flyingchairs→Flyingthings, and then experiments were performed on Flowscape. During training, 100K iterations of pre-training were set on Flyingchairs with a batch size of 4, followed by 100K iterations of training on Flyingthings with a batch size of 4. Finally, 100K iterations of training were set on the Flowscape dataset with a batch size of 2.

[0111] In the algorithm, we plan to use a widely used evaluation metric in similar tasks to quantitatively evaluate the model algorithm: EPE (endpoint error). This metric measures the Euclidean distance between the calculated optical flow vector and the true optical flow vector. The calculation formula for this evaluation metric is shown below.

[0112]

[0113] Where is the calculated optical flow vector and is the real optical flow vector.

[0114] The above steps can solve the distortion and large displacement problems in the panoramic optical flow estimation task, and it is a novel panoramic optical flow estimation method.

[0115] The above description of the present invention and its embodiments is non-limiting. The accompanying drawings illustrate only one embodiment of the present invention. In short, if a person skilled in the art is inspired by the above description and, without departing from the purpose of the present invention, designs structures and embodiments similar to the technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. A panoramic optical flow estimation method based on section processing and global features, characterized by: The following steps are involved: Step 1: Load 5000 training images containing four weather conditions: sunny, cloudy, foggy, and rainy by calling the Dataloader function. Pass the panoramic image, loop through the data in the 'train' folder, and load the equirectangular projection dataset Flowscape into the neural network. Step 2: After entering the neural network, the input tensor [B, C, H, W] is trained through the encoder; Step 3. Place 1400 test images containing four weather conditions: sunny, cloudy, foggy, and rainy in a folder named 'test' and test the test data with the trained model. Step 4: In the dimension of B, use torch.cat to concatenate the upper and lower frames to obtain a tensor of size [4, 3, 512, 1024]. After isometric transformation, it is the operation of the perspective image function. Step 5: The difference between the positions of different image regions between the spherical image and the equirectangular image is re-projected back to the spherical image, and the position information of different regions is calculated using the spherical image to obtain a tensor of size [1, 64, 32, 32, 18]. Step 6. The first convolution operation is Ks (7, 7, 1), St (2, 2, 1) and Pd (3, 3, 0), which is the BatchNorm and ReLU activation function. After the first encoding layer, the sliced tensor [4, 3, 128, 128, 18] is encoded as [4, 64, 64, 64, 18]. Step 7: Use the BatchNorm function to perform the first batch normalization on the tensor obtained from the first convolution operation of the isometric projection image. This makes the input distribution of each neuron more stable, allowing for faster convergence during training. It also improves the generalization ability of the model and suppresses overfitting, thereby improving the accuracy and reliability of the deep learning model. Step 8: Use the ReLU activation function to perform nonlinear fitting on the tensor obtained by the first convolution operation of the cross-section image; Step 9. Perform the maximum pooling operation, Ks, St, and Pd are (3, 3, 1), (2, 2, 1), and (1, 1, 0), respectively. The tensor obtained is [4, 64, 32, 32, 18]. The first encoding layer encodes the contextual relationship between the 18 patch blocks. The first encoding layer contains three blocks. The first block contains a Conv3d convolution operation, whose Ks, St, and Pd are (3, 3, 1), (1, 1, 1), and (1, 1, 0), respectively. After the BatchNorm and ReLU activation functions, it undergoes another convolution operation. The Ks, St, and Pd of this convolution operation are (3, 3, 1), (1, 1, 1), and (1, 1, 0), respectively, and the BatchNorm operation. Step 10. Use the BatchNorm function to perform the first batch normalization on the tensor obtained by the first convolution operation of the equirectangular projection image, and use the ReLU activation function for nonlinear fitting; the second convolution block, the second block contains a Conv3d convolution operation, whose Ks, St and Pd are (3, 3, 1), (1, 1, 1) and (1, 1, 0) respectively. After the BatchNorm and ReLU activation functions, it undergoes another convolution operation. The Ks, St and Pd of this convolution operation are (3, 3, 1), (1, 1, 1) and (1, 1, 0) and BatchNorm operation respectively; Step 11: Use the BatchNorm function to perform the first batch normalization on the tensor obtained by the first convolution operation of the equirectangular projection image, and use the ReLU activation function for nonlinear fitting. Step 12, the third encoding block, the third block contains a Conv3d convolution operation, its Ks, St and Pd are (3, 3, 1), (1, 1, 1) and (1, 1, 0) respectively. After the BatchNorm and ReLU activation functions, it undergoes another convolution operation. The Ks, St and Pd of this convolution operation are (3, 3, 1), (1, 1, 1) and (1, 1, 0) and BatchNorm operation respectively. Step 13: Use the BatchNorm function to perform the first batch normalization on the tensor obtained from the first convolution operation of the equirectangular projection image, and use the ReLU activation function for nonlinear fitting; after encoding through the first encoding layer, a tensor of size [4, 64, 32, 32, 18] is obtained. The position information of the stereo embedding module (SE) is embedded into the encoding layer result of the first layer to obtain a tensor of size [4, 64, 32, 32, 18]; Step 14: The second encoding layer. The second layer contains three blocks. Each block undergoes a convolution operation, a BatchNorm operation, and a ReLU activation function operation. After encoding in the second encoding layer, a tensor of size [4, 128, 16, 16, 18] is obtained. Step 15. The third encoding layer contains 6 convolution blocks. Each block undergoes a convolution operation, a BatchNorm operation, and a ReLU activation function operation. After passing through the third encoding layer, a tensor of size [4, 256, 8, 8, 18] is obtained. Step 16. After reshape deformation, the tensor is [4, 36, 64, 128]. After a convolution operation with Ks and St as (1, 1, 1) and (1, 1, 1), the tensor is [4, 256, 64, 128]. Step 17. Use torch.split to separate the tensor [4, 256, 64, 128] into feature_list1 and feature_list2, whose tensors are [2, 256, 64, 128] and [2, 256, 64, 128] respectively. Step 18. After the encoding layer, the tensors of feature_list1 and feature_list2 are [2, 256, 64, 128]. Entering the transformer module, first expand the feature_list1 and feature_list2 tensors to [B, H*W, C], and concatenate feature_list1 and feature_list2 in the dimension of B to obtain [2B, H*W, C], that is, the tensor is [4, 8192, 256]. Step 19: The Q, K, and V matrices of self-attention are obtained by passing image1 through the Linear fully connected layer. The dimensions of Q, K, and V are all [4, 8192, 256]. Calculate the transpose multiplication of the Q matrix and the K matrix, and obtain the attention score through torch.softmax. The attention score is multiplied by the V matrix to obtain the self-attention enhanced tensor. The tensor obtained after self-attention is [4, 8192, 256]. Step 20: Use the output of self-attention as the input of cross-attention. Cross-attention has two inputs, one from self-attention and the other is the initial tensor obtained by concatenating feature_list1 and feature_list2 in the dimension of B. The Q matrix is calculated by the output of self-attention, and the K and V matrices are calculated using the tensor obtained by concatenating feature_list1 and feature_list2. The Q, K and V matrices are all calculated through the Linear layer. The transpose of the Q and K matrices is multiplied, and the attention score is obtained by torch.softmax and multiplied with the V matrix to obtain the cross-attention enhanced tensor. Step 21: MLP layer, including Linear layer, ReLU and Linear, and finally a Layernorm normalization, which can effectively alleviate the problem of network convergence difficulties. After corss-attention, the tensor is obtained as [4,8192,256], and then the size is restored to the input size, that is, [2,256,64,128] of fmap1 and fmap2. Step 22: In the cnet module, use the initial image1 to learn the upper and lower relationships of the image areas, with image1 as input, the tensor is [2, 3, 512, 1024], the input tensor [2, 3, 512, 1024], and the isometric conversion to perspective image function. The parameter settings are as follows: erp_img represents the isometric image, fov represents the field of view, nrows represents the number of rows, patch_size represents the size of the cross-section image, the fov field of view is 80°, nrows is 18, and the patch_size cross-section image size is 128×128. After this function, image1 and image2 and two batches of pictures are returned, namely, four tensors perspective image (pers), xyz coordinates (xyz) in the perspective image, uv coordinates (uv) in the spherical coordinate system, and center point coordinates (center_p); Step 23: Cut the image into 18 blocks of [128, 128]. That is, the tensor after cutting is [2, 3, 128, 128, 18]. Then, send the tensor to the encoding layer. Step 24: Calculate the difference between the positions of different image regions between the spherical image and the equirectangular image, reproject the equirectangular image back into the spherical image, and use the spherical image to calculate the position information of different regions to obtain a tensor of size [1, 64, 32, 32, 18]. Step 25: The first convolution operation is with Ks (7, 7, 1), St (2, 2, 1), and Pd (3, 3, 0). It uses BatchNorm and ReLU activation functions. After the first encoding layer, the tensor becomes [2, 64, 64, 64, 18]. The BatchNorm function is used to perform the first batch normalization on the tensor obtained by the first convolution operation of the isometric projection image. The ReLU activation function is used to perform the first nonlinear fitting on the tensor obtained by the first convolution operation of the cross-section image. Step 26, the maximum pooling operation, Ks, St and Pd are (3, 3, 1), (2, 2, 1) and (1, 1, 0) respectively. The tensor obtained is [2, 64, 32, 32, 18]. The first encoding layer encodes the contextual relationship between 18 patch blocks. The first encoding layer contains three blocks. The first block contains a Conv3d convolution operation, whose Ks, St and Pd are (3, 3, 1), (1, 1, 1) and (1, 1, 0) respectively. After BatchNorm and ReLU activation functions, it undergoes another convolution operation. The Ks, St and Pd of this convolution operation are Pd is (3, 3, 1), (1, 1, 1) and (1, 1, 0) respectively and BatchNorm operation, the BatchNorm function is used to perform the first batch normalization on the tensor obtained by the first convolution operation of the isometric projection image, and the ReLU activation function is used to perform nonlinear fitting on the tensor obtained by the first convolution operation of the cross-section image; the second convolution block, the second block contains a Conv3d convolution operation, its Ks, St and Pd are (3, 3, 1), (1, 1, 1) and (1, 1, 0) respectively, after BatchNorm and ReLU activation function, and then After a convolution operation, the Ks, St and Pd of this convolution operation are (3, 3, 1), (1, 1, 1) and (1, 1, 0) respectively, and the BatchNorm operation is used. The BatchNorm function is used to perform the first batch normalization on the tensor obtained by the first convolution operation of the equirectangular projection image. The ReLU activation function is used to perform nonlinear fitting on the tensor obtained by the first convolution operation of the cross-section image. The third encoding block contains a Conv3d convolution operation, whose Ks, St and Pd are (3, 3, 1), (1, 1, 1) and (1, 1, 0) respectively. , after BatchNorm and ReLU activation functions, it undergoes another convolution operation. The Ks, St, and Pd of this convolution operation are (3, 3, 1), (1, 1, 1), and (1, 1, 0), respectively. The BatchNorm operation uses the BatchNorm function to perform the first batch normalization processing on the tensor obtained by the first convolution operation of the isometric projection image, making the input distribution of each neuron more stable, which can converge faster during the training process. At the same time, it can also improve the generalization ability of the model and suppress overfitting, thereby improving the accuracy and reliability of the deep learning model; Step 27: Use the ReLU activation function to perform the first nonlinear fitting on the tensor obtained by the first convolution operation of the cross-section image. After encoding by the first coding layer, a tensor of size [2, 64, 32, 32, 18] is obtained. The position information of the stereo embedding module (SE) is embedded into the coding layer result of the first layer to obtain a tensor of size [2, 64, 32, 32, 18]. The second coding layer also contains three blocks. Each block undergoes a convolution operation three times, a BatchNorm operation and a ReLU activation function operation. After encoding by the second coding layer, a tensor of size [2, 128, 16, 16, 18] is obtained. The third coding layer contains 6 convolution blocks. Each block undergoes a convolution operation three times, a BatchNorm operation and a ReLU activation function operation. After the third coding layer, a tensor of size [2, 256, 8, 8, 18] is obtained. Step 28. After deformation, the tensor [2, 36, 64, 128] is obtained. After a convolution operation with Ks and St of (1, 1, 1) and (1, 1, 1), the tensor cnet is obtained. The dimension of cnet is [2, 256, 64, 128]. The tensor [B, C, H, W] of image1 is operated with [B, H / / 8, W / / 8] to obtain coords0: [2, 2, 64, 128] and coords1: [2, 2, 64, 128] for subsequent optical flow calculation; Step 29: Fuse fmap1 and fmap2 into a tensor of corr:[2,648,64,128]. Subtract coords1 from coords0 to obtain a simple flow of size [2,2,64,128]. Divide cnet into two tensors net and inp, which are [2,128,64,128] and [2,128,64,128] respectively. Step 30: Fuse net, inp, corr and flow. First, fuse flow and corr. Through 5 convolution operations, the first Ks and St are (1, 1) and (1, 1). The second Ks, St and Pd are (3, 3), (1, 1) and (1, 1) respectively. The third Ks, St and Pd are (7, 7), (1, 1) and (3, 3) respectively. The fourth Ks, St and Pd are (3, 3), (1, 1) and (1, 1) respectively. The fifth Ks, St and Pd are (3, 3), (1, 1) and (1, 1) respectively. Finally, the motion_features tensor is [2, 128, 64, 128]. Step 31: Fuse motion_features and inp to obtain the inp tensor of [2,256,64,128]. Fuse net and inp through 6 convolution operations. The first Ks, St, and Pd are (1,5), (1,1), and (0,2), the second Ks, St, and Pd are (1,5), (1,1), and (0,2), the third Ks, St, and Pd are (1,5), (1,1), and (0,2), the fourth Ks, St, and Pd are (5,1), (1,1), and (2,0), the fifth Ks, St, and Pd are (5,1), (1,1), and (2,0), and the sixth Ks, St, and Pd are (5,1), (1,1), and (2,0). Finally, the net tensor is [2,128,64,128]. Step 32: Use net to calculate the optical flow and get the delta_flow tensor as [2,2,64,128]. At the same time, use net to perform convolution operations with Ks, St, and Pd as (3,3), (1,1), and (1,1), respectively. It is a ReLU activation function, and the second convolution operation with Ks and St as (1,1) and (1,1) yields a mask tensor of [2,576,64,128]. Step 33, add delta_flow to coords1 by bitwise addition to get [2,2,64,128]; take coords1-coords0 and mask as input for upsampling, name coords1-coords0 flow, adjust mask to the shape of [2,1,9,8,8,64,128], and call F.unfold method to transform flow into up_flow of [2,18,8192]. The function of F.unfold is to divide the input multi-dimensional tensor (usually an image tensor) into multiple small image blocks (again) in a sliding window manner. Each image block is expanded into a one-dimensional vector, and finally all the expanded vectors are arranged into a two-dimensional tensor according to the specified dimension; up_flow is deformed to [2,2,9,1,1,64,128]; the torch.sum function is used to operate mask and up_flow to obtain the up_flow tensor as [2,2,8,8,64,128], and the up_flow is swapped to [2,2,64,8,128,8], and after the deformation of [N,2,8*H,8*W], it is [2,2,512,1024]; the final output is [2,2,512,1024]; Step 34: Select the loss function L loss : Where γ = 0.8Ni-1 is the weight of the sequence loss, the weight increases exponentially, N is the number of optical flow predictions, and the goal of this loss function is to minimize the loss between the ground truth and the predicted optical flow; Step 35. Select Adam with default parameters as the optimizer for the training process, and use a constant learning rate of 0.000125, an image size of 512*1024, and a default number of slices of N=18; perform 100K iterations on Flowscape with a batch size of 2; set the number of GRU iterations to 12 during training and validation; use Flyingchairs→Flyingthings for pre-training and conduct experiments on Flowscape; during training, set pre-training on Flyingchairs for 100K iterations with a batch size of 4, training on Flyingthings for 100K iterations with a batch size of 4; finally, set training on the Flowscape dataset for 100K iterations with a batch size of 2; Step 36: In the algorithm, we plan to use an evaluation metric that has been widely used in similar tasks in the past to quantitatively evaluate the model algorithm. EPE (endpoint error) measures the Euclidean distance between the calculated optical flow vector and the true optical flow vector. The calculation formula of the above evaluation metric is as follows: Where (u, v) is the calculated optical flow vector, (u ， , v ， ) is the real optical flow vector.

2. The method for panoramic optical flow estimation based on slice processing and global features according to claim 1, characterized in that: In step 1, the data is loaded by calling the Dataloader function, the panoramic image is passed, the data in the 'train' folder is traversed, and the index address of each file is saved for later use; Set the number of panoramic images to be loaded each time and package these panoramic images to generate indexes and input them into the network; Use the index address stored in the panoramic image and the index generated by the package file to perform a merge operation and convert the obtained data into a tensor form.

3. The method for panoramic optical flow estimation based on slice processing and global features according to claim 1, characterized in that: After the isometric projection image enters the neural network in the second step, since the model inputs two images of the upper and lower frames; batch_size is set to 2; the image is an RGB image with 3 channels; the image size is 512*1024; so the size of the tensor of image1 obtained is [2, 3, 512, 1024], image2 is the same as image1, and the corresponding data is loaded into the neural network.

4. The method for panoramic optical flow estimation based on slice processing and global features according to claim 1, wherein: After entering the neural network in step 3, the input tensor size is first required to be [B, C, H, W]. The meaning of each parameter in the input tensor is: B represents the input batch size of 2, C represents the number of channels of the input tensor of 3, H represents the height of the input image of 512, and W represents the width of the input image of 1024. The image is downsampled using a convolution operation. The convolution operation Ks represents the size of the convolution kernel, St represents the sliding step size of the convolution operation, and Pa represents the padding of the original 3D grid space.

5. The method for panoramic optical flow estimation based on slice processing and global features according to claim 1, wherein: In step 4, the perspective image function parameters are set as follows: erp_img represents the isometric image, fov represents the field of view, nrows represents the number of rows, and patch_size represents the size of the slice image. The fov field of view is 80°, nrows is 18, and the patch_size slice image size is 128×128. This function returns image1 and image2, along with two batches of images: four tensors: the perspective image (pers), the xyz coordinates (xyz) of the perspective image, the uv coordinates (uv) in the spherical coordinate system, and the center point coordinates (center_p). The image is cut into 18 [128, 128] blocks, resulting in a tensor of [4, 3, 128, 128, 18] after cutting. This tensor is then fed into the encoding layer.

6. The method for panoramic optical flow estimation based on slice processing and global features according to claim 1, wherein: The ReLU activation function in step 21 is a smooth activation function whose derivative is differentiable over the entire real number range, thereby avoiding the problems of gradient vanishing and gradient exploding, while improving the stability of training. The Linear layer is a fully connected layer, i.e., each input feature is connected to each weight of the output feature. This fully connected structure can capture the global information in the input data, helping the model to better understand the data.

Images