Panoramic image monocular depth estimation method based on double-branch efficient fusion network

Abstract

The application relates to a panoramic image monocular depth estimation method based on a double-branch high-efficiency fusion network, and has the characteristics that a group of image data is obtained from a Matterport3D dataset platform and is transmitted into a computer; complete visual field information of panoramic images is utilized according to a double-branch network structure; a panoramic depth estimation algorithm based on grouping dilated convolution and channel attention is used; extra receptive fields provided by grouping dilated convolution and channel attention are used to extract rich features of equirectangular projection panoramic images to solve the above problems; different panoramic image projection forms are used as inputs to correct distortion caused by panoramic images; the two-pole distortion problem of panoramic images is solved, and the precision of monocular depth estimation of panoramic images is improved.

Description

Technical Field

[0001] This invention relates to a monocular depth estimation method for panoramic images based on a dual-branch efficient fusion network, belonging to the field of panoramic image 3D reconstruction. Background Technology

[0002] Depth information provides three-dimensional information about objects when computers perform computer vision tasks, avoiding inconsistencies in object proportions and errors in scene understanding. The depth information from panoramic images can be used to reconstruct a realistic three-dimensional world. Panoramic image depth information is widely used in 3D reconstruction, virtual reality, autonomous driving, and augmented reality. However, panoramic images differ from ordinary images; they exhibit varying degrees of distortion and warping, making it difficult to fully utilize their rich visual information for depth estimation. To address these issues, three main technical solutions are employed. The first is panoramic depth estimation using specialized convolutional networks, which design unique convolutional structures to compute depth for panoramic images. The second is panoramic depth estimation based on geometric priors, utilizing geometric information as supplementary data to complete the task. In recent years, panoramic depth estimation using different projection methods has begun to demonstrate its advantages, with distortion-free and full-field-of-view projection methods being introduced into the network to solve the panoramic depth estimation problem.

[0003] Image-based depth acquisition methods can be categorized into monocular, binocular, and multi-view methods based on the number of cameras required by the algorithm. Summary of the Invention

[0004] The purpose of this invention is to provide a monocular depth estimation method for panoramic images based on a dual-branch efficient fusion network, which is used for monocular depth estimation of panoramic images. The dual-branch fusion network structure enables the utilization of complete field-of-view information of panoramic images, solves the problem of polar distortion in panoramic images, and improves the accuracy of monocular depth estimation of panoramic images. The dual-branch fusion network structure can obtain fused depth information of equilateral and cubic projections.

[0005] The technical solution of this invention is implemented as follows: a panoramic image monocular depth estimation method based on a dual-branch efficient fusion network obtains a set of image data from the Matterport3D dataset platform and inputs it into the computer; the specific steps are as follows:

[0006] Step 1: Place 7829 training images in a folder named 'train', 947 validation images in a folder named 'val', and 2014 test point clouds for testing in a folder named 'test'.

[0007] Step 2: The isometric projection convolution operation requires the input tensor to be [Bs, Ic, H, W]. The meaning of each parameter in the input tensor is as follows: Bs represents the input batch size, Ic represents the number of channels in the input tensor, H represents the height of the input image, and W represents the width of the input image.

[0008] Step 3: Loop through the data in the 'train' folder or the data in the 'test' folder, and save the index address of each file for easy retrieval later; set how many depth images to load each time and pack these depth images to generate an index for input into the network; use the index address stored in the depth images and the index generated by the packing file to merge the data and convert the resulting data into tensor form, where the images are divided into isometric projection images and cube projection images;

[0009] Step 4: After the isometric projection image enters the neural network, since the model inputs 4 images at a time; the images are RGB images with 3 channels; the image size is 512*1024; therefore, the tensor size is [4, 3, 512, 1024], and the corresponding data is loaded into the neural network.

[0010] Step 5: The input tensor [4, 3, 512, 1024] undergoes the first convolution operation through the ResNet18 Block module. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the stride of the convolution operation, Pa represents the padding of the original 3D mesh space, the convolution kernel used in the convolution operation is a two-dimensional matrix, the values ​​of all elements in the matrix are randomly generated, and the range is [-1, 1], Bs is the input batch size of 4, Ks is the convolution kernel size of (7, 7), St is the stride of 2, Pa is the input of the padding of 3, and the tensor size obtained after the first convolution operation is [4, 64, 256, 512].

[0011] Step 6: Use the BatchNorm function to perform the first batch normalization on the tensor obtained from the first convolution operation of the isometric projection image, so that the input distribution of each neuron is more stable, which can converge faster during training. It can also improve the generalization ability of the model and suppress overfitting, thereby improving the accuracy and reliability of the deep learning model.

[0012] Step 7: Then, the ReLU activation function is used to perform the first nonlinear fitting on the tensor obtained from the first ResNet18 Block operation of the isometric projection image;

[0013] Step 8: Next, Maxpooling is used with input tensor size [4, 64, 256, 512], Bs batch size of 4, Ks kernel size of [3, 3], St stride of 2, Pa padding of 1 input, and output channel set to 32. The most representative information is extracted from the input feature map, and the size of the feature map is reduced. The output tensor size is [4, 32, 128, 256].

[0014] Step 9: Input the tensor [4,32,128,256] obtained from the max pooling layer into the first ECD Block. The parameters in the ECD Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B,C,H,W], keep B unchanged and split C into 4 parts on average. Perform convolution calculation on the input [B,C1,H,W] using 4 dilation convolutions with different dilation coefficients, where C1 is C divided by 4. Output the four parts [B,C1,H,W]. Pass the four parts [B,C1,H,W] through a global average pooling layer to obtain an output tensor of size [B,C,H,W]. Then pass it through 4 fast one-dimensional convolutional layers with a kernel size of 3 to obtain four outputs of size [B,C1,H,W]. Finally, multiply the four outputs by the original [B,C1,H,W] using the sigmoid function to complete the first ECD operation. Block computation outputs tensor sizes of [4, 32, 128, 256];

[0015] Step 10: Then, the ReLU activation function is used to perform a second nonlinear fitting on the tensor obtained from the first ECD Block operation of the isometric projection image;

[0016] Step 11: Input the tensor [4, 32, 128, 256] obtained from the second nonlinear fitting into the second ECDBlock module. The parameters in the ECD Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B, C, H, W], keep B unchanged and split C into 4 parts on average. Perform convolution calculation on the input [B, C1, H, W] using 4 dilation convolutions with different dilation coefficients, where C1 is C divided by 4. Output the four parts [B, C1, H, W]. Pass the four parts [B, C1, H, W] through a global average pooling layer to obtain the output tensor size [B, C, H, W]. Then pass it through 4 fast one-dimensional convolutional layers with a kernel size of 3 to obtain the four outputs, with the output tensor size [B, C1, H, W]. Finally, multiply the four outputs by the original [B, C1, H, W] using the Sigmoid function to end the second ECD. Block computation outputs tensor sizes of [4, 16, 64, 128];

[0017] Step 12: Then, the ReLU activation function is used to perform a third nonlinear fitting on the tensor obtained from the second ECD Block operation of the isometric projection image;

[0018] Step 13: Input the tensor [4, 16, 64, 128] obtained from the third nonlinear fitting into the third ECD Block module. The parameters in the ECD Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B, C, H, W], keep B unchanged, and split C into 4 parts on average. Perform convolution calculation on the input [B, C1, H, W] using 4 dilation convolutions with different dilation coefficients, where C1 is C divided by 4, and output the four parts [B, C1, H, W]. Pass the four parts of input [B, C1, H, W] through a global average pooling layer to obtain an output tensor of size [B, C, H, W]. Then pass it through 4 fast one-dimensional convolutional layers with a kernel size of 3 to obtain 4 outputs of size [B, C1, H, W]. Finally, multiply the four outputs by the original [B, C1, H, W] using the Sigmoid function, and end the third ECD. Block computation outputs tensors of size [4, 8, 32, 64].

[0019] Step 14: Then, the ReLU activation function is used to perform a fourth nonlinear fitting on the tensor obtained from the third ECD Block operation of the isometric projection image;

[0020] Step 15: Input the tensor [4, 8, 32, 64] obtained from the fourth nonlinear fitting into the fourth ECD Block module. The parameters in the ECD Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B, C, H, W], keep B unchanged and split C into 4 parts on average. Perform convolution calculation on the input [B, C1, H, W] using 4 dilation convolutions with different dilation coefficients, where C1 is C divided by 4. Output the four parts [B, C1, H, W]. Pass the four parts [B, C1, H, W] through a global average pooling layer to obtain the output tensor size [B, C, H, W]. Then pass it through 4 fast one-dimensional convolutional layers with a kernel size of 3 to obtain the four outputs, with the output tensor size [B, C1, H, W]. Finally, multiply the four outputs by the original [B, C1, H, W] using the Sigmoid function to end the fourth ECD. The block computation outputs a tensor of size [4, 4, 16, 32].

[0021] Step 16: Then, the ReLU activation function is used to perform a fifth nonlinear fitting on the tensor obtained from the fourth ECD Block operation of the isometric projection image;

[0022] Step 17: After the cube projection image is input into the neural network, since the model inputs 4 images at a time, each image has 6 faces of a cube; the image is an RGB image with 3 channels; the size of each face of the image is 256*256; therefore, the size of the tensor is [4, 3, 256, 256, 6], and the corresponding data is loaded into the neural network.

[0023] Step 18: The input tensor [4, 3, 256, 256, 6] undergoes the first convolution operation through the first ResNet18 Block module. The parameters are set as follows: Bs input batch size is 4, Ks convolution kernel size is (3, 3), St stride is 2, Pa padding input is 1. After the first convolution operation, the tensor size is [4, 64, 256, 256].

[0024] Step 19: Use the BatchNorm function to perform the first batch normalization on the tensors obtained from the first convolution operation of the cube projection image, so that the input distribution of each neuron is more stable, which can converge faster during training. It can also improve the generalization ability of the model and suppress overfitting, thereby improving the accuracy and reliability of the deep learning model.

[0025] Step 20: Then, the ReLU activation function is used to perform the first nonlinear fitting on the tensor obtained from the first ResNet18 Block operation on the cube projection image;

[0026] Step 21: Next, Maxpooling is used with input tensor size [4, 64, 256, 256], Bs batch size of 4, Ks kernel size of [3, 3], St stride of 2, Pa padding of 1 input, and output channel set to 32. The most representative information is extracted from the input feature map, and the size of the feature map is reduced. The output tensor size is [4, 32, 128, 128].

[0027] Step 22: Input the tensor [4,32,128,128] obtained from the first nonlinear fitting into the second ResNet18Block module. The parameters in the ResNet18 Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B,C,H,W], it first undergoes a convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and input padding of 1 (Pa). Then, it is normalized using the BatchNorm function. Next, it undergoes a second convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and input padding of 1 (Pa). Normalization is performed using the BatchNorm function; then, a third convolution operation is performed with a kernel size of [3, 3], a batch size of 4 (Bs), a stride of 1 (St), and an input padding of 1 (Pa). Normalization is then performed again using the BatchNorm function. Next, a fourth convolution operation is performed with a kernel size of [3, 3], a batch size of 4 (Bs), a stride of 1 (St), and an input padding of 1 (Pa). Normalization is then performed again using the BatchNorm function, ending the first ResNet18 Block computation. The output tensor size is [4, 32, 128, 128].

[0028] Step 23: Then, the ReLU activation function is used to perform a second nonlinear fitting on the tensor obtained from the second ResNet18 Block operation on the cube projection image;

[0029] Step 24: Input the tensor [4,32,128,128] obtained from the second nonlinear fitting into the third ResNet18Block module. The parameters in the ResNet18 Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B,C,H,W], it first undergoes a convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 2 (St), and a padding input of 1 (Pa). Then, it is normalized using the BatchNorm function. Next, it undergoes a second convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and a padding input of 1 (Pa). It is then normalized again using the BatchNorm function. Finally, it undergoes a third convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and a padding input of 1 (Pa). It is then normalized again using the BatchNorm function. Next, after the fourth convolution operation, the kernel size is [3, 3], the batch size is 4, the stride is 1, and the input padding is 1. Then, the BatchNorm function is used for normalization. The second ResNet18 Block calculation ends, and the output tensor size is [4, 16, 64, 64].

[0030] Step 25: Then, the ReLU activation function is used to perform a third nonlinear fitting on the tensor obtained from the third ResNet18 Block operation on the cube projection image;

[0031] Step 26: Input the tensor [4,16,64,64] obtained from the third nonlinear fitting into the fourth ResNet18 Block. The parameters in the ResNet18 Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B,C,H,W], it first undergoes a convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 2 (St), and a padding input of 1 (Pa). Then, it is normalized using the BatchNorm function. Next, it undergoes a second convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and a padding input of 1 (Pa). Then, it is normalized again using the BatchNorm function. Finally, it undergoes a third convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and a padding input of 1 (Pa). Then, it is normalized again using the BatchNorm function. Next, after the fourth convolution operation, the kernel size is [3, 3], the batch size is 4, the stride is 1, and the input padding is 1. Then, the BatchNorm function is used for normalization. The third ResNet18 Block calculation ends, and the output tensor size is [4, 8, 32, 32].

[0032] Step 27: Then, the ReLU activation function is used to perform a fourth nonlinear fitting on the tensor obtained from the fourth ResNet18 Block operation on the cube projection image;

[0033] Step 28: Input the tensor [4,8,32,32] obtained from the fourth nonlinear fitting into the fifth ResNet18Block module. The parameters in the ResNet18 Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B,C,H,W], it first undergoes a convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 2 (St), and input padding of 1 (Pa). Then, it is normalized using the BatchNorm function. Next, it undergoes a second convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and input padding of 1 (Pa). It is then normalized again using the BatchNorm function. Next, after a third convolution operation, the kernel size is [3, 3], the batch size (Bs) is 4, the stride (St) is 1, and the input padding (Pa) is 1. Then, the BatchNorm function is used for normalization. Following this, a fourth convolution operation is performed, with the kernel size [3, 3], the batch size (Bs) is 4, the stride (St) is 1, and the input padding (Pa) is 1. The BatchNorm function is used for normalization again. This completes the third ResNet18 Block computation, resulting in an output tensor size of [4, 4, 16, 16].

[0034] Step 29: Then, the ReLU activation function is used to perform a fifth nonlinear fitting on the tensor obtained from the fifth ResNet18 Block operation on the cube projection image;

[0035] Step 30: Perform the first two-branch fusion operation. The isometric projection feature map from the fifth nonlinear fitting in Step 16 is denoted as F1, with a tensor size of [4,4,16,16]. The cube projection feature map from the fifth nonlinear fitting in Step 29 is transformed into an isometric projection feature map, denoted as F2, with a tensor size of [4,4,16,32], through a C2E operation. F1 and F2 are concatenated to obtain a preliminary fused feature map F3. F3 undergoes a first convolution operation with a kernel size of [1,1], followed by a second convolution operation with a kernel size of [3,3], generating feature map F4. F4 is then pixel-wise added to F2 to obtain feature map F5. Finally, F5 and F1 are fused using the AFF module to obtain a fused feature map F6. F6 is then pixel-wise added to F5 using the CBAM module to obtain the final feature map F7, with an output tensor of [4,256,16,32].

[0036] Step 31: Use the BatchNorm function to perform the first batch normalization on the tensors obtained from the first two-branch fusion operation;

[0037] Step 32: Then, the ReLU activation function is used to perform the first nonlinear fitting on the tensor obtained from the first bi-branch fusion operation.

[0038] Step 33: Perform the second two-branch fusion operation. The isometric projection feature map from the fourth nonlinear fitting in Step 14, denoted as F1, with a tensor size of [4, 8, 32, 64], is transformed into an isometric projection feature map, denoted as F2, with a tensor size of [4, 8, 32, 32], through a C2E operation. F1 and F2 are concatenated to obtain a preliminary fused feature map F3. F3 undergoes a first convolution operation with a kernel size of [1, 1], followed by a second convolution operation with a kernel size of [3, 3], generating feature map F4. F4 is then pixel-wise added to F2 to obtain feature map F5. Finally, F5 and F1 are fused using the AFF module to obtain a fused feature map F6. F6 is then pixel-wise added to F5 using the CBAM module to obtain the final feature map F7, with an output tensor of [4, 128, 32, 64].

[0039] Step 34: Use the BatchNorm function to perform batch normalization on the tensors obtained from the second two-branch fusion operation;

[0040] Step 35: Then, the tensor obtained from the second bi-branch fusion operation is nonlinearly fitted using the ReLU activation function;

[0041] Step 36: Perform the third two-branch fusion operation. The isometric projection feature map obtained from the third nonlinear fitting in Step 12, denoted as F1, has a tensor size of [4, 16, 64, 128]. The cube projection feature map obtained from the third nonlinear fitting in Step 25 is transformed into an isometric projection feature map, denoted as F2, with a tensor size of [4, 16, 64, 64]. F1 and F2 are concatenated to obtain a preliminary fused feature map F3. F3 undergoes a first convolution operation with a kernel size of [1, 1], followed by a second convolution operation with a kernel size of [3, 3], generating feature map F4. F4 is then pixel-wise added to F2 to obtain feature map F5. Finally, F5 and F1 are fused using the AFF module to obtain a fused feature map F6. F6 is then pixel-wise added to F5 using the CBAM module to obtain the final feature map F7, with an output tensor of [4, 64, 64, 128].

[0042] Step 37: Use the BatchNorm function to perform batch normalization on the tensors obtained from the third bi-branch fusion operation;

[0043] Step 38: Then, the tensor obtained from the third bi-branch fusion operation is nonlinearly fitted using the ReLU activation function;

[0044] Step 39: Perform the fourth two-branch fusion operation. The isometric projection feature map obtained from the second nonlinear fitting in Step 10 is denoted as F1, with a tensor size of [4, 32, 128, 256]. The cube projection feature map obtained from the second nonlinear fitting in Step 23 is transformed into an isometric projection feature map, denoted as F2, with a tensor size of [4, 32, 128, 128]. F1 and F2 are concatenated to obtain a preliminary fused feature map F3. F3 undergoes a first convolution operation with a kernel size of [1, 1], followed by a second convolution operation with a kernel size of [3, 3], generating feature map F4. F4 is then added to F2 pixel by pixel to obtain feature map F5. Finally, F5 and F1 are fused through the AFF module to obtain fused feature map F6. F6 is then added to F5 pixel by pixel through the CBAM module to obtain the final feature map F7. The output tensor is [4, 32, 128, 256].

[0045] Step 40: Use the BatchNorm function to perform batch normalization on the tensors obtained from the fourth bi-branch fusion operation;

[0046] Step 41: Then, the tensor obtained from the fourth bi-branch fusion operation is nonlinearly fitted using the ReLU activation function;

[0047] Step 42: Perform the fifth bi-branch fusion operation. The isometric projection feature map obtained from the first nonlinear fitting in Step 8 is denoted as F1, with a tensor size of [4,32,128,256]. The cube projection feature map obtained from the first nonlinear fitting in step 21 is transformed into an isometric projection feature map, denoted as F2, with a tensor size of [4, 32, 128, 128]. F1 and F2 are concatenated to obtain a preliminary fused feature map F3. F3 is then subjected to a first convolution operation with a kernel size of [1, 1]. A second convolution operation is then performed on F3 with a kernel size of [3, 3] to generate feature map F4. F4 is then added to F2 pixel by pixel to obtain feature map F5. Finally, F5 and F1 are fused using the AFF module to obtain a fused feature map F6. F6 is then added to F5 pixel by pixel using the CBAM module to obtain the final feature map F7. The output tensor is [4, 16, 256, 512].

[0048] Step 43: Use the BatchNorm function to perform batch normalization on the tensors obtained from the fifth two-branch fusion operation;

[0049] Step 44: Then, the tensor obtained from the fifth bi-branch fusion operation is nonlinearly fitted using the ReLU activation function;

[0050] Step 45: Perform the first upsampling operation. Use subpixel convolution to deconvolve the fused feature map obtained from the first dual-branch fusion in Step 32. The input tensor is [4, 256, 16, 32], the kernel size is [3, 3], the batch size of Bs is 4, the stride of St is 1, and the input padding of Pa is 1; the output tensor size is [4, 128, 32, 64].

[0051] Step 46: Perform a second upsampling operation using subpixel convolution. The fused feature tensor [4, 128, 32, 64] obtained from the second dual-branch fusion in Step 35 and the output tensor [4, 128, 32, 64] obtained from the first upsampling operation in Step 45 are deconvolved using subpixel convolution. The kernel size is [3, 3], the batch size Bs is 4, the stride St is 1, the input padding Pa is 1, and the output tensor size is [4, 64, 64, 128].

[0052] Step 47: Perform a third upsampling operation using subpixel convolution. The fused feature tensor [4, 128, 32, 64] obtained from the third bi-branch fusion in Step 38 and the output tensor [4, 128, 32, 64] obtained from the second upsampling operation in Step 46 are deconvolved using subpixel convolution. The kernel size is [3, 3], the batch size (Bs) is 4, the stride (St) is 1, and the input padding (Pa) is 1. The output tensor size is [4, 32, 128, 256].

[0053] Step 48: Perform the fourth upsampling operation using subpixel convolution. The fused feature tensor [4, 128, 32, 64] obtained from the fourth bi-branch fusion in Step 41 and the output tensor [4, 128, 32, 64] obtained from the third upsampling operation in Step 47 are deconvolved using subpixel convolution. The kernel size is [3, 3], the batch size (Bs) is 4, the stride (St) is 1, and the input padding (Pa) is 1. The output tensor size is [4, 16, 256, 512].

[0054] Step 49: Perform the fifth upsampling operation using subpixel convolution. Use subpixel convolution to deconvolve the fused feature tensor [4,16,256,512] obtained from the fifth dual-branch fusion in Step 44 and the output tensor [4,16,256,512] obtained from the fourth upsampling operation in Step 48. The kernel size is [3,3], the batch size of Bs is 4, the stride of St is 1, and the input padding of Pa is 1. The output tensor size after dimensionality reduction is [4,1,512,1024].

[0055] Step 50: The selected loss function L is the improved scale-invariant loss. for , For the true value, For predicted values, It is 0.85. For the sample size, Set to a fixed value of 10;

[0056]

[0057]

[0058] Step 51: To improve the robustness of the algorithm and enable it to adapt to the influence of additional factors such as different brightness and object size, the algorithm uses the following three data augmentation methods: ① Color transformation: The contrast, saturation, hue, and brightness of the input image in the dataset are randomly transformed with a 50% probability, and the transformation range is set to 0.8 to 1.2; ② Horizontal flip: The input image and the corresponding ground truth image in the dataset are randomly flipped horizontally with a 50% probability; ③ Random yaw rotation: The input image and the corresponding depth ground truth image are moved by a random length along axis=1.

[0059] Step 52: Select Adam with default parameters as the optimizer for the training process, and use a constant learning rate of 0.00001. In the experiment, the dilation rate of the dilated convolution is set to 1×1, 1×2, 1×4, and 2×1. For the two real-world datasets, train for 100 epochs each, with the batch size set to 3. For the virtual-world dataset, train for 30 epochs, also with the batch size set to 3.

[0060] Step 53: In the algorithm, four evaluation metrics that are widely used in previous similar tasks are used to quantitatively evaluate the model algorithm, including mean absolute error, root mean square error, log mean square error, absolute relative error, and percentage of accuracy at three thresholds. The calculation formulas for the above evaluation metrics are shown below.

[0061]

[0062]

[0063]

[0064]

[0065]

[0066] In the above expression This refers to the total number of pixels in the input panoramic image. and This refers to the predicted depth value and the ground truth depth value for each pixel in the input panoramic image; the accuracy percentage uses three thresholds: 1.25, 1.25, and so on. 2 and 1.25 3 The similarity between the predicted depth and the ground truth depth under three threshold constraints is calculated. By comparing the similarity, we can intuitively see the depth estimation ability of different models. The higher the similarity, the more accurate the depth map predicted by the model.

[0067] Step 54: Using the above steps, a monocular depth estimation method for panoramic images based on a dual-branch efficient fusion network can be obtained.

[0068] The positive effect of this invention is that it realizes the utilization of complete field of view information of panoramic images based on the dual-branch network structure, solves the problem of polar distortion of panoramic images, and improves the accuracy of monocular depth estimation of panoramic images.

[0069] To address the problem of not being able to fully utilize the richer visual information in panoramic images, a panoramic depth estimation algorithm based on grouped dilated convolution and channel attention is proposed. By leveraging the additional receptive field provided by grouped dilated convolution and channel attention, the algorithm extracts rich features from isometric projection panoramic images to solve the aforementioned problem.

[0070] 2. To address the problem that isometric projection of panoramic images can introduce distortion, leading to poor depth prediction results, a dual-branch fusion network for panoramic image depth estimation is proposed. This network uses different panoramic image projection forms as inputs to correct the distortion caused by the panoramic image. Attached Figure Description

[0071] Figure 1 Flowchart of a monocular depth estimation method for panoramic images based on a dual-branch efficient fusion network.

[0072] Figure 2 Dual-branch fusion module. Detailed Implementation

[0073] The invention will now be further described with reference to the accompanying drawings: Figure 1-2 As shown, the panoramic image monocular depth estimation method based on a dual-branch efficient fusion network obtains a set of image data from the Matterport3D dataset platform and inputs it into computer 1; the specific steps are as follows:

[0074] Step 1: Place 7829 training images in a folder named 'train', 947 validation images in a folder named 'val', and 2014 test point clouds for testing in a folder named 'test'.

[0075] Step 2: The isometric projection convolution operation requires the input tensor to be [Bs, Ic, H, W]. The meaning of each parameter in the input tensor is as follows: Bs represents the input batch size, Ic represents the number of channels in the input tensor, H represents the height of the input image, and W represents the width of the input image.

[0076] Step 3: Loop through the data in the 'train' folder or the 'test' folder, saving the index address of each file for later retrieval. Set the number of depth images to load each time and package them into an index for input into the network. Merge the index addresses stored in the depth images with the index generated from the packaged files, converting the resulting data into tensor form. The images are divided into isometric projection images and cube projection images.

[0077] Step 4: After the isometric projection image enters the neural network, since the model inputs 4 images at a time; the images are RGB images with 3 channels; and the image size is 512*1024, the resulting tensor size is [4, 3, 512, 1024]. The corresponding data is then loaded into the neural network.

[0078] Step 5: The input tensor [4, 3, 512, 1024] undergoes its first convolution operation through the ResNet18 Block module. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the stride of the convolution operation, and Pa represents the padding of the original 3D mesh space. The convolution kernel used in the convolution operation is a two-dimensional matrix, and the values ​​of all elements in the matrix are randomly generated, ranging from [-1, 1]. In this invention, the input tensor is [4, 3, 512, 1024], Bs is the input batch size of 4, Ks is the kernel size of (7, 7), St is the stride of 2, and Pa is the padding of the input of 3. After the first convolution operation, the tensor size is [4, 64, 256, 512].

[0079] Step 6: Use the BatchNorm function to perform the first batch normalization on the tensors obtained from the first convolution operation of the isometric projection image. This makes the input distribution of each neuron more stable, allowing for faster convergence during training. It also improves the model's generalization ability and suppresses overfitting, thereby improving the accuracy and reliability of the deep learning model.

[0080] Step 7: Then, the ReLU activation function is used to perform the first nonlinear fitting on the tensor obtained from the first ResNet18 Block operation of the isometric projection image.

[0081] Step 8: Next, Maxpooling is applied. The input tensor size is [4, 64, 256, 512], the BS batch size is 4, the Ks kernel size is [3, 3], the St stride is 2, the Pa padding is 1, and the output channels are set to 32. The most representative information is extracted from the input feature map, and the size of the feature map is reduced, resulting in an output tensor size of [4, 32, 128, 256].

[0082] Step 9: Input the tensor [4,32,128,256] obtained from the max pooling layer into the first ECD Block. The parameters in the ECD Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B,C,H,W], keep B constant and split C into 4 parts on average. Perform convolution calculation on the input [B,C1,H,W] using 4 dilation convolutions with different dilation coefficients, where C1 is C divided by 4. Output the four parts [B,C1,H,W]. Pass the four parts [B,C1,H,W] through a global average pooling layer to obtain an output tensor of size [B,C,H,W]. Then pass it through 4 fast one-dimensional convolutional layers with a kernel size of 3 to obtain four outputs, with an output tensor of size [B,C1,H,W]. The four outputs are then multiplied by the initial [B,C1,H,W] using the Sigmoid function, ending the first ECD Block calculation. The output tensor size is [4, 32, 128, 256].

[0083] Step 10: Then, the ReLU activation function is used to perform a second nonlinear fitting on the tensor obtained from the first ECD Block operation of the isometric projection image.

[0084] Step 11: Input the tensor [4, 32, 128, 256] obtained from the second nonlinear fitting into the second ECDBlock module. The parameters in the ECD Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B, C, H, W], keep B constant and split C into 4 parts on average. Perform convolution calculation on the input [B, C1, H, W] using 4 dilation convolutions with different dilation coefficients, where C1 is C divided by 4. Output the four parts [B, C1, H, W]. Pass the four parts [B, C1, H, W] through a global average pooling layer to obtain an output tensor of size [B, C, H, W]. Then pass it through 4 fast one-dimensional convolutional layers with a kernel size of 3 to obtain four outputs, with an output tensor of size [B, C1, H, W]. The four outputs are then multiplied by the original [B,C1,H,W] using the Sigmoid function, ending the second ECD Block calculation. The output tensor size is [4, 16, 64, 128].

[0085] Step 12: Then, the ReLU activation function is used to perform a third nonlinear fitting on the tensor obtained from the second ECD Block operation of the isometric projection image.

[0086] Step 13: Input the tensor [4, 16, 64, 128] obtained from the third nonlinear fitting into the third ECD Block module. The parameters in the ECD Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B, C, H, W], keep B constant and split C into 4 parts on average. Perform convolution calculation on the input [B, C1, H, W] using 4 dilation convolutions with different dilation coefficients, where C1 is C divided by 4. Output the four parts [B, C1, H, W]. Pass the four parts [B, C1, H, W] through a global average pooling layer to obtain an output tensor of size [B, C, H, W]. Then pass it through 4 fast one-dimensional convolutional layers with a kernel size of 3 to obtain four outputs, with an output tensor of size [B, C1, H, W]. The four outputs are then multiplied by the original [B,C1,H,W] using the Sigmoid function, ending the third ECD Block calculation. The output tensor size is [4, 8, 32, 64].

[0087] Step 14: Then, the ReLU activation function is used to perform a fourth nonlinear fitting on the tensor obtained from the third ECD Block operation of the isometric projection image.

[0088] Step 15: Input the tensor [4, 8, 32, 64] obtained from the fourth nonlinear fitting into the fourth ECD Block module. The parameters in the ECD Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B, C, H, W], keep B unchanged and split C into 4 parts on average. Perform convolution calculation on the input [B, C1, H, W] using 4 dilation convolutions with different dilation coefficients, where C1 is C divided by 4. Output the four parts [B, C1, H, W]. Pass the four parts [B, C1, H, W] through a global average pooling layer to obtain an output tensor of size [B, C, H, W]. Then pass it through 4 fast one-dimensional convolutional layers with a kernel size of 3 to obtain four outputs, with an output tensor of size [B, C1, H, W]. The four outputs are then multiplied by the original [B,C1,H,W] using the Sigmoid function, ending the fourth ECD Block calculation. The output tensor size is [4, 4, 16, 32].

[0089] Step 16: Then, the ReLU activation function is used to perform a fifth nonlinear fitting on the tensor obtained from the fourth ECD Block operation of the isometric projection image.

[0090] Step 17: After the cube projection image is input into the neural network, since the model inputs 4 images at a time, each image has 6 faces of a cube; the image is an RGB image with 3 channels; the size of each face of the image is 256*256. Therefore, the size of the tensor obtained is [4, 3, 256, 256, 6]. The corresponding data is then loaded into the neural network.

[0091] Step 18: The input tensor [4, 3, 256, 256, 6] undergoes its first convolution operation through the first ResNet18 Block module. The parameters are set as follows: Bs input batch size is 4, Ks convolution kernel size is (3, 3), St stride is 2, and Pa padding is 1. After the first convolution operation, the tensor size is [4, 64, 256, 256].

[0092] Step 19: Use the BatchNorm function to perform the first batch normalization on the tensors obtained from the first convolution operation of the cube projection image. This makes the input distribution of each neuron more stable, allowing for faster convergence during training. It also improves the generalization ability of the model and suppresses overfitting, thereby improving the accuracy and reliability of the deep learning model.

[0093] Step 20: Then, the ReLU activation function is used to perform the first nonlinear fitting on the tensor obtained from the first ResNet18 Block operation on the cube projection image;

[0094] Step 21: Next, Maxpooling is applied with an input tensor size of [4, 64, 256, 256], a batch size of 4 (Bs), a kernel size of [3, 3] (Ks), a stride of 2 (St), an input padding of 1 (Pa), and an output channel of 32. The most representative information is extracted from the input feature map, and the feature map size is reduced, resulting in an output tensor size of [4, 32, 128, 128].

[0095] Step 22: Input the tensor [4,32,128,128] obtained from the first nonlinear fitting into the second ResNet18Block module. The parameters in the ResNet18 Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B,C,H,W], it first undergoes a convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and input padding of 1 (Pa). Then, it is normalized using the BatchNorm function. Next, it undergoes a second convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and input padding of 1 (Pa). Then, it is normalized again using the BatchNorm function. Finally, it undergoes a third convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and input padding of 1 (Pa). Finally, it is normalized again using the BatchNorm function. Then, after the fourth convolution operation, the kernel size is [3, 3], the batch size is 4, the stride is 1, and the input padding is 1. Normalization is then performed using the BatchNorm function. This completes the first ResNet18 Block computation, with an output tensor size of [4, 32, 128, 128].

[0096] Step 23: Then, the ReLU activation function is used to perform a second nonlinear fitting on the tensor obtained from the second ResNet18 Block operation on the cube projection image;

[0097] Step 24: Input the tensor [4,32,128,128] obtained from the second nonlinear fitting into the third ResNet18Block module. The parameters in the ResNet18 Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B,C,H,W], it first undergoes a convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 2 (St), and a padding input of 1 (Pa). Then, it is normalized using the BatchNorm function. Next, it undergoes a second convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and a padding input of 1 (Pa). Then, it is normalized again using the BatchNorm function. Finally, it undergoes a third convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and a padding input of 1 (Pa). Finally, it is normalized again using the BatchNorm function. Next, a fourth convolution operation is performed with a kernel size of [3, 3], a batch size of 4 (Bs), a stride of 1 (St), and an input padding of 1 (Pa). Normalization is then performed using the BatchNorm function. This completes the second ResNet18 Block computation, resulting in an output tensor size of [4, 16, 64, 64].

[0098] Step 25: Then, the ReLU activation function is used to perform a third nonlinear fitting on the tensor obtained from the third ResNet18 Block operation on the cube projection image;

[0099] Step 26: Input the tensor [4,16,64,64] obtained from the third nonlinear fitting into the fourth ResNet18Block module. The parameters in the ResNet18 Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B,C,H,W], it first undergoes a convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 2 (St), and a padding input of 1 (Pa). Then, it is normalized using the BatchNorm function. Next, it undergoes a second convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and a padding input of 1 (Pa). Then, it is normalized again using the BatchNorm function. Finally, it undergoes a third convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and a padding input of 1 (Pa). Finally, it is normalized again using the BatchNorm function. Next, a fourth convolution operation is performed with a kernel size of [3, 3], a batch size of 4 (Bs), a stride of 1 (St), and an input padding of 1 (Pa). Normalization is then performed using the BatchNorm function. This completes the third ResNet18 Block computation, resulting in an output tensor size of [4, 8, 32, 32].

[0100] Step 27: Then, the ReLU activation function is used to perform a fourth nonlinear fitting on the tensor obtained from the fourth ResNet18 Block operation of the cube projection image;

[0101] Step 28: Input the tensor [4,8,32,32] obtained from the fourth nonlinear fitting into the fifth ResNet18Block module. The parameters in the ResNet18 Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B,C,H,W], it first undergoes a convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 2 (St), and a padding input of 1 (Pa). Then, it is normalized using the BatchNorm function. Next, it undergoes a second convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and a padding input of 1 (Pa). Then, it is normalized again using the BatchNorm function. Finally, it undergoes a third convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and a padding input of 1 (Pa). Finally, it is normalized again using the BatchNorm function. Next, a fourth convolution operation is performed with a kernel size of [3, 3], a batch size of 4 (Bs), a stride of 1 (St), and an input padding of 1 (Pa). Normalization is then performed using the BatchNorm function. This completes the third ResNet18 Block computation, resulting in an output tensor size of [4, 4, 16, 16].

[0102] Step 29: Then, the ReLU activation function is used to perform a fifth nonlinear fitting on the tensor obtained from the fifth ResNet18 Block operation on the cube projection image.

[0103] Step 30: Perform the first two-branch fusion operation. The isometric projection feature map from the fifth nonlinear fitting in Step 16 is denoted as F1, with a tensor size of [4,4,16,16]. The cube projection feature map from the fifth nonlinear fitting in Step 29 is transformed into an isometric projection feature map, denoted as F2, with a tensor size of [4,4,16,32], through a C2E operation. F1 and F2 are concatenated to obtain a preliminary fused feature map F3. F3 undergoes a first convolution operation with a kernel size of [1,1], followed by a second convolution operation with a kernel size of [3,3], generating feature map F4. F4 is then pixel-wise added to F2 to obtain feature map F5. Finally, F5 and F1 are fused using the AFF module to obtain a fused feature map F6. F6 is then pixel-wise added to F5 using the CBAM module to obtain the final feature map F7, with an output tensor of [4,256,16,32].

[0104] Step 31: Use the BatchNorm function to perform the first batch normalization on the tensors obtained from the first two-branch fusion operation.

[0105] Step 32: Then, the ReLU activation function is used to perform the first nonlinear fitting on the tensor obtained from the first bi-branch fusion operation.

[0106] Step 33: Perform the second two-branch fusion operation. The isometric projection feature map from the fourth nonlinear fitting in Step 14 is denoted as F1, with a tensor size of [4, 8, 32, 64]. The cube projection feature map from the fourth nonlinear fitting in Step 27 is transformed into an isometric projection feature map, denoted as F2, with a tensor size of [4, 8, 32, 32], through a C2E operation. F1 and F2 are concatenated to obtain a preliminary fused feature map F3. F3 undergoes a first convolution operation with a kernel size of [1, 1], followed by a second convolution operation with a kernel size of [3, 3], generating feature map F4. F4 is then pixel-wise added to F2 to obtain feature map F5. Finally, F5 and F1 are fused using the AFF module to obtain a fused feature map F6. F6 is then pixel-wise added to F5 using the CBAM module to obtain the final feature map F7, with an output tensor of [4, 128, 32, 64].

[0107] Step 34: Use the BatchNorm function to perform batch normalization on the tensors obtained from the second two-branch fusion operation.

[0108] Step 35: Then, the tensor obtained from the second bi-branch fusion operation is nonlinearly fitted using the ReLU activation function.

[0109] Step 36: Perform the third two-branch fusion operation. The isometric projection feature map obtained from the third nonlinear fitting in Step 12 is denoted as F1, with a tensor size of [4, 16, 64, 128]. The cube projection feature map obtained from the third nonlinear fitting in Step 25 is transformed into an isometric projection feature map, denoted as F2, with a tensor size of [4, 16, 64, 64], through a C2E operation. F1 and F2 are concatenated to obtain a preliminary fused feature map F3. F3 undergoes a first convolution operation with a kernel size of [1, 1], followed by a second convolution operation with a kernel size of [3, 3], generating feature map F4. F4 is then pixel-wise added to F2 to obtain feature map F5. Finally, F5 and F1 are fused using the AFF module to obtain a fused feature map F6. F6 is then pixel-wise added to F5 using the CBAM module to obtain the final feature map F7, with an output tensor of [4, 64, 64, 128].

[0110] Step 37: Use the BatchNorm function to perform batch normalization on the tensors obtained from the third bi-branch fusion operation.

[0111] Step 38: Then, the tensor obtained from the third bi-branch fusion operation is nonlinearly fitted using the ReLU activation function.

[0112] Step 39: Perform the fourth two-branch fusion operation. The isometric projection feature map obtained from the second nonlinear fitting in Step 10 is denoted as F1, with a tensor size of [4, 32, 128, 256]. The cube projection feature map obtained from the second nonlinear fitting in Step 23 is transformed into an isometric projection feature map, denoted as F2, with a tensor size of [4, 32, 128, 128], through a C2E operation. F1 and F2 are concatenated to obtain a preliminary fused feature map F3. F3 undergoes a first convolution operation with a kernel size of [1, 1], followed by a second convolution operation with a kernel size of [3, 3], generating feature map F4. F4 is then pixel-wise added to F2 to obtain feature map F5. Finally, F5 and F1 are fused using the AFF module to obtain a fused feature map F6. F6 is then pixel-wise added to F5 using the CBAM module to obtain the final feature map F7, with an output tensor of [4, 32, 128, 256].

[0113] Step 40: Use the BatchNorm function to perform batch normalization on the tensors obtained from the fourth bi-branch fusion operation.

[0114] Step 41: Then, use the ReLU activation function to perform nonlinear fitting on the tensor obtained from the fourth bi-branch fusion operation.

[0115] Step 42: Perform the fifth two-branch fusion operation. The isometric projection feature map obtained from the first nonlinear fitting in Step 8 is denoted as F1, with a tensor size of [4, 32, 128, 256]. The cube projection feature map obtained from the first nonlinear fitting in Step 21 is transformed into an isometric projection feature map, denoted as F2, with a tensor size of [4, 32, 128, 128], through a C2E operation. F1 and F2 are concatenated to obtain a preliminary fused feature map F3. F3 undergoes a first convolution operation with a kernel size of [1, 1], followed by a second convolution operation with a kernel size of [3, 3], generating feature map F4. F4 is then pixel-wise added to F2 to obtain feature map F5. Finally, F5 and F1 are fused using the AFF module to obtain a fused feature map F6. F6 is then pixel-wise added to F5 using the CBAM module to obtain the final feature map F7, with an output tensor of [4, 16, 256, 512].

[0116] Step 43: Use the BatchNorm function to perform batch normalization on the tensors obtained from the fifth bi-branch fusion operation.

[0117] Step 44: Then, use the ReLU activation function to perform nonlinear fitting on the tensor obtained from the fifth bi-branch fusion operation.

[0118] Step 45: Perform the first upsampling operation. Use subpixel convolution to deconvolve the fused feature map obtained from the first dual-branch fusion in Step 32. The input tensor is [4, 256, 16, 32], the kernel size is [3, 3], the batch size is 4, the stride is 1, and the padding input is 1. The output tensor size is [4, 128, 32, 64].

[0119] Step 46: Perform a second upsampling operation using subpixel convolution. The fused feature tensor [4, 128, 32, 64] obtained from the second bi-branch fusion in Step 35 and the output tensor [4, 128, 32, 64] obtained from the first upsampling operation in Step 45 are deconvolved using subpixel convolution. The kernel size is [3, 3], the batch size (Bs) is 4, the stride (St) is 1, and the input padding (Pa) is 1. The output tensor size is [4, 64, 64, 128].

[0120] Step 47: Perform a third upsampling operation using subpixel convolution. The fused feature tensor [4, 128, 32, 64] obtained from the third bi-branch fusion in Step 38 and the output tensor [4, 128, 32, 64] obtained from the second upsampling operation in Step 46 are deconvolved using subpixel convolution. The kernel size is [3, 3], the batch size (Bs) is 4, the stride (St) is 1, and the input padding (Pa) is 1. The output tensor size is [4, 32, 128, 256].

[0121] Step 48: Perform the fourth upsampling operation using subpixel convolution. The fused feature tensor [4, 128, 32, 64] obtained from the fourth bi-branch fusion in Step 41 and the output tensor [4, 128, 32, 64] obtained from the third upsampling operation in Step 47 are deconvolved using subpixel convolution. The kernel size is [3, 3], the batch size (Bs) is 4, the stride (St) is 1, and the input padding (Pa) is 1. The output tensor size is [4, 16, 256, 512].

[0122] Step 49: Perform the fifth upsampling operation using subpixel convolution. The fused feature tensor [4, 16, 256, 512] obtained from the fifth bi-branch fusion in Step 44 and the output tensor [4, 16, 256, 512] obtained from the fourth upsampling operation in Step 48 are deconvolved using subpixel convolution. The kernel size is [3, 3], the batch size (Bs) is 4, the stride (St) is 1, and the input padding (Pa) is 1. The output tensor size after dimensionality reduction is [4, 1, 512, 1024].

[0123] Step 50: The loss function selected in this invention is an improved scale-invariant loss. for , It is 0.85. Set to a fixed value of 10.

[0124]

[0125]

[0126]

[0127] Step 51: To improve the robustness of the algorithm and enable it to adapt to the influence of additional factors such as different brightness and object size, the algorithm uses the following three data augmentation methods: ① Color transformation: With a 50% probability, the contrast, saturation, hue, and brightness of the input image in the dataset are randomly transformed, and the transformation range is set to 0.8 to 1.2; ② Horizontal flip: With a 50% probability, the input image and the corresponding ground truth image in the dataset are randomly flipped horizontally; ③ Random yaw rotation: The input image and the corresponding depth ground truth image are moved along the axis axis=1 by a random length.

[0128] Step 52: Select Adam with default parameters as the optimizer for the training process, and use a constant learning rate of 0.00001. In the experiment, the dilation rates of the dilated convolutions are set to 1×1, 1×2, 1×4, and 2×1. For the two real-world datasets, train for 100 epochs each, with a batch size of 3. For the virtual-world dataset, train for 30 epochs, also with a batch size of 3.

[0129] Step 53: In the algorithm, four evaluation metrics that are widely used in previous similar tasks are used to quantitatively evaluate the model algorithm, including mean absolute error, root mean square error, log mean square error, absolute relative error, and percentage of accuracy at three thresholds. The calculation formulas for the above evaluation metrics are shown below.

[0130]

[0131]

[0132]

[0133]

[0134]

[0135] In the above expression This refers to the total number of pixels in the input panoramic image. and This refers to the predicted depth value and the ground truth depth value for each pixel in the input panoramic image; the accuracy percentage uses three thresholds: 1.25, 1.25, and so on.2 and 1.25 3 The similarity between the predicted depth and the ground truth depth under three threshold constraints is calculated. By comparing the similarity, we can intuitively see the depth estimation ability of different models. The higher the similarity, the more accurate the depth map predicted by the model.

[0136] Step 54: Using the above steps, a monocular depth estimation method for panoramic images based on a dual-branch efficient fusion network can be obtained.

Claims

1. A monocular depth estimation method for panoramic images based on a dual-branch efficient fusion network, characterized in that: Obtain a set of image data from the Matterport3D dataset platform and import it into the computer; the specific steps are as follows: Step 1: Place 7829 training images in a folder named 'train', 947 validation images in a folder named 'val', and 2014 test point clouds for testing in a folder named 'test'. Step 2: The isometric projection convolution operation requires the input tensor to be [Bs, Ic, H, W]. The meaning of each parameter in the input tensor is as follows: Bs represents the input batch size, Ic represents the number of channels in the input tensor, H represents the height of the input image, and W represents the width of the input image. Step 3: Loop through the data in the 'train' folder or the data in the 'test' folder, and save the index address of each file for easy retrieval later; Configure the program to load a number of depth images each time and package these depth images to generate an index for input into the network; The index address stored in the depth image and the index generated by the packaged file are merged to convert the resulting data into tensor form, where the image is divided into isometric projection image and cube projection image. Step 4: After the isometric projection image enters the neural network, since the model inputs 4 images at a time; the images are RGB images with 3 channels; the image size is 512*1024; therefore, the tensor size is [4, 3, 512, 1024], and the corresponding data is loaded into the neural network. Step 5: The input tensor [4, 3, 512, 1024] undergoes the first convolution operation through the ResNet18 Block module. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the stride of the convolution operation, Pa represents the padding of the original 3D mesh space, the convolution kernel used in the convolution operation is a two-dimensional matrix, the values ​​of all elements in the matrix are randomly generated, and the range is [-1, 1], Bs is the input batch size of 4, Ks is the convolution kernel size of (7, 7), St is the stride of 2, Pa is the input of the padding of 3, and the tensor size obtained after the first convolution operation is [4, 64, 256, 512]. Step 6: Use the BatchNorm function to perform the first batch normalization on the tensor obtained from the first convolution operation of the isometric projection image; Step 7: Then, the ReLU activation function is used to perform the first nonlinear fitting on the tensor obtained from the first ResNet18 Block operation of the isometric projection image; Step 8: Next, Maxpooling is used with input tensor size [4, 64, 256, 512], Bs batch size of 4, Ks kernel size of [3, 3], St stride of 2, Pa padding of 1 input, and output channel set to 32. The most representative information is extracted from the input feature map, and the size of the feature map is reduced. The output tensor size is [4, 32, 128, 256]. Step 9: Input the tensor [4,32,128,256] obtained from the max pooling layer into the first ECD Block. The parameters in ECDBlock are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B,C,H,W], keep B unchanged and split C into 4 parts on average. Perform convolution calculation on the input [B,C1,H,W] using 4 dilation convolutions with different dilation coefficients, where C1 is C divided by 4, outputting four parts [B,C1,H,W]. Pass the four parts of input [B,C1,H,W] through a global average pooling layer to obtain an output tensor of size [B,C,H,W]. Then pass it through 4 fast one-dimensional convolutional layers with a kernel size of 3 to obtain four outputs of size [B,C1,H,W]. Finally, multiply the four outputs by the original [B,C1,H,W] using the sigmoid function, ending the first ECD operation. Block computation outputs tensor sizes of [4, 32, 128, 256]; Step 10: Then, the ReLU activation function is used to perform a second nonlinear fitting on the tensor obtained from the first ECD Block operation of the isometric projection image; Step 11: Input the tensor [4, 32, 128, 256] obtained from the second nonlinear fitting into the second ECD Block module. The parameters in the ECD Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B, C, H, W], keep B unchanged and split C into 4 parts on average. Perform convolution calculation on the input [B, C1, H, W] using 4 dilation convolutions with different dilation coefficients, where C1 is C divided by 4, outputting four parts [B, C1, H, W]. Pass the four parts of input [B, C1, H, W] through a global average pooling layer to obtain an output tensor of size four [B, C, H, W]. Then pass it through 4 fast one-dimensional convolutional layers with a kernel size of 3 to obtain four outputs of size [B, C1, H, W]. Finally, multiply the four outputs by the original [B, C1, H, W] using the Sigmoid function to end the second ECD. Block computation outputs tensor sizes of [4, 16, 64, 128]; Step 12: Then, the ReLU activation function is used to perform a third nonlinear fitting on the tensor obtained from the second ECD Block operation of the isometric projection image; Step 13: Input the tensor [4, 16, 64, 128] obtained from the third nonlinear fitting into the third ECD Block module. The parameters in the ECD Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B, C, H, W], keep B unchanged, and split C into 4 parts on average. Perform convolution calculation on the input [B, C1, H, W] using 4 dilation convolutions with different dilation coefficients, where C1 is C divided by 4, outputting four parts [B, C1, H, W]. Pass the four parts of input [B, C1, H, W] through a global average pooling layer to obtain an output tensor of size [B, C, H, W]. Then pass it through 4 fast one-dimensional convolutional layers with a kernel size of 3 to obtain four outputs of size [B, C1, H, W]. Finally, multiply the four outputs by the original [B, C1, H, W] using the Sigmoid function, ending the third ECD step. Block computation outputs tensor sizes of [4, 8, 32, 64]; Step 14: Then, the ReLU activation function is used to perform a fourth nonlinear fitting on the tensor obtained from the third ECD Block operation of the isometric projection image; Step 15: Input the tensor [4, 8, 32, 64] obtained from the fourth nonlinear fitting into the fourth ECD Block module. The parameters in the ECD Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B, C, H, W], keep B unchanged and split C into 4 parts on average. Perform convolution calculation on the input [B, C1, H, W] using 4 dilation convolutions with different dilation coefficients, where C1 is C divided by 4, outputting four parts [B, C1, H, W]. Pass the four parts of input [B, C1, H, W] through a global average pooling layer to obtain an output tensor of size [B, C, H, W]. Then pass it through 4 fast one-dimensional convolutional layers with a kernel size of 3 to obtain four outputs of size [B, C1, H, W]. Finally, multiply the four outputs by the original [B, C1, H, W] using the Sigmoid function to end the fourth ECD. The block computation outputs a tensor of size [4, 4, 16, 32]. Step 16: Then, the ReLU activation function is used to perform a fifth nonlinear fitting on the tensor obtained from the fourth ECD Block operation of the isometric projection image; Step 17: After the cube projection image is input into the neural network, since the model inputs 4 images at a time, each image has 6 faces of a cube; the image is an RGB image with 3 channels; the size of each face of the image is 256*256; therefore, the size of the tensor is [4, 3, 256, 256, 6], and the corresponding data is loaded into the neural network. Step 18: The input tensor [4, 3, 256, 256, 6] undergoes the first convolution operation through the first ResNet18 Block module. The parameters are set as follows: Bs input batch size is 4, Ks convolution kernel size is (3, 3), St stride is 2, Pa padding input is 1. After the first convolution operation, the tensor size is [4, 64, 256, 256]. Step 19: Use the BatchNorm function to perform the first batch normalization on the tensors obtained from the first convolution operation of the cube projection image; Step 20: Then, the ReLU activation function is used to perform the first nonlinear fitting on the tensor obtained from the first ResNet18 Block operation on the cube projection image; Step 21: Next, Maxpooling is used with input tensor size [4, 64, 256, 256], Bs batch size of 4, Ks kernel size of [3, 3], St stride of 2, Pa padding of 1 input, and output channel set to 32. The most representative information is extracted from the input feature map, and the size of the feature map is reduced. The output tensor size is [4, 32, 128, 128]. Step 22: Input the tensor [4,32,128,128] obtained from the first nonlinear fitting into the second ResNet18Block module. The parameters in the ResNet18 Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B,C,H,W], it first undergoes a convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and input padding of 1 (Pa). Then, it is normalized using the BatchNorm function. Next, it undergoes a second convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and input padding of 1 (Pa). Normalization is performed using the BatchNorm function; then, a third convolution operation is performed with a kernel size of [3, 3], a batch size of 4 (Bs), a stride of 1 (St), and an input padding of 1 (Pa). Normalization is then performed again using the BatchNorm function. Next, a fourth convolution operation is performed with a kernel size of [3, 3], a batch size of 4 (Bs), a stride of 1 (St), and an input padding of 1 (Pa). Normalization is then performed again using the BatchNorm function, ending the first ResNet18 Block computation. The output tensor size is [4, 32, 128, 128]. Step 23: Then, the ReLU activation function is used to perform a second nonlinear fitting on the tensor obtained from the second ResNet18 Block operation on the cube projection image; Step 24: Input the tensor [4,32,128,128] obtained from the second nonlinear fitting into the third ResNet18Block module. The parameters in the ResNet18 Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B,C,H,W], it first undergoes a convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 2 (St), and input padding of 1 (Pa). Then, it is normalized using the BatchNorm function, followed by a second convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and input padding of 1 (Pa). Normalization is performed using the BatchNorm function; then, a third convolution operation is performed with a kernel size of [3, 3], a batch size of 4 (Bs), a stride of 1 (St), and an input padding of 1 (Pa), followed by normalization using the BatchNorm function; then a fourth convolution operation is performed with a kernel size of [3, 3], a batch size of 4 (Bs), a stride of 1 (St), and an input padding of 1 (Pa), followed by normalization using the BatchNorm function; finally, the second ResNet18 Block computation ends, and the output tensor size is [4, 16, 64, 64]. Step 25: Then, the ReLU activation function is used to perform a third nonlinear fitting on the tensor obtained from the third ResNet18 Block operation on the cube projection image; Step 26: Input the tensor [4,16,64,64] obtained from the third nonlinear fitting into the fourth ResNet18 Block. The parameters in the ResNet18 Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B,C,H,W], it first undergoes a convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 2 (St), and input padding of 1 (Pa). Then, it is normalized using the BatchNorm function. Next, it undergoes a second convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and input padding of 1 (Pa). Normalization is performed using the BatchNorm function; then, a third convolution operation is performed with a kernel size of [3, 3], a batch size of 4 (Bs), a stride of 1 (St), and an input padding of 1 (Pa), followed by normalization using the BatchNorm function; then a fourth convolution operation is performed with a kernel size of [3, 3], a batch size of 4 (Bs), a stride of 1 (St), and an input padding of 1 (Pa), followed by normalization using the BatchNorm function; finally, the third ResNet18 Block computation ends, and the output tensor size is [4, 8, 32, 32]. Step 27: Then, the ReLU activation function is used to perform a fourth nonlinear fitting on the tensor obtained from the fourth ResNet18 Block operation on the cube projection image; Step 28: Input the tensor [4,8,32,32] obtained from the fourth nonlinear fitting into the fifth ResNet18 Block. The parameters in the ResNet18 Block are as follows: B is the input batch size, C is the number of channels, H is the image height, and W is the image width. For a given input [B,C,H,W], it first undergoes a convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 2 (St), and input padding of 1 (Pa). Then, it is normalized using the BatchNorm function. Next, it undergoes a second convolution operation with a kernel size of [3,3], a batch size of 4 (Bs), a stride of 1 (St), and input padding of 1 (Pa). Normalization is performed using the BatchNorm function; then, a third convolution operation is performed with a kernel size of [3, 3], a batch size of 4 (Bs), a stride of 1 (St), and an input padding of 1 (Pa). Normalization is then performed again using the BatchNorm function. A fourth convolution operation is then performed with the same kernel size of [3, 3], a batch size of 4 (Bs), a stride of 1 (St), and an input padding of 1 (Pa). Normalization is then performed again using the BatchNorm function. The third ResNet18 Block computation ends, and the output tensor size is [4, 4, 16, 16]. Step 29: Then, the ReLU activation function is used to perform a fifth nonlinear fitting on the tensor obtained from the fifth ResNet18 Block operation on the cube projection image; Step 30: Perform the first two-branch fusion operation. The isometric projection feature map from the fifth nonlinear fitting in Step 16 is denoted as F1, with a tensor size of [4,4,16,16]. The cube projection feature map from the fifth nonlinear fitting in Step 29 is transformed into an isometric projection feature map, denoted as F2, with a tensor size of [4,4,16,32], through a C2E operation. F1 and F2 are concatenated to obtain a preliminary fused feature map F3. F3 undergoes a first convolution operation with a kernel size of [1,1], followed by a second convolution operation with a kernel size of [3,3], generating feature map F4. F4 is then pixel-wise added to F2 to obtain feature map F5. Finally, F5 and F1 are fused using the AFF module to obtain a fused feature map F6. F6 is then pixel-wise added to F5 using the CBAM module to obtain the final feature map F7, with an output tensor of [4,256,16,32]. Step 31: Use the BatchNorm function to perform the first batch normalization on the tensors obtained from the first two-branch fusion operation; Step 32: Then, the ReLU activation function is used to perform the first nonlinear fitting on the tensor obtained from the first bi-branch fusion operation; Step 33: Perform the second two-branch fusion operation. The isometric projection feature map from the fourth nonlinear fitting in Step 14, denoted as F1, has a tensor size of [4, 8, 32, 64]. The cube projection feature map from the fourth nonlinear fitting in Step 27 is transformed into an isometric projection feature map, denoted as F2, with a tensor size of [4, 8, 32, 32]. F1 and F2 are concatenated to obtain a preliminary fused feature map F3. F3 undergoes a first convolution operation with a kernel size of [1, 1], followed by a second convolution operation with a kernel size of [3, 3], generating feature map F4. F4 is then pixel-wise added to F2 to obtain feature map F5. Finally, F5 and F1 are fused using the AFF module to obtain fused feature map F6. F6 is then pixel-wise added to F5 using the CBAM module to obtain the final feature map F7, with an output tensor of [4, 128, 32, 64]. Step 34: Use the BatchNorm function to perform batch normalization on the tensors obtained from the second two-branch fusion operation; Step 35: Then, the tensor obtained from the second bi-branch fusion operation is nonlinearly fitted using the ReLU activation function; Step 36: Perform the third bi-branch fusion operation. The isometric projection feature map obtained from the third nonlinear fitting in Step 12, denoted as F1, has a tensor size of [4, 16, 64, 128]. The cube projection feature map obtained from the third nonlinear fitting in Step 25 is transformed into an isometric projection feature map, denoted as F2, with a tensor size of [4, 16, 64, 64]. F1 and F2 are concatenated to obtain a preliminary fused feature map F3. F3 undergoes a first convolution operation with a kernel size of [1, 1], followed by a second convolution operation with a kernel size of [3, 3], generating feature map F4. F4 is then pixel-wise added to F2 to obtain feature map F5. Finally, F5 and F1 are fused using the AFF module to obtain a fused feature map F6. F6 is then pixel-wise added to F5 using the CBAM module to obtain the final feature map F7, with an output tensor of [4, 64, 64, 128]. Step 37: Use the BatchNorm function to perform batch normalization on the tensors obtained from the third bi-branch fusion operation; Step 38: Then, the tensor obtained from the third bi-branch fusion operation is nonlinearly fitted using the ReLU activation function; Step 39: Perform the fourth bi-branch fusion operation. The isometric projection feature map obtained from the second nonlinear fitting in Step 10 is denoted as F1, with a tensor size of [4, 32, 128, 256]. The cube projection feature map obtained from the second nonlinear fitting in Step 23 is transformed into an isometric projection feature map, denoted as F2, with a tensor size of [4, 32, 128, 128]. F1 and F2 are concatenated to obtain the preliminary fused feature map F3. F3 performs the first convolution operation with a kernel size of [1, 1]. Then, F3 performs a second convolution operation with a kernel size of [3, 3] to generate feature map F4. F4 is then added to F2 pixel by pixel to obtain feature map F5. Finally, F5 and F1 are fused through the AFF module to obtain fused feature map F6. F6 is then added to F5 pixel by pixel through the CBAM module to obtain the final feature map F7. The output tensor is [4, 32, 128, 256]. Step 40: Use the BatchNorm function to perform batch normalization on the tensors obtained from the fourth bi-branch fusion operation; Step 41: Then, the tensor obtained from the fourth bi-branch fusion operation is nonlinearly fitted using the ReLU activation function; Step 42: Perform the fifth two-branch fusion operation. The isometric projection feature map obtained from the first nonlinear fitting in Step 8 is denoted as F1, with a tensor size of [4,32,128,256]. The cube projection feature map obtained from the first nonlinear fitting in Step 21 is transformed into an isometric projection feature map, denoted as F2, with a tensor size of [4,32,128,128]. F1 and F2 are concatenated to obtain the preliminary fused feature map F3.

3. Perform the first convolution operation with a kernel size of [1, 1]. Then perform the second convolution operation on F3 with a kernel size of [3, 3] to generate feature map F4. F4 is then added to F2 pixel by pixel to obtain feature map F5. Finally, F5 and F1 are fused through the AFF module to obtain fused feature map F6. F6 is then added to F5 pixel by pixel through the CBAM module to obtain the final feature map F7. The output tensor is [4, 16, 256, 512]. Step 43: Use the BatchNorm function to perform batch normalization on the tensors obtained from the fifth two-branch fusion operation; Step 44: Then, the tensor obtained from the fifth bi-branch fusion operation is nonlinearly fitted using the ReLU activation function; Step 45: Perform the first upsampling operation. Use subpixel convolution to deconvolve the fused feature map obtained from the first dual-branch fusion in Step 32. The input tensor is [4, 256, 16, 32], the kernel size is [3, 3], the batch size of Bs is 4, the stride of St is 1, and the input padding of Pa is 1; the output tensor size is [4, 128, 32, 64]. Step 46: Perform a second upsampling operation using subpixel convolution. The fused feature tensor [4, 128, 32, 64] obtained from the second dual-branch fusion in Step 35 and the output tensor [4, 128, 32, 64] obtained from the first upsampling operation in Step 45 are deconvolved using subpixel convolution. The kernel size is [3, 3], the batch size Bs is 4, the stride St is 1, the input padding Pa is 1, and the output tensor size is [4, 64, 64, 128]. Step 47: Perform a third upsampling operation using subpixel convolution. The fused feature tensor [4, 128, 32, 64] obtained from the third bi-branch fusion in Step 38 and the output tensor [4, 128, 32, 64] obtained from the second upsampling operation in Step 46 are deconvolved using subpixel convolution. The kernel size is [3, 3], the batch size Bs is 4, the stride St is 1, and the input padding Pa is 1. The output tensor size is [4, 32, 128, 256]. Step 48: Perform the fourth upsampling operation using subpixel convolution. Use subpixel convolution to deconvolve the fused feature tensor [4, 128, 32, 64] obtained from the fourth bi-branch fusion in Step 41 and the output tensor [4, 128, 32, 64] obtained from the third upsampling operation in Step 47. The kernel size is [3, 3], the batch size Bs is 4, the stride St is 1, and the input padding Pa is 1. The output tensor size is [4, 16, 256, 512]. Step 49: Perform the fifth upsampling operation using subpixel convolution. Use subpixel convolution to deconvolve the fused feature tensor [4,16,256,512] obtained from the fifth dual-branch fusion in Step 44 and the output tensor [4,16,256,512] obtained from the fourth upsampling operation in Step 48. The kernel size is [3,3], the batch size of Bs is 4, the stride of St is 1, and the input padding of Pa is 1. The output tensor size after dimensionality reduction is [4,1,512,1024]. Step 50: The selected loss function L is the improved scale-invariant loss. for , For the true value, For predicted values, It is 0.

85. For the sample size, Set to a fixed value of 10; ; ; Step 51: To improve the robustness of the algorithm and enable it to adapt to the influence of additional factors such as different brightness and object size, the algorithm uses the following three data augmentation methods: ① Color transformation: The contrast, saturation, hue, and brightness of the input image in the dataset are randomly transformed with a 50% probability, and the transformation range is set to 0.8 to 1.2; ② Horizontal flip: The input image and the corresponding ground truth image in the dataset are randomly flipped horizontally with a 50% probability. ③ Random yaw rotation: Move the input image and the corresponding depth ground truth image along the axis axis=1 by a random length; Step 52: Select Adam with default parameters as the optimizer for the training process and use a constant learning rate. 0.00001. In the experiment, the dilation rate of the dilated convolution was set to 1×1, 1×2, 1×4 and 2×1. For the two real-world datasets, the training was carried out for 100 rounds with the batch size set to 3. For the virtual world dataset, the training was carried out for 30 rounds with the batch size also set to 3. Step 53: In the algorithm, four evaluation metrics that are widely used in previous similar tasks are used to quantitatively evaluate the model algorithm, including mean absolute error, root mean square error, log mean square error, absolute relative error, and percentage of accuracy at three thresholds. The calculation formulas for the above evaluation metrics are shown below. ; ; ; ; ; In the above expression This refers to the total number of pixels in the input panoramic image. and This refers to the predicted depth value and the ground truth depth value for each pixel in the input panoramic image; where, The precision percentage uses three thresholds: 1.25, 1.25, and so on. 2 and 1.25 3 The similarity between the predicted depth and the ground truth depth under three threshold constraints is calculated. By comparing the similarity, we can intuitively see the depth estimation ability of different models. The higher the similarity, the more accurate the depth map predicted by the model. Using the above steps, a monocular depth estimation method for panoramic images based on a dual-branch efficient fusion network can be obtained.

Images