Panoramic image restoration method based on two-stage generative adversarial network

Abstract

The invention discloses a panoramic image restoration method based on a two-stage generative adversarial network, which realizes complete view information utilization of a panoramic image according to a two-stage generative adversarial network structure, solves the problem of two-stage distortion of the panoramic image, and improves the precision of panoramic image restoration. Aiming at the problem that a panoramic image has richer visual information and is difficult to fully utilize, the invention provides a convolution group with a jump connection structure, a context feature aggregation module and an attention gating module, and the extraction capability of a model on image features under different scales is enhanced through cavity convolution with different voidage rates. In order to solve the above problems, aiming at the problem that a panoramic image of equidistant projection causes distortion and a repairing result is not ideal, a two-stage generative adversarial network is provided, information in two formats of a cube and an equal rectangle is utilized, and feature information of a stereoscopic view angle and an overall view angle is combined, so that the repairing result is not ideal. Therefore, distortion caused by the panoramic image can be corrected.

Description

Technical Field

[0001] The present invention relates to the technical field of panoramic image three-dimensional reconstruction, and in particular to a panoramic image restoration method based on a two-stage generative adversarial network. Background Art

[0002] Image inpainting, when computers perform computer vision tasks, utilizes existing information in an image to repair damaged or missing parts, making the restored image appear visually consistent with the original. Currently, demand for panoramic image inpainting is growing, and it is widely used in fields such as panoramic photography, virtual reality, tourism, and medical image processing. However, due to varying degrees of distortion and distortion in panoramic images and their high resolution, traditional inpainting methods often struggle to achieve comprehensive and accurate restorations and may even cause memory shortages. To address these issues, a novel two-stage network architecture is designed to repair damaged panoramic images. The first stage uses a cubic panoramic image format with minimal distortion as input to restore some basic structure and shape. In the second stage, the restored cubic image is converted to an equirectangular format, where the repair of damaged areas is further refined, resulting in a more refined restoration of the panoramic image. Finally, the generated image is verified by a discriminant network for both cubic and equirectangular formats to ensure image integrity.

[0003] In recent years, panoramic image restoration has begun to demonstrate its superiority. Different convolution categories and intentional mechanisms have been introduced into the network to finely repair the defective parts of panoramic images. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the above technical defects and provide a panoramic image restoration method based on a two-stage generative adversarial network.

[0005] In order to solve the above problem, the technical solution of the present invention is that a set of image data is obtained from a 360-degree panoramic street view dataset and transmitted to a computer, comprising the following steps:

[0006] S1, encoding the public 360-degree panoramic street view dataset and the random mask, and converting the data into a cubic panorama format for processing;

[0007] S2, placing the training images in the cubic panorama format data into a folder named 'training', and placing the test images for testing into a folder named 'testing';

[0008] S3, looping through the data in the 'training' folder or the data in the 'testing' folder, saving the index address of each file for easy later search and use, setting up to load several panoramic images each time and packaging these panoramic images to generate an index input into the network, performing a merging operation using the index address stored in the panoramic image and the index generated by the packaged file, and converting the obtained panoramic image into a tensor form;

[0009] S4. The tensor size of one face of the cube format image is [B, F, 3, H, W], and the mask tensor size of one face of the cube format image is [B, F, 1, H, W];

[0010] S5. Since the cube format panoramic image has six faces, the number of channels will be multiplied sixfold when they are combined. The faces are respectively recorded as the cube panoramic image feature map F1, and the F1 tensor size is [B, F, 18, H, W]. The faces are respectively recorded as the cube mask image feature map F2, and the F2 tensor size is [B, F, 6, H, W]. The shapes of F1 and F2 are transformed and spliced to obtain a feature map F3, and the F3 tensor size is [B, 24, 256, 256].

[0011] S6. Input the F3 tensor of size [B, 24, 256, 256] in step S5 into the cube generation network, and the tensor size obtained after the two convolution operations in the first residual block is [1, 64, 256, 256];

[0012] S7, performing a first batch normalization process on the tensor obtained by the first convolution operation of the cube projection image using the batch normalization function;

[0013] S8, performing a first nonlinear fitting on the tensor obtained by the first convolution operation of the cube projection image using the linear rectification activation function;

[0014] S9, passing [B, 24, 256, 256] directly to the subsequent layers of the network through an additional convolutional layer to retain the original input information, adding it to the tensor [1, 64, 256, 256] obtained after the previous convolution through the residual connection to obtain the tensor [1, 64, 256, 256];

[0015] S10, the tensor [1, 64, 256, 256] obtained after passing through the residual block is input into the first residual convolution block, and the size of the output tensor is [1, 128, 128, 128];

[0016] S11, the tensor [1, 128, 128, 128] obtained after passing through the first residual block is input into the second residual convolution block, and the size of the output tensor is [1, 256, 64, 64];

[0017] S12, the tensor [1, 256, 64, 64] obtained after passing through the second residual block is input into the third residual convolution block, and the output tensor size is [1, 512, 32, 32];

[0018] S13, inputting the tensor [1, 512, 32, 32] obtained after passing through the third residual convolution block into the first upsampling module, and returning the upsampled tensor [1, 512, 64, 64] as the output of the module;

[0019] S14, the tensor [1, 512, 64, 64] obtained after the first upsampling module is input into the dilated spatial pyramid pooling module, the tensor does not change after passing through the dilated spatial pyramid pooling module, and the output tensor remains [1, 512, 64, 64];

[0020] S15, concatenating the output tensor [1, 256, 64, 64] obtained in step S11 and the output tensor [1, 512, 64, 64] obtained in step S14 to obtain a tensor [1, 768, 64, 64];

[0021] S16, inputting the concatenated tensor [1,768,64,64] into the fourth residual convolution block, and the size of the output tensor is [1,256,64,64];

[0022] S17, inputting the tensor [1, 256, 64, 64] obtained after passing through the fourth residual convolution block into the second upsampling module, and returning the upsampled tensor [1, 256, 128, 128] as the output of the module;

[0023] S18, concatenating the output tensor [1, 256, 128, 128] obtained in step 17 and the output tensor [1, 128, 128, 128] obtained in step 10 to obtain the tensor [1, 384, 128, 128];

[0024] S19, inputting the concatenated tensor [1, 384, 128, 128] into the fifth residual convolution block, and the size of the output tensor is [1, 128, 128, 128];

[0025] S20, inputting the tensor [1, 128, 128, 128] obtained after passing through the fifth residual convolution block into the third upsampling module, and returning the upsampled tensor [1, 128, 256, 256] as the output of the module;

[0026] S21, concatenating the output tensor [1, 128, 256, 256] obtained in step 20 and the output tensor [1, 64, 256, 256] obtained in step 6 to obtain the tensor [1, 192, 256, 256];

[0027] S22, inputting the concatenated tensor [1, 192, 256, 256] into the sixth residual convolution block, and the size of the output tensor is [1, 64, 256, 256];

[0028] S23, the tensor [1, 64, 256, 256] obtained after the sixth residual convolution block is input to the output layer module, and the size of the tensor obtained after the processing is [1, 18, 256, 256];

[0029] S24, inputting the output tensor of step S23 into the hyperbolic tangent function activation layer module for activation, and obtaining the final tensor of the cubic format generation network, wherein the size of the tensor is [1, 18, 256, 256];

[0030] S25, rearranging the output tensor in step S24 into a new shape, wherein the dimension is [1, 6, 3, 256, 256];

[0031] S26, performing calculation using a cubic panoramic image calculation formula to obtain a final restored cubic panoramic image;

[0032] S27, converting the final restored cubic panoramic image, the original cubic panoramic image, and the masked cubic image from a cubic format to an equirectangular format, so that the sizes of the converted tensors are [1, 3, 256, 512], [1, 3, 256, 512], and [1, 1, 256, 512], respectively;

[0033] S28, the panoramic image converted into an equirectangular format is input into an equirectangular format generation network for finer restoration, the input tensor size is [1, 3, 256, 512], and the images are respectively input into four residual blocks, and different features are extracted using a variety of convolution kernel sizes. Each residual block first undergoes a batch normalization, then a layer of rectangular convolution, then a linear rectification activation function, and then another layer of rectangular convolution. A skip connection is added to the output, and the output tensor size after the four residual blocks is [1, 8, 256, 512].

[0034] S29, concatenating the four tensors in step S28 to obtain a tensor size of [1, 32, 256, 512], passing the concatenated tensor through a first context feature aggregation module to obtain a final aggregated feature map;

[0035] S30, inputting the aggregated feature map in step S29 into an attention gating mechanism, multiplying the original input tensor and the attention weight to obtain a weighted input tensor;

[0036] S31, the tensor size is set to [1, 32, 256, 512], and the convolution group composed of four residual blocks is entered again, and the output tensor size after the four residual blocks are all [1, 16, 256, 512];

[0037] S32, enter the first layer of the encoder, input the output tensor in step S31 to the first layer of the encoder, and the obtained tensor size is [1, 128, 128, 256];

[0038] S33, entering the second layer of the encoder, inputting the output tensor in step S32 into the second layer of the encoder, and the obtained tensor size is [1, 256, 128, 256];

[0039] S34, entering the third layer of the encoder, inputting the output tensor in step S33 into the third layer of the encoder, and the obtained tensor size is [1, 256, 128, 256];

[0040] S35, entering the first layer of the decoder, inputting the output tensor in step S34 into the first layer of the decoder, and the obtained tensor size is [1, 128, 256, 512];

[0041] S36, enter the second layer of the decoder, input the output tensor in step S35 into the second layer of the decoder, and the obtained tensor size is [1, 128, 256, 512];

[0042] S37, entering the third layer of the decoder, inputting the output tensor in step S36 into the third layer of the decoder, and the obtained tensor size is [1, 64, 256, 512];

[0043] S38, inputting the output tensor in S37 into a second context feature aggregation module, and obtaining a final aggregated feature map;

[0044] S39, inputting the aggregated feature map in step S38 into the convolution layer, generating pixel-level prediction results, and obtaining a tensor size of [1, 3, 256, 512];

[0045] S40, performing calculation using an equirectangular panoramic image calculation formula to obtain a final restored equirectangular panoramic image, wherein the tensor remains unchanged after the calculation;

[0046] S41, stitching, stitching the original equirectangular panoramic image and the image that has not been repaired by the equirectangular generation network together to obtain a true equirectangular panoramic image, wherein the obtained tensor size is [1, 6, 256, 512];

[0047] S42, inputting the true equirectangular panoramic image of the output tensor in step S41 into the equirectangular discriminant network, passing through a module consisting of a convolutional layer, a batch normalization layer, and a leaky rectifier unit layer four times, and obtaining a tensor size of [1, 1, 30, 62];

[0048] S43, inputting the equirectangular panoramic image generated by the output tensor in step S42 into the equirectangular discriminant network, and passing through a module consisting of four convolutional layers, batch normalization layers, and leaky rectifier unit layers, so that the tensor size obtained is [1, 1, 30, 62];

[0049] S44, deleting the portion of the cubic panoramic image blocked by the mask, and recording the image with the remaining portions unchanged as the cubic panoramic image with the blocked region removed, wherein the tensor size of the cubic panoramic image with the blocked region removed is [1, 6, 3, 256, 256], and the tensor size of the original cubic panoramic image is [1, 6, 3, 256, 256];

[0050] S45, stitching and reshaping the cube panoramic image with the occlusion area removed in step S44 and the original cube panoramic image to obtain a real cube panoramic image, wherein the tensor size of the real cube panoramic image is [1, 36, 256, 256]. Stitching and reshaping the tensor repaired and rearranged by the cube generation network in step S25 and the cube panoramic image with the occlusion area removed to obtain a generated cube panoramic image, wherein the tensor size of the cube panoramic image is [1, 36, 256, 256].

[0051] S46, inputting the tensor of the cube panoramic image in step S45 into the cube discriminant network, wherein the tensor is of size [1, 6] after passing through a module composed of a convolutional layer, a batch normalization layer, and a leaky rectifier unit layer multiple times;

[0052] S47, inputting the tensor of the real cube panoramic image in step S45 into the cube discriminant network, wherein the tensor is of size [1, 6] after passing through a module composed of multiple convolutional layers, batch normalization layers, and leaky rectifier unit layers;

[0053] S48, loss function, the loss function includes perceptual loss L PREC , style loss L style , binary loss function L BCEL and L1 loss function;

[0054] S49. The formula of the overall loss function used is as follows:

[0055] L total =λ prec L prec +λ style L style +λ BCEL L BCEL +λ1L1

[0056] The λ prec ,λ style ,λ BCEL and λ1 represent hyperparameters for weighting the loss function, the L total Represents the overall loss function.

[0057] S50, selecting the adaptive moment estimation optimizer with default parameters as the optimizer for the training process, using a constant learning rate of 0.0001, training the panoramic image street view dataset for 30 rounds, and setting the batch size to 1;

[0058] S51. In the algorithm, we plan to use two evaluation indicators that have been widely used in similar tasks in the past to quantitatively evaluate the model algorithm, including Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). The calculation formulas of the evaluation indicators are as follows:

[0059]

[0060] In PSNR, MAX represents the maximum possible value of an image pixel (e.g., for an 8-bit grayscale image, MAX=255), and MSE stands for mean square error, which represents the average of the squared differences of each pixel between the original image and the distorted image.

[0061] In SSIM, μ x and μ y Represent the average values of image x and y respectively, and Represents the variance of images x and y, σ xy represents the covariance of the image and. c1 and c2 represent constants used for stability, usually set to c1 = (k1L) 2 and c2=(k2L) 2 , where k1 and k2 represent constants and L represents the maximum possible value of a pixel.

[0062] Furthermore, the calculation formula of the medium rectangular panoramic image in step S40 is as follows:

[0063] E out =E G ×M E +(1―M E )×E

[0064] The E out Represents the final restored equirectangular panoramic image, E G The image representing the generated repaired part, M E represents the equirectangular mask, and E represents the original equirectangular panoramic image.

[0065] Furthermore, in step S48, L style The formula is as follows:

[0066]

[0067] The style loss L style Used to describe the difference between the generated image and the real image in the feature space, the φ i Represents a given image I * After that, the activation map of the i-th pooling layer from the VGG-19 network, the i OUT Represents the generated image, the I GTrepresents a real image, the ∥·∥1 represents the L1 norm, and the Gram represents a matrix composed of the inner product of a set of vectors.

[0068] Furthermore, the formula of L1 in step S48 is as follows:

[0069]

[0070] N represents the number of samples, and y i represents the true value, Represents the predicted value.

[0071] Furthermore, in step S20, the upsampling module implements upsampling through a transposed convolution operation, the convolution kernel size Ks is (2, 2), the step size St is 2, the input of Pa padding is a convolution layer of 1, and the transposed convolution operation expands the size of the input tensor to obtain an upsampled output tensor.

[0072] Furthermore, the cubic panoramic image calculation formula in step S26 is as follows:

[0073] C out =C G ×M c +(1―M c )×C

[0074] The C out Represents the final restored cube panoramic image, C G The image representing the generated repaired part, M c represents the cube mask, and C represents the original cube panorama image.

[0075] Furthermore, in step S48, L PREC The formula is as follows:

[0076]

[0077] The L PREC For evaluating the global structure of an image, N represents the number of samples, and φ i Represents a given image I * After that, the activation map of the i-th pooling layer from the VGG-19 network, the i OUT Represents the generated image, the I GT represents a real image, and the ∥·∥1 represents the L1 norm, which is used to measure the distance between two vectors.

[0078] Furthermore, in step S48, L BCEL The formula is as follows:

[0079] L BCEL=-W n [y n ·logσ(p n )+(1―y n )·log(1―σ(p n ))]

[0080] The p represents the probability that the sample χ is predicted to be a positive example, y represents the true label of the sample χ, y is usually 0 or 1, representing whether the sample χ is a positive example, the σ represents the Sigmoid function, which converts the predicted value of the sample χ into a probability mapped to the range of 0 to 1, and the W represents the weight, which can be manually adjusted for the loss of each sample.

[0081] Furthermore, the atrous spatial pyramid pooling module in step S14 controls the receptive field size of the convolution kernel on the input tensor for feature extraction by using convolution layers with atrous rates of 6, 12, and 18, respectively. Each convolution layer is followed by a linear rectification activation function and a batch normalization layer. The outputs of the three convolution layers are concatenated along the channel dimension to obtain a tensor with multiple feature channels. A 1×1 convolution layer is used to compress the number of channels to the number of output channels. The tensor does not change after passing through the atrous spatial pyramid pooling module.

[0082] The advantages of the present invention compared with the existing technology are:

[0083] 1. This paper provides a panoramic image restoration method based on a two-stage generative adversarial network. To address the problem of panoramic images having richer visual information but struggling to fully utilize it, this paper proposes a convolutional group with a skip connection structure, a contextual feature aggregation module, and an attention gating module. By using dilated convolutions with different dilation rates, the model's ability to extract image features at different scales is enhanced, allowing for better acquisition of the image's structural and texture features, addressing this issue.

[0084] 2. The present invention provides a panoramic image restoration method based on a two-stage generative adversarial network. To address the problem that panoramic images with equidistant projections may cause distortion, resulting in unsatisfactory restoration results, a two-stage generative adversarial network is proposed. It utilizes information in both cubic and equirectangular formats, combined with feature information of stereoscopic and overall perspectives, to correct the distortion caused by panoramic images. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 It is a flow chart of the present invention.

[0086] Figure 2 It is a schematic diagram of the cube generation network module of the present invention.

[0087] Figure 3 It is a schematic diagram of the equirectangular generation network module of the present invention. DETAILED DESCRIPTION

[0088] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0089] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0090] Example 1:

[0091] like Figures 1 to 3 As shown, this embodiment proposes a panoramic image restoration method based on a two-stage generative adversarial network. A set of image data is obtained from a 360-degree panoramic street view dataset and input into a computer, including the following steps:

[0092] S1, encode the public 360 panoramic street view dataset and random mask data into data containing cubic panorama format for processing;

[0093] S2. Place the 15,200 training images in the cubic panorama format data into a folder named 'training', and place the 3,800 test images used for testing into a folder named 'testing';

[0094] S3. Loop through the data in the 'training' folder or the data in the 'testing' folder, save the index address of each file for easy reference later; set the number of panoramic images to be loaded each time and package these panoramic images to generate an index input to the network; use the index address stored in the panoramic image and the index generated by the package file to perform a merge operation, and convert the obtained panoramic image into a tensor form;

[0095] S4. The tensor size of one face of a cube format image is [B, F, 3, H, W], and the mask tensor size of one face of a cube format image is [B, F, 1, H, W], where B represents the input batch size, F represents the number of cubes, 3 and 1 represent the number of channels, H represents the height of the input image, and W represents the width of the input image.

[0096] S5. Since the cube format panoramic image has six faces, the number of channels will be multiplied six times when combined together, which are respectively recorded as the cube panoramic image feature map F1, with a tensor size of [B, F, 18, H, W], and the cube mask image feature map F2, with a tensor size of [B, F, 6, H, W]. Then, the shapes of F1 and F2 are transformed and spliced to obtain the feature map F3, with a tensor size of [B, 24, 256, 256] (resolution is 256×256), and it is input into the cube generation network, where B represents the input batch size, F represents the number of cubes, and H represents the height of the input image; W represents the width of the input image, 18, 6, and 24 represent the number of channels, 256 represents the width of the image, and 256 represents the height of the image;

[0097] S6. Input the tensor [B, 24, 256, 256] into the cube generation network, first passing through the first residual block with the following parameters: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, Pa represents the padding of the original three-dimensional grid space, the convolution kernel used in the convolution operation is a two-dimensional matrix, the input batch size is 1, the Ks convolution kernel size is (3, 3), the step size of St is 1, and the input in the Pa padding is 1. After two convolution operations in the residual block, the resulting tensor size is [1, 64, 256, 256]; B, 1 represents the input batch size, 24, 64 represents the width of the input image, 256 represents the width of the image, and 256 represents the height of the image;

[0098] S7. Use the batch normalization function to perform the first batch normalization processing on the tensor obtained from the first convolution operation of the cube projection image. This makes the input distribution of each neuron more stable, allowing for faster convergence during training. It also improves the generalization ability of the model and suppresses overfitting, thereby improving the accuracy and reliability of the deep learning model.

[0099] S8, then using a linear rectification activation function to perform a first nonlinear fitting on the tensor obtained by the first convolution operation of the cube projection image;

[0100] S9. Pass [B, 24, 256, 256] directly to the subsequent layers of the network through an additional convolutional layer to preserve the original input information. It is added to the tensor [1, 64, 256, 256] obtained after the previous convolution through a residual connection to obtain the tensor [1, 64, 256, 256] where B, 1, 1 represents the input batch size, 24, 64, 64 represents the width of the input image, 256 represents the width of the image, and 256 represents the height of the image.

[0101] S10. Input the tensor [1, 64, 256, 256] obtained after the residual block into the first residual convolution block. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, and Pa represents the padding of the original three-dimensional grid space. For a given input, it first passes through a batch normalization function, then a linear rectification activation function, and then passes through a convolution layer with a Ks convolution kernel size of (3, 3), a St step size of 2, and an input of 1 in the Pa padding. Then, it passes through these three layers again in the same order, and the initial input tensor and the output tensor of the second convolution operation are added to form a jump connection. The calculation of the first residual convolution block ends, and the size of the output tensor is [1, 128, 128, 128], where 1, 1 represents the input batch size, 64, 128 represents the number of channels, 256, 128 represents the width of the input image, and 256, 128 represents the height of the image;

[0102] S11. Input the tensor [1, 128, 128, 128] obtained after the first residual block into the second residual convolution block. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, and Pa represents the padding of the original three-dimensional grid space. For a given input, it first passes through a batch normalization function, then a linear rectification activation function, and then passes through a convolution layer with a Ks convolution kernel size of (3, 3), a St step size of 2, and an input of 1 in the Pa padding. Then, it passes through these three layers again in the same order, adds the initial input tensor and the output tensor of the second convolution operation, forms a jump connection, and ends the calculation of the second residual convolution block. The size of the output tensor is [1, 256, 64, 64], where 1, 1 represents the input batch size, 128, 256 represents the number of channels, 128, 64 represents the width of the input image, and 128, 64 represents the height of the image;

[0103] S12. Input the tensor [1, 256, 64, 64] obtained after the second residual block into the third residual convolution block. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, and Pa represents the padding of the original three-dimensional grid space. For a given input, it first passes through a batch normalization function, then a linear rectification activation function, and then passes through a convolution layer with a Ks convolution kernel size of (3, 3), a step size of St of 2, and an input of 1 in Pa padding. Then, it passes through these three layers again in the same order, adds the initial input tensor and the output tensor of the second convolution operation to form a jump connection, and ends the calculation of the third residual convolution block. The size of the output tensor is [1, 512, 32, 32], where 1, 1 represents the input batch size, 256, 512 represents the number of channels, 64, 32 represents the width of the input image, and 64, 32 represents the height of the image.

[0104] S13. The tensor [1, 512, 32, 32] obtained after the third residual convolution block is input into the first upsampling module. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, and Pa represents the padding of the original three-dimensional grid space. For a given input, the upsampling module implements upsampling through a transposed convolution operation. The Ks convolution kernel size is (2, 2), the step size of St is 2, and the input in the Pa padding is a convolution layer of 1. The transposed convolution operation expands the size of the input tensor to obtain an upsampled output tensor. Finally, the upsampled tensor [1, 512, 64, 64] is returned as the output of the module, where 1, 1 represents the input batch size, 512, 512 represents the number of channels, 32, 64 represents the width of the input image, and 32, 64 represents the height of the image.

[0105] S14. The tensor [1, 512, 64, 64] obtained after the first upsampling module is input into the dilated spatial pyramid pooling module. This module performs feature extraction by using convolutional layers with dilation rates of 6, 12, and 18 to control the receptive field size of the convolution kernel on the input tensor. Then, each convolutional layer is followed by a linear rectification activation function and a batch normalization layer. Then, the outputs of the three convolutional layers are concatenated along the channel dimension to obtain a tensor with multiple feature channels. Finally, a 1×1 convolutional layer is used to compress the number of channels to the number of output channels. After passing through the dilated spatial pyramid pooling module, the tensor does not change, and the output tensor remains [1, 512, 64, 64], where 1, 1 represents the input batch size, 512, 512 represents the number of channels, 64, 64 represents the width of the input image, and 64, 64 represents the height of the image.

[0106] S15. Concatenate the output tensor [1, 256, 64, 64] obtained by S11 and the output tensor [1, 512, 64, 64] obtained by S14 to obtain the tensor [1, 768, 64, 64], where 1, 1, 1 represents the input batch size, 256, 512, 768 represents the number of channels, 64, 64, 64 represents the width of the input image, and 64, 64, 64 represents the height of the image.

[0107] S16. Input the concatenated tensor [1,768,64,64] into the fourth residual convolution block. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, and Pa represents the padding of the original three-dimensional grid space. For a given input, it first undergoes batch normalization, then a linear rectification activation function, and then a convolution layer with a Ks convolution kernel size of (3, 3), a St step size of 1, and an input of 1 in the Pa padding. Then, it passes through these three layers again in the same order, and the initial input tensor and the output tensor of the second convolution operation are added to form a jump connection. The calculation of the fourth residual convolution block ends, and the size of the output tensor is [1, 256, 64, 64], where 1, 1 represents the input batch size, 768, 256 represents the number of channels, 64, 64 represents the width of the input image, and 64, 64 represents the height of the image;

[0108] S17. Input the tensor [1, 256, 64, 64] obtained after the fourth residual convolution block into the second upsampling module. The parameter settings are as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, and Pa represents the padding of the original three-dimensional grid space. For a given input, the upsampling module implements upsampling through a transposed convolution operation. The Ks convolution kernel size is (2, 2), the step size of St is 2, and the input in the Pa padding is a convolution layer with 1. The transposed convolution operation expands the size of the input tensor to obtain an upsampled output tensor. Finally, the upsampled tensor [1, 256, 128, 128] is returned as the output of the module, where 1, 1 represents the input batch size, 256, 256 represents the number of channels, 64, 128 represents the width of the input image, and 64, 128 represents the height of the image.

[0109] S18. Concatenate the output tensor [1, 256, 128, 128] obtained by S17 and the output tensor [1, 128, 128, 128] obtained by S10 to obtain the tensor [1, 384, 128, 128], where 1, 1, 1 represents the input batch size, 256, 128, 384 represents the number of channels, 128, 128, 128 represents the width of the input image, and 128, 128, 128 represents the height of the image.

[0110] S19. Input the concatenated tensor [1, 384, 128, 128] into the fifth residual convolution block. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, and Pa represents the padding of the original three-dimensional grid space. For a given input, it first undergoes batch normalization, then a linear rectification activation function, and then a convolution layer with a Ks convolution kernel size of (3, 3), a St step size of 1, and an input of 1 in the Pa padding. Then, it passes through these three layers again in the same order, and adds the initial input tensor and the output tensor of the second convolution operation to form a jump connection. End the calculation of the fifth residual convolution block, and the size of the output tensor is [1, 128, 128, 128], where 1, 1 represents the input batch size, 384, 128 represents the number of channels, 128, 128 represents the width of the input image, and 128, 128 represents the height of the image.

[0111] S20. The tensor [1, 128, 128, 128] obtained after the fifth residual convolution block is input into the third upsampling module. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, and Pa represents the padding of the original three-dimensional grid space. For a given input, the upsampling module implements upsampling through a transposed convolution operation. The Ks convolution kernel size is (2, 2), the step size of St is 2, and the input in the Pa padding is a convolution layer with 1. The transposed convolution operation expands the size of the input tensor to obtain an upsampled output tensor. Finally, the upsampled tensor [1, 128, 256, 256] is returned as the output of the module, where 1, 1 represents the input batch size, 128, 128 represents the number of channels, 128, 256 represents the width of the input image, and 128, 256 represents the height of the image.

[0112] S21. Concatenate the output tensor [1, 128, 256, 256] obtained by S20 and the output tensor [1, 64, 256, 256] obtained by S6 to obtain the tensor [1, 192, 256, 256], where 1, 1, 1 represents the input batch size, 128, 64, 192 represents the number of channels, 256, 256, 256 represents the width of the input image, and 256, 256, 256 represents the height of the image.

[0113] S22. Input the concatenated tensor [1, 192, 256, 256] into the sixth residual convolution block. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, and Pa represents the padding of the original three-dimensional grid space. For a given input, it first undergoes batch normalization, then a linear rectification activation function, and then a convolution layer with a Ks convolution kernel size of (3, 3), a St step size of 1, and an input of 1 in the Pa padding. Then, it passes through these three layers again in the same order, adds the initial input tensor and the output tensor of the second convolution operation to form a jump connection, and ends the calculation of the sixth residual convolution block. The size of the output tensor is [1, 64, 256, 256], where 1, 1 represents the input batch size, 192, 64 represents the number of channels, 256, 256 represents the width of the input image, and 256, 256 represents the height of the image.

[0114] S23, the output layer module processes the input tensor through a module containing a 1×1 convolutional layer to compress the number of channels. The resulting tensor size is [1, 18, 256, 256], where 1 represents the input batch size, 18 represents the number of channels, 256 represents the width of the input image, and 256 represents the height of the image.

[0115] S24. Input the tensor into the hyperbolic tangent function activation layer module for activation. It maps each element in the input tensor to the interval (-1, 1), which helps to avoid the gradient disappearance or gradient explosion problem, thereby more stably training the neural network. In addition, using the hyperbolic tangent function as the output layer activation function in the generator network can limit the output to the range between -1 and 1. The output in this range is easier to match the real data, and the final tensor of the cubic format generation network is obtained [1, 18, 256, 256], where 1 represents the input batch size, 18 represents the number of channels, 256 represents the width of the input image, and 256 represents the height of the image;

[0116] S25. Rearrange the final tensor [1, 18, 256, 256] generated by the cube format network output into a new shape with dimensions [1, 6, 3, 256, 256], where 1, 1 represents the input batch size, 6 represents the number of cubes, 18, 3 represents the number of channels, 256, 256 represents the width of the input image, and 256, 256 represents the height of the image;

[0117] S26, performing calculations using a cubic panoramic image calculation formula to obtain a final restored cubic panoramic image. The cubic panoramic image calculation formula in step S26 is as follows:

[0118] C out =C G ×Mc +(1―M c )×C

[0119] The C out Represents the final restored cube panoramic image, C G The image representing the generated repaired part, M c represents the cube mask, and C represents the original cube panoramic image;

[0120] S27. Convert the final restored cubic panoramic image, the original cubic panoramic image, and the masked cubic image from the cubic format to an equirectangular format, so that the sizes of the converted tensors are [1, 3, 256, 512], [1, 3, 256, 512], and [1, 1, 256, 512], respectively, where 1, 1, 1 represents the input batch size, 256, 3, 1 represents the number of channels, 512, 512, 512 represent the width of the input image, and 256, 256, 256 represent the height of the image.

[0121] S28. The panoramic image converted into an equirectangular format is input into the equirectangular format generation network for more refined restoration. The input tensor size is [1, 3, 256, 512], and enters four residual blocks respectively. Different features are extracted using a variety of convolution kernel sizes. Each residual block first undergoes a batch normalization, then a layer of rectangular convolution, then a linear rectification activation function, and then another layer of rectangular convolution. Then, the jump connection is added to the output to end the calculation of the residual block. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, and Pa represents the filling of the original three-dimensional grid space. The Ks convolution kernel size of the first residual block is (3, 9), the step size of St is 1, and the input of the Pa filling is a convolution layer of (1, 4). , the Ks convolution kernel size of the second residual block is (5, 11), the step size of St is 1, and the input in Pa padding is a convolution layer of (2, 5). The Ks convolution kernel size of the third residual block is (5, 7), the step size of St is 1, and the input in Pa padding is a convolution layer of (2, 3). The Ks convolution kernel size of the fourth residual block is (7, 7), the step size of St is 1, and the input in Pa padding is a convolution layer of 3. Then the ELU activation function is applied to all four outputs to introduce nonlinearity. The input tensor [1, 3, 256, 512] has an output tensor size of [1, 8, 256, 512] after the four residual blocks are operated. Among them, 1, 1 represents the input batch size, 3, 8 represents the number of channels, 512, 512 represents the width of the input image, and 256, 256 represents the height of the image.

[0122] S29. Concatenate the four tensors to obtain a tensor size of [1, 32, 256, 512]. Then pass it through the first context feature aggregation module, which performs multi-scale feature aggregation on the input feature map. It obtains feature information at different scales by using dilated convolutions with different expansion rates, and then calculates the weight of each scale feature. Then, multiply these weights with the corresponding scale features and add the results to obtain the final aggregated feature map, where 1 represents the input batch size, 32 represents the number of channels, 512 represents the width of the input image, and 256 represents the height of the image.

[0123] S30. The aggregated feature map is input into an attention gating mechanism to dynamically adjust the weights of each element in the input tensor. In this module, the input tensor is first flattened through a fully connected layer, and then the flattened tensor is converted into attention weights through a normalized exponential function. Finally, the original input tensor and the attention weights are multiplied to obtain a weighted input tensor, so that the model pays more attention to important elements, thereby improving model performance.

[0124] S31, the tensor size is [1, 32, 256, 512], and it enters the convolution group composed of four residual blocks again, using a variety of convolution kernel sizes to extract different features. Each residual block first undergoes a batch normalization, then a layer of rectangular convolution, then a linear rectification activation function, and then another layer of rectangular convolution. Then, the jump connection is added to the output to end the calculation of the residual block. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, and Pa represents the filling of the original three-dimensional grid space. The Ks convolution kernel size of the first residual block is (3, 9), the step size of St is 1, and the input of Pa filling is a convolution layer of (1, 4). The Ks convolution kernel size of the second residual block is (3 ,7), the step size of St is 1, the input of Pa padding is the convolution layer of (1,3), the Ks convolution kernel size of the third residual block is (3,5), the step size of St is 1, the input of Pa padding is the convolution layer of (1,2), the Ks convolution kernel size of the fourth residual block is (5,5), the step size of St is 1, the input of Pa padding is the convolution layer of 2, and then the exponential linear unit activation function is applied to these four outputs to introduce nonlinearity. The input tensor [1,32,256,512] is calculated by four residual blocks, and the output tensor size is [1,16,256,512], where 1,1 represents the input batch size, 32,16 represents the number of channels, 512,512 represents the width of the input image, and 256,256 represents the height of the image;

[0125] S32, enter the first layer of the encoder, the parameters are set as follows, where Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, Pa represents the padding of the original three-dimensional grid space, and the tensor [1, 16, 256, 512] is input to the convolution layer with a Ks convolution kernel size of (3, 3), a step size of 2 for St, and an input of 1 in the padding of Pa. Then, the exponential linear unit activation function is applied to introduce nonlinearity, and the resulting tensor size is [1, 128, 128, 256], where 1, 1 represents the input batch size, 32, 16 represents the number of channels, 512, 512 represents the width of the input image, and 256, 256 represents the height of the image;

[0126] S33, enter the second layer of the encoder, the parameters are set as follows, where Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, Pa represents the padding of the original three-dimensional grid space, and the tensor [1, 128, 128, 256] is input to the convolution layer with a Ks convolution kernel size of (3, 3), a step size of 1 for St, and an input of 1 in the padding of Pa. Then, the exponential linear unit activation function is applied to introduce nonlinearity, and the resulting tensor size is [1, 256, 128, 256], where 1, 1 represents the input batch size, 128, 256 represents the number of channels, 256, 256 represents the width of the input image, and 128, 128 represents the height of the image;

[0127] S34, enter the third layer of the encoder, the parameters are set as follows, where Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, Pa represents the padding of the original three-dimensional grid space, and the tensor [1, 256, 128, 256] is input to the convolution layer with a Ks convolution kernel size of (3, 3), a step size of 1 for St, and an input of 1 in the padding of Pa. Then, the exponential linear unit activation function is applied to introduce nonlinearity, and the resulting tensor size is [1, 256, 128, 256], where 1, 1 represents the input batch size, 256, 256 represents the number of channels, 256, 256 represents the width of the input image, and 128, 128 represents the height of the image;

[0128] S35, enter the first layer of the decoder, and set the parameters as follows, where Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, and Pa represents the padding of the original three-dimensional grid space. The tensor [1, 256, 128, 256] is input to the transposed convolution layer, the Ks convolution kernel size is (4, 4), the step size of St is 2, and the input of Pa padding is a convolution layer with 1. Then, the exponential linear unit activation function is applied to introduce nonlinearity. The resulting tensor size is [1, 128, 256, 512], where 1, 1 represents the input batch size, 256, 128 represents the number of channels, 256, 512 represents the width of the input image, and 128, 256 represents the height of the image.

[0129] S36, enter the second layer of the decoder, the parameters are set as follows, where Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, Pa represents the padding of the original three-dimensional grid space, and the tensor [1, 128, 256, 512] is input to the transposed convolution layer, the Ks convolution kernel size is (5, 5), the step size of St is 1, and the input of Pa padding is a convolution layer of 2. Then, the exponential linear unit activation function is applied to introduce nonlinearity. The resulting tensor size is [1, 128, 256, 512], where 1, 1 represents the input batch size, 128, 128 represents the number of channels, 512, 512 represents the width of the input image, and 256, 256 represents the height of the image.

[0130] S37, enter the third layer of the decoder, and set the parameters as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, Pa represents the padding of the original three-dimensional grid space, and the tensor [1, 128, 256, 512] is input to the transposed convolution layer. The Ks convolution kernel size is (1, 1), the step size of St is 1, and the input of Pa padding is a convolution layer with 0. Then, the exponential linear unit activation function is applied to introduce nonlinearity. The resulting tensor size is [1, 64, 256, 512], where 1, 1 represents the input batch size, 128, 64 represents the number of channels, 512, 512 represents the width of the input image, and 256, 256 represents the height of the image.

[0131] S38. Pass the tensor [1, 64, 256, 512] through the second context feature aggregation module. The function of this module is to perform multi-scale feature aggregation on the input feature map. It obtains feature information at different scales by using dilated convolutions with different expansion rates, and then calculates the weight of each scale feature. Then, these weights are multiplied by the corresponding scale features and the results are added to obtain the final aggregated feature map, where 1 represents the input batch size, 64 represents the number of channels, 512 represents the width of the input image, and 256 represents the height of the image.

[0132] S39. Pass the tensor [1, 64, 256, 512] through a convolution layer with the following parameters: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, Pa represents the padding of the original three-dimensional grid space, the Ks convolution kernel size is (3, 3), the step size of St is 1, and the input of Pa padding is 1. The purpose of this convolution layer is to further process the feature map and generate pixel-level prediction results. The tensor size is [1, 3, 256, 512], where 1 represents the input batch size, 3 represents the number of channels, 512 represents the width of the input image, and 256 represents the height of the image.

[0133] S40, performing calculations using an equirectangular panoramic image calculation formula to obtain a final restored equirectangular panoramic image. After the calculations, the tensor remains unchanged. The equirectangular panoramic image calculation formula in step S40 is as follows:

[0134] E out =E G ×M E +(1―M E )×E

[0135] The E out Represents the final restored equirectangular panoramic image, E G The image representing the generated repaired part, M E represents the equirectangular mask, and E represents the original equirectangular panoramic image;

[0136] S41. Stitching the original equirectangular panoramic image and the image that have not been repaired by the equirectangular generation network together to obtain a true equirectangular panoramic image, where the tensor size is [1, 6, 256, 512]. Stitching the image that has not been repaired by the equirectangular generation network and the image that has been repaired by the equirectangular generation network together to obtain a generated equirectangular panoramic image, where the tensor size is [1, 6, 256, 512], where 1, 1 represents the input batch size, 6, 6 represents the number of channels, 512, 512 represents the width of the input image, and 256, 256 represents the height of the image.

[0137] S42. Input the real equirectangular panoramic image with tensor size of [1, 6, 256, 512] into the equirectangular discriminant network, and pass it through four modules consisting of convolution layer, batch normalization layer, and leaky rectifier unit layer. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, Pa represents the filling of the original three-dimensional grid space, Ks convolution kernel size is (4, 4), St step size is 2, Pa fills the convolution layer with input of 1, and the leaky rectifier unit layer is used as an activation function. To help the neural network train better, reduce the gradient vanishing problem and the occurrence of "dead neurons", and improve the stability and generalization ability of the model, a convolution layer with a Ks convolution kernel size of (4, 4), a St step size of 1, and an input of 1 in Pa padding is used as the final output layer. After processing by the equirectangular discriminant network, the prediction result of the real sample is obtained. The tensor size is [1,1,30,62], where 1 represents the input batch size, 1 represents the number of channels, 62 represents the width of the input image, and 30 represents the height of the image.

[0138] S43. The generated equirectangular panoramic image with a tensor size of [1, 6, 256, 512] is input into the equirectangular discriminant network, and passes through four modules consisting of convolutional layers, batch normalization layers, and leaky rectifier unit layers. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, Pa represents the filling of the original three-dimensional grid space, the Ks convolution kernel size is (4, 4), the step size of St is 2, the input of Pa filling is a convolution layer of 1, and the leaky rectifier unit layer is used as an activation function. To help the neural network train better, reduce the gradient vanishing problem and the occurrence of "dead neurons", and improve the stability and generalization ability of the model, a convolution layer with a Ks convolution kernel size of (4, 4), a St step size of 1, and an input of 1 in Pa padding is used as the final output layer. After processing by the equirectangular discriminant network, the prediction result of the generated sample is obtained, and the tensor size is [1, 1, 30, 62], where 1 represents the input batch size, 1 represents the number of channels, 62 represents the width of the input image, and 30 represents the height of the image;

[0139] S44. Obtain a cubic panoramic image in which the portion blocked by the mask is deleted while the other portions remain unchanged, and record the image as the cubic panoramic image with the blocked region removed, with a tensor size of [1, 6, 3, 256, 256], and the original cubic panoramic image, with a tensor size of [1, 6, 3, 256, 256], where 1, 1 represents the input batch size, 6, 6 represents the number of cubes, 3, 3 represents the number of channels, 256, 256 represents the width of the input image, and 256, 256 represents the height of the image;

[0140] S45. Splice and reshape the cube panoramic image with the original cube panoramic image after removing the occlusion area to obtain a true cube panoramic image, and the tensor size is [1, 36, 256, 256]. Splice and reshape the tensor repaired and rearranged by the cube generation network in S25 with the cube panoramic image with the occlusion area removed to obtain a generated cube panoramic image, and the tensor size is [1, 36, 256, 256], where 1, 1 represents the input batch size, 36, 36 represents the number of channels, 256, 256 represents the width of the input image, and 256, 256 represents the height of the image;

[0141] S46. Input the generated cube panoramic image with a tensor size of [1, 36, 256, 256] into the cube discriminant network, and pass through multiple modules consisting of convolution layers, batch normalization layers, and leaky rectifier unit layers. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, Pa represents the padding of the original three-dimensional grid space, the Ks convolution kernel size is (4, 4), the step size of St is 2, and the input of Pa padding is a convolution layer of 1. The leaky rectifier unit layer, as an activation function, can help the neural network train better, reduce the gradient disappearance problem and the occurrence of "dead neurons", and improve the stability and generalization ability of the model. After processing by the cube discriminant network, the prediction result of the generated sample is obtained. The tensor size is [1, 6], where 1 represents the input batch size and 6 represents the separate score of the discrimination results of the 6 faces of the input cube panoramic image. The score value can be used to judge the authenticity and quality of each face.

[0142] S47. Input a real cube panoramic image with a tensor size of [1, 36, 256, 256] into the cube discriminant network, and pass through a module consisting of multiple convolution layers, batch normalization layers, and leaky rectifier unit layers. The parameters are set as follows: Ks represents the size of the convolution kernel, St represents the sliding step of the convolution operation, Pa represents the padding of the original three-dimensional grid space, the Ks convolution kernel size is (4, 4), the step size of St is 2, and the input of Pa padding is a convolution layer of 1. The leaky rectifier unit layer, as an activation function, can help the neural network train better, reduce the gradient disappearance problem and the occurrence of "dead neurons", and improve the stability and generalization ability of the model. After processing by the cube discriminant network, the prediction result of the real sample is obtained. The tensor size is [1, 6], where 1 represents the input batch size and 6 represents the separate score of the discrimination results of the 6 faces of the input cube panoramic image. The score value can be used to judge the authenticity and quality of each face.

[0143] S48, loss function, the loss function includes perceptual loss L PREC , style loss L style , binary loss function L BCEL and L1 loss function;

[0144] S49. The formula of the overall loss function used is as follows:

[0145] L total =λ prec L prec +λ style L style +λ BCEL L BCEL +λ1L1

[0146] The λprec ,λ style ,λ BCEL and λ1 represent hyperparameters for weighting the loss function, the L total Represents the overall loss function;

[0147] S50, selecting the adaptive moment estimation optimizer with default parameters as the optimizer for the training process, using a constant learning rate of 0.0001, training the panoramic image street view dataset for 30 rounds, and setting the batch size to 1;

[0148] S51. In the algorithm, we plan to use two evaluation indicators that have been widely used in similar tasks in the past to quantitatively evaluate the model algorithm, including Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). The calculation formulas of the evaluation indicators are as follows:

[0149]

[0150] In PSNR, MAX represents the maximum possible value of an image pixel (e.g., for an 8-bit grayscale image, MAX=255), and MSE stands for mean square error, which represents the average of the squared differences of each pixel between the original image and the distorted image.

[0151] In SSIM, μ x and μ y Represent the average values of image x and y respectively, and Represents the variance of images x and y, σ xy represents the covariance of the image and. c1 and c2 represent constants used for stability, usually set to c1 = (k1L) 2 and c2=(k2L) 2 , where k1 and k2 represent constants and L represents the maximum possible value of a pixel.

[0152] Furthermore, in step S48, L PREC The formula is as follows:

[0153]

[0154] The L PREC For evaluating the global structure of an image, N represents the number of samples, and φ i Represents a given image I * After that, the activation map of the i-th pooling layer from the VGG-19 network, the i OUT Represents the generated image, the I GT represents a real image, and the ∥·∥1 represents the L1 norm, which is used to measure the distance between two vectors.

[0155] Furthermore, in step S48, L style The formula is as follows:

[0156]

[0157] The style loss L style Used to describe the difference between the generated image and the real image in the feature space, the φ i Represents a given image I * After that, the activation map of the i-th pooling layer from the VGG-19 network, the i OUT Represents the generated image, the I GT represents a real image, the ∥·∥1 represents the L1 norm, and the Gram represents a matrix composed of the inner product of a set of vectors.

[0158] Furthermore, it calculates the loss by comparing the differences between the matrices of activation maps from different layers. The matrix is calculated by flattening the activation map and performing transpose multiplication. The final style loss is obtained by calculating the L1 norm between each Gram matrix and summing up the losses of all layers. The calculation of the Gram matrix is first to flatten the feature map F of size C×H×W into a two-dimensional matrix F' with a size of C×(H×W). This is to allow each channel in the feature map to be regarded as a feature vector. Next, F'F' is calculated. T , get the Gram matrix G, size C×C, each element G in the Gram matrix ij represents the correlation between feature channels i and j. Finally, to obtain the normalized Gram matrix, we divide each element of G by the size of the feature map H×W×C to ensure that the matrix values are in a suitable range. The complete style loss calculation is to combine the differences of the Gram matrices of all layers and add the L1 norm of these differences.

[0159] Furthermore, the formula of L1 in step S48 is as follows:

[0160]

[0161] N represents the number of samples, and y i represents the true value, Represents the predicted value.

[0162] It should be noted that, in this document, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0163] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention. The scope of the invention is defined by the appended claims and their equivalents.

[0164] The present invention and its embodiments are described above. This description is not restrictive. What is shown in the accompanying drawings is only one of the embodiments of the present invention. The actual structure is not limited to this. In short, if ordinary technicians in this field are inspired by it and do not depart from the purpose of the invention, they can creatively design structural methods and embodiments similar to the technical solution, which should all fall within the scope of protection of the present invention.

Claims

1. A panoramic image restoration method based on a two-stage generative adversarial network, characterized by: The step of obtaining a set of image data from a 360° panoramic street view dataset and transferring the data to a computer comprises the following steps: S1, encoding the public 360-degree panoramic street view dataset and the random mask, and converting the data into a cubic panorama format for processing; S2, placing the training images in the cubic panorama format data into a folder named 'training', and placing the test images for testing into a folder named 'testing'; S3, looping through the data in the 'training' folder or the data in the 'testing' folder, saving the index address of each file for easy later search and use, setting up to load several panoramic images each time and packaging these panoramic images to generate an index input into the network, performing a merging operation using the index address stored in the panoramic image and the index generated by the packaged file, and converting the obtained panoramic image into a tensor form; S4. The tensor size of one face of the cube format image is [B, F, 3, H, W], and the mask tensor size of one face of the cube format image is [B, F, 1, H, W]; S5. Since the cube format panoramic image has six faces, the number of channels will be multiplied sixfold when they are combined. The faces are respectively recorded as the cube panoramic image feature map F1, and the F1 tensor size is [B, F, 18, H, W]. The faces are respectively recorded as the cube mask image feature map F2, and the F2 tensor size is [B, F, 6, H, W]. The shapes of F1 and F2 are transformed and spliced to obtain a feature map F3, and the F3 tensor size is [B, 24, 256, 256]. S6. Input the F3 tensor of size [B, 24, 256, 256] in step S5 into the cube generation network, and the tensor size obtained after the two convolution operations in the first residual block is [1, 64, 256, 256]; S7, performing a first batch normalization process on the tensor obtained by the first convolution operation of the cube projection image using the batch normalization function; S8, performing a first nonlinear fitting on the tensor obtained by the first convolution operation of the cube projection image using the linear rectification activation function; S9, passing [B, 24, 256, 256] directly to the subsequent layers of the network through an additional convolutional layer to retain the original input information, adding it to the tensor [1, 64, 256, 256] obtained after the previous convolution through the residual connection to obtain the tensor [1, 64, 256, 256]; S10, the tensor [1, 64, 256, 256] obtained after passing through the residual block is input into the first residual convolution block, and the size of the output tensor is [1, 128, 128, 128]; S11, the tensor [1, 128, 128, 128] obtained after passing through the first residual block is input into the second residual convolution block, and the size of the output tensor is [1, 256, 64, 64]; S12, the tensor [1, 256, 64, 64] obtained after passing through the second residual block is input into the third residual convolution block, and the output tensor size is [1, 512, 32, 32]; S13, inputting the tensor [1, 512, 32, 32] obtained after passing through the third residual convolution block into the first upsampling module, and returning the upsampled tensor [1, 512, 64, 64] as the output of the module; S14, the tensor [1, 512, 64, 64] obtained after the first upsampling module is input into the dilated spatial pyramid pooling module, the tensor does not change after passing through the dilated spatial pyramid pooling module, and the output tensor remains [1, 512, 64, 64]; S15, concatenating the output tensor [1, 256, 64, 64] obtained in step S11 and the output tensor [1, 512, 64, 64] obtained in step S14 to obtain a tensor [1, 768, 64, 64]; S16, inputting the concatenated tensor [1,768,64,64] into the fourth residual convolution block, and the size of the output tensor is [1,256,64,64]; S17, inputting the tensor [1, 256, 64, 64] obtained after passing through the fourth residual convolution block into the second upsampling module, and returning the upsampled tensor [1, 256, 128, 128] as the output of the module; S18, concatenating the output tensor [1, 256, 128, 128] obtained in step 17 and the output tensor [1, 128, 128, 128] obtained in step 10 to obtain the tensor [1, 384, 128, 128]; S19, inputting the concatenated tensor [1, 384, 128, 128] into the fifth residual convolution block, and the size of the output tensor is [1, 128, 128, 128]; S20, inputting the tensor [1, 128, 128, 128] obtained after passing through the fifth residual convolution block into the third upsampling module, and returning the upsampled tensor [1, 128, 256, 256] as the output of the module; S21, concatenating the output tensor [1, 128, 256, 256] obtained in step 20 and the output tensor [1, 64, 256, 256] obtained in step 6 to obtain the tensor [1, 192, 256, 256]; S22, inputting the concatenated tensor [1, 192, 256, 256] into the sixth residual convolution block, and the size of the output tensor is [1, 64, 256, 256]; S23, the tensor [1, 64, 256, 256] obtained after the sixth residual convolution block is input to the output layer module, and the size of the tensor obtained after the processing is [1, 18, 256, 256]; S24, inputting the output tensor of step S23 into the hyperbolic tangent function activation layer module for activation, and obtaining the final tensor of the cubic format generation network, wherein the size of the tensor is [1, 18, 256, 256]; S25, rearranging the output tensor in step S24 into a new shape, wherein the dimension is [1, 6, 3, 256, 256]; S26, performing calculation using a cubic panoramic image calculation formula to obtain a final restored cubic panoramic image; S27, converting the final restored cubic panoramic image, the original cubic panoramic image, and the masked cubic image from a cubic format to an equirectangular format, so that the sizes of the converted tensors are [1, 3, 256, 512], [1, 3, 256, 512], and [1, 1, 256, 512], respectively; S28, the panoramic image converted into an equirectangular format is input into an equirectangular format generation network for finer restoration, the input tensor size is [1, 3, 256, 512], and the images are respectively input into four residual blocks, and different features are extracted using a variety of convolution kernel sizes. Each residual block first undergoes a batch normalization, then a layer of rectangular convolution, then a linear rectification activation function, and then another layer of rectangular convolution. A skip connection is added to the output, and the output tensor size after the four residual blocks is [1, 8, 256, 512]. S29, concatenating the four tensors in step S28 to obtain a tensor size of [1, 32, 256, 512], passing the concatenated tensor through a first context feature aggregation module to obtain a final aggregated feature map; S30, inputting the aggregated feature map in step S29 into an attention gating mechanism, multiplying the original input tensor and the attention weight to obtain a weighted input tensor; S31, the tensor size is set to [1, 32, 256, 512], and the convolution group composed of four residual blocks is entered again, and the output tensor size after the four residual blocks are all [1, 16, 256, 512]; S32, enter the first layer of the encoder, input the output tensor in step S31 to the first layer of the encoder, and the obtained tensor size is [1, 128, 128, 256]; S33, entering the second layer of the encoder, inputting the output tensor in step S32 into the second layer of the encoder, and the obtained tensor size is [1, 256, 128, 256]; S34, entering the third layer of the encoder, inputting the output tensor in step S33 into the third layer of the encoder, and the obtained tensor size is [1, 256, 128, 256]; S35, entering the first layer of the decoder, inputting the output tensor in step S34 into the first layer of the decoder, and the obtained tensor size is [1, 128, 256, 512]; S36, enter the second layer of the decoder, input the output tensor in step S35 into the second layer of the decoder, and the obtained tensor size is [1, 128, 256, 512]; S37, entering the third layer of the decoder, inputting the output tensor in step S36 into the third layer of the decoder, and the obtained tensor size is [1, 64, 256, 512]; S38, inputting the output tensor in S37 into a second context feature aggregation module, and obtaining a final aggregated feature map; S39, inputting the aggregated feature map in step S38 into the convolution layer, generating pixel-level prediction results, and obtaining a tensor size of [1, 3, 256, 512]; S40, performing calculation using an equirectangular panoramic image calculation formula to obtain a final restored equirectangular panoramic image, wherein the tensor remains unchanged after the calculation; S41, stitching, stitching the original equirectangular panoramic image and the image that has not been repaired by the equirectangular generation network together to obtain a true equirectangular panoramic image, wherein the obtained tensor size is [1, 6, 256, 512]; S42, inputting the true equirectangular panoramic image of the output tensor in step S41 into the equirectangular discriminant network, passing through a module consisting of a convolutional layer, a batch normalization layer, and a leaky rectifier unit layer four times, and obtaining a tensor size of [1, 1, 30, 62]; S43, inputting the equirectangular panoramic image generated by the output tensor in step S42 into the equirectangular discriminant network, and passing through a module consisting of four convolutional layers, batch normalization layers, and leaky rectifier unit layers, so that the tensor size obtained is [1, 1, 30, 62]; S44, deleting the portion of the cubic panoramic image blocked by the mask, and recording the image with the remaining portions unchanged as the cubic panoramic image with the blocked region removed, wherein the tensor size of the cubic panoramic image with the blocked region removed is [1, 6, 3, 256, 256], and the tensor size of the original cubic panoramic image is [1, 6, 3, 256, 256]; S45, stitching and reshaping the cube panoramic image with the occlusion area removed in step S44 and the original cube panoramic image to obtain a real cube panoramic image, wherein the tensor size of the real cube panoramic image is [1, 36, 256, 256]. Stitching and reshaping the tensor repaired and rearranged by the cube generation network in step S25 and the cube panoramic image with the occlusion area removed to obtain a generated cube panoramic image, wherein the tensor size of the cube panoramic image is [1, 36, 256, 256]. S46, inputting the tensor of the cube panoramic image in step S45 into the cube discriminant network, wherein the tensor is of size [1, 6] after passing through a module composed of a convolutional layer, a batch normalization layer, and a leaky rectifier unit layer multiple times; S47, inputting the tensor of the real cube panoramic image in step S45 into the cube discriminant network, wherein the tensor is of size [1, 6] after passing through a module composed of multiple convolutional layers, batch normalization layers, and leaky rectifier unit layers; S48, loss function, the loss function includes perceptual loss L PREC , style loss L style , binary loss function L BCEL and L1 loss function; S49. The formula of the overall loss function used is as follows: L total =λ prec L prec +λ style L style +λ BCEL L BCEL +λ1L1 The λ prec ,λ style ,λ BCEL and λ1 represent hyperparameters for weighting the loss function, the L total Represents the overall loss function; S50, selecting the adaptive moment estimation optimizer with default parameters as the optimizer for the training process, using a constant learning rate of 0.0001, training the panoramic image street view dataset for 30 rounds, and setting the batch size to 1; S51. In the algorithm, we plan to use two evaluation indicators that have been widely used in similar tasks in the past to quantitatively evaluate the model algorithm, including Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). The calculation formulas of the evaluation indicators are as follows: In PSNR, MAX represents the maximum possible value of an image pixel (e.g., for an 8-bit grayscale image, MAX=255), and MSE stands for mean square error, which represents the average of the squared differences of each pixel between the original image and the distorted image. In SSIM, μ x and μ y Represent the average values of image x and y respectively, and Represents the variance of images x and y, σ xy represents the covariance of the image and. c1 and c2 represent constants used for stability, usually set to c1 = (k1L) 2 and c2=(k2L) 2 , where k1 and k2 represent constants and L represents the maximum possible value of a pixel.

2. The panoramic image restoration method based on a two-stage generative adversarial network according to claim 1, characterized in that: The cubic panoramic image calculation formula in step S26 is as follows: C out =C G ×M c +(1―M c )×C The C out Represents the final restored cube panoramic image, C G The image representing the generated repaired part, M c represents the cube mask, and C represents the original cube panorama image.

3. The panoramic image restoration method based on a two-stage generative adversarial network according to claim 1, characterized in that: The calculation formula of the medium rectangular panoramic image in step S40 is as follows: E out =E G ×M E +(1―M E )×E The E out Represents the final restored equirectangular panoramic image, E G The image representing the generated repaired part, M E represents the equirectangular mask, and E represents the original equirectangular panoramic image.

4. The panoramic image restoration method based on a two-stage generative adversarial network according to claim 1, characterized in that: In step S48, L PREC The formula is as follows: The L PREC For evaluating the global structure of an image, N represents the number of samples, and φ i Represents a given image I * After that, the activation map of the i-th pooling layer from the VGG-19 network, the i OUT Represents the generated image, the I GT represents a real image, and the ∥·∥1 represents the L1 norm, which is used to measure the distance between two vectors.

5. The panoramic image restoration method based on a two-stage generative adversarial network according to claim 1, characterized in that: In step S48, L style The formula is as follows: The style loss L style Used to describe the difference between the generated image and the real image in the feature space, the φ i Represents a given image I * After that, the activation map of the i-th pooling layer from the VGG-19 network, the i OUT Represents the generated image, the I GT represents a real image, the ∥·∥1 represents the L1 norm, and the Gram represents a matrix composed of the inner product of a set of vectors.

6. The panoramic image restoration method based on a two-stage generative adversarial network according to claim 1, characterized in that: In step S48, L BCEL The formula is as follows: L BCEL =―W n [y n ·logσ(p n )+(1−y n )·log(1−σ(p n ))] The p represents the probability that the sample χ is predicted to be a positive example, y represents the true label of the sample χ, y is usually 0 or 1, representing whether the sample χ is a positive example, the σ represents the Sigmoid function, which converts the predicted value of the sample χ into a probability mapped to the range of 0 to 1, and the W represents the weight, which can be manually adjusted for the loss of each sample.

7. The panoramic image restoration method based on a two-stage generative adversarial network according to claim 1, characterized in that: The formula of L1 in step S48 is as follows: N represents the number of samples, and y i represents the true value, Represents the predicted value.

8. The panoramic image restoration method based on a two-stage generative adversarial network according to claim 1, characterized in that: In step S14, the atrous spatial pyramid pooling module controls the receptive field size of the convolution kernel on the input tensor for feature extraction by using convolution layers with dilation rates of 6, 12, and 18, respectively. Each convolution layer is followed by a linear rectification activation function and a batch normalization layer. The outputs of the three convolution layers are concatenated along the channel dimension to obtain a tensor with multiple feature channels. A 1×1 convolution layer is used to compress the number of channels to the number of output channels. The tensor does not change after passing through the atrous spatial pyramid pooling module.

9. The panoramic image restoration method based on a two-stage generative adversarial network according to claim 1, characterized in that: In step S20, the upsampling module implements upsampling through a transposed convolution operation, the convolution kernel size Ks is (2, 2), the step size St is 2, the input of Pa padding is a convolution layer of 1, and the transposed convolution operation expands the size of the input tensor to obtain an upsampled output tensor.

Images