A dual generator panorama image inpainting method based on cubic projection format

Abstract

The application discloses a kind of image processing technical field's based on cube projection format's double generator panorama image repair method, to solve the problem of image repair in prior art will have artifact, distortion and over-smoothing etc., it includes obtaining panorama image to be repaired, and its format is converted into cube map projection format;The front, rear, left and right four faces of panorama image to be repaired are spliced, and the rectangular image to be repaired is obtained;The rectangular image to be repaired is repaired using four-face generator, and the reconstructed rectangular image is obtained;The reconstructed rectangular image is spliced with upper and lower two faces according to channel, and the multi-channel image to be repaired is obtained;The image to be repaired is repaired using six-face generator, and the reconstructed panorama image is obtained;The format of reconstructed panorama image is converted into equirectangular projection format, which is the panorama image after repair.The application can ensure that the content of repair area is reasonable, and reduce the phenomenon of artifact, distortion and over-smoothing in repair area.

Description

A Dual-Generator Panoramic Image Restoration Method Based on Cube Projection Format Technical Field

[0001] This invention relates to a dual-generator panoramic image restoration method based on cube projection format, belonging to the field of image restoration technology. Background Technology

[0002] Panoramic images suffer from issues such as information loss, damage, and occlusion during acquisition, storage, transmission, and post-processing. These problems lead to poor visual perception, and the loss and blurring of panoramic image information also affect human analysis and understanding of the image content. Therefore, panoramic image restoration algorithms are needed to fill in missing areas of panoramic images.

[0003] Zhu et al. proposed a panoramic image inpainting technique that projects a panoramic image onto a sphere and uses matrix calculations to repair missing regions. However, this algorithm only solves the problem of repairing the lower half of the panoramic image and is limited to the repair of simple texture images.

[0004] With the emergence and advancement of Convolutional Neural Networks (CNNs), the performance of image inpainting has been greatly improved. Akimoto et al. designed an inpainting method based on a Generative Adversarial Network (GAN) that utilizes the symmetry of buildings in panoramic images. This network uses the symmetry of buildings to recover missing parts. However, this work uses panoramic images in an isometric cylindrical projection format, which often leads to severe artifacts at the poles, resulting in artifacts and distortion in the inpainting results. Han et al. designed a panoramic image inpainting network that uses panoramic images in a cube projection format as input. This network achieves higher quality results by avoiding the distortion effects of the isometric cylindrical projection format. However, this method treats all pixels in the image as valid pixels, including those in the missing parts, which can lead to artifacts in the inpainting results.

[0005] Patent publication number CN116091330A discloses an image inpainting method based on generative adversarial networks (GANs), comprising the following steps: Step 1, generating an image to be inpainted based on a real image using a mask, training a GAN using the image to be inpainted, and establishing a generator loss function and a discriminator loss function; the adversarial network includes a generator and a discriminator, wherein the generator consists of a coarse inpainting network composed of several sets of convolutional layers and a fine inpainting network based on U-Net, and the discriminator consists of a global discriminator and a local discriminator to ensure the consistency of global structure and local details in the image; Step 2, extracting the trained generator as the image inpainting network, and passing the damaged image through the image inpainting network to obtain the image inpainting result. However, due to some inherent characteristics, the CNN used in this method performs poorly in understanding global structure, which may cause artifacts and distortions in the inpainting result.

[0006] Current technologies often result in artifacts, distortions, and oversmoothing in the restored area, significantly impacting the human visual experience. Therefore, reducing these artifacts, distortions, and oversmoothing is a pressing issue. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a dual-generator panoramic image restoration method based on cube projection format, which reduces artifacts, distortions and over-smoothing phenomena in the restored area while ensuring the reasonableness of the content of the restored area.

[0008] To achieve the above objectives, the present invention employs the following technical solution: a dual-generator panoramic image restoration method based on cube projection format, characterized by comprising the following steps:

[0009] Obtain the panoramic image to be repaired;

[0010] Convert the format of the panoramic image to be repaired from equidistant cylindrical projection format to cube map projection format;

[0011] The front, back, left, and right faces of the panoramic image to be repaired in cube map projection format are stitched together to obtain the rectangular image to be repaired.

[0012] The tetrahedron generator is used to repair the rectangular image to be repaired, resulting in a reconstructed rectangular image.

[0013] By stitching the complete rectangular image with the top and bottom surfaces along channels, a multi-channel image to be repaired is obtained.

[0014] A six-sided generator is used to repair the multi-channel image to be repaired, resulting in a reconstructed panoramic image.

[0015] The reconstructed panoramic image is obtained by converting the reconstructed panoramic image into an equidistant cylindrical projection format.

[0016] Furthermore, the four-sided generator and the six-sided generator have the same structure, both including a feature extractor, a Transformer-based U-shaped network, and a reconstruction network.

[0017] Furthermore, the feature extractor includes two gated convolutional layers, each with a 4×4 kernel, a stride of 2, and a padding count of 1.

[0018] The Transformer-based U-shaped network consists of 5 Transformer blocks;

[0019] The reconstructed network includes two gated transposed convolutional layers, each with a 3×3 kernel, a stride of 1, and a padding of 1.

[0020] Furthermore, a tetrahedral generator is used to repair the rectangular image to be repaired, resulting in a reconstructed rectangular image. A hexahedral generator is also used to repair the multi-channel image to be repaired, resulting in a reconstructed panoramic image, including:

[0021] The feature extractor is used to extract features from the rectangular image or multi-channel image to be repaired. The expression is as follows:

[0022]

[0023] in, and For two different convolution kernels, It is the ReLU activation function. yes function, It is the element-wise multiplication symbol;

[0024] Feature processing is performed using a Transformer-based U-shaped network, as shown in the following expression:

[0025]

[0026] in, The tensor output by the feature extractor; For a one-dimensional flattened tensor; This represents the tensor tiling operation, which converts a multi-dimensional tensor into a one-dimensional tensor. This represents positional encoding of a one-dimensional tensor. This represents multi-head attention computation on a one-dimensional tensor. Represents a fully connected network; Tensors output from a fully connected network;

[0027] Image reconstruction using a reconstruction network can yield either a reconstructed rectangular image or a reconstructed panoramic image.

[0028] Furthermore, before restoring the image to be restored, a panoramic image restoration model needs to be constructed and pre-trained, specifically including:

[0029] a. Based on the original real panoramic images and their corresponding masks, construct a panoramic image dataset with holes. The dataset is divided into a training set and a test set.

[0030] b. Convert the format of the panoramic images with holes in the training set from the equidistant cylindrical projection format to the cube map projection format;

[0031] c. Stitch together the front, back, left, and right faces of the panoramic image with holes in the cube map projection format to form a rectangular image with holes.

[0032] d. Construct a tetrahedron generator, taking a rectangular image with holes as input and a repaired rectangular image as output, and train the tetrahedron generator by combining the generator's loss function;

[0033] e. Construct a six-sided generator by stitching the reconstructed rectangular image with the top and bottom faces along the channels and using it as input. The panoramic image in the repaired cube map projection format is used as output. The six-sided generator is trained by combining the generator's loss function.

[0034] f. Construct a panoramic image restoration model based on the four-sided generator and the six-sided generator. Take a panoramic image in the form of a cube map projection with holes as input and the original real panoramic image as output. Train the panoramic image restoration model by combining the discriminator and the global loss function to obtain the panoramic image restoration model.

[0035] g. Test the panoramic image restoration model using the test set and obtain the test results. If the test results are not good, repeat steps c to f until the test results are good.

[0036] Furthermore, the loss function of the generator includes a style loss function, a perceptual loss function, and a reconstruction loss function;

[0037] The style loss function is obtained by calculating the Gram matrix for each convolutional layer, calculating the Euclidean distance between symmetric convolutional layers, and summing the Euclidean distances of different convolutional layers to obtain the style loss, which is expressed as follows:

[0038]

[0039] in, For the loss of style, It is in 2-normal form. For Gram matrices, For the repaired image, It is a real image;

[0040] The perceptual loss function compares the features obtained by convolving the real image with the features obtained by convolving the reconstructed image, and its expression is:

[0041]

[0042] in, In order to perceive loss, For the pre-trained VGG-16 network, the VGG-16 network adopts... As the network layer for feature extraction, j is the j-th network layer. Let N be the number of elements in the j-th network layer, and N be the number of network layers.

[0043] The expression for the reconstruction loss function is:

[0044]

[0045] in, To rebuild the losses, It is in the 1 normal form.

[0046] Furthermore, the number of discriminators is 2, namely a global discriminator and a face discriminator. The global discriminator consists of 4 convolutional layers with 4×4 kernels, stride of 2 and padding of 1, and 1 linear layer, which is used to discriminate the correlation of the six faces of the stereo image.

[0047] The face discriminator consists of 5 convolutional layers with 4×4 kernels, a stride of 2, and a padding of 1, and 1 linear layer. It is used to identify the authenticity of the four faces of a rectangle.

[0048] Furthermore, the expression for the adversarial loss function is:

[0049]

[0050] in, To combat the loss, x is the real image. The image is the restored image, and D is the discriminator. For the expectation, This represents the distribution of image data output by the generator. Represents the distribution of real image data. This indicates the distribution of the restored image data. This indicates a gradient calculation operation.

[0051] Furthermore, the expression for the global loss function is:

[0052]

[0053] in, For the overall loss, To rebuild the losses, To combat the losses, In order to perceive loss, For the loss of style, , , , These are the corresponding hyperparameters.

[0054] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0055] This invention uses panoramic images in a cube projection format with less distortion, which can reduce the impact on the effective features of the network;

[0056] This invention utilizes a dual generator to restore panoramic images, enabling the use of more effective information and resulting in higher-quality restored images. Specifically, it employs gated convolution to extract effective features, reducing the impact of invalid features on the restoration process. By using a Transformer-based U-shaped network and combining self-attention and multilayer perceptron mechanisms, it can learn global contextual information in the image, generating more accurate and natural restoration results, thereby effectively repairing missing or damaged parts of the image. Attached Figure Description

[0057] Figure 1 is a flowchart illustrating a dual-generator panoramic image restoration method based on cube projection format in one embodiment of the present invention.

[0058] Figure 2 is a schematic diagram of the framework of a dual-generator panoramic image restoration method based on cube projection format in one embodiment of the present invention. Detailed Implementation

[0059] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0060] As shown in Figures 1 and 2, this embodiment of the invention provides a dual-generator panoramic image restoration method based on cube projection format, which can be divided into a training phase and a testing phase.

[0061] Training phase: Construct a panoramic image restoration model and pre-train it.

[0062] The specific steps are as follows:

[0063] a. The real panoramic image data used in this embodiment is SUN360 panoramic image data. At the same time, randomly generated rectangular mask data is used. The real panoramic image and the mask are multiplied element by element to obtain a panoramic image dataset with holes. The dataset is divided into training set and test set.

[0064] b. Convert the format of panoramic images with holes in the training set from Equidistant Cylindrical Projection (ERP) to Cubic Map Projection (CMP).

[0065] c. Stitch together the front, back, left, and right faces of the panoramic image with holes in the cube map projection format to obtain a rectangular image with holes.

[0066] d. Construct a four-sided generator, taking a rectangular image with holes as input and reconstructing a rectangular image as output, and train the four-sided generator by combining the generator's loss function.

[0067] e. Construct a six-sided generator by stitching the reconstructed rectangular image with the top and bottom faces along the channels and using it as input. The panoramic image in the complete cube map projection format is used as output. The six-sided generator is trained by combining the generator's loss function.

[0068] In this embodiment, the four-sided generator and the six-sided generator have the same structure, differing only in their input and output parameters. Both include a feature extractor, a Transformer-based U-shaped network, and a reconstruction network. The feature extractor includes two gated convolutional layers with 4×4 kernels, a stride of 2, and 1 padding. The reconstruction network includes two gated transposed convolutional layers with 3×3 kernels, a stride of 1, and 1 padding.

[0069] The generator's loss function includes the style loss function, the perceptual loss function, and the reconstruction loss function.

[0070] Style loss function: First, calculate the Gram matrix of each convolutional layer. Then, calculate the Euclidean distance between corresponding convolutional layers. Finally, sum the Euclidean distances of different layers to obtain the style loss, which is expressed as follows:

[0071]

[0072] in, For the loss of style, It is in 2-normal form. For Gram matrices, For the repaired image, This is a real image.

[0073] Perceptual loss function: compares the features obtained by convolving the real image with the features obtained by convolving the reconstructed image. Its expression is:

[0074]

[0075] in, In order to perceive loss, For the pre-trained VGG-16 network, the VGG-16 network adopts... As the network layer for feature extraction, j is the j-th network layer. Let N be the number of elements in the j-th network layer, and N be the number of network layers.

[0076] The expression for the reconstruction loss function is:

[0077]

[0078] in, To rebuild the losses, It is in the 1 normal form.

[0079] f. Construct a panoramic image restoration model based on the four-sided generator and the six-sided generator. Take a panoramic image in the form of a cube map projection with holes as input and the original real panoramic image as output. Train the panoramic image restoration model by combining the discriminator and the global loss function to obtain the panoramic image restoration model.

[0080] The discriminator consists of two layers: a global discriminator and a face discriminator. The global discriminator is composed of four convolutional layers with 4×4 kernels, a stride of 2, and a padding of 1, and one linear layer. It is used to identify the correlation of the six faces of a stereo image. The face discriminator is composed of five convolutional layers with 4×4 kernels, a stride of 2, and a padding of 1, and one linear layer. It is used to identify the authenticity of the four faces of a rectangle.

[0081] The expression for the adversarial loss function is:

[0082]

[0083] in, For the overall loss, To rebuild the losses, To combat the losses, In order to perceive loss, For the loss of style, , , , These are the corresponding hyperparameters.

[0084]

[0085] in, To combat the loss, x is the real image. The image is the restored image, and D is the discriminator. For the expectation, This represents the distribution of image data output by the generator. Represents the distribution of real image data. This indicates the distribution of the restored image data. This indicates a gradient calculation operation.

[0086] The expression for the global loss function is:

[0087]

[0088] in, For the overall loss, To rebuild the losses, To combat the losses, In order to perceive loss, For the loss of style, , , , These are the corresponding hyperparameters.

[0089] g. Test the panoramic image restoration model using the test set and obtain the test results. If the test results are not good, repeat steps c to f until the test results are good.

[0090] Testing phase:

[0091] Obtain the panoramic image to be repaired.

[0092] The format of the panoramic image to be repaired is converted from equidistant cylindrical projection format to cube map projection format.

[0093] The front, back, left, and right faces of the panoramic image to be repaired in cube map projection format are stitched together to obtain the rectangular image to be repaired.

[0094] The tetrahedron generator is used to repair the rectangular image to be repaired, resulting in a reconstructed rectangular image, as follows:

[0095] The feature extractor is used to extract features from the rectangular image to be repaired, as shown in the following expression:

[0096]

[0097] in, and For two different convolution kernels, It is the ReLU activation function. yes function, It is the symbol for element-wise multiplication.

[0098] Feature processing is performed using a Transformer-based U-shaped network, as shown in the following expression:

[0099]

[0100] in, The tensor output by the feature extractor; For a one-dimensional flattened tensor; This represents the tensor tiling operation, which converts a multi-dimensional tensor into a one-dimensional tensor. This represents positional encoding of a one-dimensional tensor. This represents multi-head attention computation on a one-dimensional tensor. Represents a fully connected network; Tensors output from a fully connected network.

[0101] Image reconstruction is performed using a reconstruction network, resulting in a reconstructed rectangular image.

[0102] By stitching the complete rectangular image with the top and bottom surfaces along channels, a multi-channel image to be repaired is obtained.

[0103] The six-sided generator is used to repair the multi-channel image to be repaired, and a reconstructed panoramic image is obtained. The specific operation is the same as that of the four-sided generator mentioned above.

[0104] The reconstructed panoramic image is obtained by converting the reconstructed panoramic image into an equidistant cylindrical projection format.

[0105] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for panoramic image restoration based on a dual-generator cube projection format, characterized in that, Includes the following steps: Obtain the panoramic image to be repaired; The format of the panoramic image to be repaired is converted from equidistant cylindrical projection format to cube map projection format; the front, back, left and right faces of the panoramic image to be repaired in cube map projection format are stitched together to obtain the rectangular image to be repaired. The rectangular image to be repaired is repaired using a four-sided generator to obtain a reconstructed rectangular image. The reconstructed rectangular image is then stitched together with the top and bottom faces by channel to obtain a multi-channel image to be repaired. The multi-channel image to be repaired is then repaired using a six-sided generator to obtain a reconstructed panoramic image. The reconstructed panoramic image is then converted into a reconstructed panoramic image in an equidistant cylindrical projection format, which is the repaired panoramic image.

2. The method for dual-generator panoramic image restoration based on cube projection format according to claim 1, characterized in that, The four-sided generator and the six-sided generator have the same structure, both including a feature extractor, a Transformer-based U-shaped network, and a reconstruction network.

3. The dual-generator panoramic image restoration method based on cube projection format according to claim 2, characterized in that, The feature extractor includes two gated convolutional layers with a 4×4 kernel, a stride of 2, and a padding of 1. The Transformer-based U-shaped network includes five Transformer blocks. The reconstruction network includes two gated transposed convolutional layers with a 3×3 kernel, a stride of 1, and a padding of 1.

4. The dual-generator panoramic image restoration method based on cube projection format according to claim 1, characterized in that, The process involves using a tetrahedral generator to repair the rectangular image to be repaired, resulting in a reconstructed rectangular image, and using a hexahedral generator to repair the multi-channel image to be repaired, resulting in a reconstructed panoramic image. This includes using a feature extractor to extract features from either the rectangular image to be repaired or the multi-channel image to be repaired, as shown in the following expression: ； ； ;in, and For two different convolution kernels, It is the ReLU activation function. It is the sigmoid function. This is the element-wise multiplication symbol; feature processing is performed using a Transformer-based U-shaped network, as shown in the following expression: ； ； ;in, The tensor output by the feature extractor; Flatten represents a one-dimensional flattened tensor; Flatten represents the tensor flattening operation, which converts a multi-dimensional tensor into a one-dimensional tensor. This represents positional encoding of a one-dimensional tensor. This represents multi-head attention computation on a one-dimensional tensor. Represents a fully connected network; The tensor output by the fully connected network; the image is reconstructed using the reconstruction network to obtain a reconstructed rectangular image or a reconstructed panoramic image.

5. The method for dual-generator panoramic image restoration based on cube projection format according to claim 1, characterized in that, Before restoring the image to be repaired, a panoramic image restoration model needs to be constructed and pre-trained. Specifically, this includes: a) constructing a dataset of panoramic images with holes based on the original real panoramic images and their corresponding masks, with the dataset divided into training and testing sets; b) converting the format of the panoramic images with holes in the training set from equidistant cylindrical projection to cube map projection format; c) stitching the front, back, left, and right faces of the panoramic images with holes in the cube map projection format into a rectangular image with holes; d) constructing a four-face generator, using the rectangular image with holes as input and the restored rectangular image as output, and training the four-face generator using the generator's loss function. e. Construct a six-sided generator, which takes the reconstructed rectangular image and the top and bottom faces as input, and the restored panoramic image in cube map projection format as output. Train the six-sided generator using the generator's loss function. f. Construct a panoramic image restoration model based on the four-sided and six-sided generators, taking the panoramic image in cube map projection format with holes as input and the original real panoramic image as output. Train the panoramic image restoration model using a discriminator and a global loss function. g. Test the panoramic image restoration model using a test set and obtain the test results. If the test results are not good, repeat steps c to f until the test results are good.

6. The dual-generator panoramic image restoration method based on cube projection format according to claim 5, characterized in that, The generator's loss function includes a style loss function, a perceptual loss function, and a reconstruction loss function. The style loss function is calculated by obtaining the Gram matrix for each convolutional layer, calculating the Euclidean distance between symmetric convolutional layers, and summing the Euclidean distances of different convolutional layers to obtain the style loss, expressed as: ;in, For the loss of style, It is in 2-normal form. For Gram matrices, For the repaired image, The image is a real image; the perceptual loss function compares the features obtained by convolving the real image with the features obtained by convolving the reconstructed image, and its expression is: ;in, In order to perceive loss, For the pre-trained VGG-16 network, the VGG-16 network adopts... As the network layer for feature extraction, j is the j-th network layer. Let N be the number of elements in the j-th network layer, and N be the number of network layers; the expression for the reconstruction loss function is: ;in, To rebuild the losses, It is in the 1 normal form.

7. The dual-generator panoramic image restoration method based on cube projection format according to claim 5, characterized in that, The discriminator consists of two layers: a global discriminator and a face discriminator. The global discriminator is composed of four convolutional layers with 4×4 kernels, a stride of 2, and a padding of 1, and one linear layer. It is used to identify the correlation of the six faces of a stereo image. The face discriminator is composed of five convolutional layers with 4×4 kernels, a stride of 2, and a padding of 1, and one linear layer. It is used to identify the authenticity of the four faces of a rectangle.

8. The method for dual-generator panoramic image restoration based on cube projection format according to claim 5, characterized in that, The expression for the global loss function is: ;in, For the overall loss, To rebuild the losses, To combat the losses, In order to perceive loss, For the loss of style, 、 、 、 These are the corresponding hyperparameters.

9. The dual-generator panoramic image restoration method based on cube projection format according to claim 8, characterized in that, The expression for the adversarial loss function is: ;in, To combat the losses, For real images, The image is the restored image, and D is the discriminator. As expected, This represents the distribution of image data output by the generator. Represents the distribution of real image data. This indicates the distribution of the restored image data. This indicates a gradient calculation operation.

Images