Panoramic image restoration method, device and medium

Abstract

The present invention discloses a panoramic image restoration method, device, and medium. The method maps the annotated points of an object to be removed to an ERP image to generate a mask region for the object to be removed. Global features and local texture features are extracted from the input panoramic image, and the extracted features are fused to generate a feature parameter map. A normal map of the panoramic image is calculated using a feature fusion method that combines equirectangular projection and cubic projection. Position encoding is performed on the mask region, and the resulting encoding result, the feature parameter map, and the normal map are input into a multilayer perceptron to reconstruct the coordinate pixel values of the mask region to obtain a restored image. This application scheme can reduce distortion in panoramic image restoration and improve restoration effectiveness.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a panoramic image restoration method, device and medium. Background Art

[0002] With the development of augmented reality and virtual reality technologies, consumer-grade panoramic cameras are gradually entering the market. Compared to traditional cameras, these cameras can capture a wider field of view, providing users with an immersive visual experience. 360° panoramic images have been widely used in various fields such as tourism, real estate, and education. As an important virtual reality resource, they provide rich scene information and greatly enhance the viewer experience. Panoramic image restoration aims to restore missing areas in panoramic images caused by object deletion or damage by simulating the human visual restoration process, thereby improving image integrity and realism.

[0003] Existing image restoration methods mostly apply to two-dimensional images, with the ERP (Equirectangular Projection) format being the most commonly used projection format. However, these 2D-based methods often overlook the distortion introduced by the ERP format in panoramic images. Unlike traditional two-dimensional images, panoramic images exhibit varying degrees of distortion during the sampling process from the equator to the poles. Therefore, deep learning models trained on 2D images are not suitable for panoramic images, resulting in distortion and poor restoration results. Summary of the Invention

[0004] In order to solve the above problems, the present invention proposes a panoramic image restoration method, device and medium to reduce the distortion of panoramic image restoration and improve the restoration effect.

[0005] An embodiment of the present invention provides a panoramic image restoration method, the method comprising:

[0006] Mapping the marked points of the object to be removed to the ERP image to generate a mask area of the object to be removed;

[0007] Extract global features and local texture features from the input panoramic image, fuse the extracted features, and generate a feature parameter map;

[0008] Calculating a normal map of the panoramic image by adopting a feature fusion method of a mixture of equirectangular projection and cubic projection;

[0009] Position encoding is performed on the mask area, and the obtained encoding result, the feature parameter map, and the normal map are input into a multi-layer perceptron to reconstruct the coordinate pixel values of the mask area to obtain a repaired image.

[0010] Preferably, mapping the annotation points of the object to be removed to the ERP image to generate a mask area of the object to be removed includes:

[0011] Obtaining the marked points of the object to be removed marked by the user in the three-dimensional view;

[0012] Mapping the annotation points in the three-dimensional view to two-dimensional pixel coordinates in the ERP image;

[0013] The two-dimensional pixel coordinates are calculated using a pre-trained semantic segmentation model to generate a mask area for the object to be removed.

[0014] As a preferred solution, global features and local texture features are extracted from the input panoramic image, and the extracted features are fused to generate a feature parameter map, including:

[0015] Resampling the panoramic image and extracting features of the image through a fully convolutional encoder to obtain a feature map;

[0016] Performing a fast Fourier transform on the feature map to extract global context information of the image and obtain a frequency domain representation;

[0017] Processing the frequency domain representation through a convolutional layer and performing an inverse fast Fourier transform to obtain the global feature;

[0018] Performing local texture extraction on the feature map to obtain local texture features;

[0019] Cascading the global features and the local texture features to obtain a spatial attention feature map;

[0020] The spatial attention feature map is upsampled to obtain the feature parameter map.

[0021] Furthermore, the spatial attention feature map ;

[0022] Among them, 9× is the scale factor, F is the feature map, is the global feature, is the local texture feature, Indicates a cascade operation.

[0023] Preferably, the normal map of the panoramic image is calculated by adopting a feature fusion method of a mixture of equirectangular projection and cubic projection, including:

[0024] fusing the features of the equirectangular projection and the features of the cubic projection of the panoramic image to obtain fused features;

[0025] performing weighted calculation on the fusion features according to the latitude of the panoramic image, and correcting the coordinates of the fusion features;

[0026] Generate a predicted normal map based on the corrected fusion features and determine the predicted normal of each pixel in the image;

[0027] The main direction of the panoramic image is determined, and the predicted normal is corrected according to the main direction to obtain the normal map.

[0028] Furthermore, determining the main direction of the panoramic image and correcting the predicted normal according to the main direction to obtain the normal map includes:

[0029] Performing edge detection on the panoramic image to extract edge points;

[0030] Projecting the pixels of the edge points to spherical coordinates, generating a prediction vector corresponding to the 3D projection plane by randomly sampling two points and calculating the cross product, calculating the optimal vanishing point using the RANSAC algorithm, and finding the main direction of the panoramic image;

[0031] Align the main direction with the predicted normal, and determine the main direction closest to the predicted normal as the corrected normal;

[0032] Calculate the similarity between the predicted normal and the corrected normal;

[0033] When the calculated similarity is greater than a preset correction threshold, the number of corrections is calculated, the correction threshold is updated, the main direction of the panoramic image is re-determined, the predicted normal is corrected according to the main direction, and the similarity between the predicted normal and the corrected normal is recalculated until the calculated similarity is no greater than the updated correction threshold;

[0034] When the calculated similarity is not greater than the correction threshold, determining the normal map using the corrected normal;

[0035] Among them, the updated correction threshold , and is a preset hyperparameter, is the number of corrections.

[0036] Preferably, position encoding is performed on the mask area, and the obtained encoding result, the feature parameter map, and the normal map are input into a multi-layer perceptron to reconstruct the coordinate pixel values of the mask area to obtain a repaired image, comprising:

[0037] Encoding the pixel coordinates of the mask area using a position encoding function to obtain the encoding result;

[0038] Inputting the encoding result, the feature parameter map, and the normal map into the multilayer perceptron to reconstruct the pixel coordinate value of each coordinate position of the mask area;

[0039] A cascade operation is performed on the pixel value of each coordinate position of the mask area and the panoramic image to obtain the repaired image.

[0040] An embodiment of the present invention further provides a panoramic image restoration device, comprising:

[0041] A mask generation module, configured to map the annotation points of the object to be removed to the ERP image to generate a mask region of the object to be removed;

[0042] The aggregation module is used to extract global features and local texture features of the input panoramic image, and fuse the extracted features to generate a feature parameter map;

[0043] a normal estimation module, configured to calculate a normal map of the panoramic image by adopting a feature fusion method of a mixture of equirectangular projection and cubic projection;

[0044] The restoration module is used to perform position encoding on the mask area, and input the obtained encoding result, the feature parameter map and the normal map into a multi-layer perceptron to reconstruct the coordinate pixel values of the mask area to obtain a restored image.

[0045] An embodiment of the present invention also provides a panoramic image restoration device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, a panoramic image restoration method as described in any of the above embodiments is implemented.

[0046] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the panoramic image restoration method as described in any of the above embodiments.

[0047] The present invention provides a panoramic image restoration method, device, and medium. These methods generate a mask region for the object to be removed by mapping the annotated points of the object to be removed to an ERP image; extract global features and local texture features from the input panoramic image, and fuse the extracted features to generate a feature parameter map; calculate a normal map for the panoramic image using a feature fusion method that combines equirectangular projection and cubic projection; positionally encode the mask region, and input the resulting encoding, the feature parameter map, and the normal map into a multilayer perceptron to reconstruct the coordinate pixel values of the mask region and obtain a restored image. This application scheme can reduce distortion in panoramic image restoration and improve restoration effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 1 is a flow chart of a panoramic image restoration method provided by an embodiment of the present invention;

[0049] Figure 2 1 is a schematic structural diagram of a panoramic image restoration device provided by an embodiment of the present invention;

[0050] Figure 3 This is another structural schematic diagram of a panoramic image restoration device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0052] With the continuous expansion of panoramic multimedia content, research on panoramic image restoration has gradually attracted the attention of numerous scholars. Existing methods mostly apply to 2D images, with the ERP (Equirectangular Projection) format being the most commonly used projection format. However, these 2D-based methods often overlook the distortion introduced by the ERP format in panoramic images. Unlike traditional 2D images, panoramic images exhibit varying degrees of distortion during sampling from the equator to the poles. Therefore, deep learning models trained on 2D images are not suitable for panoramic images.

[0053] Although existing panoramic image restoration methods have taken the distortion problem of ERP format into consideration, CMP images or polyhedron projection images are usually used to reduce the distortion of panoramic images. However, these methods fail to fully exploit the geometric information in panoramic images, which will lead to distortion of panoramic images and affect the restoration effect.

[0054] In order to solve the above technical problems, this application proposes a panoramic image restoration method. Figure 1 , is a flow chart of a panoramic image restoration method provided by an embodiment of the present invention, the method comprising steps S1 to S4;

[0055] Step S1, mapping the marked points of the object to be removed to the ERP image to generate a mask area of the object to be removed;

[0056] Step S2, extracting global features and local texture features from the input panoramic image, and fusing the extracted features to generate a feature parameter map;

[0057] Step S3, calculating a normal map of the panoramic image by adopting a feature fusion method of a mixture of equirectangular projection and cubic projection;

[0058] Step S4: Position encoding is performed on the mask area, and the obtained encoding result, the feature parameter map, and the normal map are input into a multi-layer perceptron to reconstruct the coordinate pixel values of the mask area to obtain a repaired image.

[0059] In this embodiment, the area of the object to be removed is annotated through user interaction. The user can mark points in the 3D view, and then map these marked points back to the ERP image to generate a masked area of the object to be removed. This interactive method improves the flexibility and accuracy of annotation and enhances the user experience.

[0060] By combining global context with local spatial information through Fast Fourier Transform (FFT) and convolution operations, global and local texture features are extracted and fused to generate a feature parameter map. This effectively extracts both high- and low-frequency features, improving image restoration. Capturing both global and local features within a 360° panoramic image ensures more consistent structure and texture within the restored area.

[0061] In order to solve the problem of unsatisfactory results caused by geometric distortion in the panoramic image restoration process, a preset dual-branch normal prediction network is used to estimate the normal map of the panoramic image through hybrid projection fusion, thereby enhancing the three-dimensional sense and lighting effect of the image, and effectively improving the geometric structure consistency and realism of the restoration area.

[0062] In order to optimize the processing efficiency of high-resolution panoramic images, during image restoration, an image is generated only within the mask area, the mask area is position-encoded, and the resulting encoding result, the feature parameter map, and the normal map are input into a multi-layer perceptron to reconstruct the pixel value at each coordinate of the mask area to obtain a restored image.

[0063] Pixel generation is performed through coordinate query to reduce unnecessary computational burden, and the generation effect of high-frequency details is enhanced through coordinate encoding and multi-layer perceptron network.

[0064] The solution of this application can solve the problems of geometric distortion and high-frequency detail retention encountered by traditional methods when processing panoramic images. The restoration effect is improved through coordinate query and spherical normal constraints. Compared with the traditional cube projection, this case effectively solves the distortion problem of panoramic images caused by projection by introducing spherical normal estimation. Spherical normals can capture the geometric characteristics in the image, thereby more accurately restoring the three-dimensional structure of the scene, and can enhance the lighting effect to avoid the distortion in the polar regions affecting the restoration results. Pixel query and generation are only performed within the mask area, avoiding unnecessary full-image calculations, greatly optimizing memory usage and computing efficiency, and are particularly suitable for high-resolution panoramic image restoration scenarios.

[0065] In another embodiment of the present invention, the specific implementation of step S1 includes:

[0066] Obtaining the marked points of the object to be removed marked by the user in the three-dimensional view;

[0067] Mapping the annotation points in the three-dimensional view to two-dimensional pixel coordinates in the ERP image;

[0068] The two-dimensional pixel coordinates are calculated using a pre-trained semantic segmentation model to generate a mask area for the object to be removed.

[0069] During the specific implementation of this embodiment, since the panoramic image is usually presented in the form of equirectangular projection, the two-dimensional image pixels need to be mapped to the three-dimensional coordinates on the spherical surface to simulate the view and interaction effects in the 3D space.

[0070] The user can interact with the panoramic image in the 3D view and mark the objects that they want to remove. The user marks the location of the objects to be removed in the 3D view, and these marks will be mapped back to the 2D pixel coordinates in the ERP image. .

[0071] For the two-dimensional pixel coordinates in the ERP image , which is converted to spherical coordinates using the following formula and Cartesian coordinates , realizing the conversion between two-dimensional pixel coordinates and three-dimensional viewing angle coordinates.

[0072] ;

[0073] ;

[0074] in, are the three coordinates of the spherical coordinate system, and are the width and height of the ERP image respectively. Then the spherical coordinates are converted to Cartesian coordinates. To simulate the effect in 3D space. Through coordinate transformation, we get , , .

[0075] It should be noted that the conversion between two-dimensional pixel coordinates and three-dimensional viewing angle coordinates can be inferred based on the above formula.

[0076] The pre-trained semantic segmentation model (SAM) automatically generates the mask area of the object to be removed .

[0077] in, , is the input panoramic image. SAM() is the semantic segmentation model function.

[0078] The generated mask is then fed into subsequent modules of the framework to guide image inpainting and restoration. This coordinate-query-based mask generation method not only improves interactivity but can also be combined with efficient feature extraction and inpainting mechanisms to achieve more refined and natural inpainting of the masked area.

[0079] In another embodiment provided by the present invention, the step S2 specifically includes:

[0080] Resampling the panoramic image and extracting features of the image through a fully convolutional encoder to obtain a feature map;

[0081] Performing a fast Fourier transform on the feature map to extract global context information of the image and obtain a frequency domain representation;

[0082] Processing the frequency domain representation through a convolutional layer and performing an inverse fast Fourier transform to obtain the global feature;

[0083] Performing local texture extraction on the feature map to obtain local texture features;

[0084] Cascading the global features and the local texture features to obtain a spatial attention feature map;

[0085] The spatial attention feature map is upsampled to obtain the feature parameter map.

[0086] When this embodiment is implemented, the input panoramic image It is resampled to a fixed low resolution to reduce the amount of computation, and the features of the input panoramic image are extracted through a fully convolutional encoder. The extracted feature map is represented as ,in is the height, is the width, and C is the number of channels.

[0087] The feature map is processed using Fast Fourier Transform (FFT) to capture the global context information of the image. Fast Fourier Transform can better process global features in the frequency domain.

[0088] First, the feature map Perform a 1×1 convolution, batch normalization, and ReLU activation to obtain an enhanced feature map Apply two-dimensional fast Fourier transform FFT2d, which only retains half of the true value of the spectrum to reduce the amount of calculation. The calculation process is .

[0089] The real and imaginary parts after the Fourier transform are combined to form the final frequency domain representation.

[0090] The frequency domain representation is restored to the spatial structure through a 1×1 convolution layer and inverse fast Fourier transform IFFT to obtain the global features. .

[0091] Fourier convolution can capture global information, but relying solely on frequency domain information is insufficient for repairing large masked areas, especially when detail textures are involved. Therefore, this embodiment also designs a local texture extraction unit to specifically address these details and perform local texture extraction on the feature map.

[0092] Local texture features Extracted from the original feature map, the texture details of the repaired area are retained, ensuring that high-frequency detail areas (such as edges and object surfaces) are well restored.

[0093] Global Features and local texture features are concatenated to synthesize a spatial attention feature .

[0094] In order to restore the original resolution, the fused feature map is upsampled to obtain a feature parameter map with the same resolution as the input image. , used for subsequent mask area pixel query and repair.

[0095] By combining global and local features, we ensure that the restored image can retain high-frequency structure and low-frequency texture, thereby improving the overall restoration quality and detail retention.

[0096] In another embodiment of the present invention, the spatial attention feature map ;

[0097] Among them, 9× is the scale factor, F is the feature map, is the global feature, is the local texture feature, Indicates a cascade operation.

[0098] In the specific implementation of this embodiment, the global feature and local texture features Spatial attention features obtained by concatenation ;

[0099] in, represents the cascade operation, and 9× is the scale factor, which ensures that the global and local features are fused to adapt to the feature expression of high-resolution images.

[0100] In another embodiment provided by the present invention, the step S3 specifically includes:

[0101] fusing the features of the equirectangular projection and the features of the cubic projection of the panoramic image to obtain fused features;

[0102] performing weighted calculation on the fusion features according to the latitude of the panoramic image, and correcting the coordinates of the fusion features;

[0103] Generate a predicted normal map based on the corrected fusion features and determine the predicted normal of each pixel in the image;

[0104] The main direction of the panoramic image is determined, and the predicted normal is corrected according to the main direction to obtain the normal map.

[0105] In the specific implementation of this embodiment, in order to better process the geometric characteristics of the panoramic image, the present application scheme designs a dual-branch normal prediction network, which combines the advantages of the two projection formats of equirectangular projection (ERP) and cubic projection (CMP), and fuses the features of the equirectangular projection and the features of the cubic projection of the panoramic image.

[0106] The features of the two projections are combined and fused at different stages of the decoder, leveraging their complementary strengths to accurately recover the image normal information. Since the latitude and longitude coordinates of ERP images produce different sampling rates in polar regions, a spherical weighting mechanism is introduced to weight features based on the image's latitude to avoid overfitting in these areas.

[0107] The fusion feature is weighted according to the latitude of the panoramic image, and the coordinates of the fusion feature are corrected. The calculation formula is: ;

[0108] in, Represents the pixel coordinates of the image. The weighted features can better handle geometric distortion problems at different latitudes.

[0109] The fused features are used to generate a normal map through a network module called NormalHead The surface normal of each pixel in the panoramic image is predicted. Due to the geometric characteristics of the spherical surface, especially in the polar regions, the normal prediction may be inaccurate or noisy, so further correction is required.

[0110] In order to correct the predicted normals, the module introduces a main direction correction mechanism to determine the main direction of the panoramic image and correct the predicted normals according to the main direction to obtain the normal map.

[0111] This application solution is applicable to 360° panoramic images, particularly those using the equirectangular projection (ERP) format. Due to the spherical geometry of these images, severe geometric distortion occurs in the polar regions. Conventional 2D image restoration methods are unable to accurately restore the surface information of these distorted regions. By estimating the image's surface normals, these geometric distortions can be effectively corrected, and the lighting effect after object removal can be enhanced, improving the three-dimensional perception.

[0112] In another embodiment of the present invention, the process of correcting the prediction vector according to the main direction specifically includes the following steps:

[0113] Performing edge detection on the panoramic image to extract edge points;

[0114] Projecting the pixels of the edge points to spherical coordinates, generating a prediction vector corresponding to the 3D projection plane by randomly sampling two points and calculating the cross product, calculating the optimal vanishing point using the RANSAC algorithm, and finding the main direction of the panoramic image;

[0115] Align the main direction with the predicted normal, and determine the main direction closest to the predicted normal as the corrected normal;

[0116] Calculate the similarity between the predicted normal and the corrected normal;

[0117] When the calculated similarity is greater than a preset correction threshold, the number of corrections is calculated, the correction threshold is updated, the main direction of the panoramic image is re-determined, the predicted normal is corrected according to the main direction, and the similarity between the predicted normal and the corrected normal is recalculated until the calculated similarity is no greater than the updated correction threshold;

[0118] When the calculated similarity is not greater than the correction threshold, determining the normal map using the corrected normal;

[0119] Among them, the updated correction threshold , and is a preset hyperparameter, is the number of corrections.

[0120] In the specific implementation of this embodiment, in order to correct the predicted normals, a principal direction correction mechanism is introduced based on the Manhattan World assumption that surfaces in indoor scenes (such as walls, floors, and ceilings) generally follow a principal axis direction (vertical or horizontal).

[0121] Extract edge points by performing Canny edge detection on the panoramic image;

[0122] The edge points are projected onto spherical coordinates, and the normal vector corresponding to the 3D projection plane is generated by randomly sampling two points and calculating their cross product.

[0123] Use the RANSAC algorithm to calculate the optimal vanishing point and find the main direction of the image , there are usually multiple main directions, such as perpendicular to the ground or parallel to the wall.

[0124] By main direction With predicted normal Align and calculate predicted normals With the main direction The cosine similarity between

[0125] Select the main direction closest to the predicted normal as the corrected normal , ;

[0126] When the calculated similarity is greater than the preset correction threshold, the number of corrections is calculated, the correction threshold is updated, the main direction of the panoramic image is re-determined, and the predicted normal is corrected according to the main direction. Gradually increase: ;in, and is a hyperparameter that controls the sensitivity of the normal correction.

[0127] When the calculated similarity is not greater than the correction threshold, determining the normal map using the corrected normal;

[0128] In order to optimize the normal estimation module during training, a normal constraint loss function was designed in the study. , which measures the difference between the predicted normal and the true normal:

[0129] ;

[0130] in, is the number of pixels, is the main direction mask, is the true normal.

[0131] In another embodiment provided by the present invention, the step S4 specifically includes:

[0132] Encoding the pixel coordinates of the mask area using a position encoding function to obtain the encoding result;

[0133] Inputting the encoding result, the feature parameter map, and the normal map into the multilayer perceptron to reconstruct the pixel coordinate value of each coordinate position of the mask area;

[0134] A cascade operation is performed on the pixel value of each coordinate position of the mask area and the panoramic image to obtain the repaired image.

[0135] In the specific implementation of this embodiment, the input is a high-resolution panoramic image and its mask image. First, frequency-space feature aggregation is performed to generate a feature parameter map The feature parameter map contains all the high- and low-frequency information in the input image that needs to be repaired. To enhance the processing of high-frequency details, the input coordinates are first encoded by the position encoding module and then input into the MLP (Multi-Layer Perceptron) for further processing.

[0136] Positional encoding function Encode the pixel coordinates of the masked area, is the two-dimensional pixel coordinate, is the height and width of the image. The result of position encoding will be combined with the normal map and feature parameter map The data are input to the MLP layer to ensure the processing of high-frequency information.

[0137] After position encoding, the coordinates of each pixel in the mask area are input into the MLP (Multi-layer Perceptron) for query. The MLP reconstructs the value of each pixel based on the input mask area coordinates, feature parameter map and normal map: .

[0138] in, Represents the position of each pixel obtained by the query The pixel value of .

[0139] The corresponding pixel value is generated pixel by pixel within the masked area, and the normal map and feature parameter map are combined to calculate each pixel to ensure that the high-frequency details and geometric structure of the repaired area are accurately restored. This pixel-by-pixel generation method can effectively handle the boundary details and complex textures of the masked area, ensuring a smooth transition of the repair.

[0140] Finally, the pixel values of the mask area With the initial input image , generate a complete repaired image :

[0141] ;

[0142] Pixel coordinate queries for masked regions utilize position encoding and an MLP network, combined with normal maps and feature parameter maps, to accurately generate pixel-by-pixel representations of masked regions. This implicit representation not only reduces unnecessary full-image computations but also effectively handles details in high-resolution images, significantly improving computational efficiency and restoration quality.

[0143] By querying the pixel coordinates within the mask area, the pixel values within the repair area are selectively generated, avoiding redundant calculations of the entire image, thereby greatly improving memory usage efficiency and computing speed.

[0144] This application's solution is based on an evaluation on the SD dataset, using commonly used image quality assessment metrics to measure inpainting performance. The SD dataset is constructed from the Structured3D dataset through a series of preprocessing steps. The training set of this dataset contains 14,527 images, and the validation and test sets each contain 2,000 images. The experiment used the following three commonly used image quality assessment metrics to measure inpainting performance. PSNR (Peak Signal-to-Noise Ratio) measures the quality of image reconstruction. Higher values indicate smaller differences between the inpainted image and the ground-truth image, indicating better inpainting results. SSIM (Structural Similarity Index) assesses the similarity between two images in terms of brightness, contrast, and structure. Values closer to 1 indicate a closer approximation of the image's structure to the ground-truth scene. LPIPS (Perceptual Image Patch Similarity) measures the perceptual quality of image inpainting. Lower values indicate a perceptual closer approximation of the inpainted image to the ground-truth image. The proposed model was compared with six existing baseline models: CoMod-GAN, Palette, Inst, CoordFill, LII, and PIINET. The results of the model performance comparison are given in Table 1. It is obvious from this table that the model proposed in this application achieves good performance on the SD dataset compared with other benchmark models.

[0145] Table 1 Performance comparison evaluation table

[0146]

[0147] Compared with existing models, the present invention demonstrates significant advantages in evaluation indicators. Existing models such as CoMod-GAN and Palette have certain performance in image structural similarity, but poor perceptual quality. Inst and PIINET have shortcomings in repairing high-frequency details and complex geometric structures, and CoordFill performs averagely in terms of accuracy and structural restoration. In contrast, the present invention surpasses these methods in key indicators such as PSNR, SSIM, and LPIPS, demonstrating higher restoration accuracy, detail preservation capabilities, and more natural visual perception. In particular, when processing complex panoramic images, the present invention can better maintain geometric consistency and light and shadow transitions, and the overall restoration effect is better than existing technologies.

[0148] In addition, when Palette handles large-area image restoration, the lighting transition appears abrupt, and the visual quality of the repaired area is not natural enough. Inst performs poorly in complex background filling and texture consistency, and is prone to artifacts and discontinuous textures. Although PIINET uses cubic projection to reduce geometric distortion, it still suffers from distortion and blurring when repairing polar regions and edge details. Although CoordFill can maintain a certain degree of structural consistency, it is not ideal in preserving high-frequency details and texture transitions, and the transition between the repaired area and the surrounding area is relatively abrupt.

[0149] In contrast, the restored image produced by our method is highly visually consistent with the undamaged area, and the restored area appears remarkably natural in terms of detail, structure, and lighting transitions. This demonstrates that our method outperforms existing technologies in addressing structural consistency, texture detail, lighting and shadow consistency, and geometric distortion in extreme regions. Our method demonstrates a particularly strong advantage in terms of the naturalness of the restored image and the preservation of high-frequency detail, providing a superior solution for panoramic image restoration.

[0150] In another embodiment provided by the present invention, see Figure 2 , is a schematic structural diagram of a panoramic image restoration device provided by an embodiment of the present invention, the device comprising:

[0151] A mask generation module, configured to map the annotation points of the object to be removed to the ERP image to generate a mask region of the object to be removed;

[0152] The aggregation module is used to extract global features and local texture features of the input panoramic image, and fuse the extracted features to generate a feature parameter map;

[0153] a normal estimation module, configured to calculate a normal map of the panoramic image by adopting a feature fusion method of a mixture of equirectangular projection and cubic projection;

[0154] The restoration module is used to perform position encoding on the mask area, and input the obtained encoding result, the feature parameter map and the normal map into a multi-layer perceptron to reconstruct the coordinate pixel values of the mask area to obtain a restored image.

[0155] The panoramic image restoration device provided in this embodiment can execute all the steps and functions of the panoramic image restoration method provided in any of the above embodiments, and the specific functions of the device will not be described in detail here.

[0156] See also Figure 3 , is another structural diagram of a panoramic image restoration device provided by an embodiment of the present invention. The panoramic image restoration device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a panoramic image restoration program. When the processor executes the computer program, the steps of each of the above-mentioned panoramic image restoration method embodiments are implemented, such as Figure 1 Alternatively, the processor implements the functions of the modules in the above-mentioned device embodiments when executing the computer program.

[0157] Exemplarily, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the panoramic image restoration device. For example, the computer program can be divided into various modules, and the specific functions of each module have been described in detail in the panoramic image restoration method provided in any of the above embodiments. The specific functions of the device are not repeated here.

[0158] The panoramic image restoration device can be a computing device such as a desktop computer, laptop, PDA, or cloud server. The panoramic image restoration device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the schematic diagram is merely an example of a panoramic image restoration device and does not constitute a limitation on the device. The device may include more or fewer components than shown, or a combination of certain components, or different components. For example, the panoramic image restoration device may also include input and output devices, network access devices, buses, and the like.

[0159] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the panoramic image restoration device, and utilizes various interfaces and lines to connect various parts of the panoramic image restoration device.

[0160] The memory can be used to store the computer programs and / or modules. The processor implements the various functions of the panoramic image restoration device by running or executing the computer programs and / or modules stored in the memory and accessing the data stored in the memory. The memory may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function or an image playback function); the data storage area may store data generated based on the use of the mobile phone (such as audio data, a phone book, etc.). Furthermore, the memory may include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0161] If the integrated module of the panoramic image restoration device is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the process steps in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium.

[0162] It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A panoramic image restoration method, characterized in that: The method comprises: Mapping the marked points of the object to be removed to the ERP image to generate a mask area of the object to be removed; Extract global features and local texture features from the input panoramic image, fuse the extracted features, and generate a feature parameter map; Calculating a normal map of the panoramic image by adopting a feature fusion method of a mixture of equirectangular projection and cubic projection; Performing position encoding on the masked area, inputting the obtained encoding result, the feature parameter map, and the normal map into a multi-layer perceptron to reconstruct the coordinate pixel values of the masked area to obtain a repaired image; The normal map of the panoramic image is calculated by using a feature fusion method of a mixture of equirectangular projection and cubic projection, including: fusing the features of the equirectangular projection and the features of the cubic projection of the panoramic image to obtain fused features; performing weighted calculation on the fusion features according to the latitude of the panoramic image, and correcting the coordinates of the fusion features; Generate a predicted normal map based on the corrected fusion features and determine the predicted normal of each pixel in the image; The main direction of the panoramic image is determined, and the predicted normal is corrected according to the main direction to obtain the normal map.

2. The panoramic image restoration method according to claim 1, wherein: Mapping the annotation points of the object to be removed to the ERP image to generate a mask area of the object to be removed includes: Obtaining the marked points of the object to be removed marked by the user in the three-dimensional view; Mapping the annotation points in the three-dimensional view to two-dimensional pixel coordinates in the ERP image; The two-dimensional pixel coordinates are calculated using a pre-trained semantic segmentation model to generate a mask area for the object to be removed.

3. The panoramic image restoration method according to claim 1, wherein: Extract global features and local texture features from the input panoramic image, fuse the extracted features, and generate a feature parameter map, including: Resampling the panoramic image and extracting features of the image through a fully convolutional encoder to obtain a feature map; Performing a fast Fourier transform on the feature map to extract global context information of the image and obtain a frequency domain representation; Processing the frequency domain representation through a convolutional layer and performing an inverse fast Fourier transform to obtain the global feature; Performing local texture extraction on the feature map to obtain local texture features; Cascading the global features and the local texture features to obtain a spatial attention feature map; The spatial attention feature map is upsampled to obtain the feature parameter map.

4. The panoramic image restoration method according to claim 3, wherein: The spatial attention feature map ; Among them, 9× is the scale factor, F is the feature map, is the global feature, is the local texture feature, Indicates a cascade operation.

5. The panoramic image restoration method according to claim 1, wherein: Determining a main direction of the panoramic image and correcting the predicted normal according to the main direction to obtain the normal map includes: Performing edge detection on the panoramic image to extract edge points; Projecting the pixels of the edge points to spherical coordinates, generating a prediction vector corresponding to the 3D projection plane by randomly sampling two points and calculating the cross product, calculating the optimal vanishing point using the RANSAC algorithm, and finding the main direction of the panoramic image; Align the main direction with the predicted normal, and determine the main direction closest to the predicted normal as the corrected normal; Calculate the similarity between the predicted normal and the corrected normal; When the calculated similarity is greater than a preset correction threshold, the number of corrections is calculated, the correction threshold is updated, the main direction of the panoramic image is re-determined, the predicted normal is corrected according to the main direction, and the similarity between the predicted normal and the corrected normal is recalculated until the calculated similarity is no greater than the updated correction threshold; When the calculated similarity is not greater than the correction threshold, determining the normal map using the corrected normal; Among them, the updated correction threshold , and is a preset hyperparameter, is the number of corrections.

6. The panoramic image restoration method according to claim 1, wherein: Position encoding is performed on the mask area, and the obtained encoding result, the feature parameter map, and the normal map are input into a multi-layer perceptron to reconstruct the coordinate pixel values of the mask area to obtain a repaired image, including: Encoding the pixel coordinates of the mask area using a position encoding function to obtain the encoding result; Inputting the encoding result, the feature parameter map, and the normal map into the multilayer perceptron to reconstruct the pixel coordinate value of each coordinate position of the mask area; A cascade operation is performed on the pixel value of each coordinate position of the mask area and the panoramic image to obtain the repaired image.

7. A panoramic image restoration device, characterized in that: The device comprises: A mask generation module, configured to map the annotation points of the object to be removed to the ERP image to generate a mask region of the object to be removed; The aggregation module is used to extract global features and local texture features of the input panoramic image, and fuse the extracted features to generate a feature parameter map; a normal estimation module, configured to calculate a normal map of the panoramic image by adopting a feature fusion method of a mixture of equirectangular projection and cubic projection; a restoration module, configured to perform position encoding on the masked area, and input the obtained encoding result, the feature parameter map, and the normal map into a multilayer perceptron to reconstruct the coordinate pixel values of the masked area to obtain a restored image; The normal estimation module is specifically used for: fusing the features of the equirectangular projection and the features of the cubic projection of the panoramic image to obtain fused features; performing weighted calculation on the fusion features according to the latitude of the panoramic image, and correcting the coordinates of the fusion features; Generate a predicted normal map based on the corrected fusion features and determine the predicted normal of each pixel in the image; The main direction of the panoramic image is determined, and the predicted normal is corrected according to the main direction to obtain the normal map.

8. A panoramic image restoration device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the panoramic image restoration method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the panoramic image restoration method according to any one of claims 1 to 6.

Images