Panoramic image restoration method and device and medium

Abstract

The invention discloses a panoramic image restoration method and device and a medium, and the method comprises the steps: generating a mask region of a to-be-removed object through mapping a marking point of the to-be-removed object to an ERP image; extracting global features and local texture features of the input panoramic image, and fusing the extracted features to generate a feature parameter graph; calculating a normal graph of the panoramic image by adopting a feature fusion mode of mixing equidistant columnar projection and cubic projection; and carrying out position coding on the mask area, and inputting an obtained coding result, the characteristic parameter graph and the normal graph into a multi-layer perceptron to reconstruct a coordinate pixel value of the mask area so as to obtain a repaired image. According to the scheme, distortion of panoramic image restoration can be reduced, and the restoration effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a panoramic image restoration method, device and medium. Background Art

[0002] With the development of augmented reality and virtual reality technologies, consumer-grade panoramic cameras have gradually entered the market. Compared with traditional cameras, these panoramic cameras can capture a wider field of view and bring users an immersive visual experience. 360° panoramic images have been widely used in many fields such as tourism, real estate, and education. As an important virtual reality resource, they provide rich scene information and greatly enhance the viewer's experience. Panoramic image restoration aims to restore the missing areas in panoramic images caused by object deletion or damage by simulating the human visual restoration process, thereby improving the integrity and realism of the image.

[0003] Most existing image restoration methods are applied to two-dimensional images, and the ERP (EquiRectangularProjection) format is currently the most commonly used projection form. However, these methods based on two-dimensional images often ignore the distortion problem caused by the ERP format in panoramic images. Unlike traditional two-dimensional plane images, panoramic images will have different degrees of distortion during the sampling process from the equator to the poles. Therefore, the deep learning model trained on 2D images is not suitable for panoramic images, which will cause distortion of panoramic images and affect the restoration effect. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a panoramic image restoration method, device and medium to reduce the distortion of panoramic image restoration and improve the restoration effect.

[0005] An embodiment of the present invention provides a panoramic image restoration method, the method comprising: Mapping the marked points of the object to be removed to the ERP image to generate a mask area of ​​the object to be removed; Extract global features and local texture features of the input panoramic image, fuse the extracted features, and generate a feature parameter map; Calculating the normal map of the panoramic image by using a feature fusion method that combines equirectangular projection and cubic projection; Position encoding is performed on the mask area, and the obtained encoding result, the feature parameter map and the normal map are input into a multi-layer perceptron to reconstruct the coordinate pixel values ​​of the mask area to obtain a repaired image.

[0006] Preferably, mapping the annotation points of the object to be removed to the ERP image to generate a mask area of ​​the object to be removed includes: Obtaining the marked points of the object to be removed marked by the user in the three-dimensional view; Mapping the annotation points in the three-dimensional view to two-dimensional pixel coordinates in the ERP image; The two-dimensional pixel coordinates are calculated using a pre-trained semantic segmentation model to generate a mask area of ​​the object to be removed.

[0007] As a preferred solution, global features and local texture features are extracted from the input panoramic image, and the extracted features are fused to generate a feature parameter map, including: Resampling the panoramic image, and extracting features of the image through a fully convolutional encoder to obtain a feature map; Performing a fast Fourier transform on the feature map to extract global context information of the image and obtain a frequency domain representation; Processing the frequency domain representation through a convolution layer and performing an inverse fast Fourier transform to obtain the global feature; Performing local texture extraction on the feature map to obtain local texture features; Cascading the global feature and the local texture feature to obtain a spatial attention feature map; The spatial attention feature map is upsampled to obtain the feature parameter map.

[0008] Furthermore, the spatial attention feature map ; Among them, 9× is the scale factor, F is the feature map, is the global feature, is the local texture feature, Indicates a cascade operation.

[0009] Preferably, the normal map of the panoramic image is calculated by using a feature fusion method of a mixture of equirectangular projection and cubic projection, including: Fusing the features of the equirectangular projection and the features of the cubic projection of the panoramic image to obtain fused features; Performing weighted calculation on the fusion feature according to the latitude of the panoramic image, and correcting the coordinates of the fusion feature; Generate a predicted normal map based on the corrected fusion features to determine the predicted normal of each pixel in the image; The main direction of the panoramic image is determined, and the predicted normal is corrected according to the main direction to obtain the normal map.

[0010] Further, determining the main direction of the panoramic image and correcting the predicted normal according to the main direction to obtain the normal map includes: Performing edge detection on the panoramic image to extract edge points; Projecting the pixels of the edge points to spherical coordinates, generating a prediction vector corresponding to the 3D projection plane by randomly sampling two points and calculating the cross product, calculating the best vanishing point using the RANSAC algorithm, and finding the main direction of the panoramic image; Align the main direction with the predicted normal, and determine the main direction closest to the predicted normal as the corrected normal; Calculate the similarity between the predicted normal and the corrected normal; When the calculated similarity is greater than a preset correction threshold, the number of corrections is calculated, the correction threshold is updated, the main direction of the panoramic image is re-determined, the predicted normal is corrected according to the main direction, and the similarity between the predicted normal and the corrected normal is recalculated until the calculated similarity is no greater than the updated correction threshold; When the calculated similarity is not greater than the correction threshold, determining the normal map with the corrected normal; Among them, the updated correction threshold , and is a preset hyperparameter, is the number of corrections.

[0011] Preferably, position encoding is performed on the mask area, and the obtained encoding result, the feature parameter map and the normal map are input into a multi-layer perceptron to reconstruct the coordinate pixel values ​​of the mask area to obtain a repaired image, including: Encoding the pixel coordinates of the mask area using a position encoding function to obtain the encoding result; Inputting the encoding result, the feature parameter map and the normal map into the multi-layer perceptron to reconstruct the pixel coordinate value of each coordinate position of the mask area; A cascade operation is performed on the pixel value of each coordinate position of the mask area and the panoramic image to obtain the repaired image.

[0012] The embodiment of the present invention further provides a panoramic image restoration device, the device comprising: A mask generation module, used for mapping the annotation points of the object to be removed to the ERP image to generate a mask area of ​​the object to be removed; The aggregation module is used to extract global features and local texture features of the input panoramic image, and fuse the extracted features to generate a feature parameter map; A normal estimation module, used for calculating a normal map of the panoramic image by using a feature fusion method of a mixture of equirectangular projection and cubic projection; The restoration module is used to perform position encoding on the mask area, and input the obtained encoding result, the feature parameter map and the normal map into a multi-layer perceptron to reconstruct the coordinate pixel values ​​of the mask area to obtain a restored image.

[0013] An embodiment of the present invention also provides a panoramic image restoration device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, a panoramic image restoration method as described in any of the above embodiments is implemented.

[0014] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the panoramic image restoration method as described in any of the above embodiments.

[0015] The present invention provides a panoramic image restoration method, device and medium, which generates a mask area of ​​the object to be removed by mapping the annotation points of the object to be removed to an ERP image; extracting global features and local texture features of the input panoramic image, and fusing the extracted features to generate a feature parameter map; calculating the normal map of the panoramic image by a feature fusion method that mixes equirectangular projection and cubic projection; position encoding the mask area, and inputting the obtained encoding result, the feature parameter map and the normal map into a multi-layer perceptron to reconstruct the coordinate pixel values ​​of the mask area to obtain a restored image. The present application scheme can reduce the distortion of panoramic image restoration and improve the restoration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flowchart of a panoramic image restoration method provided by an embodiment of the present invention; Figure 2 is a structural schematic diagram of a panoramic image restoration device provided by an embodiment of the present invention; Figure 3 It is another structural schematic diagram of a panoramic image restoration device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] With the continuous enrichment of panoramic multimedia content, the research on panoramic image restoration has gradually attracted the attention of many scholars. Most of the existing methods are applied to two-dimensional images, and the ERP (EquiRectangular Projection) format is currently the most commonly used projection form. However, these two-dimensional image-based methods often ignore the distortion problem caused by the ERP format in panoramic images. Unlike traditional two-dimensional plane images, panoramic images will have different degrees of distortion during the sampling process from the equator to the poles, so the deep learning model trained on 2D images is not suitable for panoramic images.

[0019] Although existing panoramic image restoration methods have taken into account the distortion problem of the ERP format, and usually use CMP images or polyhedron projection images to reduce the distortion of panoramic images, these methods fail to fully exploit the geometric information in panoramic images, which will cause distortion of panoramic images and affect the restoration effect.

[0020] In view of the above technical problems, this application proposes a panoramic image restoration method. Figure 1 , is a flow chart of a panoramic image restoration method provided by an embodiment of the present invention, the method comprising steps S1 to S4; Step S1, mapping the marked points of the object to be removed to the ERP image to generate a mask area of ​​the object to be removed; Step S2, extracting global features and local texture features from the input panoramic image, and fusing the extracted features to generate a feature parameter map; Step S3, calculating the normal map of the panoramic image by adopting a feature fusion method of a mixture of equirectangular projection and cubic projection; Step S4, position encoding is performed on the mask area, and the obtained encoding result, the feature parameter map and the normal map are input into a multi-layer perceptron to reconstruct the coordinate pixel values ​​of the mask area to obtain a repaired image.

[0021] In the specific implementation of this embodiment, the area of ​​the object to be removed is marked by user interaction. The user can mark points in the 3D view, and then map these marked points back to the ERP image to generate the mask area of ​​the object to be removed. This interactive method improves the flexibility and accuracy of the marking and enhances the user experience.

[0022] By combining global context and local spatial information through Fast Fourier Transform (FFT) and convolution operations, global features and local texture features are extracted and fused to generate feature parameter maps. This effectively extracts high-frequency and low-frequency features and improves the effect of image restoration. It can capture global and local features in 360° panoramic images to ensure that the structure and texture of the repaired area are more consistent.

[0023] In order to solve the problem of unsatisfactory effects caused by geometric distortion in the restoration process of panoramic images, the normal map of the panoramic image is estimated through hybrid projection fusion according to the preset dual-branch normal prediction network, thereby enhancing the three-dimensional sense and lighting effects of the image, and effectively improving the geometric structure consistency and realism of the restoration area.

[0024] In order to optimize the processing efficiency of high-resolution panoramic images, an image is generated only in the mask area during image restoration, the mask area is position-encoded, and the obtained encoding result, the feature parameter map, and the normal map are input into a multi-layer perceptron to reconstruct the pixel value of each coordinate of the mask area to obtain a restored image.

[0025] Pixel generation is performed through coordinate query to reduce unnecessary computational burden, and the generation effect of high-frequency details is enhanced through coordinate encoding and multi-layer perceptron network.

[0026] The present application solution can solve the problems of geometric distortion and high-frequency detail retention encountered by traditional methods when processing panoramic images. The restoration effect is improved by coordinate query and spherical normal constraints. Compared with the traditional cube projection, this case effectively solves the distortion problem of panoramic images caused by projection by introducing spherical normal estimation. Spherical normals can capture the geometric characteristics in the image, thereby more accurately restoring the three-dimensional structure of the scene, and can enhance the lighting effect to avoid the distortion in the polar regions affecting the restoration results. Pixel query and generation are only performed within the mask area, avoiding unnecessary full-image calculations, greatly optimizing memory usage and computing efficiency, and are particularly suitable for high-resolution panoramic image restoration scenarios.

[0027] In another embodiment provided by the present invention, the step S1, when implemented specifically, includes: Obtaining the marked points of the object to be removed marked by the user in the three-dimensional view; Mapping the annotation points in the three-dimensional view to two-dimensional pixel coordinates in the ERP image; The two-dimensional pixel coordinates are calculated using a pre-trained semantic segmentation model to generate a mask area of ​​the object to be removed.

[0028] In the specific implementation of this embodiment, since the panoramic image is usually presented in the form of equirectangular projection, the two-dimensional image pixels need to be mapped to the three-dimensional coordinates on the spherical surface to simulate the view and interaction effects in the 3D space.

[0029] The user can interact with the panoramic image in the 3D view and mark the objects that he wants to remove. The user marks the location of the objects to be removed in the 3D view, and these marks will be mapped back to the 2D pixel coordinates in the ERP image. .

[0030] For the two-dimensional pixel coordinates in the ERP image , which is converted to spherical coordinates using the following formula and Cartesian coordinates , realize the conversion between two-dimensional pixel coordinates and three-dimensional viewing angle coordinates.

[0031] ; ; in, are the three coordinates of the spherical coordinate system, and are the width and height of the ERP image respectively. Then the spherical coordinates are converted to Cartesian coordinates To simulate the effect in 3D space. Through coordinate transformation, we get , , .

[0032] It should be noted that the conversion between two-dimensional pixel coordinates and three-dimensional viewing angle coordinates can be inferred according to the above formula.

[0033] The pre-trained semantic segmentation model (SAM) automatically generates the mask area of ​​the object to be removed. .

[0034] in, , is the input panoramic image. SAM() is the semantic segmentation model function.

[0035] The generated mask is then input into the subsequent modules of the entire framework to guide the inpainting and restoration of the image. This coordinate query-based mask generation method not only improves interactivity, but can also be combined with efficient feature extraction and inpainting mechanisms to make the inpainting of the masked area more refined and natural.

[0036] In another embodiment provided by the present invention, the step S2 specifically includes: Resampling the panoramic image, and extracting features of the image through a fully convolutional encoder to obtain a feature map; Performing a fast Fourier transform on the feature map to extract global context information of the image and obtain a frequency domain representation; Processing the frequency domain representation through a convolution layer and performing an inverse fast Fourier transform to obtain the global feature; Performing local texture extraction on the feature map to obtain local texture features; Cascading the global feature and the local texture feature to obtain a spatial attention feature map; The spatial attention feature map is upsampled to obtain the feature parameter map.

[0037] In the specific implementation of this embodiment, the input panoramic image It is resampled to a fixed low resolution to reduce the amount of computation, and the features of the input panoramic image are extracted through a fully convolutional encoder. The extracted feature map is represented as ,in is the height, is the width, and C is the number of channels.

[0038] The feature map is processed using Fast Fourier Transform (FFT) to capture the global context information of the image. Fast Fourier Transform can better process global features in the frequency domain.

[0039] First, the feature map Perform a 1×1 convolution, batch normalization, and ReLU activation to obtain an enhanced feature map . Apply two-dimensional fast Fourier transform FFT2d, which only retains half of the true value of the spectrum to reduce the amount of calculation. The calculation process is .

[0040] The real and imaginary parts of the Fourier transformed are combined to form the final frequency domain representation.

[0041] The frequency domain representation is restored to the spatial structure through a 1×1 convolution layer and inverse fast Fourier transform IFFT to obtain the global features. .

[0042] Fourier convolution can capture global information, but for repairing large masked areas, especially when it comes to detailed textures, simply relying on frequency domain information is not enough. Therefore, this embodiment also designs a local texture extraction unit to specifically handle these detailed problems and perform local texture extraction on the feature map.

[0043] Local texture features Extracted from the original feature map, the texture details of the repaired area are retained, ensuring that high-frequency detail areas (such as edges and object surfaces) are well restored.

[0044] Global Features Local texture features are concatenated to form a spatial attention feature .

[0045] In order to restore the original resolution, the fused feature map is upsampled to obtain a feature parameter map with the same resolution as the input image. , used for subsequent mask area pixel query and repair.

[0046] By combining global and local features, it ensures that the restored image can retain high-frequency structure and low-frequency texture, thereby improving the overall restoration quality and detail retention.

[0047] In another embodiment provided by the present invention, the spatial attention feature map ; Among them, 9× is the scale factor, F is the feature map, is the global feature, is the local texture feature, Indicates a cascade operation.

[0048] In the specific implementation of this embodiment, the global feature Local texture features Spatial attention features obtained by concatenation ; in, represents the cascade operation, and 9× is the scale factor, which ensures that the feature expression of the global and local features is adapted to the high-resolution image after fusion.

[0049] In another embodiment provided by the present invention, the step S3 specifically includes: Fusing the features of the equirectangular projection and the features of the cubic projection of the panoramic image to obtain fused features; Performing weighted calculation on the fusion feature according to the latitude of the panoramic image, and correcting the coordinates of the fusion feature; Generate a predicted normal map based on the corrected fusion features to determine the predicted normal of each pixel in the image; The main direction of the panoramic image is determined, and the predicted normal is corrected according to the main direction to obtain the normal map.

[0050] In the specific implementation of this embodiment, in order to better process the geometric characteristics of the panoramic image, the present application scheme designs a dual-branch normal prediction network, which combines the advantages of the two projection formats of equirectangular projection (ERP) and cubic projection (CMP), and fuses the features of the equirectangular projection and the features of the cubic projection of the panoramic image.

[0051] The features of the two projections are combined and fused at different stages of the decoder, thereby taking advantage of the complementary advantages of the two projections to accurately restore the normal information of the image. Since the latitude and longitude coordinates of the ERP image will produce different sampling rates in the polar regions, in order to avoid overfitting of the network in these areas, a spherical weighting mechanism is introduced to weight the features according to the latitude of the image.

[0052] The fusion feature is weighted according to the latitude of the panoramic image to correct the coordinates of the fusion feature. The calculation formula is: ; in, Represents the pixel coordinates of the image. The weighted features can better handle geometric distortion problems at different latitudes.

[0053] The fused features are used to generate a normal map through a network module called NormalHead. The surface normal of each pixel of the panoramic image is predicted. Due to the geometric characteristics of the spherical surface, especially in the polar regions, the normal prediction may be inaccurate or noisy, so further correction is required.

[0054] In order to correct the predicted normals, the module introduces a main direction correction mechanism to determine the main direction of the panoramic image and correct the predicted normals according to the main direction to obtain the normal map.

[0055] This application scheme is applicable to 360° panoramic images, especially images in equirectangular projection ERP format, which will produce serious geometric distortion in the polar regions of the image due to its spherical geometric characteristics. Ordinary two-dimensional image restoration methods cannot accurately restore the surface information of these distorted areas. By estimating the surface normal of the image, these geometric distortions can be effectively corrected, and the lighting effect after the object is removed can be enhanced to improve the three-dimensional sense.

[0056] In another embodiment of the present invention, the process of correcting the prediction vector by the main direction specifically includes the following steps: Performing edge detection on the panoramic image to extract edge points; Projecting the pixels of the edge points to spherical coordinates, generating a prediction vector corresponding to the 3D projection plane by randomly sampling two points and calculating the cross product, calculating the best vanishing point using the RANSAC algorithm, and finding the main direction of the panoramic image; Align the main direction with the predicted normal, and determine the main direction closest to the predicted normal as the corrected normal; Calculate the similarity between the predicted normal and the corrected normal; When the calculated similarity is greater than a preset correction threshold, the number of corrections is calculated, the correction threshold is updated, the main direction of the panoramic image is re-determined, the predicted normal is corrected according to the main direction, and the similarity between the predicted normal and the corrected normal is recalculated until the calculated similarity is no greater than the updated correction threshold; When the calculated similarity is not greater than the correction threshold, determining the normal map with the corrected normal; Among them, the updated correction threshold , and is a preset hyperparameter, is the number of corrections.

[0057] In the specific implementation of this embodiment, in order to correct the predicted normals, a principal direction correction mechanism is introduced based on the Manhattan World assumption that surfaces in indoor scenes (such as walls, floors and ceilings) usually follow the principal axis direction (vertical or horizontal).

[0058] Extract edge points by performing Canny edge detection on the panoramic image; The edge points are projected onto spherical coordinates, and the normal vector corresponding to the 3D projection plane is generated by randomly sampling two points and calculating their cross product.

[0059] Use the RANSAC algorithm to calculate the best vanishing point and find the main direction of the image , there are usually multiple main directions, such as perpendicular to the ground or parallel to the wall.

[0060] By main direction With predicted normal Align and calculate predicted normals With the main direction The cosine similarity between ; Select the main direction closest to the predicted normal as the corrected normal , ; When the calculated similarity is greater than the preset correction threshold, the number of corrections is calculated, the correction threshold is updated, the main direction of the panoramic image is re-determined, and the predicted normal is corrected according to the main direction. Gradually increase: ;in, and is a hyperparameter that controls the sensitivity of the normal correction.

[0061] When the calculated similarity is not greater than the correction threshold, determining the normal map with the corrected normal; In order to optimize the normal estimation module during training, a normal constraint loss function was designed in the study. , which measures the difference between the predicted normal and the true normal: ; in, is the number of pixels, is the main direction mask, is the true normal.

[0062] In another embodiment provided by the present invention, the step S4 specifically includes: Encoding the pixel coordinates of the mask area using a position encoding function to obtain the encoding result; Inputting the encoding result, the feature parameter map and the normal map into the multi-layer perceptron to reconstruct the pixel coordinate value of each coordinate position of the mask area; A cascade operation is performed on the pixel value of each coordinate position of the mask area and the panoramic image to obtain the repaired image.

[0063] In the specific implementation of this embodiment, the input is a high-resolution panoramic image and its mask image. First, frequency-space feature aggregation is performed to generate a feature parameter map ,The feature parameter map contains all the high and low frequency information in the input image that needs to be repaired.,In order to enhance the processing of high frequency details, the input coordinates are first,encoded through the position encoding module and then input to the MLP (Multi-layer Perceptron) for,further processing.

[0064] Positional Encoding Function Encode the pixel coordinates of the masked area, is the two-dimensional pixel coordinate, is the height and width of the image. The result of position encoding will be combined with the normal map and feature parameter map They are input together into the MLP layer to ensure the processing of high-frequency information.

[0065] After position encoding, the coordinates of each pixel in the mask area are input into the MLP (Multi-layer Perceptron) for query. The MLP reconstructs the value of each pixel based on the input mask area coordinates, feature parameter map and normal map: .

[0066] in, Represents the position of each pixel obtained by the query The pixel value of .

[0067] The corresponding pixel value is generated pixel by pixel in the mask area, and the normal map and feature parameter map are combined to calculate each pixel point to ensure that the high-frequency details and geometric structure of the repair area can be accurately restored. This pixel-by-pixel generation method can effectively handle the boundary details and complex textures of the mask area, ensuring a natural transition of the repair.

[0068] Finally, the pixel values ​​of the masked area With the initial input image , generate a complete repaired image : ; The pixel coordinate query of the mask area introduces position encoding and MLP network, combined with normal map and feature parameter map, to accurately generate the mask area pixel by pixel. This implicit representation method not only reduces unnecessary full image calculations, but also can effectively handle the details in high-resolution images, greatly improving the calculation efficiency and restoration quality.

[0069] By querying the pixel coordinates in the mask area, the pixel values ​​in the repair area are selectively generated, avoiding redundant calculations of the entire image, thereby greatly improving memory usage efficiency and calculation speed.

[0070] This application scheme is based on the evaluation on the SD dataset, and the restoration performance is measured by commonly used image quality evaluation indicators. The SD dataset is constructed from the Structured3D dataset after a series of preprocessing steps. The training set of this dataset contains 14,527 images, and the validation set and test set each contain 2000 images. The experiment uses the following three commonly used image quality evaluation indicators to measure the restoration performance. PSNR (peak signal-to-noise ratio) is used to measure the quality of image reconstruction. The higher the value, the smaller the difference between the restored image and the real image, and the better the restoration effect. SSIM (structural similarity index) evaluates the similarity of two images in terms of brightness, contrast and structure. The closer the value is to 1, the closer the structure of the image is to the real scene. LPIPS (perceptual image patch similarity) is used to measure the perceptual quality of image restoration. The lower the value, the closer the restored image is to the real image in perception. The model proposed in this application is compared with 6 existing benchmark models, namely CoMod-GAN, Palette, Inst, CoordFill, LII, and PIINET. The results of the model performance comparison are given in Table 1. It can be clearly seen from this table that the model proposed in this application scheme achieves good performance on the SD dataset compared with other benchmark models.

[0071] Table 1 Performance comparison evaluation table

[0072] Compared with existing models, the present invention shows significant advantages in evaluation indicators. Existing models such as CoMod-GAN and Palette have certain performance in image structure similarity, but the perceived quality is poor. Inst and PIINET have deficiencies in repairing high-frequency details and complex geometric structures, and CoordFill performs generally in terms of accuracy and structural restoration. In contrast, the present invention surpasses these methods in key indicators such as PSNR, SSIM and LPIPS, showing higher restoration accuracy, detail retention ability and more natural visual perception. Especially when processing complex panoramic images, the present invention can better maintain geometric consistency and light and shadow transition, and the overall restoration effect is better than the existing technology.

[0073] In addition, when Palette processes large-area image restoration, the lighting transition appears abrupt, and the visual effect of the repaired area is not natural enough. Inst performs poorly in complex background filling and texture consistency, and is prone to artifacts and discontinuous textures. Although PIINET uses cubic projection to reduce geometric distortion, there are still distortions and blurring in the restoration of polar areas and edge details. Although CoordFill can maintain a certain structural consistency, it is not as ideal in retaining high-frequency details and texture transition, and the transition between the repaired area and the surrounding area is relatively abrupt.

[0074] In contrast, the restored image of the present invention is highly consistent with the undamaged area visually, and the restored area is extremely natural in terms of details, structure, and lighting transition, which also shows that the present invention is superior to the existing technology in dealing with structural consistency, texture details, lighting and shadow consistency, and geometric distortion in polar areas. In particular, the present invention shows a strong advantage in terms of the naturalness of the restored image and the preservation of high-frequency details, providing a better solution for panoramic image restoration tasks.

[0075] In another embodiment provided by the present invention, see Figure 2 , is a schematic diagram of the structure of a panoramic image restoration device provided by an embodiment of the present invention, the device comprising: A mask generation module, used for mapping the annotation points of the object to be removed to the ERP image to generate a mask area of ​​the object to be removed; The aggregation module is used to extract global features and local texture features of the input panoramic image, and fuse the extracted features to generate a feature parameter map; A normal estimation module, used for calculating a normal map of the panoramic image by using a feature fusion method of a mixture of equirectangular projection and cubic projection; The restoration module is used to perform position encoding on the mask area, and input the obtained encoding result, the feature parameter map and the normal map into a multi-layer perceptron to reconstruct the coordinate pixel values ​​of the mask area to obtain a restored image.

[0076] The panoramic image restoration device provided in this embodiment can execute all the steps and functions of the panoramic image restoration method provided in any of the above embodiments, and the specific functions of the device are not described in detail here.

[0077] See also Figure 3 , is another structural diagram of a panoramic image restoration device provided by an embodiment of the present invention. The panoramic image restoration device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a panoramic image restoration program. When the processor executes the computer program, the steps in each of the above-mentioned panoramic image restoration method embodiments are implemented, such as Figure 1 Alternatively, the processor implements the functions of each module in the above-mentioned device embodiments when executing the computer program.

[0078] Exemplarily, the computer program may be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the panoramic image restoration device. For example, the computer program may be divided into various modules, and the specific functions of each module have been described in detail in a panoramic image restoration method provided in any of the above embodiments, and the specific functions of the device will not be repeated here.

[0079] The panoramic image restoration device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The panoramic image restoration device may include, but is not limited to, a processor and a memory. A person skilled in the art may understand that the schematic diagram is only an example of a panoramic image restoration device and does not constitute a limitation on a panoramic image restoration device. The panoramic image restoration device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the panoramic image restoration device may also include input and output devices, network access devices, buses, etc.

[0080] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the panoramic image restoration device, and uses various interfaces and lines to connect the various parts of the entire panoramic image restoration device.

[0081] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the panoramic image restoration device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0082] Wherein, if the module integrated in the panoramic image restoration device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0083] It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A panoramic image restoration method, characterized in that: The method comprises: Mapping the marked points of the object to be removed to the ERP image to generate a mask area of ​​the object to be removed; Extract global features and local texture features of the input panoramic image, fuse the extracted features, and generate a feature parameter map; Calculating the normal map of the panoramic image by using a feature fusion method that combines equirectangular projection and cubic projection; Position encoding is performed on the mask area, and the obtained encoding result, the feature parameter map and the normal map are input into a multi-layer perceptron to reconstruct the coordinate pixel values ​​of the mask area to obtain a repaired image.

2. The panoramic image restoration method according to claim 1, characterized in that: Mapping the annotation points of the object to be removed to the ERP image to generate a mask area of ​​the object to be removed includes: Obtaining the marked points of the object to be removed marked by the user in the three-dimensional view; Mapping the annotation points in the three-dimensional view to two-dimensional pixel coordinates in the ERP image; The two-dimensional pixel coordinates are calculated using a pre-trained semantic segmentation model to generate a mask area of ​​the object to be removed.

3. The panoramic image restoration method according to claim 1, characterized in that: Extract global features and local texture features of the input panoramic image, fuse the extracted features, and generate a feature parameter map, including; Resampling the panoramic image, and extracting features of the image through a fully convolutional encoder to obtain a feature map; Performing a fast Fourier transform on the feature map to extract global context information of the image and obtain a frequency domain representation; Processing the frequency domain representation through a convolution layer and performing an inverse fast Fourier transform to obtain the global feature; Performing local texture extraction on the feature map to obtain local texture features; Cascading the global feature and the local texture feature to obtain a spatial attention feature map; The spatial attention feature map is upsampled to obtain the feature parameter map.

4. The panoramic image restoration method according to claim 3, characterized in that: The spatial attention feature map ； Among them, 9× is the scale factor, F is the feature map, is the global feature, is the local texture feature, Indicates a cascade operation.

5. The panoramic image restoration method according to claim 1, characterized in that: The normal map of the panoramic image is calculated by using a feature fusion method of a mixture of equirectangular projection and cubic projection, including: Fusing the features of the equirectangular projection and the features of the cubic projection of the panoramic image to obtain fused features; Performing weighted calculation on the fusion feature according to the latitude of the panoramic image, and correcting the coordinates of the fusion feature; Generate a predicted normal map based on the corrected fusion features to determine the predicted normal of each pixel in the image; The main direction of the panoramic image is determined, and the predicted normal is corrected according to the main direction to obtain the normal map.

6. The panoramic image restoration method according to claim 5, characterized in that: Determining the main direction of the panoramic image and correcting the predicted normal according to the main direction to obtain the normal map includes: Performing edge detection on the panoramic image to extract edge points; Projecting the pixels of the edge points to spherical coordinates, generating a prediction vector corresponding to the 3D projection plane by randomly sampling two points and calculating the cross product, calculating the best vanishing point using the RANSAC algorithm, and finding the main direction of the panoramic image; Align the main direction with the predicted normal, and determine the main direction closest to the predicted normal as the corrected normal; Calculate the similarity between the predicted normal and the corrected normal; When the calculated similarity is greater than a preset correction threshold, the number of corrections is calculated, the correction threshold is updated, the main direction of the panoramic image is re-determined, the predicted normal is corrected according to the main direction, and the similarity between the predicted normal and the corrected normal is recalculated until the calculated similarity is no greater than the updated correction threshold; When the calculated similarity is not greater than the correction threshold, determining the normal map with the corrected normal; Among them, the updated correction threshold , and is a preset hyperparameter, is the number of corrections.

7. The panoramic image restoration method according to claim 1, characterized in that: Position encoding is performed on the mask area, and the obtained encoding result, the feature parameter map, and the normal map are input into a multi-layer perceptron to reconstruct the coordinate pixel values ​​of the mask area to obtain a repaired image, including: Encoding the pixel coordinates of the mask area using a position encoding function to obtain the encoding result; Inputting the encoding result, the feature parameter map and the normal map into the multi-layer perceptron to reconstruct the pixel coordinate value of each coordinate position of the mask area; A cascade operation is performed on the pixel value of each coordinate position of the mask area and the panoramic image to obtain the repaired image.

8. A panoramic image restoration device, characterized in that: The device comprises: A mask generation module, used for mapping the annotation points of the object to be removed to the ERP image to generate a mask area of ​​the object to be removed; The aggregation module is used to extract global features and local texture features of the input panoramic image, and fuse the extracted features to generate a feature parameter map; A normal estimation module, used for calculating a normal map of the panoramic image by using a feature fusion method of a mixture of equirectangular projection and cubic projection; The restoration module is used to perform position encoding on the mask area, and input the obtained encoding result, the feature parameter map and the normal map into a multi-layer perceptron to reconstruct the coordinate pixel values ​​of the mask area to obtain a restored image.

9. A panoramic image restoration device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the panoramic image restoration method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the panoramic image restoration method according to any one of claims 1 to 7.

Images