A virtual restoration method for murals based on hyperspectral images

Abstract

The present invention provides a method for virtual restoration of murals based on hyperspectral images, which is used to solve the technical problems that the existing CNN-based image restoration algorithm has difficulty in quickly identifying the area of ​​the mural that needs to be restored when restoring the mural, and the unnatural restoration effect will appear when repairing in a complex background or texture. The method for virtual restoration of murals based on hyperspectral images provided by the present invention uses a hyperspectral camera to collect hyperspectral mural images, obtains the mask of the area to be restored by the mural with the help of the classification algorithm of the hyperspectral mural images, and replaces the traditional convolution layer in the U‑Net neural network model with a Fourier partial convolution layer to increase the receptive field of the network, which can restore the original appearance of the mural image to the greatest extent and realize digital archiving; at the same time, the U‑Net neural network model can capture local and global features at the same time through the fusion of multiple groups of features of local branches and global branches in the Fourier partial convolution layer, thereby realizing data complementarity.

Description

Technical Field

[0001] The invention relates to a mural virtual restoration method, and in particular to a mural virtual restoration method based on hyperspectral images. Background Art

[0002] Virtual restoration of murals is to Figure 1 The method of digitizing some of the murals that have been polluted, eroded or damaged by humans and then performing image restoration on their digital images has great application prospects in the field of cultural relics protection.

[0003] Before the popularization of deep learning technology, image restoration mainly relied on traditional computer vision algorithms, mainly including texture-based restoration, structure-based restoration and sparse representation-based restoration. Texture-based restoration uses texture information in the image for restoration, such as texture synthesis technology, which fills the missing area by copying and pasting similar textures. Structure-based restoration uses structural information in the image for restoration, such as edge detection and structure-guided interpolation. Sparse representation-based restoration uses sparse representation of the image to repair missing parts through sparse coding methods. Although these methods are effective in some cases, they are often not accurate or natural enough when dealing with complex scenes and details.

[0004] Convolutional neural network (CNN) is a type of deep learning model that is particularly good at processing image data. CNN extracts different features of an image through multiple convolutional layers, and can automatically learn how to perform image restoration through training. CNN-based image virtual restoration technology usually includes feature extraction, context modeling, and image generation processes, mainly using convolutional layers to extract high-level features in images. Representative CNN models include fully convolutional networks, generative adversarial networks, and free-form image restoration (Deepfill).

[0005] At present, CNN-based image restoration algorithms still face the following two major problems when repairing murals: first, it is difficult to quickly identify the areas of the murals that need to be repaired; second, when repairing in complex backgrounds or textures, unnatural effects will appear. Summary of the invention

[0006] The purpose of the present invention is to solve the technical problems that the existing CNN-based image restoration algorithm has difficulty in quickly identifying the area of ​​the mural that needs to be restored when restoring the mural, and that unnatural restoration effects will occur when repairing in complex backgrounds or textures, and to provide a mural virtual restoration method based on hyperspectral images.

[0007] In order to achieve the above purpose, the technical solution provided by the present invention is as follows:

[0008] A virtual restoration method for murals based on hyperspectral images is special in that it includes the following steps:

[0009] Step 1: Collect the hyperspectral mural image and generate the mask of the mural area to be restored:

[0010] A hyperspectral camera is used to collect the murals to obtain a hyperspectral mural image; the hyperspectral mural image is subjected to shape adaptive reconstruction, image spectrum dimension reduction processing and feature classification in sequence to extract the area to be restored in the hyperspectral mural image and generate a mask of the area to be restored in the mural;

[0011] Step 2: Build a U-Net neural network model;

[0012] The convolution layer in the U-Net neural network model is a Fourier partial convolution layer;

[0013] Step 3: Use the CelebA dataset as the image dataset and the NVIDIA Irregular MaskDataset dataset as the mask dataset to pre-train the U-Net neural network model built in step 2.

[0014] Step 4, constructing a training set, wherein the training set includes multiple high-definition mural images; inputting the high-definition mural images in the training set into the pre-trained U-Net neural network model for further training, and transferring the common features and patterns learned by the U-Net neural network model on the CelebA dataset and the NVIDIA Irregular Mask Dataset dataset to the target task through transfer learning;

[0015] Step 5, converting the hyperspectral mural image acquired in step 1 into a mural RGB image;

[0016] Step 6, input the mural RGB image obtained in step 5 and the mask of the mural area to be repaired generated in step 1 into the U-Net neural network model that has undergone pre-training and transfer learning for processing. The U-Net neural network model outputs a virtual repair image to complete the virtual restoration of the mural.

[0017] Further, in step 2, the U-Net neural network model includes an encoder, an intermediate layer and a decoder;

[0018] The encoder includes an input layer, a plurality of Fourier partial convolution layers, and a plurality of pooling layers corresponding to the plurality of Fourier partial convolution layers in sequence according to input and output;

[0019] The input layer is used to receive image data input into the U-Net neural network model; the input end of each pooling layer is connected to the output end of the corresponding Fourier partial convolution layer in the encoder, and its output end is connected to the input end of the next Fourier partial convolution layer, and the output end of the last pooling layer is connected to the input end of the intermediate layer; the pooling layer is used to reduce the feature dimension of the output of the corresponding Fourier partial convolution layer;

[0020] The Fourier partial convolution layer includes two parallel branches, namely a local branch and a global branch; the local branch includes a first local convolution branch and a second local convolution branch in parallel, both of which use Fourier partial convolution to downsample the input image data to extract local features of the image data in the time domain; the global branch includes a third local convolution branch and a spectral convolution branch in parallel, the third local convolution branch uses Fourier partial convolution to downsample the input image data to extract boundary features of the image data, and the spectral convolution branch is used to convert the input image data from the time domain to the frequency domain, and perform a convolution operation in the frequency domain to capture the global context information of the image data in the frequency domain;

[0021] The local features of the image data extracted by the first local convolution branch of the local branch are fused with the boundary features of the image data extracted by the third local convolution branch of the global branch, and the result is processed by the activation function as the output of the local branch; the global context information of the image data in the frequency domain extracted by the spectrum convolution branch of the global branch is fused with the local features of the image data extracted by the second local convolution branch of the local branch, and the result is processed by the activation function as the output of the global branch; finally, the outputs of the local branch and the global branch are fused to generate a feature image;

[0022] The decoder includes a plurality of adjacent upsampling layers, a plurality of Fourier partial convolution layers corresponding to the plurality of adjacent upsampling layers, a plurality of skip connection structures, and an output layer; the output end of the intermediate layer is connected to the input end of the first adjacent upsampling layer in the decoder, and the output end of each adjacent upsampling layer is connected to the input end of the corresponding Fourier partial convolution layer in the decoder, so as to gradually increase the spatial resolution of the feature map through the plurality of adjacent upsampling layers and the plurality of Fourier partial convolution layers, and restore the detail information of the image;

[0023] The plurality of jump connection structures are used to respectively connect the feature images output by each Fourier partial convolution layer in the encoder with the feature images output by the corresponding Fourier partial convolution layer in the decoder to transfer detail information;

[0024] The output layer is connected to the output end of the last Fourier partial convolution layer in the decoder, and is used to generate and output a virtual restoration image of the mural according to the feature image output by the last Fourier partial convolution layer in the decoder.

[0025] Furthermore, step 1 is specifically as follows:

[0026] Step 1.1, using a hyperspectral camera to collect the murals and obtain a hyperspectral mural image;

[0027] Step 1.2, shape adaptive reconstruction of the hyperspectral mural image;

[0028] A shape-adaptive region is generated by local polynomial approximate filtering and the intersection confidence interval rule. The surrounding pixels closest to the central pixel are selected in the shape-adaptive region, and the correlation between the central pixel and the surrounding pixels is estimated by calculating the Pearson correlation coefficient. The obtained correlation coefficient is then updated as a weight vector to reconstruct the spectrum of the central pixel.

[0029] Step 1.3, using principal component analysis to perform spectral dimension reduction on the hyperspectral mural image after shape adaptive reconstruction, and obtain the principal component features of the hyperspectral mural image;

[0030] Step 1.4, using support vector machine to classify the principal component features of the hyperspectral mural image;

[0031] Step 1.5, based on the classification result of the support vector machine, extract the area to be restored of the hyperspectral mural image and generate a mask of the area to be restored of the mural.

[0032] Furthermore, in step 2, batch normalization is used after the second to N-1 Fourier partial convolution layers in the U-Net neural network model to enhance the stability and nonlinear characteristics of the model, where N is the total number of Fourier partial convolution layers in the encoder and decoder; wherein, only an activation function is set after the first Fourier partial convolution layer, and only batch normalization is set after the Nth Fourier partial convolution layer.

[0033] Furthermore, in step 2, the activation function corresponding to each Fourier partial convolution layer in the encoder is a rectified linear unit, and the activation function corresponding to each Fourier partial convolution layer in the decoder is a leaky rectified linear unit.

[0034] Furthermore, in step 1.4, when the support vector machine classifies the principal component features, the Sigmoid function is used as the kernel function.

[0035] Furthermore, step 5 is specifically as follows: three bands are selected from the hyperspectral mural image acquired in step 1 to construct three channels of the RGB image, and then the pixel values ​​of the three channels are fused to obtain the mural RGB image.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The present invention provides a method for virtual restoration of murals based on hyperspectral images. A hyperspectral camera is used to collect hyperspectral mural images. A mask of the area to be restored of the mural is obtained with the help of a classification algorithm of the hyperspectral mural images. Meanwhile, a Fourier partial convolution layer is used to replace the traditional convolution layer in the U-Net neural network model to increase the receptive field of the network, thereby restoring the original appearance of the mural image to the greatest extent and realizing digital archiving.

[0038] 2. The U-Net neural network model of the present invention adopts the Fourier partial convolution layer for feature extraction. By fusing multiple sets of features of local branches and global branches in the Fourier partial convolution layer, local and global features can be captured simultaneously to achieve data complementarity.

[0039] 3. The present invention uses local convolution and appropriate masking at the image boundary to replace the traditional filling method, ensuring that the embedded content at the image boundary will not be affected by invalid values ​​outside the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of existing murals that have been polluted, eroded or damaged by human activities;

[0041] Figure 2 It is a flow chart of an embodiment of a method for virtual restoration of murals based on hyperspectral images of the present invention;

[0042] Figure 3 This is a schematic diagram of the Fourier partial convolution layer in step 2 of an embodiment of the present invention;

[0043] Figure 4 This is the hyperspectral mural image collected in step 1 of this method in the experiment;

[0044] Figure 5 This is a schematic diagram of the mask of the area to be restored of the mural obtained by step 1 of this method in the experiment;

[0045] Figure 6 This is the virtual restoration image of the mural in the experiment. DETAILED DESCRIPTION

[0046] The technical solution of the present invention will be clearly and completely described below in combination with the embodiments of the present invention and the accompanying drawings. Obviously, the described embodiments are not limitations of the present invention.

[0047] A virtual restoration method for murals based on hyperspectral images, such as Figure 2 As shown, the specific steps include:

[0048] Step 1: Collect hyperspectral mural images and generate masks of the mural areas to be restored.

[0049] Step 1.1, use the linear sensor of a high-resolution push-broom linear array camera to scan the mural line by line to obtain a hyperspectral mural image. The hyperspectral mural image contains both the spectral information and the spatial information of the mural.

[0050] Step 1.2, perform shape adaptive reconstruction on the hyperspectral mural image.

[0051] Shape adaptive reconstruction is used to process image regions of different shapes and structures, making the feature classification process more flexible and accurate; the invention introduces shape adaptive reconstruction as an effective tool for using spatial features as a preprocessing step. In this embodiment, a shape adaptive region is generated by local polynomial approximate filtering and an intersection confidence interval rule; the surrounding pixels closest to the central pixel are selected in the shape adaptive region, and the correlation between the central pixel and the surrounding pixels is estimated by calculating the Pearson correlation coefficient, and then the obtained correlation coefficient is updated as a weight vector to reconstruct the spectrum of the central pixel.

[0052] The present invention uses shape adaptive reconstruction as preprocessing, and spatial information is further utilized, and then image spectrum dimension reduction is performed through principal component analysis.

[0053] Step 1.3: Image spectrum dimension reduction.

[0054] Since hyperspectral mural images have a large number of wavelength and spectral features, and often contain redundant information, mapping the original data into a lower dimensional space through image spectral dimensionality reduction can reduce the number of features while retaining the most important and representative information.

[0055] Adjacent bands of hyperspectral mural images are highly correlated and usually convey almost the same target information. Therefore, principal component analysis is used in this embodiment to reduce the dimensionality of image spectra. After principal component analysis, each pixel in the hyperspectral mural image is represented as a feature vector, and each dimension represents the number of principal components. Principal component analysis is used to transform the original data to eliminate the correlation between bands. In this process, principal component analysis identifies the direction of the maximum variance in the data (principal component) by calculating the covariance matrix. In band fusion, selecting the first few principal components can effectively combine and simplify the information of the original bands, reduce redundancy, and improve data processing efficiency, while retaining the principal component characteristics of the hyperspectral mural image.

[0056] Step 1.4, classify the principal component features of the hyperspectral mural image.

[0057] Since different categories of materials or objects usually show different characteristics in the spectrum, the principal component features of the hyperspectral mural image are classified by analyzing the spectral characteristics of each pixel point or region in the hyperspectral mural image, so that each pixel point or region in the hyperspectral mural image can be classified into a predefined category.

[0058] In this embodiment, support vector machine is used to classify the main component features of the hyperspectral mural image. In the previous step, each pixel of the hyperspectral mural image is represented as a feature vector, which can be any attribute that can describe the sample, such as pixel value, spectral reflectance, etc. When the support vector machine is classified, the Sigmoid function is used as the kernel function, which maps the data from the original feature space to a higher-dimensional feature space, so that it is easier to perform linear or nonlinear classification in the new feature space. Using the existing labeled training samples, a classification model is trained by the support vector machine. In the training stage, the support vector machine will find an optimal hyperplane to separate samples of different categories and maximize the interval between the support vector and the hyperplane as much as possible. In the test stage, the trained support vector machine is used to make classification decisions on unlabeled new samples, that is, the new samples are projected onto the hyperplane obtained in the training stage, and the category to which the new samples belong is determined according to the position of the new samples in the hyperplane, that is, whether it belongs to the pixel in the mural area to be repaired. In this embodiment, the pixels belonging to the mural area to be repaired are assigned a value of 0, and the pixels not belonging to the mural area to be repaired are assigned a value of 255.

[0059] Step 1.5, based on the classification result of the support vector machine, extract the area to be restored of the hyperspectral mural image, and generate a mask of the area to be restored of the mural.

[0060] Step 2: Build a U-Net neural network model.

[0061] The U-Net neural network model in the present invention is a U-Net neural network model based on a Fourier partial convolutional layer, which includes an encoder, an intermediate layer and a decoder.

[0062] The encoder includes an input layer, eight Fourier partial convolution layers and eight pooling layers. The eight Fourier partial convolution layers are sequentially recorded as a first Fourier partial convolution layer, a second Fourier partial convolution layer, a third Fourier partial convolution layer, a fourth Fourier partial convolution layer, a fifth Fourier partial convolution layer, a sixth Fourier partial convolution layer, a seventh Fourier partial convolution layer and an eighth Fourier partial convolution layer, and the eight pooling layers are sequentially recorded as a first pooling layer, a second pooling layer, a third pooling layer, a fourth pooling layer, a fifth pooling layer, a sixth pooling layer, a seventh pooling layer and an eighth pooling layer. The first Fourier partial convolution layer, the first pooling layer, the second Fourier partial convolution layer, the second pooling layer, the third Fourier partial convolution layer, the third pooling layer, the fourth Fourier partial convolution layer, the fourth pooling layer, the fifth Fourier partial convolution layer, the fifth pooling layer, the sixth Fourier partial convolution layer, the sixth pooling layer, the seventh Fourier partial convolution layer, the seventh pooling layer, the eighth Fourier partial convolution layer, and the eighth pooling layer are sequentially arranged in the order of input and output. The input layer is used to receive the image data input into the U-Net neural network model, and its output end is connected to the input end of the first Fourier partial convolution layer. The pooling layer is used to reduce the feature dimension of the output of the corresponding Fourier partial convolution layer. After each Fourier partial convolution layer and the corresponding pooling layer, the spatial resolution of the image will gradually decrease, and the number of channels of the feature image will gradually increase, thereby capturing more feature information.

[0063] The Fast Fourier Convolution operator allows the use of global context in early layers and is based on the channel-wise Fast Fourier Transform, with a receptive field covering the entire image. Figure 3 As shown, in the present invention, the Fourier partial convolution layer is divided into two parallel branches, namely, a local branch and a global branch. Among them, the local branch includes a parallel first local convolution branch and a second local convolution branch, and both the first local convolution branch and the second local convolution branch use Fourier partial convolution to downsample the input image data to extract the local features of the image data in the time domain. The global branch includes a parallel third local convolution branch and a spectral convolution branch. The third local convolution branch uses Fourier partial convolution to downsample the input image data. The spectral convolution branch first converts the input image data to the frequency domain through a fast real Fourier transform, and then performs point-by-point multiplication of the image and the convolution kernel in the frequency domain. Since the convolution in the frequency domain becomes a simple point-by-point multiplication, this makes the operation more efficient. Finally, the convolution result is converted back to the spatial domain through an inverse real Fourier transform to capture the global context information of the image data in the frequency domain. At the same time, the spectral convolution branch uses a feedforward resnet-based connection to ensure the generalization performance of the model.

[0064] The local features of the image data extracted by the first local convolution branch of the local branch are fused with the boundary features of the image data extracted by the third local convolution branch of the global branch, and are used as the output of the local branch after being processed by the activation function; the global context information of the image data in the frequency domain extracted by the spectral convolution branch of the global branch is fused with the local features of the image data extracted by the second local convolution branch of the local branch, and are used as the output of the global branch after being processed by the activation function; finally, the outputs of the local branch and the global branch are fused to generate a feature image, and the generated feature image contains the global information and semantic information of the image.

[0065] The decoder is configured correspondingly to the encoder, and in this embodiment includes eight adjacent upsampling layers, eight Fourier partial convolution layers, eight skip connection structures, and an output layer. The eight adjacent upsampling layers are respectively recorded as the first adjacent upsampling layer, the second adjacent upsampling layer, the third adjacent upsampling layer, the fourth adjacent upsampling layer, the fifth adjacent upsampling layer, the sixth adjacent upsampling layer, the seventh adjacent upsampling layer, and the eighth adjacent upsampling layer; the eight Fourier partial convolution layers have the same structure as the Fourier partial convolution layer in the encoder, and the eight Fourier partial convolution layers are respectively recorded as the ninth Fourier partial convolution layer, the tenth Fourier partial convolution layer, the eleventh Fourier partial convolution layer, the twelfth Fourier partial convolution layer, the thirteenth Fourier partial convolution layer, the fourteenth Fourier partial convolution layer, the fifteenth Fourier partial convolution layer, and the sixteenth Fourier partial convolution layer; the eight jump connection structures are respectively recorded as the first jump connection structure, the second jump connection structure, the third jump connection structure, the fourth jump connection structure, the fifth jump connection structure, the sixth jump connection structure, the seventh jump connection structure, and the eighth jump connection structure. The first adjacent upsampling layer, the ninth Fourier partial convolution layer, the second adjacent upsampling layer, the tenth Fourier partial convolution layer, the third adjacent upsampling layer, the eleventh Fourier partial convolution layer, the fourth adjacent upsampling layer, the twelfth Fourier partial convolution layer, the fifth adjacent upsampling layer, the thirteenth Fourier partial convolution layer, the sixth adjacent upsampling layer, the fourteenth Fourier partial convolution layer, the seventh adjacent upsampling layer, the fifteenth Fourier partial convolution layer, the eighth adjacent upsampling layer, the sixteenth Fourier partial convolution layer and the output layer are arranged in sequence according to the input and output order. The input end of the intermediate layer is connected to the output end of the eighth pooling layer in the encoder, and its output end is connected to the input end of the first adjacent upsampling layer in the decoder, which is used to gradually increase the spatial resolution of the feature image through eight adjacent upsampling layers and eight Fourier partial convolution layers, and restore the detail information of the image. The output layer is used to generate and output a virtual restoration image of the mural according to the feature image output by the sixteenth Fourier partial convolution layer in the decoder.

[0066] The eight skip connection structures are used to connect the feature images output by the eight Fourier partial convolutional layers in the encoder with the feature images output by the corresponding eight Fourier partial deconvolutional layers in the decoder, so as to help transmit detail information and improve the restoration effect.

[0067] Batch normalization and activation functions are used after each Fourier partial convolution layer in the U-Net neural network model to enhance the stability and nonlinear characteristics of the model. It is worth noting that batch normalization is not set after the first Fourier partial convolution layer, and no activation function is set after the sixteenth Fourier partial convolution layer. The activation function corresponding to each Fourier partial convolution layer in the encoder of this embodiment is a rectified linear unit, and the activation function corresponding to each Fourier partial convolution layer in the decoder is a leaky rectified linear unit. The specific parameters of the sixteen Fourier partial convolution layers, eight adjacent upsampling layers, and eight jump connection structures are shown in Table 1.

[0068] Table 1 Parameters of each layer of the U-Net neural network model

[0069]

[0070] Step 3: Pre-training of U-Net neural network model.

[0071] The CelebA dataset is used as the image dataset, and the NVIDIA Irregular Mask Dataset dataset is used as the mask dataset to pre-train the U-Net neural network model built in step 2. During the pre-training process, the U-Net neural network model extracts and represents data by learning features and patterns from the CelebA dataset and the NVIDIA Irregular Mask Dataset dataset. The features learned at different levels by the U-Net neural network model have different levels of abstraction, from simple edges and textures to more complex objects and semantic information.

[0072] Step 4: Transfer learning of U-Net neural network model.

[0073] A training set is constructed based on high-definition mural images taken from archaeological sites. The collected high-definition mural images are used as the objects of image virtual restoration. The high-definition mural images correspond one-to-one with the pixels of the hyperspectral mural images in the spatial dimension. The high-definition mural images in the training set are input into the pre-trained U-Net neural network model for further training. Through transfer learning, the common features and patterns learned by the U-Net neural network model on the CelebA dataset and the NVIDIA Irregular Mask Dataset dataset are transferred to the target task, thereby accelerating the training process of the target task and improving the performance of the U-Net neural network model.

[0074] During transfer learning, the parameters of the front layers of the pre-trained U-Net neural network model are kept unchanged, and only the back layers are fine-tuned to accelerate the learning of this task by using the knowledge of the source image when there is less image restoration data. In this embodiment, since the first few layers of the Fourier partial convolution layer usually learn some basic low-level features (such as edges, textures, etc.), these features are common to most tasks, the mean and variance parameters of the first five layers of the Fourier partial convolution layer can be frozen, while keeping their weights unchanged to avoid recalculation of batch normalization. The decoder maps the low-level features extracted by the encoder back to the high-resolution structure of the input space, and uses fine-tuning operations to adjust the stride or kernel size of the deconvolution through manual parameter adjustment to obtain better restoration effects.

[0075] Step 5: Select three bands from the hyperspectral mural image collected in step 1 to construct three channels of the RGB image, and then fuse the pixel values ​​of the three channels to obtain the mural RGB image.

[0076] In step 6, the RGB image of the mural obtained in step 5 and the mask of the mural generated in step 1 are input into a U-Net neural network model that has undergone pre-training and transfer learning for processing. The U-Net neural network model outputs a virtual restoration image of the mural to complete the virtual restoration of the mural.

[0077] The effect of this method is further illustrated by specific experiments below.

[0078] The hyperspectral mural images in this experiment were collected using an Aphis 2.0-I spectrometer. The experiment was conducted on an H3C UniServerR5300 G3 computer equipped with an NVIDIAGeForce RTX 3090 graphics card. The programming environment included software such as Anaconda5.3.0 (Python 3.7.0) and Matlab R2018b.

[0079] The hyperspectral mural image collected by step 1 of this method is as follows Figure 4 As shown, the obtained mask of the mural to be restored area is as follows Figure 5 As shown. Figure 4 The hyperspectral mural image is converted into a mural RGB image. Figure 5 The mask of the mural to be restored area is input into the trained U-Net neural network model, and the following output can be obtained: Figure 6 The virtual restoration image of the mural to be restored is shown.

[0080] The above description is only used to illustrate the technical solution of the present invention rather than to limit it. For ordinary professional and technical personnel in the field, the specific technical solution recorded in the above embodiment can be modified, or some of the technical features therein can be replaced by equivalents, and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution protected by the present invention.

Claims

1. A method for virtual restoration of murals based on hyperspectral images, characterized in that: The following steps are involved: Step 1: Collect the hyperspectral mural image and generate the mask of the area to be restored: A hyperspectral camera is used to collect the murals to obtain a hyperspectral mural image; the hyperspectral mural image is subjected to shape adaptive reconstruction, image spectrum dimension reduction processing and feature classification in sequence to extract the area to be restored in the hyperspectral mural image and generate a mask of the area to be restored in the mural; Step 2: Build a U-Net neural network model; The convolution layer in the U-Net neural network model is a Fourier partial convolution layer; the Fourier partial convolution layer includes two parallel branches, namely a local branch and a global branch; the local branch includes a first local convolution branch and a second local convolution branch in parallel, both of which use Fourier partial convolution to downsample the input image data to extract local features of the image data in the time domain; the global branch includes a third local convolution branch and a spectral convolution branch in parallel, the third local convolution branch uses Fourier partial convolution to downsample the input image data to extract boundary features of the image data, and the spectral convolution branch is used to convert the input image data from the time domain to the frequency domain, and perform a convolution operation in the frequency domain to capture the global context information of the image data in the frequency domain; The local features of the image data extracted by the first local convolution branch of the local branch are fused with the boundary features of the image data extracted by the third local convolution branch of the global branch, and the result is processed by the activation function as the output of the local branch; the global context information of the image data in the frequency domain extracted by the spectrum convolution branch of the global branch is fused with the local features of the image data extracted by the second local convolution branch of the local branch, and the result is processed by the activation function as the output of the global branch; finally, the outputs of the local branch and the global branch are fused to generate a feature image; Step 3: Use the CelebA dataset as the image dataset and the NVIDIA Irregular Mask Dataset as the mask dataset to pre-train the U-Net neural network model built in step 2. Step 4, constructing a training set, wherein the training set includes multiple high-definition mural images; inputting the high-definition mural images in the training set into the pre-trained U-Net neural network model for further training, and transferring the common features and patterns learned by the U-Net neural network model on the CelebA dataset and the NVIDIAIrregular Mask Dataset dataset to the target task through transfer learning; Step 5, converting the hyperspectral mural image acquired in step 1 into a mural RGB image; Step 6, input the mural RGB image obtained in step 5 and the mask of the mural area to be repaired generated in step 1 into the U-Net neural network model that has undergone pre-training and transfer learning for processing. The U-Net neural network model outputs a virtual repair image to complete the virtual restoration of the mural.

2. The method for virtual restoration of murals based on hyperspectral images according to claim 1, characterized in that: In step 2, the U-Net neural network model includes an encoder, an intermediate layer and a decoder; The encoder includes an input layer, a plurality of Fourier partial convolution layers, and a plurality of pooling layers corresponding to the plurality of Fourier partial convolution layers in sequence according to input and output; The input layer is used to receive image data input into the U-Net neural network model; the input end of each pooling layer is connected to the output end of the corresponding Fourier partial convolution layer in the encoder, and its output end is connected to the input end of the next Fourier partial convolution layer, and the output end of the last pooling layer is connected to the input end of the intermediate layer; the pooling layer is used to reduce the feature dimension of the output of the corresponding Fourier partial convolution layer; The decoder includes a plurality of adjacent upsampling layers, a plurality of Fourier partial convolution layers corresponding to the plurality of adjacent upsampling layers, a plurality of skip connection structures, and an output layer; the output end of the intermediate layer is connected to the input end of the first adjacent upsampling layer in the decoder, and the output end of each adjacent upsampling layer is connected to the input end of the corresponding Fourier partial convolution layer in the decoder, so as to gradually increase the spatial resolution of the feature map through the plurality of adjacent upsampling layers and the plurality of Fourier partial convolution layers, and restore the detail information of the image; The plurality of jump connection structures are used to respectively connect the feature images output by each Fourier partial convolution layer in the encoder with the feature images output by the corresponding Fourier partial convolution layer in the decoder to transfer detail information; The output layer is connected to the output end of the last Fourier partial convolution layer in the decoder, and is used to generate and output a virtual restoration image of the mural according to the feature image output by the last Fourier partial convolution layer in the decoder.

3. The method for virtual restoration of murals based on hyperspectral images according to claim 2 is characterized in that: Step 1 is as follows: Step 1.1, using a hyperspectral camera to collect the murals and obtain a hyperspectral mural image; Step 1.2, shape adaptive reconstruction of the hyperspectral mural image; A shape-adaptive region is generated by local polynomial approximate filtering and the intersection confidence interval rule. The surrounding pixels closest to the central pixel are selected in the shape-adaptive region, and the correlation between the central pixel and the surrounding pixels is estimated by calculating the Pearson correlation coefficient. The obtained correlation coefficient is then updated as a weight vector to reconstruct the spectrum of the central pixel. Step 1.3, using principal component analysis to perform spectral dimension reduction on the hyperspectral mural image after shape adaptive reconstruction, and obtain the principal component features of the hyperspectral mural image; Step 1.4, using support vector machine to classify the principal component features of the hyperspectral mural image; Step 1.5, based on the classification result of the support vector machine, extract the area to be restored of the hyperspectral mural image and generate a mask of the area to be restored of the mural.

4. The method for virtual restoration of murals based on hyperspectral images according to claim 3 is characterized by: In step 2, batch normalization is used after the second to N-1 Fourier partial convolution layers in the U-Net neural network model to enhance the stability and nonlinear characteristics of the model, where N is the total number of Fourier partial convolution layers in the encoder and decoder; among them, only the activation function is set after the first Fourier partial convolution layer, and only batch normalization is set after the Nth Fourier partial convolution layer.

5. The method for virtual restoration of murals based on hyperspectral images according to claim 4 is characterized by: In step 2, the activation function corresponding to each Fourier partial convolution layer in the encoder is a rectified linear unit, and the activation function corresponding to each Fourier partial convolution layer in the decoder is a leaky rectified linear unit.

6. The method for virtual restoration of murals based on hyperspectral images according to claim 3 is characterized by: In step 1.4, when the support vector machine classifies the principal component features, the Sigmoid function is used as the kernel function.

7. The method for virtual restoration of murals based on hyperspectral images according to claim 3 is characterized in that: Step 5 is as follows: Select three bands from the hyperspectral mural image collected in step 1 to construct three channels of the RGB image, and then fuse the pixel values ​​of the three channels to obtain the mural RGB image.

Images