A tangka image inpainting method based on iterative network

Abstract

The present application relates to the technical field of image processing, and more particularly to a Thangka image repairing method based on an iterative network. The Thangka image repairing method based on the iterative network comprises the following steps: (1) constructing an iterative image repairing network module: repairing the damaged area of the Thangka image in an iterative manner from outside to inside in the network; (2) optimizing the image repairing network module; and (3) inputting the damaged Thangka image into the optimized image repairing network module to obtain a repaired target Thangka image. The present application combines the texture and structure features of the Thangka image to design a texture repairing network and a structure repairing network, uses a feature self-adaptive fusion module to fuse the image features, and designs a loss function matched with the network according to the network characteristics, so as to effectively repair the damaged Thangka image by using the prior information of the Thangka image and greatly improve the training efficiency of the Thangka image and the repairing performance of the damaged Thangka image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a Thangka image inpainting method based on an iterative network. BACKGROUND

[0002] Image inpainting is an important research direction in the field of computer vision. Its research content is to enable the computer to automatically fill in the missing or damaged areas of the image, so that the output result retains the information of the original image and is as real and natural as possible. The key to solving the image inpainting problem lies in how to enable the computer to "understand" the information of natural images from the shallow to the deep level, and use these information to obtain the image inpainting result.

[0003] Thangka is a unique form of painting art in Tibetan culture. Tibetan culture has a long history, and over time, many precious Thangka murals have been damaged. Studying Thangka mural image inpainting can not only restore the damaged parts of the mural as much as possible and protect the cultural heritage of the Tibetan people, but also further explore the cultural and economic value of Thangka murals by obtaining complete mural content after inpainting. Traditional non-digital manual repair methods not only require professional Thangka painting skills, but also are extremely likely to cause irreversible repair results or even permanent damage to the Thangka. The existing digital repair methods rarely repair the structure and texture characteristics of Thangka images, and the performance needs to be improved. SUMMARY

[0004] In order to overcome the above problems, the purpose of the present application is to provide a Thangka image inpainting method based on an iterative network, which can effectively repair damaged Thangka images by using the prior information of Thangka images according to the image characteristics of Thangka images, and greatly improve the training efficiency of Thangka images and the performance of repairing damaged Thangka images.

[0005] A Thangka image inpainting system based on an iterative network includes an image input module, an image inpainting network module, a total loss calculation module, and an image output module.

[0006] The image input module is used to input the damaged Thangka image into the image inpainting network module.

[0007] The image inpainting network module includes an encoder unit, a region identification unit, a texture repair unit and a structure repair unit, a feature adaptive fusion unit, and a decoder unit, which are connected to form a neural network.

[0008] The total loss calculation module is used to train the image inpainting network module to iteratively optimize the parameters of the image inpainting network module. The total loss function of the total loss calculation module is composed of a pixel reconstruction loss function, a content loss function, and a style loss function:

[0009] The expression of the pixel reconstruction loss is:

[0010]

[0011] In formula (1), X i and X′ i are pixels on the original Thangka image and the corresponding repaired Thangka image respectively, m is the number of all pixels of the sample image, and c is a minimum constant used to stabilize the calculation process;

[0012] The expression of the content loss is:

[0013]

[0014] In formula (2), Φ i (·) represents a function used to convert an image to a feature space; i is an output layer in the function;

[0015] The expression of the style loss is:

[0016]

[0017] In formula (3), G(·) is a Gram matrix calculation operation on a feature map;

[0018] The total loss function is:

[0019]

[0020] In formula (4), L is the pixel reconstruction loss, is the content loss, is the style loss, λ Char , λ content and λ style are parameters for balancing the weights of the loss functions;

[0021] An image output module is configured to output a repaired Thangka image generated by the image repairing network module after being trained by the total loss calculation module.

[0022] As a preferred solution, the encoder unit and the decoder unit in the image repairing network module are both multi-layer convolutional layer structures; the convolutional layers of the encoder unit all use partial convolution, the convolutional layers of the decoder unit use transpose convolution and partial convolution, and a normalization layer is constructed using BatchNorm at the lower layer of each convolutional layer.

[0023] As a preferred solution, the region identification unit in the image inpainting network module is composed of partial convolution cascades. The partial convolution distinguishes the effective area and the hollow area of the damaged Thangka image by using the binary mask map of the damaged Thangka image, so that the convolution of the damaged Thangka image only depends on the effective area. The formula for generating new feature points x' by partial convolution is as follows:

[0024]

[0025] In formula (5), W is the convolution kernel parameter of ordinary convolution, b is the bias, X is the sliding window corresponding to the current convolution kernel on the input feature map, M is the binary mask map corresponding to the sliding window, and represents the matrix element point multiplication operation.

[0026] If the sliding window corresponding to the current convolution kernel is all from the hollow area, the feature points generated by the window will be directly set to 0.

[0027] As a preferred solution, the texture repair unit and the structure repair unit in the image inpainting network module are composed of a down-sampling layer, a feature repair layer and an up-sampling layer.

[0028] In the texture repair unit, the down-sampling layer is composed of three layers of convolution, which are one layer of convolution with a step size of 2 and a convolution kernel size of 3*3, and two layers of convolution with a step size of 1 and a convolution kernel size of 3*3; the feature repair layer is composed of three layers of dilated convolution, which are three layers of convolution with a convolution kernel size of 3*3 and a step size of 1, and the dilated rates are 2, 4 and 8 respectively; and the up-sampling layer is composed of three layers of transposed convolution, which are three layers of transposed convolution with a convolution kernel size of 4*4 and a step size of 2.

[0029] In the structure repair unit, the down-sampling layer is composed of three layers of convolution, which are three layers of convolution with a convolution kernel size of 3*3 and a step size of 2; the feature repair layer is composed of three layers of dilated convolution, which are three layers of convolution with a convolution kernel size of 3*3 and a step size of 1, and the dilated rates are all 2; and the up-sampling layer is composed of three layers of transposed convolution, which are three layers of transposed convolution with a convolution kernel size of 4*4 and a step size of 2.

[0030] As a preferred solution, the dilated convolution formula of the feature repair layer in the texture repair unit and the structure repair unit is as follows:

[0031]

[0032] Let the input feature map be x, where each pixel coordinate is i, the convolution kernel be w, the dilated rate be r, and the output feature map be y.

[0033] As a preferred solution, the expression of feature self-adaptive fusion in the image inpainting network module is as follows:

[0034]

[0035] In formula (7), M i is the occlusion map of the i-th iteration; F i is the feature map generated in the i-th iteration; and represents the matrix element point multiplication operation. out is the final fused feature map.

[0036] A method for repairing a Thangka image by using a Thangka image repairing system based on an iterative network, comprising the steps of:

[0037] 1. constructing an image repairing network module of the iterative network;

[0038] 1a. an encoder unit extracts texture detail features and global structure features from a damaged Thangka image to obtain preliminary image high-level semantic information and sends the information to a region identification unit;

[0039] 1b. a region identification unit extracts a region to be repaired in the current iteration to obtain a feature map, and sends the feature map to a texture repairing unit and a structure repairing unit for repairing respectively. After each iteration, the region identification unit extracts the region to be repaired in the current iteration from the feature map generated by a feature adaptive fusion unit in the previous iteration, and sends the region to the texture repairing unit and the structure repairing unit for repairing again;

[0040] 1c. the feature map is sent to the texture repairing unit and the structure repairing unit respectively, and two repaired feature maps are generated, which are sent to a feature adaptive fusion unit for combination to generate a result of one iteration and save the result, and then the result is sent to the region identification unit for the next iteration;

[0041] 1d. in the texture repairing unit, a down-sampling layer performs down-sampling processing on the feature map output by the region identification unit and sends the feature map to a feature repairing layer; the feature repairing layer repairs image texture detail information to obtain a corresponding repaired feature map and sends the feature map to an up-sampling layer; the up-sampling layer restores the feature map output by the feature repairing layer to the size of the feature map before inputting into the down-sampling layer;

[0042] 1e. in the structure repairing unit, a down-sampling layer performs down-sampling processing on the feature map output by the region identification unit and sends the feature map to a feature repairing layer; the feature repairing layer repairs image macro-structure information to obtain a corresponding repaired feature map and sends the feature map to an up-sampling layer; the up-sampling layer restores the feature map output by the feature repairing layer to the size of the feature map before inputting into the down-sampling layer;

[0043] 1f. a feature adaptive fusion unit fuses the feature maps generated by the texture repairing unit and the structure repairing unit in each iteration to form a final feature map of one iteration and save the feature map; the feature adaptive fusion unit fuses the feature maps generated in each iteration to generate a final fused feature map and sends the feature map to a decoder unit;

[0044] 1g, The final fused feature map of the decoder unit is restored to the target Thangka image;

[0045] 2. Optimize the image restoration network module;

[0046] 2a. Establish a training set to train the image inpainting network module and output the training results;

[0047] Each training sample in the training set includes: the original Thangka image, a mask image representing the missing area, and a damaged Thangka image obtained by fusing the original Thangka image and the mask image representing the missing area; the training result corresponding to each training sample is: the repaired Thangka image obtained by repairing the damaged Thangka image in the training sample using the image restoration network module in step 1;

[0048] 2b. Construct an image loss calculation module to calculate the total loss function between the original Thangka image and the corresponding restored Thangka image;

[0049] 2c. Optimize the parameters of the image inpainting network module using the total loss function to obtain the optimized image inpainting network module;

[0050] 3. The damaged Thangka image is input into the optimized image restoration network module through the image input module, and the target Thangka image is output by the image output module.

[0051] A device, characterized in that it comprises:

[0052] At least one processor; and,

[0053] A memory communicatively connected to the at least one processor; wherein,

[0054] The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform the above-described method for restoring Thangka images.

[0055] A computer-readable storage medium storing a computer program, which, when executed by a processor, describes a method for restoring the aforementioned Thangka image.

[0056] The advantages of this invention are as follows: Compared with traditional image restoration methods, this invention has the following advantages: Based on the idea of ​​iterative networks, this invention combines the texture and structural features of Thangka images to design texture restoration networks and structural restoration networks, and uses a feature adaptive fusion module to fuse image features. Furthermore, it designs a loss function that matches the network characteristics, effectively utilizing the prior information of Thangka images to restore damaged Thangka images, greatly improving the training efficiency of Thangka images and the performance of restoring damaged Thangka images. Attached Figure Description

[0057] Figure 1 This is a flowchart of the method for restoring Thangka images using the iterative network-based Thangka image restoration system in this embodiment;

[0058] Figure 2 This is a schematic diagram showing the connection relationship between the modules in the Thangka image restoration system based on iterative networks in this embodiment;

[0059] Figure 3 This is a schematic diagram of the image restoration network module in this embodiment;

[0060] Figure 4 This is a flowchart illustrating the construction of the total loss calculation module in this embodiment;

[0061] Figure 5 This is a rendering of the damaged Thangka image and the target Thangka image in this embodiment. Detailed Implementation

[0062] To better understand the present invention, the invention will be described in detail below with reference to the accompanying drawings and specific examples.

[0063] Combination Figure 1 and Figure 2 As shown, the Thangka image restoration method based on iterative networks provided in this embodiment includes...

[0064] (1) Construct an iterative image restoration network module: repair the damaged areas of the Thangka image step by step from the outside to the inside in an iterative manner in the network;

[0065] (2) Optimize the image restoration network module;

[0066] (3) Input the damaged Thangka image into the optimized image restoration network module through the image input module, and output the target Thangka image through the image output module.

[0067] The following details the process of Thangka restoration:

[0068] Step (1), Combining Figure 3 As shown, an iterative image inpainting network module is constructed, specifically including:

[0069] From top to bottom, the encoder unit, region recognition unit, texture restoration unit and structure restoration unit, feature adaptive fusion unit and decoder unit are connected to form a neural network.

[0070] 11. Both the encoder unit and the decoder unit are multi-layer convolutional layer structures, specifically including: the convolutional layers of the encoder unit all use partial convolution; the convolutional layers of the decoder unit use transposed convolution and partial convolution; the lower layer of each convolutional layer uses BatchNorm to construct a normalization layer.

[0071] 12. In this embodiment, the region identification unit specifically includes:

[0072] The region recognition unit is composed of partially convolutional cascades. The partially convolutional layer utilizes the binary mask image of the damaged image (marked as 1 if a pixel is within the valid region, and 0 otherwise) to distinguish between the valid and void regions of the damaged image, making the convolution on the damaged image dependent only on the valid region. Let W be the kernel parameters of a normal convolution, b be the bias, X be the sliding window corresponding to the current convolution kernel on the input feature map, M be the binary occlusion image corresponding to the sliding window, and ⊙ represent the matrix element-wise multiplication operation. Then, the formula for generating a new feature point x′ through partially convolution can be expressed as:

[0073]

[0074] If the sliding window corresponding to the current convolution kernel is entirely derived from the hole region, then the feature points generated by that window will be directly set to 0. In this way, the feature map output by part of the convolution can completely mask the interference from the hole region. Similarly, while performing convolution operations on the image, a similar convolution operation is performed on the mask image and it is re-binarized. The updated mask image is then passed to the next convolutional layer. The update formula for a point m′ on the new occlusion map is as follows:

[0075]

[0076] The mask image generated by partial convolution has the same size as the feature map, and its update is parallel to the feature map update. They correspond one-to-one from the input partial convolution layer. As the partial convolution continues, the feature map size shrinks, and the hole region on the mask image also shrinks accordingly, ultimately achieving convolution of the effective feature region of the image.

[0077] After each iteration, the region identification unit sends the results obtained from the previous texture restoration unit and structure restoration unit into the region identification unit, extracts the region restored in this iteration, and sends it back to the texture restoration unit and structure restoration unit for restoration.

[0078] 13. In this embodiment, the texture restoration unit and the structure restoration unit are composed of a downsampling layer, a feature restoration layer, and an upsampling layer, specifically including:

[0079] The downsampling layer, feature restoration layer, and upsampling layer are all multi-layer convolutional layer structures;

[0080] Each convolutional layer is followed by a normalization layer and an activation function layer.

[0081] In this embodiment, the downsampling layer in the texture restoration unit consists of three convolutional layers. From top to bottom, these three layers consist of one convolution with a stride of 2 and a kernel size of 3*3, and two convolutions with a stride of 1 and a kernel size of 3*3. In the structure restoration unit, these three convolutions are three convolutions with a kernel size of 3*3 and a stride of 2. The downsampling layer's function is to downsample the primary features obtained from the decoder unit to obtain further image features, which are then sent to the feature restoration layer. Since texture information is largely contained within the high-resolution feature map, excessive downsampling would destroy the texture information. Therefore, in the texture restoration unit, to ensure that the texture restoration unit can capture a wider range of texture details, only one downsampling operation with a stride of 2 and a kernel size of 3*3 is performed on the feature map. The subsequent two convolutions with a stride of 1 and a kernel size of 3*3 do not change the size of the feature map.

[0082] The feature restoration layer consists of three dilated convolutional layers. Dilated convolutions increase the dilation rate compared to ordinary convolutions, which is the distance between adjacent points in the kernel matrix, but the actual size of the kernel matrix remains unchanged. This increases the receptive field of the model without increasing computational cost. Let the input feature map be x, where each pixel coordinate is i, the convolution kernel is w, the dilation rate is r, and the output feature map be y. The dilated convolution formula is defined as follows:

[0083]

[0084] In the texture restoration unit, the three convolutional layers are 3x3 kernels with a stride of 1, and their dilation rates are 2, 4, and 8, respectively. In the structure restoration unit, the three convolutional layers are also 3x3 kernels with a stride of 1, and their dilation rate is 2 for all layers. Using progressively increasing dilation rates effectively ensures that the texture restoration unit has a comprehensive grasp of both the overall texture and its details, while the structure restoration unit focuses on the macroscopic semantic structure of the Thangka image.

[0085] The upsampling layer restores the feature map from the feature restoration layer to the same size as the feature map before it was input to the downsampling layer. The upsampling layer consists of three transposed convolutional layers with a kernel size of 4*4 and a stride of 2.

[0086] 14. In this embodiment, the lower layer of each convolutional layer in the texture restoration unit and the structure restoration unit is normalized using BatchNorm, and then ReLU is used as the activation function.

[0087] 15. In this embodiment, the feature adaptive fusion unit specifically includes:

[0088] The feature adaptive fusion unit fuses and saves the feature maps generated by the texture repair branch and the structure repair branch through each iteration of the iterative network structure, and fuses the feature maps of each round to generate the final fused feature map.

[0089] The expression for adaptive feature fusion is:

[0090]

[0091] In formula (7), M i It is the occlusion map of the i-th iteration; F i It is the feature map generated in the i-th iteration; ⊙ represents the element-wise multiplication operation of the matrix; F out This is the final fused feature map.

[0092] 16. Combining Figure 3 As shown, the process of repairing a damaged Thangka image using the image restoration network module described above is as follows:

[0093] 16a. The encoder unit extracts texture detail features and global structural features from the damaged Thangka image to obtain preliminary high-level semantic information of the image, and sends it to the region recognition unit.

[0094] 16b. The region identification unit extracts the region repaired in the current iteration to obtain a feature map, and sends the feature map to the texture repair unit and the structure repair unit for repair respectively. After each iteration, the region identification unit extracts the region repaired in the current iteration from the feature map generated by the feature adaptive fusion unit in the previous round, and sends it to the texture repair unit and the structure repair unit for repair again;

[0095] 16c. The feature map is processed by the texture restoration unit and the structure restoration unit respectively, and two restored feature maps are generated. They are then sent to the feature adaptive fusion unit for combination, generating the result of one round of iteration and saving it. The result is then sent to the region recognition unit for the next round of iteration.

[0096] In the 16d texture restoration unit, the downsampling layer downsamples the feature map output by the region recognition and sends it to the feature restoration layer; the feature restoration layer restores the image texture details to obtain the corresponding restored feature map and sends it to the upsampling layer; the upsampling layer restores the feature map output by the feature restoration layer to the size of the feature map before it was input to the downsampling layer.

[0097] 16e. In the structural repair unit, the downsampling layer downsamples the feature map output by the region recognition and sends it to the feature repair layer; the feature repair layer repairs the macroscopic structural information of the image to obtain the corresponding repaired feature map and sends it to the upsampling layer; the upsampling layer restores the feature map output by the feature repair layer to the size of the feature map before it was input to the downsampling layer.

[0098] 16f. The feature adaptive fusion unit fuses the feature maps generated by the texture restoration unit and the structure restoration unit in each iteration to form the final feature map of one iteration and saves it; it fuses the feature maps generated in each iteration to generate the final fused feature map and sends it to the decoder unit.

[0099] The final fused feature map of the 16g decoder unit is restored to the target Thangka image;

[0100] (2) Combination Figure 4 As shown, the image inpainting network module has been optimized:

[0101] 2a. First, a training set is established to train the image restoration network module, and the training results are output. Each training sample in the training set includes: the original Thangka image (derived from an undamaged Thangka wall image), a mask image representing the missing area, and a damaged Thangka image obtained by fusing the original Thangka image and the mask image representing the missing area. The corresponding training result for each training sample is: the restored Thangka image obtained by restoring the damaged Thangka image using the method in step 1.

[0102] The specific process is as follows:

[0103] First, preprocess the training samples: Before inputting each damaged Thangka image from the training samples into the image restoration network module, training samples of different sizes need to be scaled to a uniform size. In this embodiment, the training set is preprocessed to make the images in the training samples have a resolution of 256*256.

[0104] Then, each damaged Thangka image in the training samples is input into the image restoration network module: its operation process is the same as step 1, and will not be repeated here.

[0105] 2b. Construct a total loss calculation module using the VGG16 model;

[0106] In this embodiment, the VGG16 model consists of 5 convolutional layers, each with 2-4 convolutional kernels and corresponding ReLU and pooling layers.

[0107] The total loss function in the total loss calculation module is composed of the pixel reconstruction loss function, content loss function, and style loss function superimposed:

[0108] The expression for the pixel reconstruction loss is:

[0109]

[0110] In formula (1), X i With X i , respectively, are the pixels on the original Thangka image and the corresponding restored Thangka image, m is the total number of pixels in the sample image, and c is a minimal constant used for the stabilization calculation process;

[0111] The expression for the content loss is:

[0112]

[0113] In formula (2), Φ i (·) represents the function used to transform the image to the feature space; i is a certain output layer in the function.

[0114] In this embodiment, a pre-trained VGG16 network is used as a feature extractor to extract feature maps from the image. The original Thangka image and the corresponding restored Thangka image are input into the image loss calculation module, and the content loss function is calculated using the feature maps output by VGG16's conv1_2, conv2_2, and conv3_3.

[0115] The style loss expression is as follows:

[0116]

[0117] In formula (3), G(·) is the Gram matrix calculation operation performed on the feature map.

[0118] In the specific implementation process, a pre-trained VGG16 network is used as a feature extractor to extract feature maps from the image. The original Thangka image and the corresponding restored Thangka image are input into the image loss calculation module, and the style loss function is calculated using the feature maps output by VGG16's conv1_2, conv2_2, and conv3_3.

[0119] The total loss function is:

[0120]

[0121] In formula (4), For pixel reconstruction loss, For content loss, For style loss, λ Char , λ content and λ style The parameters are used to balance the weights of the various loss functions.

[0122] 2c. The total loss function is minimized by using the adaptive moment estimator (Adam) optimizer, and the parameters of the image inpainting network module are iteratively optimized to obtain the optimized image inpainting network module.

[0123] The optimized parameters are as follows: learning rate of 0.0001, batch size (number of samples selected in one training iteration) of 6, epochs (single training iteration of all batches in forward and backward propagation) of 7, number of iterations of the iterative network of 6, and the size of the Tangka dataset of approximately 90,000. In actual training, the total loss basically converged when the training was 2 / 3 complete, so appropriately reducing the number of epochs is feasible and can improve efficiency.

[0124] (3) Input the damaged Thangka image into the optimized image restoration network module to obtain the restored target Thangka image.

[0125] Combination Figure 5 As shown, five damaged Thangka images are input, and the corresponding target Thangka image is output by the Thangka image restoration method based on iterative network in this embodiment. It can be seen that the output target Thangka image is complete and has distinct layers. It has good effects in terms of overall structural completion and texture detail restoration of the damaged area of ​​the image, which conforms to human visual semantics.

Claims

1. A Thangka image restoration system based on iterative networks, characterized in that, It includes an image input module, an image inpainting network module, a total loss calculation module, and an image output module; The image input module is used to input the image of the damaged Thangka into the image restoration network module; The image inpainting network module consists of an encoder unit, a region recognition unit, a texture inpainting unit, a structure inpainting unit, a feature adaptive fusion unit, and a decoder unit, which are connected to form a neural network. The total loss calculation module is used to train the image inpainting network module to iteratively optimize the parameters of the image inpainting network module; The construction process of the total loss calculation module is as follows: First, a training set is established. Each training sample in the training set includes: the original Thangka image, a mask image representing the missing area, and a damaged Thangka image obtained by fusing the original Thangka image and the mask image representing the missing area. The training result corresponding to each training sample is: the restored Thangka image obtained by restoring it using the image restoration network module. Then, the total loss calculation module is constructed using the VGG16 model. The total loss function of the total loss calculation module is composed of the pixel reconstruction loss function, the content loss function, and the style loss function superimposed. The expression for the pixel reconstruction loss is: In formula (1), and , respectively, are the pixels on the original Thangka image and the corresponding restored Thangka image, m is the total number of pixels in the sample image, and c is a minimal constant used for the stabilization calculation process; The expression for the content loss is: In formula (2), This represents the function used to transform an image into a feature space; It is a certain output layer in this function; The style loss expression is as follows: In formula (3), To perform Gram matrix calculation operations on the feature map; The total loss function is: In formula (4), For pixel reconstruction loss, For content loss, For the loss of style, , and Parameters to balance the weights of the various parts of the loss function; The image output module is used to output the restored Thangka image generated by the image restoration network module after being trained by the total loss calculation module; The encoder and decoder units in the image restoration network module are both multi-layer convolutional layer structures; the convolutional layers of the encoder unit all use partial convolution, and the convolutional layers of the decoder unit use transposed convolution and partial convolution. The lower layer of each convolutional layer uses BatchNorm to construct a normalization layer. The region recognition unit in the image restoration network module is composed of partially convolutional cascades. The partially convolutional units utilize the binary mask image of the damaged Thangka image to distinguish between the effective and hollow regions of the damaged Thangka image, ensuring that convolution on the damaged Thangka image depends only on the effective region. New feature points are generated through the partially convolutional units. The formula is expressed as: In formula (5), These are the kernel parameters for a regular convolution. For deviation, This represents the sliding window corresponding to the current convolutional kernel on the input feature map. This is the binary occlusion map corresponding to the sliding window. Represents the element-wise multiplication operation of a matrix; If the sliding window corresponding to the current convolution kernel is entirely from the hole region, then the feature points generated by that window will be directly set to 0; The texture restoration unit and structure restoration unit in the image restoration network module are composed of a downsampling layer, a feature restoration layer and an upsampling layer; In the texture restoration unit, the downsampling layer consists of three convolutional layers, which from top to bottom are one convolution with a stride of 2 and a kernel size of 3×3, and two convolutional layers with a stride of 1 and a kernel size of 3×3; the feature restoration layer consists of three dilated convolutional layers, which are three convolutional layers with a kernel size of 3×3 and a stride of 1, with dilation rates of 2, 4, and 8 respectively; the upsampling layer consists of three transposed convolutional layers, which are three transposed convolutional layers with a kernel size of 4×4 and a stride of 2. In the structural repair unit, the downsampling layer consists of three convolutional layers with a kernel size of 3×3 and a stride of 2; the feature repair layer consists of three dilated convolutional layers with a kernel size of 3×3 and a stride of 1, all with a dilation rate of 2; and the upsampling layer consists of three transposed convolutional layers with a kernel size of 4×4 and a stride of 2.

2. The Thangka image restoration system based on iterative networks according to claim 1, characterized in that, The dilated convolution formulas for the feature restoration layers in the texture restoration unit and structure restoration unit are as follows: Let the input feature map be The coordinates of each pixel are The convolution kernel is expansion rate The output feature map is .

3. The Thangka image restoration system based on iterative networks according to claim 2, characterized in that, The expression for feature adaptive fusion in the image inpainting network module is: In formula (7), It is the first The occlusion map of the next iteration; It is the first Feature maps generated in the next iteration; Represents the element-wise multiplication operation of a matrix; This is the final fused feature map.

4. A method for restoring Thangka images using the Thangka image restoration system based on iterative networks as described in any one of claims 1 to 3, characterized in that, Including the following steps:

1. Construct an iterative image inpainting network module; 1a. The encoder unit extracts texture detail features and global structural features from the damaged Thangka image to obtain preliminary high-level semantic information of the image, and sends it to the region recognition unit. 1b. The region identification unit extracts the region repaired in the current iteration to obtain a feature map, and sends the feature map to the texture repair unit and the structure repair unit for repair. After each iteration, the region identification unit extracts the region repaired in the current iteration from the feature map generated by the feature adaptive fusion unit in the previous round, and sends it to the texture repair unit and the structure repair unit for repair again. 1c. The feature map is processed by the texture restoration unit and the structure restoration unit respectively, and two restored feature maps are generated. They are then sent to the feature adaptive fusion unit for combination, generating the result of one round of iteration and saving it. The result is then sent to the region recognition unit for the next round of iteration. 1d. In the texture restoration unit, the downsampling layer downsamples the feature map output by the region recognition and sends it to the feature restoration layer; The feature restoration layer restores the image texture details, obtains the corresponding restored feature map, and sends it to the upsampling layer; the upsampling layer restores the feature map output by the feature restoration layer to the size of the feature map before it was input to the downsampling layer; 1e. In the structural repair unit, the downsampling layer downsamples the feature map output by the region identification and sends it to the feature repair layer; The feature restoration layer restores the macroscopic structural information of the image, obtains the corresponding restored feature map, and sends it to the upsampling layer; the upsampling layer restores the feature map output by the feature restoration layer to the size of the feature map before it was input to the downsampling layer; 1f. The feature adaptive fusion unit fuses the feature maps generated by the texture restoration unit and the structure restoration unit in each iteration to form the final feature map of one iteration and saves it; it fuses the feature maps generated in each iteration to generate the final fused feature map and sends it to the decoder unit. 1g, The final fused feature map of the decoder unit is restored to the target Thangka image; 2. Optimize the image restoration network module; 2a. Establish a training set to train the image inpainting network module and output the training results; Each training sample in the training set includes: the original Thangka image, a mask image representing the missing area, and a damaged Thangka image obtained by fusing the original Thangka image and the mask image representing the missing area; the training result corresponding to each training sample is: the repaired Thangka image obtained by repairing the damaged Thangka image in the training sample using the image restoration network module in step 1; 2b. Construct an image loss calculation module to calculate the total loss function between the original Thangka image and the corresponding restored Thangka image; 2c. Optimize the parameters of the image inpainting network module using the total loss function to obtain the optimized image inpainting network module; 3. The damaged Thangka image is input into the optimized image restoration network module through the image input module, and the target Thangka image is output by the image output module.

5. A device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as claimed in claim 4.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of claim 4.

Images