Improved loss function SinGAN method for reconstructing missing information between tomb mural blocks

Abstract

The present invention discloses a method for reconstructing missing information between tomb mural blocks by improving the loss function SinGAN, comprising the following steps: taking any mural block image I in the mural set r , and then downsample it to {I r0 ,I r1 ,I r2 ,I r3 ,I r4}, where I m4 The mural is downsampled 4 times after masking; input I to the bottom layer of the generator m4 , the generator G4 generates the extensional mural G4 (I m4 ), change G4(I m4 ) and I r4 The input is sent to the discriminator D4 for comparison and discrimination, and the weight parameters of the layer are updated according to the loss function; the weight parameters of each layer in the generator are obtained; the generator is trained using the training set and the discriminator so that the generator and the discriminator reach a Nash equilibrium, and then the trained generator is used to reconstruct the missing information between the blocks of the tomb murals. This method can realize the reconstruction of the extensional information between the blocks of the murals.

Description

Technical Field

[0001] The invention belongs to the field of digital image restoration and relates to a method for reconstructing missing information between tomb mural blocks by improving the loss function SinGAN. Background Art

[0002] Tomb murals have dual important significance for ancient Chinese history and art history. They truly reflect the living conditions, social customs and artistic interests of ancient Chinese people, as well as people's religious beliefs, funeral culture and many other aspects. Because tomb murals are different from palace murals and cave temple murals, they have been buried deep underground for thousands of years and were completely closed spaces before excavation, so the residual information is highly reliable. In addition, tomb murals are large in size. Take the polo picture in the tomb of Prince Zhanghuai as an example: the mural is nearly 7 meters long and 3 meters high. Under the conditions of archaeological excavation in the 1970s and 1980s, only the method of removing them in blocks could be adopted. This resulted in a large number of precious tomb murals preserved in blocks, and the process of removing them also caused the murals to have information missing between blocks, affecting the coherence and integrity of the entire mural.

[0003] At present, the computer-aided digital information restoration method of ancient murals is digital image inpaiting, which retrieves the edge of the missing area of ​​the image information, uses the residual information of the image to diffuse from the edge and fill it into the defective area, and finally completes the reconstruction of the mural information. This type of technology is mainly completed from two technical directions. On the one hand, it is a PDE model based on pixel diffusion. It completes the information filling by calculating the high-order partial derivative function of the filling front edge and using different diffusion equations. The disadvantage is that when the picture information is missing, it will produce blurring; on the other hand, it is a texture synthesis model based on sample filling. It completes the information completion in a certain filling order by comparing the correlation between the retained samples and the filling front edge. The disadvantage is that the filling of a large number of similar samples will produce a mosaic effect. These two methods cannot achieve a good restoration effect of tomb murals when the missing area is large, and they only find available information from the existing residual information. They have great limitations. For block murals, they can only fill the holes inward, and cannot meet the reconstruction requirements of the extensional information between block murals. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a method for reconstructing missing information between blocks of tomb murals with an improved loss function SinGAN, which can realize the reconstruction of extensional information between blocks of murals.

[0005] To achieve the above object, the method for reconstructing missing information between tomb mural blocks using the improved loss function SinGAN of the present invention comprises the following steps:

[0006] 1) Collect images of multiple blocks of a mural excavated in the tomb passage, obtain mural block images, add masks to the mural block images and perform min-max normalization, then construct an image set based on the normalized mural block images, and use the image set as a training set for SinGAN to generate extensional information between murals;

[0007] 2) Build a generative network based on SinGAN that includes generators and discriminators of 5 scales, and increase the reconstruction loss L rec , pixel reconstruction loss and texture loss L texture The loss function of

[0008] 3) Take any mural block image I in the mural collection r , and then downsample it to {I r0 ,I r1 ,I r2 ,I r3 ,I r4}, where I m4 This is a mural that was downsampled 4 times after masking;

[0009] 4) Input I to the bottom layer of the generator m4 , the generator G4 generates the extensional mural G4 (I m4 ), change G4(I m4 ) and I r4 The input is sent to the discriminator D4 for comparison and discrimination, and the weight parameters of the layer are updated according to the loss function;

[0010] 5) Repeat step 4) to obtain the weight parameters of each layer in the generator;

[0011] 6) Use the training set and the discriminator to train the generator so that the generator and the discriminator reach Nash equilibrium, and then use the trained generator to reconstruct the missing information between the blocks of the tomb murals.

[0012] The pixel reconstruction loss is used to capture the overall pixel information around the mural generation area, i.e.

[0013]

[0014] Where M is a binary mask, I rn is the downsampled mural of the real data, G n To generate the mural images generated by the network, Multiply the pixels of the mural;

[0015] The texture loss is designed using the Gram matrix, namely

[0016]

[0017] Among them, G is the Gram matrix, l n is the real mural image I rn Through the output of the nth layer of the multi-scale generator, I n is the multi-scale generator G n The generated mural, g represents the generation network;

[0018] The designed reconstruction loss is:

[0019] L rec =||G n (Noize+I mn )-I rn || 2 (4)

[0020] Among them, G n is the nth layer of the multi-scale generator, (Noize+I mn ) is the noise of each layer in the multi-scale generator superimposed on the output mural of the previous layer, I rn It is a real mural.

[0021] The reconstruction loss L rec , pixel reconstruction loss and texture loss L texture The loss functions are:

[0022]

[0023] L rec =||G n (Noize+I mn )-I rn ‖ (6)

[0024] L G = xL rec +L 2+texture (7)

[0025]

[0026] Among them, L 2+texture is the sum of the pixel reconstruction loss and texture loss of the mural, L D is the loss function of the discriminator, L D Used to update the discriminator weights, χ is an adjustable parameter, and L is adjusted by adjusting the generator's loss function 2+texture .

[0027] Design the multi-scale generator and discriminator structure, let {I r0 ,I r1 ,I r2 ...I rn} are a group of mural block images obtained by downsampling the real mural, I mnThe mural block image is downsampled n times after masking, and the input I is at the bottom layer of the multi-scale generator. mn , after the generator G n Generate mural G n (I mn ), and then input to the discriminator D n Zhong and I rn Compare and judge, and update the weight parameters of this layer, and then G n (I mn ) is upsampled and superimposed with noise of equal size as the generator G n-1 The input image is looped until the image is of its original size.

[0028] SSIM is used to measure the fidelity and similarity between two murals, where

[0029]

[0030] in, Original mural image I r The mean of For the extended mural I m The mean of Original mural I r The variance of To extend the mural I m variance, For I r ,I m The covariance of b1 and b2 is used to prevent overflow caused by the SSIM denominator being zero. SSIM is between [0,1]. The larger the SSIM, the more similar the contents of the two images are.

[0031] The present invention has the following beneficial effects:

[0032] The method for reconstructing missing information between tomb mural blocks using the improved loss function SinGAN described in the present invention, during specific operation, uses the SinGAN after the improved loss function to generate extensional information on both sides of the mural blocks, thereby realizing the reconstruction of missing parts between mural blocks unearthed from underground tombs. Compared with traditional digital image restoration that uses image completion (image inpainting) to fill in information on tomb murals, the present invention is more suitable for the outward extension of mural blocks to generate picture information. It should be noted that the present invention can realize the reconstruction of a large amount of information between the tomb mural blocks that have been removed by blocks, and compared with inpainting filling, it can effectively ensure the consistency and coherence of global information. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Generate a discrimination flow chart in the present invention;

[0034] Figure 2 is the mural image after training in the present invention;

[0035] Figure 3 This is the mural painting of the Central Plains of the present invention;

[0036] Figure 4 This is a calculation result diagram of the SSIM value in the present invention;

[0037] Figure 5 This is a result map of the mural generation in the present invention. DETAILED DESCRIPTION

[0038] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0039] refer to Figure 1 The method for reconstructing missing information between tomb mural blocks using the improved loss function SinGAN of the present invention comprises the following steps:

[0040] 1) Collect high-definition images of tomb murals, add masks to the images and perform min-max standardization, then construct a set of 8000 256*256 images through the standardized murals, and use this mural set as the training set. Among them, high-definition tomb murals are collected through split shots, and when selecting standardized murals, it is necessary to ensure that the redundancy of effective information in the split shots is maximized.

[0041] Among them, the tomb mural polo picture dataset, the murals in the original data form a training set. Its preprocessing process is given a training mural I r , normalize the mural to I r ∈[0,1] 256*256*3 , the formula for min-max normalization is:

[0042]

[0043] Among them, x is the value of the sample point, Min is the minimum value of the sample data, and Max is the maximum value of the sample data.

[0044] Subtract the minimum value from the sample point value and divide it by the difference between the maximum and minimum values ​​of the sample point value to improve the performance of the neural network.

[0045] During training, I r For the original mural, define mask M∈[0,1] 256*256*3 , so that M ij It can mask the central part of the image, where the mask is a binary mask M (mask) using only two values ​​0 and 1. A value of 1 indicates that the corresponding part of the image is retained, and a value of 0 indicates that the corresponding part of the image needs to be completed.

[0046] 2) Construct five multi-scale generators and discriminators of SinGAN, and increase the loss function of optimizing the iteration degree;

[0047] Five multi-scale generators and discriminators are designed. Both the generator and the discriminator are equipped with 6 convolutional layers with a convolution kernel of 4*4. Among them, there are two layers of dilated convolution to complete the mural image. The discriminator consists of 6 layers of full convolution. The generator at the bottom layer can generate the completeness information of the mural. With each upward layer, the generator generates more details until the size generated by the last layer generator G0 is 256*256 and the content and style of the mural become more and more realistic.

[0048] The pixel reconstruction loss is used to capture the overall pixel information around the mural generation area, and its expression is:

[0049]

[0050] Where M is a binary mask, I rn Some downsampled murals of real data, G n To generate the mural images generated by the network, Multiply the pixels of the mural.

[0051] The texture loss is designed through the Gram matrix, namely:

[0052]

[0053] Among them, G is the Gram matrix, l n is the real mural image I rn Through the output of the nth layer of the network, I n is the multi-scale generator G n The generated mural, g represents the generation network, which extracts the gap between the texture features of each layer so that the epitaxial mural image has the same local texture as the global one to enhance the realism of the epitaxial mural.

[0054] Design the reconstruction loss, i.e.

[0055] L rec =‖G n (Noize+I mn )-I rn || 2 (4)

[0056] Among them, G n is the nth layer of the multi-scale generator, (Noize+I mn ) is the noise of each layer superimposed on the output mural of the previous layer, I rn It is a real mural.

[0057] 3) Take a mural from the mural collection I r , downsample it to {I r0 ,I r1 ,I r2 ,I r3 ,I r4}, I m4 The mural is downsampled 4 times after masking, and the input I is at the bottom layer of the multi-scale generator. m4 , through generator G4, generates mural G4 (I m4 );

[0058] 4) Place G4(I m4 ) and I r4 The input is sent to the discriminator D4 for comparison and discrimination, and the weight parameters of the layer are updated according to the training loss function. This is repeated until the G0 layer. Since the weight parameters of each layer are updated, the generated murals will become more and more realistic.

[0059] The generator can extract the mural features to generate murals, and the training set murals I r Downsampling 4 times, we get {I r0 ,I r1 ,I r2 ,I r3 ,I r4}, define the masked I m We also downsample 4 times to get I m4 First, input a mural I into the generator G4 m4 , the generator can generate the overall information of the mural. m4 ) and I r4 Input to the discriminator D4 for discrimination, and update the weight parameters of the layer according to the loss function, and then obtain G4 (I m4 ) is upsampled to obtain I m3 , and superimposed with noise of the same size, the superimposed mural is input to the G3 generator, and G3(I m3 ) and I r3 Input the discriminator D3 for discrimination, calculate the loss and update the parameters, and repeat until the G0 generator. At this time, the generated mural has the richest details and content, which greatly improves the quality of the generated mural. The generation and discrimination flow chart is as follows: Figure 1 As shown:

[0060] 5) After 200 epochs, the generator and the discriminator reach Nash equilibrium, completing the model training. The trained model is then used to reconstruct the missing information between the blocks of the tomb murals.

[0061] Both the generator and the discriminator are trained through the loss function, which is:

[0062]

[0063] L rec =||G n (Noize+I mn )-I rn || (6)

[0064] L G = xL rec +L 2+texture (7)

[0065]

[0066] L 2+texture is the sum of the pixel reconstruction loss and texture loss of the mural, which is used to update the generator weights to train the generator, L D is the loss function of the discriminator, and then according to L D To update the discriminator weights, the discriminator and the generator are finally trained against each other. χ is an adjustable parameter, set to 0.96. By adjusting L G The loss function of the generator is used to adjust L 2+texture , where the weight parameters are fixed after each layer of training is completed, and then the next layer is trained.

[0067] After training, the generated images are evaluated using the SSIM method. The value is between [0,1]. The larger the value, the higher the similarity with the original image. SSIM is used to measure the fidelity and similarity between the two murals. SSIM is not only used to perceive errors, but also to observe the degree of distortion in the perception-related parts of the visual system. It can evaluate the similarity between two murals of different sizes. The formula is:

[0068]

[0069] in, Original mural image I r The mean of For the extended mural I m The mean of Original mural I r variance, To extend the mural I m variance, For I r ,I m The covariance of b1 and b2 is used to prevent the overflow caused by the SSIM denominator being zero. The SSIM score is between [0,1]. When the contents of two images are exactly the same, the SSIM score is equal to 1.

[0070] Take a trained mural and compare it with the original mural to get Figure 4 The SSIM value mentioned above shows that the model is very rich in generating the content and texture of the mural. The trained weights are saved and a mural image is input to generate the result as follows: Figure 5 shown.

Claims

1. A method for reconstructing missing information between tomb mural blocks using an improved loss function SinGAN, characterized in that: The following steps are involved: 1) Collect images of multiple blocks of a mural excavated in the tomb passage, obtain mural block images, add masks to the mural block images and perform min-max normalization, then construct an image set based on the normalized mural block images, and use the image set as a training set for SinGAN to generate extensional information between murals; 2) Build a generative network based on SinGAN that includes generators and discriminators of 5 scales, and increase the reconstruction loss L rec , pixel reconstruction loss and texture loss L texture The loss function of 3) Take any mural block image I in the mural collection r , and then downsample them to get {I r0 ,I r1 ,I r2 ,I r3 ,I r4 }, where I r4 is the information obtained by downsampling 4 times, for I r Add mask I m ,I m4 This is the information obtained by adding a mask to the mural and then downsampling it 4 times; 4) Input I to the bottom layer of the generator m4 , the generator G4 generates the extensional mural G4 (I m4 ), change G4(I m4 ) and I r4 The input is sent to the discriminator D4 for comparison and discrimination, and the weight parameters of the layer are updated according to the loss function; 5) Repeat step 4) to obtain the weight parameters of each layer in the generator; 6) Use the training set and the discriminator to train the generator so that the generator and the discriminator reach Nash equilibrium, and then use the trained generator to reconstruct the missing information between the blocks of the tomb murals; The pixel reconstruction loss is used to capture the overall pixel information around the mural generation area, i.e. Where M is a binary mask, I rn is the downsampled mural of the real data, G n To generate the mural images generated by the network, Multiply the pixels of the mural; The texture loss is designed using the Gram matrix, namely Among them, G is the Gram matrix, l n is the real mural image I rn Through the output of the nth layer of the multi-scale generator, I n is the multi-scale generator G n The generated mural, g represents the generation network; The designed reconstruction loss is: L rec =||G n (Noise+I mn )-I rn || 2 (4) Among them, G n is the nth layer of the multi-scale generator, (Noize+I mn ) is the noise of each layer in the multi-scale generator superimposed on the output mural of the previous layer, I rn For real murals; The reconstruction loss L rec , pixel reconstruction loss and texture loss L texture The loss functions are: L rec =||G n (Noise+I mn )-I rn || (6) L G =χL rec +L 2+texture (7) Among them, L 2+texture is the sum of the pixel reconstruction loss and texture loss of the mural, L D is the loss function of the discriminator, L D Used to update the discriminator weights, χ is an adjustable parameter, and L is adjusted by adjusting the generator's loss function 2+texture .

2. The method for reconstructing missing information between tomb mural blocks using the improved loss function SinGAN according to claim 1, characterized in that: Design the multi-scale generator and discriminator structure, let {I r0 ,I r1 ,I r2 ...I rn } are a group of mural block images obtained by downsampling the real mural, I mn The mural block image is downsampled n times after masking, and the input I is at the bottom layer of the multi-scale generator. mn , after the generator G n Generate mural G n (I mn ), and then input to the discriminator D n Zhong and I rn Compare and judge, and update the weight parameters of this layer, and then G n (I mn ) is upsampled and superimposed with noise of equal size as the generator G n-1 The input image is looped until the image is of its original size.

3. The method for reconstructing missing information between tomb mural blocks using the improved loss function SinGAN according to claim 1, characterized in that: SSIM is used to measure the fidelity and similarity between two murals, where in, Original mural image I r The mean of For the extended mural I m The mean of Original mural I r The variance of To extend the mural I m variance, For I r ,I m The covariance of b1 and b2 is used to prevent overflow caused by the SSIM denominator being zero. SSIM is between [0,1]. The larger the SSIM, the more similar the contents of the two images are.

Images