Ancient mural line drawing generation method based on diffusion model

Abstract

An ancient fresco line drawing generation method based on a diffusion model comprises the following steps: constructing an ancient fresco data set, segmenting the ancient fresco data set into a training set and a test set according to a certain proportion, carrying out image enhancement on the training data set, setting hyper-parameters of variational self-coding, and training a variational self-coding model by using the training data set. Obtaining a trained encoder and decoder; constructing and training an ancient fresco line drawing generation model; and generating the line drawing of the ancient fresco. According to the method, the hidden space filtering module is introduced into the diffusion model, so that the edge features of the mural in the hidden space can be extracted, noise components can be filtered out, and the training process of the ancient mural line drawing generation model is constrained by using the joint loss function; the ancient fresco line drawing generation model can generate accurate and clear line drawing according to the input ancient fresco, and the generated line drawing does not need to be subjected to post-processing any more. According to the method, the wall painting line drawing with a clear structure and fine lines can be generated.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a method for generating line drawings of ancient murals based on a diffusion model. Background Art

[0002] Mural line drawings play an important role in assisting the restoration of murals. However, due to the uneven thickness, complex intersections and diverse colors of the lines and contours of ancient murals, the existing image edge extraction methods will lose local feature information in the process of extracting mural line drawings, making it difficult to accurately and completely present the line drawings and composition of murals. Therefore, how to use modern technical means to achieve high-quality generation and reproduction of ancient mural line drawings has become an urgent problem to be solved in the field of cultural relics protection.

[0003] Most current methods focus on edge extraction and line drawing generation of natural images. CaixiaZhou et al. proposed an uncertainty edge detector (UAED), which can be combined with various encoder-decoder backbone networks to achieve edge extraction of natural images. Mengyang Pu et al. proposed a Transformer-based edge detector EDTER, which aims to extract clear and distinct object boundaries by simultaneously utilizing complete image context information and detailed local clues. However, the image edges or lines in line drawings obtained by these learning-based methods are too thick to be directly applied to the generation of mural line drawings. Recently, Baokai Liu et al. proposed a new edge detector based on the combination of self-attention mechanism and convolution. This method designs a residual self-attention and convolution hybrid module to merge local and global features into the feature map, and uses a densely connected backbone extraction network to effectively propagate information rich in edge features from the shallow layer to the deep layer, and finally achieves the generation of Dunhuang mural line drawings.

[0004] Although deep learning-based edge extraction or line drawing generation methods have made significant progress, when used to generate line drawings corresponding to ancient murals with rich semantics and complex texture lines, there are still problems such as inaccurate line drawings, thick edge lines, loss of local information, and inability to fully utilize the contextual information of the murals, resulting in the generated mural line drawings having unclear or irregular lines. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a method for generating line drawings of ancient murals based on a diffusion model, which has a smooth training process and generates accurate and clear lines.

[0006] The technical solution adopted to solve the above technical problems is: a method for generating line drawings of ancient murals based on a diffusion model, which consists of the following steps:

[0007] Step 1. Build an ancient mural dataset;

[0008] The ancient mural dataset includes an ancient mural image dataset and a corresponding line drawing dataset;

[0009] Step 2. Select 90% of the images in the ancient mural dataset for image enhancement as the training dataset, and the remaining 10% of the images as the test dataset;

[0010] Step 3. Configure the hyperparameters of the variational autoencoder model, use the training data set to train the variational autoencoder model and obtain the trained encoder and decoder;

[0011] Step 4. Build and train a generative model for ancient mural line drawings;

[0012] Step 4.1: The ancient mural line drawing generation model is composed of a diffusion model and an encoder and a decoder trained in step 3, wherein the output end of the encoder is connected to the input end of the diffusion model, and the output end of the diffusion model is connected to the input end of the decoder; the encoder is used to encode the input mural image into a latent space, the diffusion model is used to separate the mural edge features from the noise components in the latent space, and the decoder is used to decode the features in the latent space into mural line drawings;

[0013] Step 4.2 uses the training data set to train the generation model of the ancient mural line drawing. During the training process, a joint loss function is used to constrain and optimize the generation model of the ancient mural line drawing until the generation model of the ancient mural line drawing reaches a maximum number of iterations, that is, the training is completed;

[0014] Step 5. Generate a line drawing of an ancient mural

[0015] The ancient mural images in the test data set are input into the trained ancient mural line drawing generation model, and the line drawing of the corresponding mural is output.

[0016] As a preferred technical solution, the image enhancement method in step 2 includes an operation of randomly cropping the image and adjusting the image size to 512×512 pixels and a random horizontal flipping operation.

[0017] As a preferred technical solution, the variational autoencoder model in step 3 includes an encoder and a decoder;

[0018] The encoding method of the encoder is:

[0019] z=Gau(Qua(En(x)))

[0020] Where x and z represent the input and output of the encoder operation, En() represents the encoding operation, Qua() represents the quantization operation, and Gau() represents the diagonalized Gaussian distribution operation;

[0021] The decoding method of the decoder is:

[0022] o=De(Qua(z))

[0023] In the formula, o represents the output of the decoder, and De represents the decoding operation;

[0024] As a preferred technical solution, the diffusion model described in step 4.1 uses a U-Net network with conditional input coding as a noise estimator.

[0025] As a preferred technical solution, the U-Net network is composed of a contraction module, a first latent space filtering module, a second latent space filtering module, a first expansion module, a second expansion module, and an element-level addition operation unit; the output of the contraction module is divided into two branches, one output to the first latent space filtering module, and the other output to the second latent space filtering module, the first latent space filtering module outputs to the first expansion module, the second latent space filtering module outputs to the second expansion module, the first expansion module and the second expansion module both output to the element-level addition operation unit, and the output of the element-level addition operation unit is used as the output of the U-Net network.

[0026] As a preferred technical solution, the conditional input encoding method of the U-Net network is:

[0027] e=SwinT t (I c )

[0028] In the formula, I c represents the input ancient mural image, and e represents the image after SwinT t The code obtained after the operation, SwinT t Represents a minimal version of the Swin Transformer operation.

[0029] As a preferred technical solution, the joint loss function in step 4.2 is composed of the reconstruction loss L I , weighted binary cross entropy loss L w , distillation loss L dill According to the following formula,

[0030] L join =ρ1L I +ρ2L w +ρ3L dill

[0031] Where, Ljoin represents the joint loss function, ρ1, ρ1 and ρ3 are weight parameters;

[0032] The reconstruction loss L I Is to calculate the output of the first expansion module I u Target Line Drawing I gt The l2 norm is then combined with the output of the second expansion module n u The sum of the l2 norm of the noise distribution n~N(0,I) is calculated as:

[0033]

[0034] N(0, I) is the standard normal distribution;

[0035] The weighted binary cross entropy loss L w The cross entropy between the true mural line drawing and the mural line drawing predicted by the model is calculated:

[0036]

[0037] In the formula, w represents the weight parameter, p represents the probability of generating a realistic mural line drawing, represents the probability of the mural line drawing predicted by the model;

[0038] The distillation loss L dill Used to calculate the output I of the first expansion module u Target Line Drawing I gt KL divergence of:

[0039]

[0040] In the formula, is the LogSoftmax operation, and KL() represents the Kullback-Leibler divergence operation.

[0041] As a preferred technical solution, the first latent space filtering module is used to adaptively filter out the edge features and noise components of the mural in the frequency domain, and the calculation method is:

[0042] x o =α1*f IFFT (f comp (Amplitude(x i ), phase(x i )))+α2*f c (x i )

[0043] In the formula, x i , x o represents the input and output of the latent space filter module, fcomp () represents the operation of calculating complex tensors, f IFFT () represents the inverse fast Fourier transform operation, f c represents a convolution operation with a kernel size of 1×1, α1 and α2 represent learnable weight parameters, Amplitude() represents amplitude component operation, and phase() represents phase component operation;

[0044] The second latent spatial filtering module and the first latent spatial filtering module are calculated in the same manner.

[0045] As a preferred technical solution, the amplitude component operation Amplitude() is:

[0046] Amplitude()=f T (f A (f FFT (x i )))*f cos (f T (f P (f FFT (x i ))))

[0047] In the formula, f T () represents the radioactive transformation operation, f A () indicates the absolute value operation, f FFT () represents the fast Fourier transform operation, f P () represents the angle operation of calculating the tensor, f cos () represents the cosine function operation;

[0048] The phase component operation phase() is:

[0049] phase()=f T (f P (f FFT (x i )))*f sin (f T (f A (f FFT (x i ))))

[0050] f sin () represents the sine function operation.

[0051] As a preferred technical solution, the radiation transformation operation f T for:

[0052] x2=LReLu(BCN((ACMix(x1))))

[0053] Where x1, x2 represent the radiation transformation operation f T The input and output of ACMix() represents the self-attention and convolution mixed attention operation, BCN() represents the batch channel normalization operation, and LReLU() represents the Leaky ReLu activation function operation.

[0054] The beneficial effects of the present invention are as follows:

[0055] The present invention makes the training process more stable and reduces fluctuations during the training process by using a combination of a variational autoencoder model and a diffusion model. By transforming the mural feature map from the spatial domain to the frequency domain in the latent space filtering module, effective separation of edge features and noise features is achieved, thereby making the generated line drawing lines more accurate. The model is trained using a joint loss function, including reconstruction loss, weighted binary cross entropy loss and distillation loss, so that the generated line drawing lines are clearer. Since the generated line drawing has a high quality, it is no longer necessary to perform additional post-image processing on the generated line drawing.

[0056] The line drawing generated by the present invention has clearer structure and finer lines, can capture subtle features of murals, has the characteristic of accurately generating mural line drawings, saves time and cost of manual drawing, is helpful for the digital protection and research of cultural relics, and provides new technical support for the restoration and protection of cultural relics.

[0057] The present invention is not only applicable to the generation of line drawings of ancient murals, but can also be extended to other types of line drawing generation tasks, and has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of the method for generating line drawings of ancient murals based on the diffusion model of the present invention.

[0059] Figure 2 It is a structural schematic diagram of the ancient mural line drawing generation model of the present invention.

[0060] Figure 3 It is a structural schematic diagram of the first latent spatial filtering module of the present invention.

[0061] Figure 4 This is a comparison chart of the effects of line drawing of ancient murals. DETAILED DESCRIPTION

[0062] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples, but the present invention is not limited to the following embodiments.

[0063] exist Figure 1 In the present embodiment, the method for generating line drawings of ancient murals based on a diffusion model comprises the following steps:

[0064] Step 1. Construct ancient wall dataset;

[0065] The ancient mural dataset includes ancient mural image data and line drawing image data, and the public ancient mural dataset is obtained from the Internet, including 2687 mural images and one-to-one correspondences;

[0066] Step 2. Select 90% of the images in the ancient mural dataset for image enhancement as the training dataset, and the remaining 10% of the images as the test dataset;

[0067] The image enhancement method includes randomly cropping the image and adjusting it to 512×512 pixels and randomly flipping it horizontally.

[0068] Step 3. Configure the hyperparameters of the variational autoencoder model, use the training data set to train the variational autoencoder model and obtain the trained encoder and decoder;

[0069] The input of the variational autoencoder model during training is an ancient mural image, and the output is a line drawing image; the number of gradient accumulations in the variational autoencoder model hyperparameters is set to 4, the learning rate is set to 5e-6, the total number of training iterations is set to 150000, the batch size is set to 8, the number of channels of the intermediate features is set to 128, the resolution of the input image is set to 512×512, the number of channels of the input image is set to 3, and the number of channels of the output image is set to 1;

[0070] The variational autoencoder model consists of an encoder and a decoder;

[0071] The encoding method of the encoder is:

[0072] z=Gau(Qua(En(x)))

[0073] Where x and z represent the input and output of the encoder operation, En() represents the encoding operation, which includes two residual blocks and two downsampling operations, Qua() represents the quantization operation, which is implemented by a convolution operation with a kernel size of 3×3, and Gau() represents the diagonalized Gaussian distribution operation;

[0074] The decoding method of the decoder is:

[0075] o=De(Qua(z))

[0076] Where o represents the output of the decoder, De() represents the decoding operation, which includes two residual blocks and two upsampling operations.

[0077] Step 4. Build and train a generative model for ancient mural line drawings;

[0078] Step 4.1 The ancient mural line drawing generation model is composed of a diffusion model and the encoder and decoder trained in step 3. The output of the encoder is connected to the input of the diffusion model, and the output of the diffusion model is connected to the input of the decoder. The encoder is used to encode the input mural image into the latent space, the diffusion model is used to separate the mural edge features from the noise components in the latent space, and the decoder is used to decode the features in the latent space into mural line drawings, such as Figure 2 ;

[0079] The diffusion model uses a U-Net network with conditional input coding as a noise estimator. The U-Net network consists of a contraction module, a first latent space filter module, a second latent space filter module, a first expansion module, a second expansion module, and an element-level addition operation unit; the output of the contraction module is divided into two branches, one output to the first latent space filter module, and the other output to the second latent space filter module, the first latent space filter module outputs to the first expansion module, the second latent space filter module outputs to the second expansion module, the first expansion module and the second expansion module both output to the element-level addition operation unit, and the output of the element-level addition operation unit is used as the output of the U-Net network.

[0080] The conditional input encoding method of the U-Net network is:

[0081] e=SwinT t (I c )

[0082] In the formula, I c represents the input ancient mural image, and e represents the image after SwinT t The code obtained after the operation, SwinT t Represents a minimal version of the Swin Transformer operation.

[0083] Among them, Figure 3 , the expression of the second latent space filter module is the same as that of the first latent space filter module, both of which are used to adaptively filter out the edge features and noise components of the mural in the frequency domain. The expression of the first latent space filter module is:

[0084] x o =α1*f IFFT (f comp (Amplitude(x i ), phase(x i )))+α2*f c (x i )

[0085] In the formula, x i , x o represents the input and output of the latent space filter module, f comp() represents the operation of calculating complex tensors, f IFFT () represents the inverse fast Fourier transform operation, f c () represents a convolution operation with a kernel size of 1×1, α1 and α2 are learnable weight parameters defined using the torch.nn.parameter() method, Amplitude() represents the amplitude component operation, and phase() represents the phase component operation;

[0086] Among them, the amplitude component operation Amplitude() is:

[0087] Amplitude(x i )=f T (f A (f FFT (x i )))*f cos (f T (f P (f FFT (x i ))))

[0088] f T () represents the radioactive transformation operation, f A () indicates the absolute value operation, f FFT () represents the fast Fourier transform operation, f P () represents the angle operation of calculating the tensor, f cos () represents the cosine function operation.

[0089] Among them, the phase component operation phase() is:

[0090] phase(x i )=f T (f P (f FFT (x i )))*f sin (f T (f A (f FFT (x i ))))

[0091] f sin () represents the sine function operation;

[0092] Among them, the radiation transformation operation f T for:

[0093] x2=LReLu(BCN((ACMix(x1))))

[0094] Among them, x1, x2 represent the radiation transformation operation f TThe input and output of ACMix() represents the self-attention and convolution mixed attention operation, BCN() represents the batch channel normalization operation, and LReLu() represents the Leaky ReLu activation function operation.

[0095] Step 4.2 Use the training dataset to train the generative model of ancient mural line drawings

[0096] When training the ancient mural line drawing generation model, load the parameters of the encoder and decoder of the trained variational autoencoder model, set the number of channels of the input image to 3, the resolution to 512×512, the number of output channels to 1, the number of intermediate feature channels to 128, the time step to 1000, the sampling time step to 1, the Fourier transform scale factor to 16, the number of gradient accumulation to 4, the learning rate to 5e-6, the total number of training iterations to 100000, and the batch size to 2;

[0097] During the training process, a joint loss function is used to constrain and optimize the ancient mural line drawing generation model until the ancient mural line drawing generation model reaches a maximum number of iterations, that is, the training is completed;

[0098] Among them, the joint loss function is used to constrain the training process of the mural line drawing generation model. The joint loss function is composed of the reconstruction loss L I , weighted binary cross entropy loss L w , distillation loss L dill According to the following formula,

[0099] L join =ρ1L I +ρ2L w +ρ3L dill

[0100] Where, L join represents the joint loss function, ρ1, ρ2 and ρ3 are weight parameters, ρ1 is set to 0.5, ρ1 is set to 0.05 and ρ3 is set to 0.01.

[0101] Reconstruction loss L I Is to calculate the output of the first expansion module I u Target Line Drawing I gt The l2 norm is then combined with the output of the second expansion module n u The sum of the l2 norm of the noise distribution n~N(0,I) is calculated as:

[0102]

[0103] , N(0,I) is the standard normal distribution;

[0104] Weighted binary cross entropy loss Lw The cross entropy between the true mural line drawing and the mural line drawing predicted by the model is calculated:

[0105]

[0106] In the formula, w represents the weight parameter, w is set to 0.02, p represents the probability of generating a realistic mural line drawing, represents the probability of the mural line drawing predicted by the model;

[0107] Distillation loss L dill Used to calculate the output I of the first expansion module u Target Line Drawing I gt KL divergence of:

[0108]

[0109] In the formula, is the LogSoftmax operation, and KL() represents the Kullback-Leibler divergence operation.

[0110] Step 5 Generate a line drawing of an ancient mural

[0111] The ancient mural images in the test set are input into the trained ancient mural line drawing generation model, and the line drawing of the corresponding mural is output.

[0112] experiment

[0113] The line drawing generation method of ancient murals based on the diffusion model of the present invention is applied to the murals simultaneously with the existing Sobel and PiDiNet methods to obtain the line drawing results of the ancient murals, as shown in FIG. Figure 4 , the line drawing image of the present invention is more accurate and clear.

Claims

1. A method for generating line drawings of ancient murals based on a diffusion model, characterized in that It consists of the following steps: Step 1. Build an ancient mural dataset; The ancient mural dataset includes an ancient mural image dataset and a corresponding line drawing dataset; Step 2. Select 90% of the images in the ancient mural dataset for image enhancement as the training dataset, and the remaining 10% of the images as the test dataset; Step 3. Configure the hyperparameters of the variational autoencoder model, use the training data set to train the variational autoencoder model and obtain the trained encoder and decoder; Step 4. Build and train a generative model for ancient mural line drawings; Step 4.1: The ancient mural line drawing generation model is composed of a diffusion model and an encoder and a decoder trained in step 3, wherein the output end of the encoder is connected to the input end of the diffusion model, and the output end of the diffusion model is connected to the input end of the decoder; the encoder is used to encode the input mural image into a latent space, the diffusion model is used to separate the mural edge features from the noise components in the latent space, and the decoder is used to decode the features in the latent space into mural line drawings; Step 4.2 uses the training data set to train the generation model of the ancient mural line drawing. During the training process, a joint loss function is used to constrain and optimize the generation model of the ancient mural line drawing until the generation model of the ancient mural line drawing reaches a maximum number of iterations, that is, the training is completed; Step 5. Generate a line drawing of an ancient mural The ancient mural images in the test data set are input into the trained ancient mural line drawing generation model, and the line drawing of the corresponding mural is output.

2. The method for generating line drawings of ancient murals based on a diffusion model according to claim 1, characterized in that: The image enhancement method in step 2 includes randomly cropping the image and adjusting it to a size of 512×512 pixels and performing a random horizontal flipping operation.

3. The method for generating line drawings of ancient murals based on a diffusion model according to claim 1, characterized in that: The variational autoencoder model in step 3 includes an encoder and a decoder; The encoding method of the encoder is: z=Gau(Qua(En(x))) Where x and z represent the input and output of the encoder operation, En() represents the encoding operation, Qua() represents the quantization operation, and Gau() represents the diagonalized Gaussian distribution operation; The decoding method of the decoder is: o=De(Qua(z)) Where o represents the output of the decoder and De represents the decoding operation.

4. The method for generating line drawings of ancient murals based on a diffusion model according to claim 1, characterized in that: The diffusion model described in step 4.1 uses a U-Net network with conditional input encoding as a noise estimator.

5. The method for generating line drawings of ancient murals based on a diffusion model according to claim 4 is characterized in that: The U-Net network is composed of a contraction module, a first latent space filtering module, a second latent space filtering module, a first expansion module, a second expansion module, and an element-level addition operation unit; the output of the contraction module is divided into two branches, one branch is output to the first latent space filtering module, and the other branch is output to the second latent space filtering module, the first latent space filtering module outputs to the first expansion module, and the second latent space filtering module outputs to the second expansion module, and both the first expansion module and the second expansion module output to the element-level addition operation unit, and the output of the element-level addition operation unit is used as the output of the U-Net network.

6. The method for generating line drawings of ancient murals based on a diffusion model according to claim 4 is characterized in that: The conditional input encoding method of the U-Net network is: e=SwinT t (I c ) In the formula, I c represents the input ancient mural image, and e represents the image after SwinT t The code obtained after the operation, SwinT t Represents a minimal version of the Swin Transformer operation.

7. The method for generating line drawings of ancient murals based on a diffusion model according to claim 5, characterized in that: The joint loss function described in step 4.2 is composed of the reconstruction loss L I , weighted binary cross entropy loss L w , distillation loss L dill According to the following formula, L join =ρ1L I +p2L w +p3L dill Where, L join represents the joint loss function, ρ1, ρ1 and ρ3 are weight parameters; The reconstruction loss L I Is to calculate the output of the first expansion module I u Target Line Drawing I gt The l2 norm is then combined with the output of the second expansion module n u The sum of the l2 norm of the noise distribution n~N(0,I) is calculated as: N(0,I) is the standard normal distribution; The weighted binary cross entropy loss L w The cross entropy between the true mural line drawing and the mural line drawing predicted by the model is calculated: In the formula, w represents the weight parameter, p represents the probability of generating a realistic mural line drawing, represents the probability of the mural line drawing predicted by the model; The distillation loss L dill Used to calculate the output I of the first expansion module u Target Line Drawing I gt KL divergence of: In the formula, is the LogSoftmax operation, and KL( ) represents the Kullback-Leibler divergence operation.

8. The method for generating line drawings of ancient murals based on a diffusion model according to claim 5, characterized in that: The first latent spatial filtering module is used to adaptively filter out the edge features and noise components of the mural in the frequency domain. The calculation method is: x o =α1*f IFFT (f comp (Amplitude(x i ),phase(x i )))+α2 / f c (x i ) In the formula, x i , x o represents the input and output of the latent space filter module, f comp ( ) represents the operation of calculating complex tensors, f IFFT ( ) represents the inverse fast Fourier transform operation, f c represents a convolution operation with a kernel size of 1×1, α1 and α2 represent learnable weight parameters, Amplitude( ) represents amplitude component operation, and phase( ) represents phase component operation; The second latent spatial filtering module and the first latent spatial filtering module are calculated in the same manner.

9. The method for generating line drawings of ancient murals based on a diffusion model according to claim 8, characterized in that: The amplitude component operation Amplitude() is: Amplitude( )=f T (f A (f FFT (x i )))*f cos (f T (f P (f FFT (x i )))) In the formula, f T ( ) represents the radioactive transformation operation, f A ( ) represents the absolute value operation, f FFT ( ) represents the fast Fourier transform operation, f P ( ) represents the angle operation of calculating the tensor, f cos ( ) represents the cosine function operation; The phase component operation phase ( ) is: phase()=f T (f P (f FFT (x i )))*f sin (f T (f A (f FFT (x i )))) f sin ( ) represents the sine function operation.

10. The method for generating line drawings of ancient murals based on a diffusion model according to claim 9, characterized in that: The radiometric transformation operation f T for: x2=LReLu(BCN((ACMix(x1)))) Where x1, x2 represent the radiation transformation operation f T The input and output of , ACMix( ) represents the self-attention and convolution mixed attention operation, BCN( ) represents the batch channel normalization operation, and LReLu( ) represents the Leaky ReLu activation function operation.

Images