An ancient fresco line drawing extraction method

Abstract

The ancient mural sketch extraction method disclosed by the application carries out clipping and data enhancement on the collected ancient mural image to obtain an ancient mural image dataset; carries out feature extraction and processing on a single ancient mural image to obtain a multi-scale spatial information feature map and a cross-channel attention gradient information attention map; the two are weighted and fused to obtain an ancient mural feature map with gradient attention guidance, and then the ancient mural feature map is input into a decoder to obtain a sketch image of the ancient mural; the sketch image of the ancient mural is constrained with a pseudo ground truth depth map of the ancient mural image, that is, the extraction of the ancient mural sketch is completed. The gradient guidance and geometric constraint make the extracted mural image sketch clear and continuous, and reduce the influence of the mural image background on the sketch extraction.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to an ancient mural line drawing extraction method. BACKGROUND

[0002] Line drawing is the most important and basic visual language in Chinese painting, which can present the contour and posture of an object, and its unique artistic value has important guiding significance for the copying and archaeological research of ancient murals. Due to natural weathering and human damage, etc., a large number of unearthed murals have different degrees of diseases such as pigment peeling, scaling, pattern loss, etc., and cannot present their original appearance. Therefore, computer-aided ancient mural line drawing extraction is crucial in the digital protection of cultural heritage.

[0003] In recent years, the generative adversarial network proposed in the image translation field has shown strong performance, which aims to learn the mapping relationship between two image domains by using unpaired data sets, bringing new ideas for mural image line drawing extraction.

[0004] However, the current mural image extraction method based on generative adversarial network is not specially designed for translation from ancient mural domain to line drawing domain, and when this extraction method is used to face mural images with complex background, it is easy to appear the problems of discontinuous edges of extracted line drawing, difficulty in extracting weak edge regions of mural image, loss of detail information and background noise, etc. SUMMARY

[0005] The purpose of the application is to provide an ancient mural line drawing extraction method, which makes the extracted mural image line drawing clear and continuous through gradient guidance and geometric constraint, and reduces the influence of mural image background on line drawing extraction.

[0006] The technical scheme adopted by the application is an ancient mural line drawing extraction method, comprising the following steps:

[0007] Step 1, cropping and data enhancement are performed on the collected ancient mural image I mural to obtain an ancient mural image data set;

[0008] Step 2, feature extraction is performed on a single ancient mural image I mural in the ancient mural data set to obtain a feature map F c with multi-scale spatial information and cross-channel attention; and the ancient mural image I mural undergoing feature extraction is processed synchronously to obtain a gradient information attention map I attn ;

[0009] Step 3, the feature map F c with multi-scale spatial information and cross-channel attention and the gradient information attention map I attnWeighted fusion is performed to obtain the ancient mural feature map F with gradient attention guidance. g-attn Then, it is input into the decoder to obtain the line drawing image I of the ancient mural. sketch ;

[0010] Step 4, extract the line drawing image of the ancient mural I sketch With ancient mural images I mural The pseudo-ground truth depth map is constrained to complete the extraction of ancient mural line drawings.

[0011] The invention is further characterized in that,

[0012] Furthermore, the data augmentation in step 1 is performed by randomly rotating the data by 90°, 180°, 270° and horizontally flipping it.

[0013] Furthermore, the feature extraction in step 2 specifically includes: processing the ancient mural image I through two convolutional networks. mural The number of channels was increased to 64, and ancient mural images were processed. mural Downsampling yields ancient mural images I mural The shallow, low-level features F0; the specific calculation is shown in formula (1):

[0014] F0 = Conv v (Conv v (Conv x (I mural ))) (1);

[0015] The shallow, low-level features F0 are input into the cross-channel attention residual module to obtain multi-scale spatial information and the cross-channel attention feature map F. c .

[0016] Furthermore, the cross-channel attention residual module is divided into a single convolutional network branch and a branch composed of a convolutional network and the cross-channel attention module PSA. The single convolutional network branch is used to retain the coarse-scale feature F2, and the specific calculation is shown in formula (2):

[0017] F2 = Conv v (F0) (2);

[0018] Among them, F0 represents ancient mural image I. mural Shallow, low-level features;

[0019] The branch consisting of the convolutional network and the cross-channel attention module PSA is used to extract features F1 at different spatial scales and channel dimensions. The specific calculation is shown in formula (3):

[0020] F1 = Conv v(PSA(F0)) (3);

[0021] wherein, PSA is a cross-channel attention module;

[0022] The features F1, the coarse scale features F2 and the shallow low-level features F0 of different spatial scales and channel dimensions are integrated together by pixel-by-pixel addition to obtain a feature map F of multi-scale spatial information and cross-channel attention c .

[0023] Further, the cross-channel attention module PSA specifically includes, in a multi-branch manner, extracting spatial information of the input feature map F0, and in a parallel manner, performing calculation and processing on multiple scales, and the specific calculation is shown in formula (4):

[0024] F 1，i =Conv(k i ,G i )(F0) (4);

[0025] wherein, k i represents a convolution kernel of different scales, G i represents different group sizes, and F 1，i is a feature map of different scales;

[0026] The multi-branch spatial information calculated is obtained by cascading to obtain a multi-scale extraction feature map F all From the multi-scale extraction feature map, channel attention weight information is extracted to obtain an attention weight vector of different scales, and the specific calculation is shown in formula (5):

[0027] Z i =SEW eight (F 1，i ) (5);

[0028] wherein, Z i is an attention weight vector, and i is a scale value of a channel;

[0029] The multiple groups of attention weight vectors Z i calculated are obtained by cascading to obtain a multi-scale channel attention vector Z,

[0030] Then, the weights of the cascaded multi-scale channel attention are recalibrated using Softmax, and the specific calculation is shown in formula (6):

[0031]

[0032] wherein, Attn i is the calibrated multi-scale channel attention, and the multi-scale channel attention Attn i is combined with the feature map F of the corresponding scale.1，i The feature map Y of the multiplied multi-scale channel attention weight i is calculated as shown in formula (7):

[0033] Y i = Attn i ⊙F 1，i (7);

[0034] Wherein, ⊙ represents element-wise multiplication; i is the scale value of the channel; and the feature maps of the calculated different scale channel attention weights are concatenated and output.

[0035] Further, the ancient mural image I mural is processed synchronously to obtain the gradient information attention map I attn , and the processing specifically comprises: processing the ancient mural image I mural by a Sobel formula to obtain the gradient map I grad of the image, then inputting the gradient map I grad into a convolutional network for iteration, inputting the gradient map I in into a PReLU layer after the iteration, outputting a feature map F in after processing by the PreLU layer, and inputting the feature map F attn into a double attention module DAM to obtain the gradient information attention map I

[0036] Further, the double attention module DAM comprises a channel attention module and a multi-scale pixel attention module.

[0037] The feature map F in is input into the channel attention module, the feature map F in is multiplied by the weight Attn channel of each channel in the channel attention module to obtain a feature map F channel of channel attention; and the weight Attn channel of each channel is specifically shown in formula (8):

[0038] Attn channel = σ(Conv 1×1 (δ(Conv 1×1 (g c )))) (8);

[0039] Wherein, σ is a sigmoid activation function, g c is a global average pooling layer, and δ is a PReLU activation function.

[0040] The generated feature map F channel of channel attention is input into the multi-scale pixel attention module, a double convolutional network is adopted to process the feature map Fchannel The feature map F of the channel attention after convolution is obtained through processing. P Then, feature maps of convolutional outputs with different dilation ratios are obtained through dilated convolutional layers with different dilation ratios; the specific calculation is shown in formula (9):

[0041] F Di =DConv 3*3 (F P ,dilation) (9);

[0042] Among them, DConv 3*3 (·) represents dilation convolution with different dilation rates, where dilation represents the dilation rate, F Di These represent feature maps output by convolutions with different dilation rates; concatenating these feature maps with different dilation rates yields the dilated convolution feature map F. D For the dilation rate convolution feature map F D The feature map F of multi-scale pixel attention is obtained by processing with a convolutional network, PreLU function and sigmoid function. pixel The specific calculation is shown in formula (10):

[0043] F pixel =sigmoid(Conv v (PReLU(Conv v (FD)))) (10);

[0044] The feature map F of multi-scale pixel attention pixel Feature map F of channel attention channel After pixel-by-pixel multiplication, it is combined with the feature map F in Gradient information is obtained by pixel-by-pixel summation of attention maps I attn .

[0045] Furthermore, the weighted fusion in step 3 specifically involves adaptively combining multi-scale spatial information and cross-channel attention feature maps F using parameter α. c Attention Graph I with Gradient Information attn The weighted fusion is performed, and the specific calculation is shown in formula (11):

[0046] F g-attn =αI attn +(1-α)F c (11);

[0047] Among them, F g-attn I is a diagram illustrating the characteristics of ancient murals. attn For gradient information attention map, F c Feature maps for multi-scale spatial information and cross-channel attention;

[0048] The decoder in step 3 is composed of a deconvolution layer and a convolution layer, both of which are based on an InstanceNorm2d and a ReLU layer; the specific calculation is shown in formula (12):

[0049] I sketch =Conv q (deConv w (deConv w (F g-attn ))) (12);

[0050] Wherein, Conv q is a convolution layer, and deConv w is a deconvolution layer.

[0051] Further, the step 4 of constraining the line drawing image I sketch of the ancient mural to the pseudo ground truth depth map of the ancient mural image I mural specifically includes:

[0052] First, input the ancient mural image I murai into the pre-trained depth prediction system, and output the depth map as the pseudo ground truth depth map F(I mural ) of the ancient mural;

[0053] Then, the Inception v3 network pre-trained on ImageNet is used to extract image features from the ancient mural line drawing image I sketch , and the global generator in pix2pixHD is used as a depth prediction network to predict the depth map G geom from the line drawing image, and the distance between the pseudo ground truth depth map F(I mural ) and the depth map G geom is constrained by a depth prediction loss function; the final line drawing image is obtained; the depth prediction loss function is shown in formula (13):

[0054] L geom =||G geom (G v3 (I sketch ))-F(I mural )|| (13);

[0055] Wherein, G v3 is the Inception v3 network pre-trained on ImageNet.

[0056] Compared with the prior art, the beneficial effects of the present application are that the present application generates multi-scale spatial information and gradient information attention map of cross-channel attention of ancient mural image, and generates pseudo ground truth depth map of ancient mural by gradient guidance and generation of line drawing image, so that the finally extracted mural image line drawing is clear and continuous, and the influence of mural image background on line drawing extraction is reduced; by setting the cross-channel attention residual module, the multi-scale spatial information and cross-channel attention are integrated into the residual module, the multi-scale spatial information of the mural image can be extracted on a finer granularity level, and the training process pays more attention to the pattern part and less attention to the background area during the generation of the gradient information attention map, so that the background noise and part of the disease can be inhibited. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a flowchart of an ancient mural line drawing extraction method of the present application;

[0058] Figure 2 is a structural schematic diagram of an ancient mural line drawing extraction method of the present application;

[0059] Figure 3 is a structural schematic diagram of a cross-channel attention residual module in the embodiment;

[0060] Figure 4 is a structural schematic diagram of a cross-channel attention module in the embodiment;

[0061] Figure 5 is a structural schematic diagram of a double attention module in the embodiment;

[0062] Figure 6 is an effect diagram of the line drawing image extracted in the embodiment. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0064] The ancient mural sketch extraction method of the application is an ancient mural sketch extraction network based on a cyclic generative adversarial model, and the ancient mural sketch extraction network comprises an attention map generation branch, a feature extraction branch, a depth map prediction branch, and a self-adaptive fusion and sketch generation branch. The attention map generation branch is used to guide the feature network to pay more attention to the pattern part of the mural and ignore the background area, so as to effectively suppress the noise generated by mural diseases. The feature extraction branch is used to perform preliminary feature extraction on the ancient mural image. The depth map prediction branch is used to make the generated sketch image close to the pseudo ground truth depth map of the original mural image, that is, the generated sketch can be placed in a meaningful position, thereby improving the quality of the finally generated sketch image.

[0065] In the attention map generation branch, a double attention module is included, and in the feature extraction branch, a cross-channel attention residual module is included.

[0066] As shown in Figure 1 and Figure 2 , the ancient mural sketch extraction method of the application comprises the following steps:

[0067] Step 1: cropping and data enhancement are performed on the collected ancient mural image I mural , and the data enhancement is specifically performed by means of random rotation by 90°, 180°, 270°, and horizontal flipping, so as to obtain an ancient mural image data set;

[0068] Step 2: feature extraction is performed on a single ancient mural image I mural in the ancient mural data set, two convolutional networks are used to increase the channel of the ancient mural image I mural by 64 channels, and downsampling is performed on the ancient mural image I mural , so as to obtain the shallow low-level feature F0 of the ancient mural image I mural , and the specific calculation is shown in formula (1):

[0069] F0=Conv v (Conv v (Conv x (I mural ))) (1);

[0070] The shallow low-level feature F0 is input into a cross-channel attention residual module, and the cross-channel attention residual module comprises a single convolutional network branch and a branch composed of a convolutional network and a cross-channel attention module PSA, and the single convolutional network branch is used to reserve the coarse scale feature F2, and the specific calculation is shown in formula (2):

[0071] F2=Conv v (F0) (2);

[0072] Wherein, F0 is an ancient mural image I mural Shallow low-level features of the image I

[0073] The convolutional network and the branch composed of the cross-channel attention module PSA are used to extract features F1 of different spatial scales and channel dimensions, and the specific calculation is shown in formula (3):

[0074] F1 = Conv v (PSA(F0)) (3);

[0075] Wherein, PSA is a cross-channel attention module;

[0076] The cross-channel attention module PSA specifically includes extracting spatial information of the input feature map F0 in a multi-branch manner, and calculating and processing in a parallel manner at multiple scales, and the specific calculation is shown in formula (4):

[0077] F 1，i =Conv(k i ,G i )(F0) (4);

[0078] Wherein, k i represents a convolution kernel of different scales, G i represents different group sizes, and F 1，i is a feature map of different scales;

[0079] The multi-branch spatial information calculated is obtained by a cascading manner to obtain a multi-scale extraction feature map F all From the multi-scale extraction feature map, channel attention weight information is extracted to obtain an attention weight vector of different scales, and the specific calculation is shown in formula (5):

[0080] Z i =SEWeight(F 1，i ) (5);

[0081] Wherein, Z i is an attention weight vector, and i is a channel scale value;

[0082] The multiple groups of attention weight vectors Z i are obtained by a cascading manner to obtain a multi-scale channel attention vector Z,

[0083] Then, the weights of the cascaded multi-scale channel attention are recalibrated using Softmax, and the specific calculation is shown in formula (6):

[0084]

[0085] Wherein, Attn iFor the calibrated multi-scale channel attention, the multi-scale channel attention Attn i is multiplied with the feature map F 1，i of the corresponding scale to obtain a feature map Y i of the multi-scale channel attention weight, which is calculated as shown in equation (7):

[0086] Y i = Attn i ⊙ F 1，i (7);

[0087] where ⊙ represents element-wise multiplication; i is the scale value of the channel; and the feature maps of the channel attention weights of different scales are concatenated and outputted.

[0088] The features F1, the coarse-scale features F2 and the shallow low-level features F0 of different spatial scales and channel dimensions are integrated together by pixel-by-pixel addition to obtain a feature map F c of multi-scale spatial information and cross-channel attention.

[0089] Then, the ancient mural image I mural undergoing feature extraction is processed synchronously to obtain a gradient information attention map I attn . Specifically, the ancient mural image I mural is processed by a Sobel formula to obtain a gradient map I grad of the image, and then the gradient map I grad is inputted into a convolutional network for iteration and then into a PReLU layer, and the feature map F in is outputted after being processed by the PreLU layer. in The feature map F c is sent into a dual attention module DAM, which includes a channel attention module and a multi-scale pixel attention module.

[0090] The channel attention module has five layers, the first layer is a global average pooling layer, which converts global spatial information into channel descriptors, and the expression of the global average pooling operation is as follows:

[0091]

[0092] where X c (i,j) represents the value of the cth channel X P at the (i,j) position, and H in is a global average pooling function.

[0093] The second layer is a 1×1 convolutional layer, the third layer is a PReLU activation function, the fourth layer is a 1×1 convolutional layer, and the fifth layer is a sigmoid activation function.

[0094] The feature map F inis sent into the channel attention module, and the feature map F in is multiplied by the weight Attn channel of each channel in the channel attention module to obtain the feature map F channel of channel attention. channel Specifically, as shown in equation (8):

[0095] Attn channel = σ(Conv 1×1 (δ(Conv 1×1 (g c )))) (8);

[0096] wherein σ is a sigmoid activation function, g c is a global average pooling layer, and δ is a PReLU activation function.

[0097] The generated feature map F channel of channel attention is input into the multi-scale pixel attention module, and a double convolution network is used to process the feature map F channel of channel attention to obtain the feature map F P of channel attention after convolution, and different dilation convolution layers with different dilation rates are used to obtain feature maps with different dilation rate convolution outputs; the specific calculation is shown in equation (9):

[0098] F Di = DConv 3*3 (F P , dilation) (9);

[0099] wherein DConv 3*3 (·) represents a dilation convolution with different dilation rates, dilation represents a dilation rate, and F Di represents feature maps with different dilation rate convolution outputs; the obtained multiple feature maps with different dilation rate convolution outputs are concatenated to obtain the dilation rate convolution feature map F D , and the dilation rate convolution feature map F D is processed by a convolution network, a PreLU function and a sigmoid function to obtain the feature map F pixel of multi-scale pixel attention; the specific calculation is shown in equation (10):

[0100] F pixel = sigmoid(Conv v (PReLU(Conv v (F D )))) (10);

[0101] The feature map F pixeland channel attention feature map F channel After pixel-wise multiplication, feature map F in After pixel-wise addition, gradient information attention map I attn .

[0102] Step 3, adaptively combine multi-scale spatial information and cross-channel attention feature map F c and gradient information attention map I attn After weighted fusion, get the ancient wall painting feature map F g-attn with gradient attention guidance, the specific calculation is shown in formula (11):

[0103] F g-attn = aI attn + (1-a)F c (11);

[0104] Where F g-attn is the ancient wall painting feature map, I attn is the gradient information attention map, and F c is the multi-scale spatial information and cross-channel attention feature map.

[0105] Then input the ancient wall painting feature map F g-attn into the decoder, the decoder is composed of a deconvolution layer and a convolution layer, both of which are based on an InstanceNorm2d and a ReLU layer; the specific calculation is shown in formula (12):

[0106] I sketch = Conv q (deConv w (deConv w (F g-attn ))) (12);

[0107] Where Conv q is a convolution layer, and deConv w is a convolution layer.

[0108] Then get the line drawing image I sketch of the ancient wall painting.

[0109] Step 4, constrain the line drawing image I sketch of the ancient wall painting with the pseudo ground truth depth map of the ancient wall painting image I mural , that is, complete the extraction of the ancient wall painting line drawing. Specifically, it includes:

[0110] First, the ancient wall painting image I muralInput into the pre-trained depth prediction system, and output the depth map as the pseudo ground truth depth map F(I mural ) of the ancient mural.

[0111] Then the Inception v3 network pre-trained on ImageNet is used to extract image features from the ancient mural sketch image I sketch , and the global generator in pix2pixHD is used as the depth prediction network to predict the depth map G geom of the sketch image from the sketch image, and the distance between the pseudo ground truth depth map F(I mural ) and the depth map G geom is constrained by the depth prediction loss function; and the final sketch image is obtained.

[0112] The depth prediction loss function is shown in formula (13):

[0113] L geom =||G geom (G v3 (I sketch ))-F(I mural )|| (13);

[0114] Where G v3 is the Inception v3 network pre-trained on ImageNet.

[0115] The ancient mural sketch extraction network is optimized by using the generated adversarial loss function and the cycle consistency loss function, wherein the generated adversarial loss function is used to constrain the generated image to belong to the respective image domain, and the cycle consistency loss function is used to limit the image content of the generated sketch domain to be consistent with the image content in the original ancient mural.

[0116] Since the present application is constructed based on the cycle generative adversarial network, there are two groups of generated adversarial loss functions, one group is to translate the image in the ancient mural domain to the image in the sketch domain, and the other group is to translate the image in the sketch domain back to the image in the ancient mural domain. Therefore, the expression of the generated adversarial loss function L GAN is:

[0117] L GAN =E a～A [D A (a) 2 ]+E b～B [(1-D A (G B (b))) 2 ]+E b～B [D B (b) 2 ]+Ea～A [(1-D B (G A (a))) 2 ]

[0118] wherein a~A represents that the image belongs to the A domain (i.e. the ancient mural domain), b~B represents that the image belongs to the B domain (i.e. the sketch domain), D A , G A respectively represent the discriminator and the generator of the A domain, D B , G B respectively represent the discriminator and the generator of the B domain.

[0119] The cycle consistency loss function is specifically that when the image in the sketch domain is translated into the ancient mural domain and then translated back into the sketch domain, the original image in the sketch domain and the translated image in the sketch domain should be consistent in content. The expression of the cycle consistency loss function Lc y cle is as follows:

[0120] L cycle =||G B (G A (a))-a||+||G A (G B (b))-b||

[0121] Embodiment

[0122] Step 1, the collected ancient mural images are uniformly cropped to 256x256 size, and data augmentation is performed by random rotation of 90°, 180°, 270° and horizontal flip to obtain an ancient mural image dataset.

[0123] The processed images are divided into a non-disease dataset and a disease dataset, and the datasets of the same category have highly similar styles.

[0124] Step 2, the feature extraction branch adopts one 7x7 convolution with a step size of 1, two 3x3 convolutions with a step size of 2, and three cross-channel attention residual modules, and each convolution is followed by an InstanceNorm2d layer and a PReLU layer.

[0125] An input is an ancient mural image I mural of 3 channels and 256x256 size. First, a 3-channel feature map is converted to 64 channels through a convolution with a kernel size of 7x7; then, a 3x3 convolution with a step size of 2 is performed to downsample the feature map, halving the size of the feature map and doubling the number of channels, to obtain a shallow low-level feature F0; the specific calculation is shown in formula (1):

[0126] F0=Conv 3×3 (Conv 3×3 (Conv7×7 (I mural ))) (1);

[0127] The shallow, low-level features F0 are input into the cross-channel attention residual module to extract feature maps of different spatial scales and channel dimensions. For example... Figure 3 As shown, the single convolutional network of the cross-channel attention residual module uses 1×1 convolutions. One branch consists of the cross-channel attention module and a 1×1 convolution to extract features of different spatial scales and channel dimensions. The second branch consists of a 1×1 convolution, which aims to preserve coarse-scale features. Finally, the outputs of the two branches and the input of the entire cross-channel attention residual module are combined by adding them pixel by pixel. This yields a mural feature map that contains rich spatial and channel dimensions while preserving coarse-scale features.

[0128] The calculation of a single convolutional network branch is shown in Equation (2):

[0129] F2 = Conv 1×1 (F0) (2);

[0130] Among them, F0 represents ancient mural image I. mural Shallow, low-level features;

[0131] The calculation of the branch composed of the convolutional network and the cross-channel attention module PSA is shown in Equation (3):

[0132] F1 = Conv 1×1 (PSA(F0)) (3);

[0133] PSA stands for cross-channel attention module;

[0134] Features F1, coarse-scale features F2, and shallow low-level features F0 from different spatial scales and channel dimensions are integrated by pixel-wise addition to obtain a feature map F that combines multi-scale spatial information and cross-channel attention. c Specifically, as shown in formula (14):

[0135]

[0136] in, Representing a pixel-by-pixel addition operation, this cross-channel attention residual module uses three modules in total. The output of the last cross-channel attention residual module is represented by F. c This represents a feature map containing multi-scale spatial information and cross-channel attention.

[0137] like Figure 4As shown, the cross-channel attention module PSA contained in each cross-channel attention residual module mainly realizes in four steps. First, the spatial information of the input feature map F0 is extracted in a multi-branch manner, which can obtain more rich position information of the input tensor, and is processed in a parallel manner at multiple scales:

[0138] F 1，1 =Conv(k 3×3 , G2)(F0)

[0139] F 1，2 =Conv(k 5×5 , G4)(F0)

[0140] F 1，3 =Conv(k 7×7 , G8)(F0)

[0141] F 1，4 =Conv(k 9×9 , G 16 )(F0)

[0142] The multi-branch spatial information calculated is obtained by cascading to obtain a multi-scale extracted feature map F all , and the specific calculation formula is shown in formula (15):

[0143] F all =Cat([F 1，1 , F 1，2 , F 1，3 , F 1，4 ]) (15)

[0144] Next, the channel attention weight information is extracted from the multi-scale extracted feature map to obtain attention weight vectors of different scales. The specific calculation is shown in formula (5):

[0145] Z i =SEWeight(F 1，i ) (5);

[0146] Wherein, Z i is the attention weight vector, i is the scale value of the channel, i of the embodiment is 1, 2, 3, 4;

[0147] The SEWeight module is used to obtain the attention weight from the feature map of different scales, and in this way, the context information of different scales can be fused. Further, in order to realize the interaction of attention information, the cross-dimension vector is fused without destroying the original channel attention vector, and the whole multi-scale channel attention vector is obtained in a cascading manner, and the specific calculation is shown in formula (16):

[0148] Z=Cat([Z1, Z2, Z3, Z4]) (16)

[0149] Then, the weights of the cascaded multi-scale channel attention are recalibrated using Softmax, as shown in formula (6):

[0150]

[0151] Among them, Attn i To calibrate the multi-scale channel attention, the multi-scale channel attention Attn is... i With the corresponding scale of feature map F 1，i Multiplication yields the feature map Y of the multi-scale channel attention weights. i The specific calculation is shown in formula (7):

[0152] Y i =Attn i ⊙F 1，i (7);

[0153] Where ⊙ represents element-wise multiplication; and i is the channel scale value, which in this embodiment uses 1, 2, 3, 4.

[0154] Finally, the output of the entire cross-channel attention module PSA is shown in Equation (17):

[0155] PSA=Cat([Y1, Y2, Y3, Y4]) (17);

[0156] Further analysis of ancient mural images I mural The gradient information attention map I is obtained through processing. attn ;

[0157] First, the input ancient mural image I mural The gradient map I of the image is obtained by applying the Sobel operator. grad Specifically, as shown in formula (18):

[0158] I grad =Sobel(I mural (18);

[0159] Next, I grad The gradient information attention map I is obtained by sequentially performing 7×7 convolution, PReLU, dual attention module DAM, 3×3 convolution, and sigmoid operations in the attention map generation branch. attn .

[0160] like Figure 5As shown, DAM represents a dual attention module, which is executed 6 times. Each dual attention module is further subdivided into a channel attention module and a multi-scale pixel attention module according to its function. The specific operation steps are as follows:

[0161] After processing by the PreLU layer, the output feature map F is obtained. in , feature map F in The data is fed into a dual attention module (DAM), which includes a channel attention module and a multi-scale pixel attention module.

[0162] The feature map F in The feature map F is fed into the channel attention module. in The weight Attn of each channel in the channel attention module channei Multiplying the features to obtain the channel attention feature map F channel The weight Attn of each channel channel Specifically, as shown in formula (8):

[0163] Attn channel =σ(Conv 1×1 (δ(Conv 1×1 (g c )))) (8);

[0164] Where σ is the sigmoid activation function, g c This is a global average pooling layer, where δ is the PReLU activation function;

[0165] The generated channel attention feature map F channel The input multi-scale pixel attention module uses two 3×3 convolutional layers to focus on the feature map F for channel attention. channel The feature map F of the channel attention after convolution is obtained through processing. P The specific calculation is shown in formula (19):

[0166] F P =(Conv 3×3 (Conv 3×3 (F channel (19)

[0167] Then, feature maps of convolutional outputs with different dilation ratios are obtained by using dilated convolutional layers with different dilation ratios, as shown in the following formula:

[0168] F D1 =DConv 3*3 (F P dilation=3)

[0169] F D2 =DConv3*3 (F P , dilation=5)

[0170] F D3 = DConv 3*3 (F P , dilation=7)

[0171] F D4 = DConv 3*3 (F P , dilation=9)

[0172] where DConv 3*3 (·) represents a dilated convolution with different dilation rates, and dilation represents a dilation rate. In this embodiment, a dilation rate of 3, 5, 7 and 9 is respectively used to obtain a feature map; F Di respectively represents a feature map output by a convolution with different dilation rates; and the obtained multiple feature maps output by convolutions with different dilation rates are concatenated to obtain a dilated convolution feature map F D , and the specific calculation is shown in formula (20):

[0173] F D = Cat([F D1 , F D2 , F D3 , F D4 ]) (20)

[0174] The dilated convolution feature map F D is processed through two 1x1 convolution layers, PReLU and a sigmoid function to obtain a feature map F pixel of multi-scale pixel attention; and the specific calculation is shown in formula (10):

[0175] F pixel = sigmoid(Conv 1×1 (PReLU(Conv 1×1 (F D )))) (10);

[0176] The feature map F pixel of multi-scale pixel attention and the feature map F channel of channel attention are pixel-by-pixel multiplied, and then added pixel-by-pixel with the feature map F in to obtain a gradient information attention map I attn , and the specific calculation is shown in formula (21):

[0177]

[0178] Step 3, obtaining the gradient information attention map I attnAfter that, we adaptively fuse the multi-scale spatial information and cross-channel attention features F c with gradient information attention map I attn by parameter a, and obtain the ancient wall painting feature map F g-attn with gradient attention guidance, as shown in equation (11):

[0179] F g-attn = aI attn + (1-a)F c (11);

[0180] where F g-attn is the ancient wall painting feature map;

[0181] Finally, the ancient wall painting feature map F g-attn with gradient attention guidance is input into the decoder composed of two 3x3 deconvolution layers and a 7x7 convolution layer, each of which is followed by an InstanceNorm2d and a ReLU layer, and finally the line drawing image I sketch of the ancient wall painting is obtained, as shown in equation (12):

[0182] I sketch = Conv 7×7 (deConv 3×3 (deConv 3×3 (F g-attn ))) (12);

[0183] Step 4, first input the ancient wall painting image I mural into the pre-trained depth prediction system, and output the depth map as the pseudo ground truth depth map F(I mural );

[0184] Then use the pre-trained Inception v3 network on mageNet to extract image features from the ancient wall painting line drawing image I sketch , and use the global generator in pix2pixHD as the depth prediction network to predict the depth map G geom from the line drawing image, and constrain the distance between the pseudo ground truth depth map F(I mural ) and the depth map G geom by the depth prediction loss function; obtain the final line drawing image; the depth prediction loss function is shown in equation (13):

[0185] L geom = |||G geom (G v3 (I sketch ))-F(Imural || (13) ;

[0186] where G v3 is a pre-trained Inception v3 network on ImageNet.

[0187] As Figure 6 shown, the extraction of ancient mural sketches is completed.

Claims

1. An ancient fresco line drawing extraction method characterized by, The method comprises the following steps: Step 1, cropping and data augmentation are performed on the collected ancient mural image to obtain an ancient mural image dataset; Step 2, extracting features from a single ancient mural image in the ancient mural dataset to obtain multi-scale spatial information and cross-channel attention feature maps ; the ancient mural image subjected to feature extraction synchronously processed to obtain gradient information attention maps ; Step 3, the feature map with multi-scale spatial information and cross-channel attention and gradient information attention map is weighted and fused to obtain an ancient wall painting feature map with gradient attention guidance Then it is input into the decoder to obtain the line drawing image of the ancient wall painting ; Step 4, the line drawing image of the ancient mural is constrained with the pseudo ground truth depth map of the ancient mural image , i.e. the extraction of the line drawing of the ancient mural is completed; The feature extraction in the step 2 specifically comprises: raising the channels of the ancient mural image to 64 channels through two convolution networks, and performing down-sampling on the ancient mural image to obtain shallow low-level features of the ancient mural image. Specifically, the calculation is shown in formula (1):​​​ （1）； extracting shallow low-level features inputting the feature map into the cross-channel attention residual module to obtain multi-scale spatial information and cross-channel attention ; The cross-channel attention residual module is divided into a single convolution network branch and a branch composed of a convolution network and a cross-channel attention module PSA, and the single convolution network branch is used to retain coarse scale features The specific calculation is shown in formula (2): （2）； wherein are ancient wall painting images of shallow low-level features; The convolution network and a branch composed of a cross-channel attention module PSA are used to extract features of different spatial scales and channel dimensions The specific calculation is shown in formula (3): （3）； The PSA is a cross-channel attention module. features of different spatial scales and channel dimensions , coarse scale features , and shallow low-level features are integrated together by pixel-wise addition to obtain a feature map with multi-scale spatial information and cross-channel attention .

2. The ancient wall drawing line drawing extraction method according to claim 1, wherein, The data enhancement in step 1 is performed by randomly rotating 90°, 180°, 270° and horizontally flipping.

3. The ancient wall drawing line drawing extraction method of claim 1, wherein, The cross-channel attention module PSA specifically includes extracting the input feature map in a multi-branch manner. The spatial information is processed in parallel at multiple scales, as shown in formula (4): （4）； wherein, representing different sizes of convolution kernels, representing different group sizes, are feature maps of different sizes; The calculated multi-branch spatial information is obtained as a multi-scale extraction feature map in a cascading manner Channel attention weight information is extracted from the multi-scale extracted feature map to obtain attention weight vectors of different scales, and the specific calculation is shown in formula (5): （5）； wherein, is an attention weight vector, i is a scale value of a channel; The calculated multiple sets of attention weight vectors The multi-scale channel attention vector Z is obtained in a cascading manner, Then use The weights of the cascaded multi-scale channel attention are recalibrated, and the specific calculation is shown in formula (6): （6）； wherein, For the calibrated multi-scale channel attention, the multi-scale channel attention is multiplied by the feature map of the corresponding scale to obtain a feature map of the multi-scale channel attention weight The specific calculation is shown in formula (7): （7）； wherein, denotes element-wise multiplication; i is the scale value of the channel; and the feature maps of the calculated different scale channel attention weights are concatenated and output.

4. The ancient wall drawing line drawing extraction method of claim 1, wherein, The step 2 ancient mural image Synchronously processing to obtain gradient information attention map Specifically comprising: processing the ancient mural image by Sobel formula To obtain the gradient map of the image Then, the gradient map After convolution network iteration, input PReLU layer, and output feature map after PreLU layer processing The feature map Is sent to the double attention module DAM to obtain the gradient information attention map .

5. The ancient wall drawing line drawing extraction method according to claim 4, characterized in that, The double attention module DAM comprises a channel attention module and a multi-scale pixel attention module. The feature map is fed into a channel attention module, and the feature map is multiplied by a weight of each channel in the channel attention module to obtain a channel-attended feature map ; the weight of each channel is specifically shown in formula (8): （8）； wherein, is a sigmoid activation function, is a global average pooling layer, is a PReLU activation function; The generated channel attention feature map The input multi-scale pixel attention module adopts a double convolution network to process the channel attention feature map to obtain a convolutional channel attention feature map and then obtains feature maps of different expansion rate convolution outputs through expansion convolution layers with different expansion rates. The specific calculation is shown in formula (9): （9）； wherein, dilated convolutions representing different dilation rates, representing a dilation rate, respectively represent feature maps output by different dilation rate convolutions; the obtained multiple feature maps output by different dilation rate convolutions are concatenated to obtain a dilation rate convolution feature map , the dilation rate convolution feature map is processed by a convolutional network, a PreLU function and a sigmoid function to obtain a feature map of multi-scale pixel attention ; the specific calculation is shown in formula (10): （10）； feature maps of multi-scale pixel attention feature maps of channel attention pixel-wise multiplication pixel-wise addition to obtain gradient information attention map .

6. The ancient wall drawing line drawing extraction method of claim 1, wherein, The weighted fusion in step 3 is specifically that the parameter The feature maps of the multi-scale spatial information and the cross-channel attention are adaptively fused with the gradient information attention map The weighted fusion is specifically calculated as shown in formula (11): （11）； wherein, is a characteristic map of an ancient mural, is a gradient information attention map, is a feature map of multi-scale spatial information and cross-channel attention. The decoder in step 3 comprises an inverse convolution layer and a convolution layer, and both the inverse convolution layer and the convolution layer are based on an InstanceNorm2d and a ReLU layer; the specific calculation is shown in formula (12): （12）； wherein, is a convolutional layer, is a deconvolutional layer.

7. The ancient wall drawing line drawing extraction method according to claim 1, wherein The step 4 of constraining the line drawing image of ancient mural with the pseudo ground truth depth map of ancient mural image specifically includes: First, the ancient mural image is input to a pre-trained depth prediction system, and a depth map is output as a pseudo ground truth depth map for the ancient mural ; Then the Inception v3 network pre-trained on mageNet is adopted to extract image features from the ancient fresco sketch images , and the global generator in pix2pixHD is used as a depth prediction network to predict the depth map of the sketch image from the sketch image , and the distance between the pseudo ground truth depth map and the depth map is constrained by a depth prediction loss function ; the final sketch image is obtained; and the depth prediction loss function is shown in formula (13): （13）； wherein, is a pre-trained Inception v3 network on ImageNet.

Images