Broken mural line draft detection method based on uncertainty principle

Abstract

The invention belongs to the technical field of cultural relic protection, and particularly discloses a damaged fresco line draft detection method based on an uncertainty principle, which comprises the following steps: establishing a line draft detection network model: the line draft detection network model is a full convolutional network model taking an encoder-decoder as a basic framework; in the decoding stage of the decoders, two decoders are adopted to generate a mean value and a variance respectively; and inserting a multi-scale weight score fusion module and a deep supervision shallow layer module into the online manuscript detection network model, and capturing inherent noise in training data based on data uncertainty. According to the method, the noise information in the mural image is measured by adopting an uncertainty principle, direct removal of the noise information in the mural is realized through learning of data uncertainty, and the high-quality line draft image which is clean in background, clear in line and free of noise interference is generated.

Description

Technical Field

[0001] The invention belongs to the technical field of cultural relics protection, and in particular relates to a method for detecting damaged mural line drawings based on the uncertainty principle. Background Art

[0002] Murals are one of the oldest forms of painting art in human history. There are a huge number of them, with gorgeous colors and rich content. They have extremely high artistic and research value and are an important part of the world's cultural heritage. Using murals to decorate walls and express artistic conception is a manifestation of the combination of ancient wisdom and art. It is a large book of color and ink painted by the ancients, and it is also a precious book that is more accurate, objective and vivid than written records. Before starting to paint a mural, usually a line draft is drawn first to determine the layout, proportion and position of the theme. Then the painting is carried out step by step, gradually filling in colors and details.

[0003] Line drawings can accurately present the main content and outline structure of mural images, reduce irrelevant content, and retain the most important structural properties. In terms of cultural heritage protection, line drawings can help researchers understand the structure of murals and assist in the restoration of murals. In addition, line drawings are the skeleton of Chinese painting to shape artistic images. They are closely related to calligraphy and have special independent aesthetic connotations and artistic value. Research on line drawings is helpful for the protection and inheritance of this art form.

[0004] Traditional line drawing extraction is mostly done by experts manually copying and drawing. Although this method can accurately reflect the original appearance of the murals, it is time-consuming and labor-intensive and depends on the experience and skill level of the experts, and its efficiency cannot be guaranteed. There are a large number of mural images, and new cultural relics are unearthed every year. Relying solely on manual labor will consume huge resources and the cultural relics may not be studied and protected in a timely manner. Therefore, it is necessary to use computer technology to assist in the research and protection of murals. In addition, when detecting damaged mural images, the existing algorithms ignore the interference of noise and extract the damaged information as line drawings. Moreover, the extracted line drawing images have blurred lines, missing details, and unclean backgrounds, which do not meet the application standards and hinder the digitization and protection of cultural heritage. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects in the prior art and provide a method for detecting damaged mural line drawings based on the uncertainty principle.

[0006] The present invention provides a method for detecting damaged mural line drawings based on the uncertainty principle, comprising the following steps:

[0007] S1. Establish a line draft detection network model:

[0008] The line draft detection network model is a fully convolutional network model with an encoder-decoder as the basic framework, wherein the encoder is used for feature extraction and the decoder is used for reconstruction of the line draft image; in the decoding stage of the decoder, two decoders are used to generate a mean and a variance respectively, wherein the variance represents data uncertainty and the mean represents predicted line draft features;

[0009] A multi-scale weight score fusion module and a deep supervision shallow module are inserted into the line draft detection network model, and the inherent noise in the training data is captured based on data uncertainty; the multi-scale weight score fusion module is used to enhance the line draft features and capture the detail information; the deep supervision shallow module uses deep semantic information to supervise the shallow fine features to remove background noise;

[0010] S2. Detect the line draft of the damaged mural using the line draft detection network model.

[0011] A further solution is to divide the decoder into five stages based on the size of the output features of each stage of the decoder and the convolution step size. The scale of the output features of each stage is 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input features respectively.

[0012] The multi-scale weight score fusion module is added at the output end of each stage of the encoder;

[0013] The deep supervision shallow module is added at the output end of the multi-scale weight score fusion module.

[0014] A further solution is that the multi-scale weight score fusion module includes: a 3×3 convolution for feature dimensionality reduction, a normal convolution, three dilated convolutions with different expansion rates, and a score module; the obtained features are concatenated using cat and fused through a 3×3 convolution and a residual structure;

[0015] The dilation rates of the three dilated convolutions are 4, 8, and 12 respectively;

[0016] The score module includes a 1×1 convolution for obtaining a feature with a channel number of 1, a 3×3 convolution for learning the applicability score of the feature with a channel number of 1, and a sigmoid function is used to obtain the applicability score in the range of 0-1.

[0017] A further solution is that the process of realizing line feature enhancement and detail information capture by the multi-scale weight score fusion module includes the following steps:

[0018] Step 1: Downsample the input image to obtain the feature map F 0 , the formula is:

[0019] F 0=conv 3×3 (input)

[0020] Step 2: transform the feature map F 0 It is sent into a normal convolution and three dilated convolutions with different expansion rates to extract multi-scale features of different receptive fields. The formula is:

[0021] F 1 =F 0

[0022] F 2 =Dconv 3×3 (F 0 , r=4)

[0023] F 3 =Dconv 3×3 (F 0 , r=8)

[0024] F 4 =Dconv 3×3 (F 0 , r=12)

[0025] Among them, Dconv 3×3 (·) represents the dilated convolution with different dilation rates, r represents the dilation rate, and F 1 Represents the output feature of the first ordinary convolution, F 2 , F 3 , F 4 Represent the output features of the 1st, 2nd, and 3rd dilated convolutions respectively;

[0026] Step 3: Get the scores corresponding to different receptive field features. The formula is:

[0027] S 1 =Score(F 1 )

[0028] S 2 =Score(F 2 )

[0029] S 3 =Score(F 3 )

[0030] S 4 =Score(F 4 )

[0031] Among them, Score(·) is the score module used to obtain the score of each feature; S 1 , S 2 , S 3 , S 4 The features F 1, F 2 , F 3 , F 4 The corresponding score;

[0032] Step 4: Multiply the obtained score by the corresponding feature to obtain the weighted feature. The formula is:

[0033] F 1 ′=F 1 ×S 1

[0034] F 2 ′=F 2 ×S 2

[0035] F 3 ′=F 3 ×S 3

[0036] F 4 ′=F 4 ×S 4

[0037] Among them, F 1 ′,F 2 ′,F 3 ′,F 4 ′ represents a feature with weight;

[0038] Step 5: Finally, the weighted features of different scales are fused. The formula is:

[0039] F fuse =conv 3×3 (cat[F 1 ′,F 2 ′,F 3 ′,F 4 ′]+input)

[0040] Among them, F fuse is the final fusion feature, conv 3×3 (·) is a normal convolution with a kernel of 3, and cat(·) is a concatenation operation.

[0041] A further solution is that the deep layer supervises the shallow layer module to remove background noise, including the following steps:

[0042] Step 1: The mural image is input into the encoder of the line draft detection network model, and the mural feature map of five stages is obtained through downsampling and multi-scale weight score fusion modules; in the decoding stage, the features of the five stages are input into multiple deep-supervised shallow modules; the feature of the i-th stage is defined as f i , the characteristic of the i+1th stage is f i+1, the deep supervision shallow module is f i and f i+1 As input, f i+1 The features are upsampled to f i Same size, f i+1 The corresponding weight is obtained through the mean and sigmoid functions, and finally the weight is combined with f i Multiplying, the formula is:

[0043] f′=sigmoid(mean(f i +1))×f i

[0044] Where f′ is f i Features with weights, finally fusion f′ and f i The final result is:

[0045] f fuse =cat[f i , f′]

[0046] Among them, f fuse is the final fusion result.

[0047] A further approach is to capture the inherent noise in the training data based on data uncertainty as follows:

[0048] Introducing uncertainty as noise in the data n(x i ), the objective function formula is:

[0049]

[0050] in, The model ultimately requires supervised learning results. is the input image, and the training process continuously reduces n(x i ) The influence of noise on the result can directly remove the damaged information in the image;

[0051] In the decoding stage a decoder D U and output header H U To obtain the uncertainty σ of the data, another decoder D and prediction head H are used to generate the mean μ, the formula is:

[0052]

[0053] The obtained mean and variance are used to establish a corresponding learnable Gaussian distribution, the formula is:

[0054]

[0055] Among them, p(·) is the established Gaussian distribution, Gaussian function, μ i As a sample x i The prediction results, and σ i Represents μ i uncertainty.

[0056] A further solution is to also train the established generative adversarial network model.

[0057] A further solution is to introduce the class-balanced cross entropy loss function and the variance-balanced loss function to train the line draft detection network model, including the following steps:

[0058] The class-balanced cross entropy loss function is selected to solve the class imbalance problem. The formula is:

[0059]

[0060] Among them, Y is The corresponding truth value, α and β weights,

[0061]

[0062] Use the variance equalization loss function to speed up the training process of the stable line draft detection network model. The formula is:

[0063]

[0064] Construct a loss function to train the line draft detection network model. The loss function is:

[0065]

[0066] According to the constructed loss function, the line draft detection network model is iteratively trained until fitting, where the number of iterations is set to 20, the initial learning rate is set to 0.001, and after every 3 iterations, the learning rate is reduced using a linear attenuation strategy.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] (1) The present invention adopts the uncertainty principle to measure the noise information in the mural image. By learning the data uncertainty, the noise information in the mural is directly removed, and a high-quality line drawing image with a clean background, clear lines and no noise interference is generated.

[0069] (2) The present invention adopts the idea of ​​multi-scale feature enhancement and introduces a multi-scale weight score fusion module. While taking into account the influence of each scale feature on the result, it reduces the information loss caused by feature dimensionality reduction, achieves more accurate detection of the line draft part, and captures more detailed information.

[0070] (3) The present invention introduces a deep-layer supervised shallow-layer module, utilizes the characteristics of features at different stages, and uses deep-layer semantic information to supervise shallow-layer fine features to remove background noise.

[0071] (4) The present invention adopts the variance equalization loss function to speed up and stabilize the training process of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The following drawings are only used to illustrate and explain the present invention, and are not used to limit the scope of the present invention, wherein:

[0073] Figure 1 It is a framework diagram of the line draft detection network model of the present invention;

[0074] Figure 2 This is a structural diagram of the multi-scale weight score fusion module of the present invention;

[0075] Figure 3 This is the structural diagram of the deep supervision shallow module of the present invention;

[0076] Figure 4 This is the complete result of extracting line drawings from a mural with line drawing labels using different methods in a simulation experiment of an embodiment of the present invention;

[0077] Figure 5 This is the complete result of line drawing extraction of a mural with a wireless draft tag using different methods in a simulation experiment of an embodiment of the present invention. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solution, design method and advantages of the present invention clearer, the present invention is further described in detail by specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0079] The present invention provides a method for detecting line drafts of damaged murals based on the uncertainty principle. The multi-scale weight score fusion module takes into account the influence of each scale feature on the result while reducing the information loss caused by channel number dimensionality reduction, thereby capturing more detail information. Using a deep-supervised shallow module, the deep-level semantic information is used to supervise the shallow fine feature information to further eliminate background noise. Finally, the data uncertainty is used to represent the noise in the image, and the prediction result is converted into a learnable Gaussian distribution, so as to learn features and uncertainties at the same time. The supervision of the variance and Gaussian distribution can reduce the influence of noise information on the prediction results, and the direct removal of noise information can be achieved. At the same time, through model pruning, the lightweight of the network is simply and efficiently achieved, which greatly reduces the computing cost and promotes the digitization and protection of murals as a cultural heritage. Before performing line draft detection, a line draft detection network model is first established: such as Figure 1As shown, the line draft detection network model is a full convolutional network model with an encoder-decoder as the basic framework. The encoder is used for feature extraction and the decoder is used for reconstruction of the line draft image. In the decoding stage of the decoder, two decoders are used to generate the mean and variance respectively. The variance is represented as data uncertainty, and the mean represents the predicted line draft feature. A multi-scale weight score fusion module and a deep supervision shallow module are inserted into the line draft detection network model, and the inherent noise in the training data is captured based on data uncertainty. The multi-scale weight score fusion module is used to enhance the line draft features and capture the detail information. The deep supervision shallow module uses deep semantic information to supervise the shallow fine features to remove the background noise. The line draft detection network model is used to detect the line draft of damaged murals. Regarding the selection of encoders, Efficientnetv2S with excellent feature extraction capabilities and less resource consumption is selected as the encoder. In order not to introduce too many parameters and ensure accuracy, Unet++ is selected as the decoder. Unlike EfficientnetV2S, in order to ensure that the entire network is a fully convolutional network, the classification head containing the pooling and fully connected layers is deleted. According to the size of the output features and the convolution step size of each stage, the decoder is divided from the original 8 stages (7 stages after deleting the classification head) into 5 stages, and each stage corresponds to Unet++ one by one, such as Figure 1 As shown. For example, if the step size of the first stage of the original EfficientnetV2S is 2, the output feature map is 1 / 2 of the input feature. The step size of the second stage is 1 and the output feature map size is the same as that of the first stage. At this time, we can regard the first and second stages as one stage, and the same applies to other stages. In this way, the model is divided into five stages. After each stage, the input feature will be downsampled to 1 / 2 of the original, and finally five features of different scales will be output, which are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input feature. The purpose of doing this is to be able to fuse features of different scales in the decoding stage. The multi-scale weight score fusion module is added to the output end of each stage of the encoder; the deep supervision shallow module is added to the output end of the multi-scale weight score fusion module. As shown Figure 2 As shown, the multi-scale weight score fusion module includes: a 3×3 convolution for feature dimensionality reduction, a normal convolution, three dilated convolutions with different expansion rates, and a score module; the obtained features are concatenated using cat and fused through a 3×3 convolution and a residual structure; the expansion rates of the three dilated convolutions are 4, 8, and 12 respectively; the score module includes a 1×1 convolution for obtaining features with a channel number of 1, a 3×3 convolution for learning the applicability score of features with a channel number of 1, and a sigmoid function for obtaining a suitability score ranging from 0 to 1. Continue to refer to Figure 1In the decoding stage, in order to fully fuse the five different scale features obtained by encoding, Unet++ is selected to restore the image. In addition, the overall structure of Unet++ can be regarded as four nested Unets, which can more conveniently perform model pruning. Each Unet contains upsampling and dense skip connections. This nested and densely connected structure can effectively fuse features of different scales and restore high-resolution line drawings through progressive upsampling. In order to make full use of deep semantic information to further eliminate background noise, a deep-supervised shallow module is added at the node where the deep features are fused with the previous layer features in the decoding stage, such as Figure 1 The decoder color node uses the deep supervised shallow module for feature fusion, and other nodes use the splicing operation for feature fusion. If all nodes use the deep supervised shallow module for feature fusion, it will greatly increase the computing resources. Therefore, in order to ensure that the features of each stage can be used without introducing too much computing, in this embodiment, an upsampling line that can balance computing resources and fusion performance is selected, such as Figure 1 The decoder has four nodes marked in red. The input of each node in this circuit comes from the concatenation of the features of the next layer and the nodes of this layer, and the relationship between each node is progressive, so that the features of each stage can be used in this circuit without introducing too much calculation. The structure of the deep supervision shallow module is as follows Figure 3 As shown in Figure 1, the input of the deep supervision shallow module comes from the features of the two adjacent stages. In order to ensure that no parameters are introduced, the convolution operation is abandoned in the design, and the operations in the deep supervision shallow module only use parameter-free calculations. The feature from the deep layer is defined as f i+1 , f i+1 First, a mean operation is performed to obtain a feature with a channel number of 1 for subsequent calculations. Then, the Sigmoid function is used to obtain the global information of the feature. Then, the shallow feature f i With f i+1 After multiplying the global information of mean and sigmoid, the multiplied features are finally combined with f i Perform a concatenation operation (Cat) to obtain the final features.

[0080] After the line draft detection network model is established, it is necessary to iteratively train the line draft detection network model, including:

[0081] Construct a mural line drawing training data set, which is a paired data set, including mural images and corresponding labeled line drawing data sets: the training data used in this embodiment mainly comes from mural cultural relics images in Fengguo Temple, Qianling Mausoleum, Zhaoling Mausoleum, Dunhuang and other places, including mural images of Buddha statues and Tang tomb murals, Dunhuang Bodhisattva and other murals of different styles and ages. Select 30 images with clear line drawing and randomly crop them into 180 images for training. In order to use the data more effectively, different data enhancement strategies are adopted to expand the data. Specifically, each image is first flipped, then rotated at 16 different angles, and the brightness of the image is adjusted with four different thresholds. Therefore, after data enhancement, the data set is expanded to 160 times the original to obtain 28,800 images, and the size of each image is 512×512.

[0082] Clean line drawings are obtained by using a multi-scale weighted score fusion module, a deep-supervised shallow module, and uncertainty. First, the multi-scale weighted score fusion module is used to enhance the line drawing features and capture the detail information; then, the deep-supervised shallow module uses deep semantic information to supervise the shallow fine features to remove background noise; finally, the prediction result is converted into a learnable Gaussian distribution, so that features and uncertainties are learned at the same time, noise information is removed, and the model is pruned to achieve lightweight model.

[0083] Design a model training strategy and introduce class-balanced cross entropy loss function and variance-balanced loss function to train the model.

[0084] In the above, the process of enhancing the line drawing features and capturing the detail information through the multi-scale weight score fusion module includes the following steps:

[0085] Step 1: Downsample the input image to obtain the feature map F 0 , the formula is:

[0086] F 0 =conv 3×3 (input)

[0087] Step 2: transform the feature map F 0 It is sent to multiple dilated convolutions with different expansion rates to extract multi-scale features of different receptive fields. The formula is:

[0088] F 1 =F 0

[0089] F 2 =Dconv 3×3 (F 0 , r=4)

[0090] F 3 =Dconv 3×3 (F 0 , r=8)

[0091] F 4 =Dconv 3×3 (F 0 , r=12)

[0092] Among them, Dconv 3×3 (·) represents the dilated convolution with different dilation rates, r represents the dilation rate, and F 1 Represents the output feature of the first ordinary convolution, F 2 , F 3 , F 4 They represent the output features of the 1st, 2nd, and 3rd dilated convolutions respectively.

[0093] Step 3: Get the scores corresponding to different receptive field features. The formula is:

[0094] S 1 =Score(F 1 )

[0095] S 2 =Score(F 2 )

[0096] S 3 =Score(F 3 )

[0097] S 4 =Score(F 4 )

[0098] Among them, Score(·) is the score module used to obtain the score of each feature; S 1 , S 2 , S 3 , S 4 The features F 1 , F 2 , F 3 , F 4 The corresponding score.

[0099] Step 4: Multiply the obtained score by the corresponding feature to obtain the weighted feature. The formula is:

[0100] F 1 ′=F 1 ×S 1

[0101] F 2 ′=F 2 ×S 2

[0102] F 3 ′=F 3 ×S 3

[0103] F 4 ′=F 4 ×S 4

[0104] Among them, F 1 ′,F 2 ′,F 3 ′,F 4 ′ represents a feature with weight.

[0105] Step 5: Finally, the weighted features of different scales are fused. The formula is:

[0106] F fuse =conv 3×3 (cat[F 1 ′,F 2 ′,F 3 ′,F 4 ′]+input)

[0107] Among them, F fuse is the final fusion feature, conv 3×3 (·) is a normal convolution with a kernel of 3, and cat(·) is a concatenation operation.

[0108] In the above, the deep-layer supervision shallow-layer module removes background noise by using deep-layer semantic information to supervise shallow-layer fine features. The specific process includes the following steps:

[0109] The mural image is input to the encoder of the model. Through downsampling and multi-scale weight score fusion modules, the mural feature maps of multiple stages are obtained. In the decoding stage, the features of multiple stages are input to multiple deep-supervised shallow modules. The feature of the i-th stage is defined as f i , the characteristic of the i+1th stage is f i+1 . The module is f i and f i+1 As input, first f i+1 The features are upsampled to f i The same size. Then, f i+1 The corresponding weight is obtained through the mean and sigmoid functions, and finally the weight is combined with f i Multiplying, the formula is:

[0110] f′=sigmoid(mean(f i +1))×f i

[0111] Where f′ is f i Features with weights, finally fusion f′ and f i The final result is:

[0112] f fuse =cat[f i , f′]

[0113] Among them, f fuse is the final fusion result, and cat(·) is the concatenation operation.

[0114] By introducing uncertainty, the prediction results are converted into a learnable Gaussian distribution, so that features and uncertainties can be learned at the same time, noise information can be removed, and model pruning can be used to achieve lightweight. The specific process is as follows:

[0115] Depend on The mural line drawing dataset composed of Input image, It is a line drawing label. If the traditional strategy is adopted, If the training is carried out in the form of , the noise in the image will be ignored, resulting in poor robustness of the model and inability to effectively extract clean line drawings. Therefore, uncertainty is introduced as the noise n(x i ), then the final objective function formula is:

[0116]

[0117] in, The model ultimately needs supervised learning results. In this way, the model is trained on To supervise, n(x i ) is a penalty term during training, and the training process continuously reduces n(x i )The influence of noise on the result is realized by directly removing the damaged information in the image.

[0118] In order to introduce uncertainty into line draft detection, a decoder D is added in the decoding stage. U and output header H U To obtain the uncertainty σ of the data, another decoder D and prediction head H are used to generate the mean μ, the formula is:

[0119]

[0120] The obtained mean and variance are used to establish a corresponding learnable Gaussian distribution, the formula is:

[0121]

[0122] Among them, p(·) is the established Gaussian distribution, Gaussian function, in order to make it easier to understand the formula, the heavy parameter technique is used, the formula is:

[0123]

[0124] in, can be considered as the result of sampling from a Gaussian distribution, μ i As a sample x i The prediction results, and σ i Represents the predicted μ i uncertainty.

[0125] Furthermore, the class-balanced cross entropy loss function is selected to solve the class imbalance problem, and the formula is:

[0126]

[0127] in, is the result of sampling from the obtained Gaussian distribution, and Y is the corresponding true value. is the corresponding weight.

[0128] Use the variance equalization loss function to speed up the training process of the stable model. The formula is:

[0129]

[0130] Construct the final loss function for training, the formula is:

[0131]

[0132] The model is iteratively trained according to the constructed loss function until it is fitted, where the number of iterations is set to 20, the initial learning rate is set to 0.001, and after every 3 iterations, the learning rate is reduced using a linear decay strategy.

[0133] In the present invention, in the line drawing extraction task of damaged murals, the noise information in the uncertainty measurement data is introduced to learn the ability to remove the damaged noise information in the murals, so as to detect and obtain a clean line drawing image. The model uses an encoder-decoder as the basic framework, in which the encoder is used to extract feature extraction and the decoder is used to reconstruct the line drawing image. Given an input image, it is first sent to the encoder to obtain multiple levels of features, and then the line drawing features are enhanced and the detail information is captured through the multi-scale weight score fusion module, and then the enhanced multi-scale features are sent to the two decoders and the prediction head to generate the mean and variance. In the decoding stage, the deep supervision shallow module uses the deep semantic information to supervise the shallow fine features to remove the background noise, and finally the obtained mean and variance are generated The corresponding Gaussian distribution is sampled from the Gaussian distribution to obtain the final result for supervision, where the mean passes through the prediction head to obtain the line drawing image. In addition, in order to speed up the stable training process, this study introduces the variance equalization loss function, compares the obtained variance with the label, and improves the robustness of the model.

[0134] In order to verify the line drawing extraction result of the present invention, the effectiveness of the damaged mural line drawing detection method based on the uncertainty principle proposed by the present invention is verified by simulation below:

[0135] First, in one embodiment of the present invention, the damaged mural line draft detection method based on the uncertainty principle proposed by the present invention is used to extract the line draft of the mural data collected and produced, and the specific process is as follows:

[0136] The experiment of the present invention was conducted on a server equipped with a Linux operating system, and the server was equipped with a high-performance NVIDIA GeForce RTX4090 GPU to ensure efficient execution of computing tasks. All neural network models designed and implemented in the experiment were completed using the Pytorch deep learning framework, and Python 3.10 was selected as the development language to better complete the tasks of this study using its powerful functions and stability.

[0137] During the network training process, thanks to Adam's fast fitting ability, the Adam optimizer is used to optimize the network parameters. This experiment set 20 iterations of training, in which the initial learning rate was set to 0.001. After every 3 iterations, the learning rate was reduced using a linear decay strategy. Finally, the results of the simulation experiment line draft extraction were analyzed.

[0138] Figure 4 This is the comparative experimental result of different methods on mural images with line drawing labels in the simulation experiment. Figure 5 This is the comparison result of the mural image without the line drawing label; Figure 4The fusion result images of five contrast methods are shown in Figure 2. From the perspective of noise removal, Figure 4 In the first three lines of the image, the original image background and characters contain a lot of noise, such as cracks and peeling. The previous method will extract this noise information as line drawing information, resulting in the detected line drawing image containing a lot of irrelevant information and poor visual effect. The line drawing image extraction method proposed by the present invention can remove most of the noise in the image to obtain a complete line drawing image. Figure 4 From the fourth to sixth rows of the figure, the method of the present invention can extract images without noise interference, while other methods still have noise interference. In terms of extracting the details of the line drawing, the previous methods often ignore some small-scale important information in the image, such as the facial information of the characters. Figure 4 It can be seen from the fourth and sixth rows that facial information is missing in the character line drawings extracted by other methods, while the method of the present invention can extract these detailed information. From the clarity of the lines, it can be seen that the lines of other methods are mostly blurred and have artifacts, and the lines will stick together in areas with dense lines, while the lines extracted by the method of the present invention are clear and fine, and each line can be well distinguished in areas with dense lines. In the application of hyperspectral images, the colors and lines of some mural images are no longer clearly visible due to their age. At this time, the full picture of the image cannot be observed by the naked eye. The present invention extracts the hyperspectral image corresponding to the image, and then extracts the line drawings, such as Figure 4 The last row is the line draft detection result of the hyperspectral image. It can be seen that the method of the present invention has good generalization ability on the hyperspectral image.

[0139] In order to verify the generalization ability of the method, it was tested on mural images of different styles, different degrees of damage, and different dynasties. Figure 5 As shown, it can be seen from the results that the method of the present invention can be applied to different types of mural images. It can effectively remove noise information in images with different degrees of damage. It can also extract line drawings with clear lines and rich details for undamaged images or images containing complex patterns.

[0140] Visual evaluation and comparison can give us a more intuitive understanding of the line draft extraction results, but it is difficult to give the most accurate judgment of the line draft extraction results by relying solely on subjective evaluation. Therefore, it is necessary to combine objective indicators to evaluate the line draft extraction results. This experimental design includes multiple key performance indicators to evaluate and quantify the effectiveness of the model: global optimal threshold (ODS), local optimal threshold (OIS), peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), average precision (AP) and root mean square error (RMSE). The higher the values ​​of ODS, OIS and AP, the more accurate the detected line draft. PSNR, SSIM and RMSE are used to evaluate image quality. The higher the values ​​of PSNR and SSIM, the higher the quality of the detected line draft image and the closer it is to the label. The smaller the value of RMSE, the smaller the difference between it and the label and the less noise information it contains.

[0141] Table 1 Index evaluation results of different methods

[0142]

[0143] Table 1 lists in detail the comparative data of the method of the present invention and several other algorithms in various evaluation indicators, thus comprehensively demonstrating the performance and advantages of the research method. Compared with the current best algorithm UAED, it achieved an improvement of 1.5% and 4.1% in RMSE and SSIM, and a significant improvement of up to 4.6%, 5.3% and 10.4% in ODS, OIS and AP. Compared with other methods, whether traditional or based on deep learning, the method of the present invention has achieved a huge improvement in various indicators. This shows that the method of the present invention can extract a more complete line draft image while ensuring image quality and removing damaged noise information in the image.

[0144] In Table 1, Canny: A Computational Approach to Edge Detection (a computational method for edge detection);

[0145] FdoG: Coherent Line Drawing (edge ​​detection based on flow-based Gaussian difference)

[0146] Traditional methods based on underlying feature calculation (including Canny and FdoG) to achieve edge detection through steps such as Gaussian filtering, gradient calculation and double threshold detection;

[0147] SE: Fast edge detection using structured forests (fast edge detection based on structured forests), a learning-based method (SE) that uses machine learning methods to train and classify different underlying features to obtain edges;

[0148] Edge detection methods based on deep learning (HED, RCF, BDCN, DexiNed, PiDiNet, PiDiNet-l, UAED);

[0149] HED: Holistically-Nested Edge Detection (HED) is an edge detection method based on holistic nesting. HED divides the backbone network into five stages based on VGG16 to obtain features of different scales, and after each stage, the high-dimensional feature map is reduced to an edge prediction map for supervision, and finally the side outputs at different levels are fused to obtain the prediction result.

[0150] RCF: Richer Convolutional Features for Edge Detection. Based on HED, RCF fuses the outputs of convolutional layers at different levels in each stage to capture more feature information, further improving the accuracy of detection.

[0151] BDCN: Bi-Directional Cascade Network for Perceptual Edge Detection (Bidirectional Cascade Perceptual Network for Edge Detection);

[0152] BDCN proposes a bidirectional cascade network that combines high-level semantic information and low-level detail information to establish connections between information at different levels, and uses a scale enhancement module to enable the network to more effectively capture multi-scale edge information;

[0153] PiDiNet:Pixel Difference Networks for Efficient Edge Detection;

[0154] PiDiNet combines traditional edge operators to capture edge features through the difference between adjacent pixels in the image, and designs a lightweight network structure that can achieve better edge detection effects while maintaining low computational costs. PiDiNet-l is a lightweight version of PiDiNet.

[0155] UAED establishes the uncertainty between multiple labels and makes full use of the information of all labels to achieve more accurate edge detection.

[0156] Combining visual evaluation and objective index evaluation, the damaged mural line drawing detection method based on the uncertainty principle provided in this embodiment can detect and obtain clear, complete and accurate line drawing extraction images from damaged murals.

[0157] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used in the present invention is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed in the present invention.

Claims

1. A method for detecting damaged mural line drawings based on the uncertainty principle, characterized in that: The following steps are involved: S1. Establish a line draft detection network model: The line draft detection network model is a fully convolutional network model with an encoder-decoder as the basic framework, wherein the encoder is used for feature extraction and the decoder is used for reconstruction of the line draft image; in the decoding stage of the decoder, two decoders are used to generate a mean and a variance respectively, wherein the variance represents data uncertainty and the mean represents predicted line draft features; A multi-scale weight score fusion module and a deep supervision shallow module are inserted into the line draft detection network model, and the inherent noise in the training data is captured based on data uncertainty; the multi-scale weight score fusion module is used to enhance the line draft features and capture the detail information; the deep supervision shallow module uses deep semantic information to supervise the shallow fine features to remove background noise; S2. Detect the line draft of the damaged mural using the line draft detection network model.

2. The method for detecting damaged mural line drawings based on the uncertainty principle according to claim 1, characterized in that: Based on the size of the output features of each stage of the decoder and the convolution step size, the decoder is divided into five stages. The scale of the output features of each stage is 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input features respectively; The multi-scale weight score fusion module is added at the output end of each stage of the encoder; The deep supervision shallow module is added at the output end of the multi-scale weight score fusion module.

3. The method for detecting damaged mural line drawings based on the uncertainty principle according to claim 2, characterized in that: The multi-scale weight score fusion module includes: a 3×3 convolution for feature dimension reduction, a normal convolution, three dilated convolutions with different expansion rates, and a score module; the obtained features are concatenated using cat and fused through a 3×3 convolution and a residual structure; The dilation rates of the three dilated convolutions are 4, 8, and 12 respectively; The score module includes a 1×1 convolution for obtaining a feature with a channel number of 1, a 3×3 convolution for learning the applicability score of the feature with a channel number of 1, and a sigmoid function is used to obtain the applicability score in the range of 0-1.

4. The method for detecting damaged mural line drawings based on the uncertainty principle according to claim 3, characterized in that: The process of implementing line feature enhancement and detail information capture by the multi-scale weight score fusion module includes the following steps: Step 1: Downsample the input image to obtain the feature map F0. The formula is: F0=conv 3×3 (input) Step 2: Send the feature map F0 into a normal convolution and three dilation convolutions with different expansion rates to extract multi-scale features of different receptive fields. The formula is: F1=F0 F2=Dconv 3×3 (F0,r=4) F3=Dconv 3×3 (F0,r=8) F4=Dconv 3×3 (F0,r=12) Among them, Dconv 3×3 (·) represents dilated convolutions with different dilation rates, r represents the dilation rate, F1 represents the output feature of the first ordinary convolution, and F2, F3, and F4 represent the output features of the 1st, 2nd, and 3rd dilated convolutions, respectively; Step 3: Get the scores corresponding to different receptive field features. The formula is: S1=Score(F1) S2=Score(F2) S3=Score(F3) S4=Score(F4) Among them, Score(·) is a score module used to obtain the score of each feature; S1, S2, S3, S4 are the scores corresponding to features F1, F2, F3, F4 respectively; Step 4: Multiply the obtained score by the corresponding feature to obtain the weighted feature. The formula is: F1′=F1×S1 F2′=F2×S2 F3′=F3×S3 F4′=F4×S4 Among them, F1′, F2′, F3′, and F4′ respectively represent the weighted features corresponding to the output features under the corresponding scores; Step 5: Finally, the weighted features of different scales are fused. The formula is: F fuse =conv 3×3 (cat[F1′,F2′,F3′,F4′]+input) Among them, F fuse is the final fusion feature, conv 3×3 (·) is a normal convolution with a kernel of 3, and cat(·) is a concatenation operation.

5. The method for detecting damaged mural line drawings based on the uncertainty principle according to claim 4, characterized in that: The deep layer supervises the shallow layer module to remove background noise, which includes the following steps: Step 1: The mural image is input into the encoder of the line draft detection network model, and the mural feature maps of five stages are obtained through downsampling and multi-scale weight score fusion modules; in the decoding stage, the features of the five stages are input into multiple deep-supervised shallow modules; the feature of the i-th stage is defined as f i , the characteristic of the i+1th stage is f i+1 , the deep supervision shallow module is based on f i and f i+1 As input, f i+1 The features are upsampled to f i Same, f i+1 The corresponding weight is obtained through the mean and sigmoid functions, and finally the weight is combined with f i Multiplying, the formula is: f′=sigmoid(mean(f i +1))×f i Where f′ is f i Features with weights, finally fusion f′ and f i The final result is: f fuse =cat[f i ,f′] Among them, f fuse is the final fusion result.

6. The method for detecting damaged mural line drawings based on the uncertainty principle according to claim 5, characterized in that: The process of capturing the inherent noise in the training data based on data uncertainty is: Introducing uncertainty as noise in the data n(x i ), the objective function formula is: in, The model ultimately requires supervised learning results. is the input image, and the training process continuously reduces n(x i ) The influence of noise on the result can directly remove the damaged information in the image; In the decoding stage a decoder D U and output header H U To obtain the uncertainty σ of the data, another decoder D and prediction head H are used to generate the mean μ, the formula is: The obtained mean and variance are used to establish a corresponding learnable Gaussian distribution, the formula is: Among them, p(·) is the established Gaussian distribution, Gaussian function, μ i As a sample x i The prediction results, and σ i Represents μ i uncertainty.

7. The method for detecting damaged mural line drawings based on the uncertainty principle according to claim 6, characterized in that: It also includes training the established generative adversarial network model.

8. The method for detecting damaged mural line drawings based on the uncertainty principle according to claim 7, characterized in that: The class-balanced cross entropy loss function and variance-balanced loss function are introduced to train the line draft detection network model, which includes the following steps: The class-balanced cross entropy loss function is selected to solve the class imbalance problem. The formula is: Among them, Y is The corresponding true value, α and β are weights, Use the variance equalization loss function to speed up the training process of the stable line draft detection network model. The formula is: Construct a loss function to train the line draft detection network model. The loss function is: According to the constructed loss function, the line draft detection network model is iteratively trained until fitting, where the number of iterations is set to 20, the initial learning rate is set to 0.001, and after every 3 iterations, the learning rate is reduced using a linear attenuation strategy.

Images