Mural painting disease annotation method based on gradient trainable gabor and channel attention
By combining gradient-trainable Gabor filters and channel attention mechanisms with U-Net networks, the problem of automatic labeling of flaking defects in murals was solved, achieving efficient and accurate automatic labeling of flaking defects in murals and reducing manual intervention.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHWEST UNIV
- Filing Date
- 2022-11-24
- Publication Date
- 2026-04-21
AI Technical Summary
In existing technologies, it is difficult to automatically label flaking defects in murals, and manual labeling results are inconsistent and time-consuming.
A method for labeling mural flaking defects based on gradient-trainable Gabor filters and channel attention is adopted. The method extracts the texture features of the mural through gradient-trainable Gabor filters and introduces a channel attention mechanism in the decoder part. Combined with U-Net network and residual convolution structure, end-to-end automatic labeling is achieved.
Automatic labeling of flaking defects in murals was achieved, improving the accuracy and efficiency of labeling, reducing manual intervention, enhancing the ability to recognize texture information, and narrowing the semantic gap between the encoder and decoder.
Smart Images
Figure CN115937582B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, specifically relating to a method for annotating nail-like defects in murals based on gradient-trainable Gabor and channel attention. Background Technology
[0002] China's long history and rich culture have given birth to many cultural heritages with unique value. Cultural relics occupy an important position in the nation's cultural development, and the rich information they contain reflects the characteristics and culture of their era. Strengthening the scientific protection of cultural relics is of great significance for inheriting my country's traditional culture and enhancing national confidence. Murals, as one of China's important cultural relics, are carriers of the continuation of Chinese traditional culture and national spirit, possessing significant historical and artistic value. The overall structure of a mural consists of a support structure, a ground layer, a coarse clay layer, a fine clay layer, a base color layer, a pigment layer, and a surface coating. Due to changes in the natural environment and human damage, murals have suffered varying degrees of damage. Common mural defects include smoke damage, cracks, crazing, flaking, and pigment peeling. These defects not only weaken the mural's expressive power and affect its aesthetic appeal but also have a severely detrimental impact on the preservation of historical culture and the transmission of art. To slow down the rate of mural damage, experts are attempting to restore the murals to their original appearance while preserving their current state. Because murals are non-renewable cultural relics, experts must strictly adhere to the principle of minimal intervention during their protection. Image processing, as a non-destructive processing technique, has been applied in the field of mural conservation and has achieved good results.
[0003] To determine the best protection and restoration plan, the condition of the murals must first be studied. Therefore, marking mural damage is an indispensable part of its scientific protection, providing guidance and a basis for scientific restoration and long-term monitoring. Currently, non-contact mural damage marking is mainly performed using digital images and software such as Geographic Information Systems (GIS), AutoCAD, and Photoshop, allowing non-conservation professionals to complete the work. However, these methods are not only time-consuming, but also prone to variations in accuracy due to differences in expert experience. Summary of the Invention
[0004] In view of the rich texture information contained in mural flaking defects, this invention provides a mural flaking defect annotation method based on gradient trainable Gabor and channel attention, which solves the problem of difficulty in automatically annotating mural flaking defects in the prior art.
[0005] To achieve the above objectives, the present invention adopts the following technical solution:
[0006] A method for annotating mural flaking defects based on gradient-trainable Gabor and channel attention includes the following steps:
[0007] S1: Obtain images of ancient murals containing nail-like defects to create a training set.
[0008] S2: Input the training set into the feature extraction network, and obtain the feature maps F by training Gabor and classical convolution through gradient descent. g and feature map F c ;
[0009] S3: Transfer the feature map F g and feature map F c F is obtained by superimposing the results using encoders. gc ;
[0010] S4: Decoder for the F gc Incorporating a channel attention mechanism enhances the connection between channel information in each feature map, resulting in a segmentation prediction map R. s ;
[0011] S5: Obtain the predicted map R s Set of areas affected by nail flaking disease R e , where R e =edge(R s ), edge(·) is the edge extraction algorithm;
[0012] S6: Set the affected areas of the scale disease into R e Image fusion was performed with the original mural image to obtain a set of marked areas of flaking disease in the mural image.
[0013] Furthermore, in S2, the gradient-trainable Gabor function is obtained by modifying a Gabor filter with fixed parameters, and the original Gabor function is as shown in equation (1):
[0014]
[0015] In the formula, x′=xcos+ysin, y′=-xsin+ycos, f,θ,ψ,σ,γ are the parameters of the Gabor function, σ and γ are the scaling factor and spatial aspect ratio of the Gaussian function, respectively, f and ψ are the frequency and phase offset of the cosine carrier wave, θ is the direction of the Gabor function, and x and y are the horizontal and vertical coordinate variables, respectively. It is a constant factor;
[0016] The kernel function K of a gradient-trainable Gabor is defined as follows:
[0017]
[0018] Where i and j are pixel indices, i∈(0,1,…,h-1), j∈(0,1,…,w-1), and h and w are the height and width of the Gabor kernel, respectively. This represents the rate of change in intensity of the center pixel. The local gradient of the kernel function k*k is represented by the kernel function K, which is established based on the learnable parameters σ, γ, f, ψ, θ and formulas (1) and (2).
[0019] Gradient-trainable Gabor filters consist of Gabor filters in d directions, where each Gabor filter is represented as G(x,y; θ). k ), k = 0, 1, ..., d-1, where ψ is set to 0 in the learnable parameters, and the direction angle of the k-th Gabor filter is set to
[0020] Furthermore, in S2, classical convolution yields feature map F. c The process is as follows: input the images I of the training set. in The feature map F is obtained by passing the input through a convolutional layer with a kernel size of 3*3. c ,Right now
[0021] F c =Conv 3*3 (I in (3)
[0022] Among them, Conv 3*3 (·) refers to a convolution operation with a kernel size of 3*3.
[0023] Furthermore, in S2, the feature extraction network adopts U-Net as its basic architecture, specifically as Conv11, Conv12, Maxpooling2*2, Conv21, Conv22, Maxpooling2*2, Conv31, Conv32, Maxpooling2*2, Conv41, Conv42, Up-conv2*2, Deconv31, Deconv32, Up-conv2*2, Deconv21, Deconv22, Up-conv2*2, Deconv11, Deconv12, Conv1*1 & Sigmoid. The convolutions all use 3*3 kernel size convolutions, downsampling uses 2*2 max pooling, and upsampling uses 2*2 deconvolution.
[0024] Furthermore, in step S3, the feature map F g and feature map F c Superimposing them together via encoders means:
[0025] The first two layers of the encoder network use a stacking strategy for the first convolution Conv11 and Conv21 of each layer, while the remaining coding layers use the U-Net network structure.
[0026] Furthermore, in step S3, the codec uses residual skip connections to superimpose feature maps, specifically including:
[0027] The results of the first convolution of each layer of the encoder, namely Conv11, Conv21, and Conv31, are used as inputs. The output is obtained by residual convolution with a kernel size of 3*3. The output is then superimposed on the feature map before the first convolution of each layer of the decoder.
[0028] Furthermore, in S4, the channel attention position is after the second convolution in each layer of the decoder, that is, after Deconv32, Deconv22, and Deconv12. The specific steps of channel attention include:
[0029]
[0030] Where σ is the sigmoid function, MLP is a multilayer perceptron, AvgPool and MaxPool represent global average pooling and global max pooling, respectively, W0 and W1 share the weights of the MLP, and F is the feature map obtained by the convolution before the channel attention as indicated in the network graph, which is used as the input to the channel attention. avg F represents the feature map obtained through AvgPool. max This represents the feature map obtained through MaxPool.
[0031] Furthermore, in S5, the edge extraction algorithm is the Canny operator edge detection method.
[0032] Furthermore, S1 also includes a test set, which and the training set are obtained by dividing the mural images according to a preset ratio, and the training set and the test set contain the real labels corresponding to each image;
[0033] The test set is obtained through steps S2 to S6 to create a set of annotated areas for flaking disease in the mural image, which is used to test the annotation effect.
[0034] Compared with the prior art, the beneficial technical effects of this invention are:
[0035] (1) In view of the rich texture information of mural flaking disease, this invention proposes an end-to-end mural flaking disease annotation network, which realizes automatic annotation of mural flaking disease without manual intervention.
[0036] (2) This invention proposes a new gradient trainable Gabor to extract texture features of mural flaking disease and enhance the network’s ability to recognize texture information; the decoder part introduces a channel attention mechanism to increase the network’s attention to each channel of the feature; in order to reduce the semantic gap between the U-Net network encoder and decoder, a residual convolution structure is added at the skip connection to reduce the semantic gap between the encoder and decoder, and further introduces the texture information of the encoder stage into the decoder, thereby improving the network’s ability to extract texture features. Attached Figure Description
[0037] Figure 1 This is a network framework diagram provided by the present invention;
[0038] Figure 2 This is an example diagram of the segmentation result of flaking disease in a mural provided by the present invention;
[0039] Figure 3 This is an example diagram of the final annotation result of mural flaking disease provided by the present invention. Detailed Implementation
[0040] This invention discloses a method for annotating flaking defects in murals based on gradient-trainable Gabor filters and channel attention. Addressing the characteristic of numerous texture features in mural flaking defects, this method employs gradient-trainable Gabor filters to extract texture features from the input image in the first two layers. A channel attention mechanism is introduced in the decoder part of the network to increase the network's attention to the various channels of the features. To reduce the semantic gap between the U-Net network encoder and decoder, this invention improves the skip connections by adding residual convolutional structures at the skip connections to narrow the semantic gap between the encoder and decoder. It also further incorporates texture information from the encoder stage into the decoder, improving the network's ability to extract texture features.
[0041] The following is combined with Figure 1-3 The present invention provides a more detailed description of the mural nailing disease annotation method based on gradient trainable Gabor and channel attention, which specifically includes the following steps:
[0042] Step 1: Obtain images of ancient murals containing nail-like defects. Divide the mural dataset into training and testing sets according to a preset ratio. The training and testing sets contain the real labels corresponding to each image.
[0043] Step 2: Input the training and test sets into the feature extraction network. After passing through gradient-trainable Gabor and classical convolution, the feature maps F are obtained. g and feature map F cSpecifically, the gradient-trainable Gabor function is obtained by modifying a Gabor filter with fixed parameters, and the original Gabor function is shown in equation (1):
[0044]
[0045] In the formula, x′=xcos+ysin, y′=-xsin+ycos, f,θ,ψ,σ,γ are the parameters of the Gabor function, σ and γ are the scaling factor and spatial aspect ratio of the Gaussian function, respectively, f and ψ are the frequency and phase offset of the cosine carrier wave, θ is the direction of the Gabor function, and x and y are the horizontal and vertical coordinate variables, respectively. It is a constant factor;
[0046] The kernel function K of a gradient-trainable Gabor is defined as follows:
[0047]
[0048] Where i and j are pixel indices, i∈(0,1,…,h-1), j∈(0,1,…,w-1), and h and w are the height and width of the Gabor kernel, respectively. This represents the rate of change in intensity of the center pixel. The local gradient of the kernel function k*k is represented by the kernel function K, which is established based on the learnable parameters σ, γ, f, ψ, θ and formulas (1) and (2). The gradient can be trained by Gabor filters in d directions, where the Gabor filter is represented as G(x, y; θ). k ), k = 0, 1, ..., d-1. In the learnable parameters, ψ is set to 0, and the direction angle of the k-th Gabor filter is set to... Uppercase K represents the function name, and lowercase k represents the receptive field of the function (e.g., the receptive field of a 3*3 convolution is a square with a length and width of 3) or the k-th filter (k determines the number of filters).
[0049] Classical convolution yields F c The process is as follows: input images from the training set and the test set I in The input is passed through a convolutional layer with a kernel size of 3*3 to obtain F. c ,Right now
[0050] F c =Conv 3*3 (I in (3)
[0051] Among them, Conv 3*3 (·) refers to a convolution operation with a kernel size of 3*3;
[0052] The feature extraction network uses U-Net as its basic architecture, specifically consisting of Conv11, Conv12, Maxpooling2*2, Conv21, Conv22, Maxpooling2*2, Conv31, Conv32, Maxpooling2*2, Conv41, Conv42, Up-conv2*2, Deconv31, Deconv32, Up-conv2*2, Deconv21, Deconv22, Up-conv2*2, Deconv11, Deconv12, Conv1*1 & Sigmoid. All convolutions use a 3*3 kernel size. Downsampling uses 2*2 max pooling, and upsampling uses 2*2 deconvolution. The specific structure is shown in the attached figure. Figure 1 As shown.
[0053] Step 3: Transfer the feature map F obtained in Step 2 to... g and feature map F c F is obtained by superimposing the encoders together. gc The features F are used as input to the encoder for convolution; specifically, the features F are used as input to the encoder for convolution. g and F c Superimposed together means:
[0054] The feature map F obtained from gradient-trainable Gabor g Feature map F after 3x3 convolution c F is obtained by superimposing on the channels. gc The first convolution of each of the first two layers of the encoder network (i.e., Conv11 and Conv21) uses a stacking strategy, while the remaining coding layers use the U-Net network structure. The specific strategy is shown in the appendix. Figure 1 As shown;
[0055] The codec employs residual skip connections for feature map stacking; specifically, it includes:
[0056] The results of the first convolution of each layer of the encoder, namely Conv11, Conv21, and Conv31, are used as input. The output is obtained through residual convolution with a kernel size of 3*3. This output is then stacked channel-wise with the feature maps from before the first convolution of each layer of the decoder. The specific stacking method is shown in the attached figure. Figure 1 As shown.
[0057] Step 4: Decoder part for F gc Incorporating a channel attention mechanism enhances the connection between channel information in each feature map, ultimately yielding a segmentation prediction map R. s Specifically, as shown in the appendix Figure 1As shown, the channel attention position is after the second convolution in each layer of the decoder, that is, after Deconv32, Deconv22, and Deconv12. The specific steps of channel attention include:
[0058]
[0059] Where σ is the sigmoid function, MLP is a multilayer perceptron, AvgPool and MaxPool represent global average pooling and global max pooling, respectively, W0 and W1 share the weights of the MLP, and F is the feature map obtained by the convolution before the channel attention as indicated in the network graph, which is used as the input to the channel attention. avg F represents the feature map obtained through AvgPool. max This represents the feature map obtained through MaxPool.
[0060] Step 5: Obtain the prediction map R s Set of areas affected by nail flaking disease R e , where R e =edge(R s ), edge(·) is the edge extraction algorithm; specifically, the edge extraction algorithm is the Canny operator edge detection method.
[0061] Step 6: Perform image fusion with the set of flaking disease areas obtained in Step 5 and the original image to obtain the set of marked flaking disease areas of the mural.
[0062] This embodiment uses a test set from the murals in the Daxiong Hall of Fengguo Temple in Jinzhou, Liaoning Province, as data to verify the effectiveness of the proposed mural flaking disease annotation method. The experiment uses the PyTorch software package to train the network, performing approximately 500 iterations with a batch size of 16. This embodiment mainly compares the annotation results with those of the U-Net network.
[0063] Analysis of experimental results:
[0064] Figure 2 Example diagram showing the segmentation results of flaking disease in murals; black areas represent non-flaking areas, and white areas represent flaking disease areas; among them... Figure 2 (a) is the original image of the mural. Figure 2 (b) is the reference segmentation result. Figure 2 (c) shows the U-Net network segmentation results. Figure 2 (d) shows the segmentation result of the method of the present invention; by visually comparing the reference segmentation result and the segmentation results of the two methods, it can be seen that both methods can segment the area of the mural with flaking disease, but it is found that... Figure 2In (c), the U-Net method segmentation result in the lower left corner of the image separates some nail-like disease lesions into non-nail-like lesions, which differs significantly from the reference image. Figure 2 (d) The segmentation result of the present invention has the smallest difference from the reference segmentation result, and the flaking disease of the mural is automatically marked. Figure 3 The final annotation results of the method of the present invention are shown. Among them, Figure 3 (a) is the original image of the mural. Figure 3 (b) is the reference annotation result. Figure 3 (c) shows the annotation results of the U-Net network. Figure 3 (d) is the labeling result of the method of the present invention; the area enclosed by the line is the non-flaking area, and the rest is the flaking disease area.
[0065] Relying solely on subjective evaluation makes it difficult to provide the most accurate judgment on the results of mural flaking. Therefore, it is necessary to combine objective indicators to evaluate the annotation results. This invention uses Jaccard Similarity and Dice Coefficient, two objective evaluation indicators with high reference value in the field of image segmentation, to comprehensively evaluate the annotation results. Jaccard Similarity, also known as the Jaccard similarity coefficient or intersection-union ratio, is used to compare the similarity and differences between a finite set of samples. The larger the Jaccard coefficient value, the higher the sample similarity. Dice Coefficient is also used to measure the similarity between two sets and is an objective standard for evaluating images. Both indicators range from 0% to 100%. The closer the indicator is to 100%, the higher the similarity between the two images.
[0066] Table 1 Objective Indicators of Mural Skin Aging Disease Marking Results
[0067]
[0068]
[0069] Table 1 shows the objective indicators of the labeling results of the flaking disease in the murals of Fengguo Temple. As can be seen from Table 1, the effectiveness of each module of the present invention has been verified. All objective indicators are better than the U-Net method. That is, the labeling results of the flaking disease in the murals generated by the method of the present invention have the highest similarity with the reference labeling results. It can automatically label the flaking disease in the murals very well, and realize the labeling of flaking disease in the murals with high accuracy without human intervention.
[0070] Based on a combination of visual and objective evaluation, the method proposed in this invention can effectively mark the nail-like defects in ancient murals.
Claims
1. A mural mold disease annotation method based on gradient trainable Gabor and channel attention, characterized by, Includes the following steps: S1: Obtain images of ancient murals containing nail-like defects to create a training set. S2: input the training set into a feature extraction network, and obtain feature maps respectively through gradient trainable Gabor and classic convolution and feature maps ; The gradient-trainable Gabor function is obtained by modifying a Gabor filter with fixed parameters. The original Gabor function is shown in equation (1): (1) in the formula , These are the parameters of the Gabor function. and These are the scaling factor and spatial aspect ratio of the Gaussian function, respectively. and These are the frequency and phase offset of the cosine carrier wave, respectively. It is the direction of the Gabor function. and These are the horizontal and vertical coordinate variables, respectively. It is a constant factor; Kernel function of gradient trainable Gabor is defined as follows: (2) wherein and are pixel indices, , and are height and width of the Gabor kernel, respectively, denotes the rate of intensity change of the center pixel, denotes the local gradient of the kernel function based on learnable parameters and the kernel function is established by equations (1) and (2) , Gradient-trainable Gabor includes Gabor filters in directional directions, where the Gabor filter is represented as... Learnable parameters will Set it to 0, and set the number of... The direction angle of each Gabor filter is set to... ; S3: the feature map and feature map are stacked together by the encoder to obtain ; S4: Decoder for the Incorporating a channel attention mechanism enhances the connection between channel information in each feature map, resulting in a segmentation prediction map. ; S5: obtaining the prediction map A set of regions of the disease wherein , is an edge extraction algorithm S6: obtain the set of the initial mold disease area of the mural image Image fusion is performed with the original mural image to obtain a set of the initial mold disease area of the mural image.
2. The mural blight disease annotation method based on gradient trainable Gabor and channel attention according to claim 1, wherein, In S2, the classical convolution obtains the feature map The process is as follows: the input image of the training set As input, a convolution layer with a convolution kernel size of 3*3 obtains a feature map That is (3) in, This refers to a convolution operation with a kernel size of 3*3.
3. The mural blight disease annotation method based on gradient trainable Gabor and channel attention according to claim 2, wherein, In S2, the feature extraction network uses U-Net as its basic architecture, specifically Conv11, Conv12, Maxpooling2*2, Conv21, Conv22, Maxpooling2*2, Conv31, Conv32, Maxpooling2*2, Conv41, Conv42, Up-conv2*2, Deconv31, Deconv32, Up-conv2*2, Deconv21, Deconv22, Up-conv2*2, Deconv11, Deconv12, Conv1*1 & Sigmoid. The convolutions all use 3*3 kernel size convolutions, downsampling uses 2*2 max pooling, and upsampling uses 2*2 deconvolution.
4. The method for annotating mural flaking defects based on gradient-trainable Gabor and channel attention as described in claim 3, characterized in that, In S3, the feature maps and feature maps are stacked together by the encoder. The first two layers of the encoder network use a stacking strategy for the first convolution Conv11 and Conv21 of each layer, while the remaining coding layers use the U-Net network structure.
5. The mural blight disease annotation method based on gradient trainable Gabor and channel attention according to claim 4, wherein, In step S3, the encoder uses residual skip connections to superimpose feature maps, specifically including: The results of the first convolution of each layer of the encoder, namely Conv11, Conv21, and Conv31, are used as inputs. The output is obtained by residual convolution with a kernel size of 3*3. The output is then superimposed on the feature map before the first convolution of each layer of the decoder.
6. The mural blight disease annotation method based on gradient trainable Gabor and channel attention according to claim 5, wherein, In step S4, the channel attention is located after the second convolution in each layer of the decoder, that is, after Deconv32, Deconv22, and Deconv12. The specific steps of channel attention include: (4) wherein is a sigmoid function, MLP is a multi-layer perceptron, and represent global average pooling and global max pooling, respectively, and share the weights of MLP, F is the feature map obtained by the convolution marked in the network graph as the input of channel attention, F avg represents the feature map obtained by , F max represents the feature map obtained by .
7. The mural mold disease annotation method based on gradient trainable Gabor and channel attention according to claim 1, wherein, In S5, the edge extraction algorithm is the Canny operator edge detection method.
8. The method for labeling mural flaking disease based on gradient trainable Gabor and channel attention as described in claim 1, wherein step S1 further includes a test set, the test set and the training set are obtained by dividing the mural images according to a preset ratio, and the training set and the test set contain the real labels corresponding to each image; The test set is obtained through steps S2 to S6 to create a set of annotated areas for flaking disease in the mural image, which is used to test the annotation effect.