An automatic marking method for cracks and falling defects in murals

Abstract

This invention discloses a method for automatically labeling cracks and flaking defects in murals, aiming to address the technical problem of automatically generating mural defect masks. The method includes the following steps: obtaining original mural images of varying sizes; converting the original mural images from RGB space to HSV color space; adjusting the weight coefficients of the H, S, and V components; extracting texture and line features from the mural; enhancing the defective areas of the mural and suppressing the line features; segmenting the defective areas and performing a brightness enhancement operation; connecting discontinuous crack pixels within the segmented defective areas; generating a mural defect mask; and adding the mask to the original mural image to obtain the final labeling result. The method accurately labels flaking and crack defects in murals, achieving excellent visual labeling results.

Description

Technical Field

[0001] The invention relates to an automatic marking method for cracks and falling defects in murals, and belongs to the technical field of ancient mural cultural relic protection. Background Art

[0002] Ancient murals are treasures of human cultural heritage, possessing immense historical, cultural, and artistic research value. However, due to both natural degradation and human influence, most murals have suffered varying degrees of damage. To safeguard this ancient cultural heritage over the long term, digitally capturing mural images and restoring damaged areas to restore their original appearance are crucial and crucial research areas.

[0003] The advent of computer technology has brought new vitality to the digital restoration of ancient murals. This process first requires marking damaged areas with a mask, then digitally restoring those masked areas. Currently, most methods rely on manual demarcation of damaged and damaged areas, but this process is extremely time-consuming and labor-intensive. Utilizing modern computer technology to automatically demarcate damaged and damaged areas in digital murals can provide accurate masks for digital restoration and offer a reference for actual mural restorers.

[0004] Currently, research on mural annotation is underway. Annotating cracks and peeling defects in murals will help researchers assess the severity of mural damage and lay the foundation for subsequent digital restoration of murals. In the field of automatic annotation of mural defects, there are currently three main methods:

[0005] (1) Methods based on morphological threshold segmentation: Wu et al. proposed the “Study on Multi-scale Marking and Restoration Technology of Tang Dynasty Tomb Mural Defects” in 2016. This method uses morphological high-hat and low-hat operators to design multi-scale structural elements to mark defects, but this method is mainly suitable for marking crack features in murals. Yang et al. proposed the “Automatic Identification and Repair of Mural Image Cracks Based on Improved SOM” in 2020. This method uses multi-scale morphological edge gradient detection to extract edge information, performs adaptive threshold segmentation on the transformed image, and measures the connectivity rule of the target area with area to remove false targets, thereby realizing automatic identification and labeling of damaged pixels. However, this method is suitable for images with obvious gradient transformation between the defect cracks and the mural background.

[0006] (2) Methods based on color clustering segmentation: Ren et al. proposed "Mural Restoration Based on Color Clustering Segmentation and Improved FMM Algorithm" in 2014, using color clustering in Lαβ space to segment the damaged area of ​​the mural. Zhang et al. proposed "Multi-scale Mural Restoration Method Based on Edge Reconstruction" in 2021, which smoothed the mural and then calculated the mask through color clustering. The segmentation method based on color clustering is suitable for murals with a large contrast between the color of the damaged area and the color of the mural background.

[0007] (3) Region growing-based methods: Cao et al. proposed the “Application of an Improved Region Growing Algorithm in Marking Defects of Temple Murals” in 2018. They used a region growing algorithm fused with threshold segmentation to mark the damaged areas of murals. However, this method is relatively complex and requires analysis of the threshold range. Jaidilert et al. proposed the “Crack Detection and Images Inpainting Method for Thai Mural Painting Images” in 2018. This method requires the user to provide some seed points and then perform region growing. Summary of the Invention

[0008] In order to solve the above technical problems, the present invention provides a method for automatically marking cracks and falling defects in murals, which can simultaneously mark cracks and falling defects in murals.

[0009] The technical solutions provided by the present invention are as follows:

[0010] A method for automatically marking cracks and falling defects in murals, characterized by comprising the following steps:

[0011] 1) Collect mural images, randomly crop original mural images of different sizes, and construct a dataset;

[0012] 2) Convert the original mural image in RGB space to HSV space. The mural image in HSV space is f(x,y) = [H(x,y),S(x,y),V(x,y)] T , where H, S, and V represent hue, saturation, and brightness components respectively; adjust the weight coefficients of the three components H, S, and V; weight image The specific formula is Among them, ω H 、ω S 、ω V weight coefficients corresponding to hue, saturation and brightness components respectively;

[0013] 3) Weight images The three components H, S, and V are partially differentiated in the x and y directions to obtain the gradient image Gθ (x,y):

[0014]

[0015] where g xx 、g yy and g xy are the parameters of the gradient image, and θ(x,y) is the weight image Angle function of the direction of maximum rate of change;

[0016] Select the gradient image G θ The maximum value of each pixel in (x,y) is used to obtain the result of multi-dimensional gradient detection:

[0017] p i =max(G θ (x,y))

[0018] Among them, p i It is the final output image obtained after the original mural image is subjected to multi-dimensional gradient detection. This image is a grayscale image containing the characteristics of the mural disease.

[0019] 4) Suppressing the multidimensional gradient map p i The drawing line features in need to satisfy the following relationship:

[0020]

[0021] Among them, q i is the output image, ω k is a local window centered at k, a k 、b k are all constant coefficients;

[0022] 5) Use the following formula to perform threshold segmentation to obtain the initial mask of the mural disease area:

[0023]

[0024] Among them, I mask Represents the initial mask of the mural disease area, q i is the output image, t represents the threshold;

[0025] 6) Mask I mask Perform morphological hole filling to obtain the final mask M mask , the mask M mask Add it to the original mural image to get the final annotation result.

[0026] Furthermore, the Sobel operator is used to weight the image The three components of are partially differentiated in the x and y directions to obtain vectors u1 and u2. The specific formula is as follows:

[0027]

[0028] Among them, h, s and v are unit vectors along the H, S and V axes of the HSV color space. and are the partial differentials of H, S and V components along the x direction, and are the partial differentials of the H, S and V components along the y direction respectively;

[0029] Calculate the dot product of vectors u1 and u2 respectively to get three parameters g xx 、g yy and g xy :

[0030]

[0031] The elements of θ(x,y) are the angles of each pixel used to calculate the gradient. The calculation formula of the angle function θ(x,y) is:

[0032]

[0033] Further, solve the linear coefficient a k ,b k While maintaining local linearity, the multidimensional gradient map p i and the output image q i Minimize the difference between the two, and use the loss function E(a k ,b k )express:

[0034]

[0035] Find the optimal solution a of the loss function k and b k Get the value of the local window, the specific expression is b k =(1-a k )μ k , a pixel i may belong to multiple windows, and the output image is obtained by averaging the overlapping windows. The specific calculation formula is as follows:

[0036]

[0037] in and A k and b k The mean of .

[0038] Furthermore, the Otsu threshold segmentation method is used to segment the output image q i Segmented into foreground and background, the threshold t at this time is the optimal threshold:

[0039]

[0040] Among them, P f (t) and P b (t) are the proportions of foreground and background in the mural diseased area after threshold segmentation, m f (t) and m b (t) are the mean values ​​of the pixels in the foreground and background of the mural diseased area, and m is the mean value of the pixels in the entire image q i The mean value of the pixels in the image q, L represents the gray level, i Compared with the threshold, pixels with grayscale values ​​greater than the threshold remain unchanged and are foreground pixels of the mural diseased area; pixels with grayscale values ​​less than the threshold are set to 0 and are background pixels of the mural diseased area.

[0041] Furthermore, the initial mask I of the mural disease area mask After tensor voting, a new tensor is obtained. Since tensor voting is used to connect and expand the discontinuous crack pixels in the initial mask, the stick tensor saliency map is extracted as the result. Finally, the stick tensor saliency map and the disease feature mask I are combined. mask The union is calculated to obtain the defect feature mask with complete cracks.

[0042] Furthermore, the final mask M mask Add to the original mural f(x,y) to get the final annotation result f c , the specific calculation formula is as follows:

[0043]

[0044] The beneficial effects of the present invention are as follows:

[0045] The automatic labeling method proposed in this invention can simultaneously mark cracks and peeling defects in murals. Compared with existing technologies, this method has the following advantages:

[0046] 1. The labeling algorithm of the present invention can accurately label defects that are similar in color to the background color of the mural.

[0047] 2. When the color of the cracks is similar to the painting lines of the mural, the cracks in the mural can also be accurately marked.

[0048] 3. Able to mark small area defects in murals.

[0049] 4. The labeling method of the present invention can be applied to mural images of different tones.

[0050] 5. Fast calculation speed, high efficiency, and no need for human-computer interaction. The labeling effect of this invention on multiple mural images is better than other methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is an example diagram of cracks and falling defects in murals in the practice of the present invention;

[0052] Figure 2 This is a flow chart of automatic marking of cracks and falling defects in murals in a specific embodiment of the present invention;

[0053] Figure 3 It is the annotation result applied to the crack disease example in the implementation of the present invention;

[0054] Figure 4 These are the labeling results applied to various disease examples in the implementation of the present invention. DETAILED DESCRIPTION

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] The present invention Figure 1 Shown are two types of damage to the original appearance of the mural, Figure 1 (a) is a crack, Figure 1 (b) is a falling-off disease. The automatic marking method of the mural cracks and falling-off diseases of the present invention is as follows: Figure 2 As shown, the following steps are included:

[0057] The first step is to collect mural images, randomly crop mural images of different sizes, and construct the data set required for the experiment. Random cropping is used to generate images of different sizes, thereby increasing the number of images and expanding the data set.

[0058] The second step is to convert the original mural image in RGB space to HSV color space.

[0059] Convert the mural image in RGB color space to HSV color space, and get the mural image in HSV space f(x,y)=[H(x,y),S(x,y),V(x,y)] T , where H, S, and V represent the hue, saturation, and brightness components respectively.

[0060] The third step is to adjust the weight coefficients of H, S, and V.

[0061] Adjust the weight coefficients of the three components of hue, saturation and brightness in the HSV color space to obtain the weighted image The specific formula is as follows:

[0062]

[0063] Among them, ω H 、ω S 、ω V The weight coefficients corresponding to hue, saturation and brightness components are taken as ω H =0.1,ω S =0.8,ω V =0.1.

[0064] The fourth step is to extract the disease characteristics of the mural

[0065] Use Sobel operator to weight images The three components of are partially differentiated in the x and y directions to obtain vectors u1 and u2. The specific formula is as follows:

[0066]

[0067] Among them, h, s and v are unit vectors along the H, S and V axes of the HSV color space. and are the partial differentials of H, S and V components along the x direction, and are the partial differentials of the H, S and V components along the y direction. Next, we calculate the dot product of vectors u1 and u2 to get three parameters g xx 、g yy and g xy :

[0068]

[0069] Weight Image The direction of maximum rate of change is given by the angle function θ(x,y):

[0070]

[0071] Among them, the elements of θ(x,y) are the angles of each pixel used to calculate the gradient. In this direction, the gradient image G θ (x,y) is given by:

[0072]

[0073] In the process of extracting the features of mural disease, the gradient image G is selected θThe maximum value of each pixel in (x,y) is used to obtain the result of multi-dimensional gradient detection:

[0074] p i =max(G θ (x,y))

[0075] Among them, p i It is the final output image obtained after the original mural image undergoes multi-dimensional gradient detection. The image is a grayscale image containing the characteristics of mural disease.

[0076] Step 5: Enhance the damaged area of ​​the mural

[0077] Suppressing the multidimensional gradient map p i The drawing line features in need to satisfy the following relationship:

[0078]

[0079] Among them, q i is the output image, ω k is a local window centered at k, a k 、b k are all constant coefficients.

[0080] The output image and the multidimensional gradient map satisfy a local linear relationship. Solve the linear coefficient a k ,b k While maintaining local linearity, the multidimensional gradient map p i and the output image q i Minimize the difference between the two, and use the loss function E(a k ,b k )express:

[0081]

[0082] Find the optimal solution a of the loss function k and b k Get the value of the local window, the specific expression is b k =(1-a k )μ k , a pixel i may belong to multiple windows, and the output image is obtained by averaging the overlapping windows. The specific calculation formula is as follows:

[0083]

[0084] in and A k and b k The final output image q that suppresses the lines of the mural painting is obtainedi , achieving the enhancement of the diseased areas of the mural.

[0085] Step 6: Automatically segment the damaged area in the mural and perform brightness enhancement

[0086] The Otsu threshold segmentation method is used to obtain the initial mask of the mural disease area. The Otsu threshold segmentation algorithm can be used to i Segmented into foreground and background, the threshold t at this time is the optimal threshold:

[0087]

[0088] Among them, P f (t) and P b (t) are the proportions of foreground and background in the mural diseased area after threshold segmentation, m f (t) and m b (t) are the mean values ​​of the pixels in the foreground and background of the mural diseased area, and m is the mean value of the pixels in the entire image q i The mean value of the pixels in , L represents the gray level.

[0089] The image q i Compare with the threshold. Pixels with grayscale values ​​greater than the threshold remain unchanged and are considered foreground pixels in the mural's diseased area. Pixels with grayscale values ​​less than the threshold are set to 0 and are considered background pixels in the mural's diseased area. Threshold segmentation is used to separate the mural's diseased area. The following formula is used for threshold segmentation to calculate the initial mask for the mural's diseased area:

[0090]

[0091] Among them, I mask Represents the initial mask of the mural disease area, q i is the output image of the sixth step, and t represents the threshold. At the same time, the brightness enhancement operation is performed on the disease feature mask.

[0092] Step 7: Connect the discontinuous crack pixels in the image after threshold segmentation

[0093] First, the initial mask I of the mural disease area mask After tensor voting, a new tensor is obtained. Secondly, since tensor voting is used to connect and expand the discontinuous crack pixels in the initial mask, the stick tensor saliency map is extracted as the result. Finally, the stick tensor saliency map and the disease feature mask I are combined. mask The union is calculated to obtain the defect feature mask with complete cracks.

[0094] Step 8: Generate a mask for mural damage

[0095] The morphological holes of the complete crack damage feature mask are filled to obtain the final mask M mask .

[0096] Step 9: Add the final mask to the original mural image to get the final annotation result

[0097] The final mask M mask Add to the original mural f(x,y) to get the final annotation result f c , the specific calculation formula is as follows:

[0098]

[0099] like Figure 3 The following is the result of the annotation of the present invention for the crack disease example, where Figure 3 (a) is an image of a mural with cracks. It can be seen that the cracks in the image are very similar in color to the lines of the mural. Figure 3 (b) is the final mask generated, Figure 3 (c) is the final annotation result. Figure 3 (c) It can be seen that the labeling effect of the present invention is good. Even if the cracks are very similar in color to the painted lines of the mural, the cracks can be accurately labeled, and the labeling results of the detached areas are also very complete.

[0100] like Figure 4 Shown are the labeling results of the present invention for various disease examples. Figure 4 (a1) to (a2) are original mural images. Figure 4 Figures (b1) to (b2) show the results of the present invention marking different mural defects. As can be seen from the figures, even when the mural's peeling defect is similar in color to the mural's background and the defect is complex, the present invention can still accurately mark the defect area.

[0101] The present invention achieves rapid computation and high accuracy, fully labeling cracks and areas of detachment in murals. The present invention has been tested on various images, achieving significantly better results than other labeling methods. The present invention can automatically label cracks and detachment in input mural images, with the resulting labeling showing that both cracks and detachment areas are fully labeled. This method can be applied to applications such as road crack detection, solar panel defect detection, and crop disease detection.

[0102] The above embodiments are only used to illustrate the present invention, and are not intended to limit the present invention. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention. The scope of patent protection of the present invention should be defined by the claims.

Claims

1. A method for automatically marking cracks and falling defects in murals, characterized by: The following steps are involved: 1) Collect mural images, randomly crop original mural images of different sizes, and construct a dataset; 2) Convert the original mural image in RGB space to HSV space. The mural image in HSV space is f(x,y) = [H(x,y),S(x,y),V(x,y)] T , where H, S, and V represent hue, saturation, and brightness components respectively; adjust the weight coefficients of the three components H, S, and V; weight image The specific formula is Among them, ω H 、ω S 、ω V weight coefficients corresponding to hue, saturation and brightness components respectively; 3) Weight images respectively The three components H, S, and V are partially differentiated in the x and y directions to obtain the gradient image G θ (x,y) is given by: Select the gradient image G θ The maximum value of each pixel in (x,y) is used to obtain the result of multi-dimensional gradient detection: p i =max(G θ (x,y)) Among them, p i It is the final output image obtained after the original mural image is subjected to multi-dimensional gradient detection. This image is a grayscale image containing the characteristics of the mural disease. 4) Suppressing the multidimensional gradient map p i The drawing line features in need to satisfy the following relationship: Among them, q i is the output image, ω k is a local window centered at k, a k 、b k are all constant coefficients; 5) Use the following formula to perform threshold segmentation to obtain the initial mask of the mural disease area: Among them, I mask Represents the initial mask of the mural disease area, q i is the output image, t represents the threshold; 6) Mask I mask Perform morphological hole filling to obtain the final mask M mask , the mask M mask Add it to the original mural image to get the final annotation result.

2. The automatic marking method for cracks and falling defects in murals according to claim 1, characterized in that: Use Sobel operator to weight images The three components of are partially differentiated in the x and y directions to obtain vectors u1 and u2. The specific formula is as follows: Among them, h, s and v are unit vectors along the H, S and V axes of the HSV color space. and are the partial differentials of H, S and V components along the x direction, and are the partial differentials of the H, S and V components along the y direction respectively; Calculate the dot product of vectors u1 and u2 respectively to get three parameters g xx 、g yy and g xy : The elements of θ(x,y) are the angles of each pixel used to calculate the gradient. The calculation formula of the angle function θ(x,y) is:

3. The automatic marking method for cracks and falling defects in murals according to claim 1, characterized in that: The weight coefficient ω H =0.1,ω S =0.8,ω V =0.

1.

4. The automatic marking method for cracks and falling defects in murals according to claim 1, characterized in that: Solve for the linear coefficient a k ,b k While maintaining local linearity, the multidimensional gradient map p i and the output image q i Minimize the difference between the two, and use the loss function E(a k ,b k )express: Find the optimal solution a of the loss function k and b k Get the value of the local window, the specific expression is A pixel i may belong to multiple windows, and the output image is obtained by averaging the overlapping windows. The specific calculation formula is as follows: in and A k and b k The mean of .

5. The automatic marking method for cracks and falling defects in murals according to claim 1, characterized in that: The output image q is segmented using the Otsu threshold method. i Segmented into foreground and background, the threshold t at this time is the optimal threshold: Among them, P f (t) and P b (t) are the proportions of foreground and background in the mural diseased area after threshold segmentation, m f (t) and m b (t) are the mean values ​​of the pixels in the foreground and background of the mural diseased area, and m is the mean value of the pixels in the entire image q i The mean value of the pixels in the image q, L represents the gray level, i Compared with the threshold, pixels with grayscale values ​​greater than the threshold remain unchanged and are foreground pixels of the mural diseased area; pixels with grayscale values ​​less than the threshold are set to 0 and are background pixels of the mural diseased area.

6. The automatic marking method for cracks and falling defects in murals according to claim 1, characterized in that: The initial mask I of the mural disease area mask After tensor voting, a new tensor is obtained. Since tensor voting is used to connect and expand the discontinuous crack pixels in the initial mask, the stick tensor saliency map is extracted as the result. Finally, the stick tensor saliency map and the disease feature mask I are combined. mask The union is calculated to obtain the defect feature mask with complete cracks.

7. The automatic marking method for cracks and falling defects in murals according to claim 1, characterized in that: The final mask M mask Add to the original mural f(x,y) to get the final annotation result f c , the specific calculation formula is as follows:

Images