A method for identifying diseases of calligraphy and painting cultural relics based on object detection technology

Abstract

The present invention discloses a method for identifying diseases of calligraphy and painting cultural relics based on object detection technology, which includes the following steps: data preprocessing: obtaining dataset pictures of calligraphy and painting cultural relics to be detected, performing manual preprocessing on the obtained dataset pictures of calligraphy and painting cultural relics, and performing data augmentation on the processed dataset pictures; model training: establishing a calligraphy and painting cultural relics model based on self-adversarial training, CmBN strategy, Dropblock regularization and loss function, and training the calligraphy and painting cultural relics model through the processed dataset pictures; picture detection and localization cropping: inputting the dataset pictures of calligraphy and painting cultural relics to be detected into the trained model, performing disease detection in a sliding window manner, analyzing the dataset pictures layer by layer, and performing localization cropping on the identified viruses. Through the object detection technology of deep learning, the identification of diseases of calligraphy and painting cultural relics is realized.

Description

Technical Field

[0001] The present invention relates to the fields of object detection and calligraphy and painting cultural relic restoration, and particularly relates to a method for identifying diseases of calligraphy and painting cultural relics based on object detection technology. Background Art

[0002] With the improvement of the concept of historical and cultural heritage protection in China over the years and the increasing attention of governments at all levels, relevant protection measures for calligraphy and painting cultural relics have been gradually implemented and have achieved great development in recent years. In China, the disease detection work of calligraphy and painting cultural relics is still in the early stage of development, and its related identification methods still rely on manual work. Restoration technicians analyze and identify the material status of calligraphy and painting cultural relics through chemical or biological means, and hand-draw disease area maps or mark diseases on photos. It is of great significance to fully recognize the advantages of scientific and technological means and promote the integration and improvement of cultural relic protection with intelligent technologies. Therefore, the protection of calligraphy and painting cultural relics is crucial for the country and social and cultural construction.

[0003] In today's society, in response to the relevant policies of the state for the protection of calligraphy and painting cultural relics, various research institutes and museums have carried out a series of extensive sorting work on a large number of ancient books and paintings. With the continuous improvement of living standards and the popularization of art aesthetic awareness, the public has a high interest in collecting ancient calligraphy and painting works or other paper art works. Genuine ancient calligraphy and paintings are not easy to preserve, with a small number in existence, and are scarce and non-renewable. For example, there are only about 1,200 surviving Song Dynasty paintings, and only 1,106 of the 6,400 calligraphy and paintings recorded in the "Xuanhe Painting Catalogue" are currently in China. At the same time, due to the special composition structure of paper cultural relics, they are difficult to protect. In addition to being easily damaged by insect bites and mildew, their fibers will also degrade under the action of light, heat, and harmful gases, causing the paper to turn yellow, become brittle, and its strength to decline until it powderizes. In this case, the relevant work of identifying the damage degree of calligraphy and painting cultural relics is an essential and crucial step for laying the foundation for the subsequent restoration or grading work.

[0004] The existing restoration work of calligraphy and painting cultural relics mainly relies on manual labor supplemented by modern technology. In the work of cultural relic protection and restoration, disease investigation is the foundation. At present, disease investigation and disease map drawing mainly rely on manual labor. This method still requires restorers to manually search for diseased areas and manually mark them. This process takes a long time and a series of protection measures need to be implemented to avoid damaging the cultural relics during the identification process. The complex manual identification method also indirectly increases the appraisal cost of calligraphy and painting cultural relics. The general appraisal cost in the current market starts from 300 yuan per square meter. Moreover, due to the reliance on manual labor in the existing identification method, there are often omissions in the process of identifying and marking the diseases of calligraphy and painting cultural relics. Not only can the damaged parts not be comprehensively treated during the later restoration process, but also in some channels of private cultural relic mail-order, it is difficult to grade the product quality due to the fuzzy number of disease identifications. Therefore, the identification and positioning of the diseases of calligraphy and painting cultural relics are currently difficult problems faced in the field of calligraphy and painting cultural relic restoration. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for identifying diseases of calligraphy and painting cultural relics based on object detection technology, aiming at the problems such as difficult manual identification of disease positions in the current restoration of calligraphy and painting cultural relics. By using object detection technology and through steps such as data preprocessing, model training, picture detection, and positioning and cropping, the diseases of cultural relics are identified and positioned, solving the problem of difficult manual identification speed faced in the current field of identifying diseases of calligraphy and painting cultural relics.

[0006] The technical solution of the present invention is as follows:

[0007] A method for identifying diseases of calligraphy and painting cultural relics based on object detection technology of the present invention includes the following steps:

[0008] Data preprocessing: Obtain the dataset pictures of the calligraphy and painting cultural relics to be detected, perform manual preprocessing on the obtained dataset pictures of calligraphy and painting cultural relics, remove invalid pictures, and perform data augmentation on the processed dataset pictures;

[0009] Model training: Establish a calligraphy and painting cultural relic model based on self-adversarial training, CmBN strategy, Dropblock regularization, and loss function, and use the dataset pictures after data augmentation to train the calligraphy and painting cultural relic model; The self-adversarial training includes the FGSM algorithm, and the loss function of the FGSM algorithm is:

[0010]

[0011] Among them, J(x) represents the loss function, θ represents the parameters of model training, x represents the input, y represents the true label; α represents the weighting coefficient, and ε represents the amplitude of the perturbation;

[0012] Image detection and positioning cropping: Input the dataset images of the calligraphy and painting cultural relics to be detected into the trained model, and use the sliding window method to detect diseases. Analyze the dataset images layer by layer to output the disease position coordinates. The layer-by-layer analysis of the dataset images includes at least convolution operation, feature extraction, pooling operation, and upsampling operation; Locate and crop the identified viruses according to the output position coordinates.

[0013] Further, the specific steps of the sliding window method are as follows:

[0014] S1: Initialization and scanning preparation of the sliding window: According to the specific form of the calligraphy and painting photos, uniformly fix the scanning starting from the top or right side of the calligraphy and painting, and establish the starting position S of the sliding window; And set the initial length of the window to L according to the ratio, then the currently scanned area is L+S;

[0015] S2: Identification and marking of diseases within the sliding window: Mark and identify diseases in the area within the current window, use the trained model as an algorithm to mark diseases, and save the marking results;

[0016] S3: Adjust the size and position of the sliding window according to the integrity and effect of disease identification, and update the identification system;

[0017] Further, S3 also includes the following steps:

[0018] S3.1: For the diseases near the front-end position L+S of the window, if the disease identification area is incomplete, increase the window size by a step length ε, that is, mark the new window area [S+ε,L+S+ε];

[0019] S3.2: Determine the incomplete disease area in the current window, especially calculate the farthest distance ε' from the boundary L+S, and at this time move the boundary S of the window to S+ε';

[0020] S3.3: When moving to the farthest end position of the calligraphy and painting image, stop the movement and growth of the window.

[0021] Further, the convolution operation uses a matrix with a smaller shape, inputs the image and transforms the image according to the values of the filters. The subsequent feature map calculation formula is:

[0022] Among them, f represents the input image, h represents the kernel of the model, and m and n represent the row and column indices of the result matrix; The feature extraction: Divide the image into small connected regions, collect the direction histograms of the gradients or edges of each pixel point, and combine the histograms into a feature descriptor; The pooling operation: Compress the original data; The upsampling operation: Restore the image features to the original image size.

[0023] Further, the specific method of positioning and cropping is as follows: Obtain the center coordinates of each disease according to the output position index during the detection process, perform overall cropping on the diseases, and save them.

[0024] Further, the iteration formula of the CmBN strategy is:

[0025]

[0026] Where, represents the output feature of a certain neural network convolutional layer of the i-th sample in the t-th mini-batch; and represent smooth activations that follow a normal distribution; ε is a small integer used to ensure numerical stability.

[0027] Further, the formula of the Dropblock regularization is:

[0028]

[0029] Where, keep_prob is the probability that a unit is retained in the traditional dropout, and (feat_size - block_size + 1) 2 represents the size of the effective seed region, feat_size represents the size of the feature map, and block_size is a constant.

[0030] Further, the loss function adopts the CIOU_Loss function, which comprehensively calculates the overlapping area, the distance between the center points, and the aspect ratio of the target box and the predicted box.

[0031] Further, the data augmentation adopts the Mosaic data augmentation method. The Mosaic data augmentation method is used to randomly crop and scale four pictures, and then randomly arrange and splice them to form a picture.

[0032] Further, the dataset pictures are pictures of calligraphy and painting cultural relics taken from different angles with authorization.

[0033] Compared with the existing technologies, the beneficial effects of the present invention are:

[0034] 1. The method for identifying diseases of calligraphy and painting cultural relics based on the object detection technology of the present invention uses the object detection technology to identify and locate the diseases of cultural relics, assisting cultural relic workers to carry out cultural relic protection and restoration work, and reducing the overall cost of cultural relic restoration by saving the time cost of cultural relic workers;

[0035] 2. A method for identifying diseases of calligraphy and painting cultural relics based on object detection technology uses sliding window detection technology, image layer-by-layer analysis, and positioning and cropping technology to identify and crop the disease positions, facilitating subsequent analysis and repair by staff. It can objectively identify diseases while avoiding secondary contact between people and calligraphy and painting cultural relics, preventing secondary damage to the cultural relics. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flowchart of a method for identifying diseases of calligraphy and painting cultural relics based on object detection technology.

[0037] Figure 2 It is a step diagram of a method for identifying diseases of calligraphy and painting cultural relics based on object detection technology.

[0038] Figure 3 It is a comparison diagram of the effects of three data augmentation methods of a method for identifying diseases of calligraphy and painting cultural relics based on object detection technology.

[0039] Figure 4 Schematic diagram of the model recognition effect of a method for identifying diseases of calligraphy and painting cultural relics based on object detection technology Figure 1 .

[0040] Figure 5 Schematic diagram of the model recognition effect of a method for identifying diseases of calligraphy and painting cultural relics based on object detection technology Figure 2 .

[0041] Figure 6 Schematic diagram of the model recognition effect of a method for identifying diseases of calligraphy and painting cultural relics based on object detection technology Figure 3 .

[0042] Figure 7 Schematic diagram of the model recognition effect of a method for identifying diseases of calligraphy and painting cultural relics based on object detection technology Figure 4 . DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] It should be noted that relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variation thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, elements defined by the statement "including a..." do not exclude the existence of additional identical elements in the process, method, article or device including the said elements.

[0044] The features and performance of the present invention will be further described in detail below in conjunction with embodiments.

[0045] Embodiment 1

[0046] As Figure 1 and Figure 2 shown, a method for identifying diseases of calligraphy and painting cultural relics based on object detection technology of the present invention includes the following steps.

[0047] Data preprocessing: Obtain dataset pictures of the calligraphy and painting cultural relics to be detected. The dataset pictures are pictures of different angles of the authorized calligraphy and painting cultural relics. Perform manual preprocessing on the obtained dataset pictures of calligraphy and painting cultural relics to remove invalid pictures. The invalid pictures are low-quality pictures that are blurred, overexposed, and dark. Perform data augmentation on the processed dataset pictures.

[0048] Model training: Establish a calligraphy and painting cultural relics model based on self-adversarial training, CmBN strategy, Dropblock regularization, and loss function. Use the dataset pictures after data augmentation to train the calligraphy and painting cultural relics model. The self-adversarial training includes the FGSM algorithm. The loss function of the FGSM algorithm is:

[0049]

[0050] where J(x) represents the loss function, θ represents the parameters of model training, x represents the input, y represents the true label; α represents the weighting coefficient, and ε represents the amplitude of the perturbation. The FGSM algorithm described in the present invention is an algorithm that directly modifies the loss function of the original FGSM algorithm. Its function is to create an imaginary situation without targets on the image through adversarial attacks to facilitate subsequent disease area positioning and detection.

[0051] Picture detection and localization cropping: Input the dataset pictures of the calligraphy and painting cultural relics to be detected into the trained model, and use the sliding window method to perform disease detection. The dataset pictures are analyzed layer by layer to output the disease position coordinates. The layer-by-layer analysis of the dataset pictures includes at least convolution operation, feature extraction, pooling operation, and upsampling operation; crop the identified virus according to the output position coordinates.

[0052] The CmBN strategy, Dropblock regularization, loss function, sliding window, and layer-by-layer analysis in the above steps can be implemented by existing technologies.

[0053] Embodiment Two

[0054] Embodiment Two is a further improvement of Embodiment One. The iterative formula of the CmBN strategy is:

[0055]

[0056] Among them, represents the output feature of the convolutional layer of a certain neural network for the i-th sample in the t-th mini-batch; and represent a smooth activation that follows a normal distribution; ε is a small integer used to ensure numerical stability. The role of batch normalization CmBN is to reduce the impact of image noise and ensure the effect of final identification of calligraphy and painting disease areas.

[0057] The formula for the Dropblock regularization is:

[0058]

[0059] where keep_prob is the probability that a unit is retained in traditional dropout, and (feat_size - block_size + 1) 2 represents the size of the effective seed region, feat_size represents the size of the feature map, and block_size is a constant. The purpose is to avoid overfitting problems when the neural network learns the features of calligraphy and painting cultural relic images.

[0060] The loss function uses the CIOU_Loss function, and the CIOU_Loss function comprehensively calculates the overlapping area, center point distance, and aspect ratio of the target box and the predicted box. Through calculation, it can ensure that the identification of calligraphy and painting cultural relic disease areas can comprehensively consider important geometric elements such as overlapping area, center point distance, and aspect ratio of length and width. Thus, the accuracy and reliability of disease area identification are improved.

[0061] Embodiment III

[0062] Embodiment III is a further improvement of Embodiment I. The specific steps of the sliding window method are as follows:

[0063] S1: Initialization of the sliding window and preparation for scanning; According to the specific form of the calligraphy and painting photo, uniformly fix the start of scanning from the top or right side of the calligraphy and painting to establish the starting position S of the sliding window; and set the initial length of the window to L according to an appropriate ratio, then the currently scanned area is L + S;

[0064] S2: Identification and marking of diseases within the sliding window; Mark and identify diseases in the area within the current window, use the trained model as an algorithm for disease marking, and save the marking results;

[0065] S3: Adjust the size and position of the sliding window according to the integrity and effect of disease identification, and update the identification system;

[0066] S3.1: For the diseases at the position L+S near the front end of the window, if the disease recognition area is incomplete, increase the window size by the step size ε, that is, mark the new window area [S+ε, L+S+ε];

[0067] S3.2: Determine the incomplete disease areas in the current window, especially calculate the farthest distance ε' from the boundary L+S. At this time, move the boundary S of the window to S+ε';

[0068] S3.3: When moving to the endmost position of the calligraphy and painting image, stop the movement and growth of the window. The role of the sliding window is that it can identify calligraphy and painting cultural relics of different sizes and shapes without restricting the image acquisition technology.

[0069] The convolution operation uses a matrix with a small shape, inputs the image and transforms the image according to the values of the filter. The subsequent feature map calculation formula is:

[0070] Among them, f represents the input image, h represents the kernel of the model, and m and n represent the row and column indices of the result matrix;

[0071] The feature extraction: Divide the image into small connected regions, collect the direction histograms of the gradients or edges of each pixel point, and combine the histograms into a feature descriptor; Let z be a random variable representing the gray level, p(z i ), i - 0, 1,..., L - 1 be the corresponding normalized histograms, then the nth moment of z is: Among them, The feature extraction operation can greatly improve the detection performance of the model.

[0072] The pooling operation compresses the original data, reduces the calculation parameters of the model, and improves the operation efficiency; For the image processed by the pooling layer, the size of the output image is:

[0073]

[0074]

[0075] In the formula, W fitter and H fitter represent the width and height of the convolution kernel (filter) respectively. W in , H in , W out , H out represent the width and height of the feature map of the input and output respectively. S is the stride. The depth of the input feature map is the same as the depth of the convolution kernel.

[0076] The upsampling operation restores the image features to the original image size.

[0077] The specific method of positioning and cropping is as follows: Obtain the center coordinates of each disease according to the output position index during the detection process, and use OpenCV to perform overall cropping of the disease and save it.

[0078] The process of cropping a certain disease is as follows:

[0079] ① Obtain the center coordinates according to the position index output during the detection process.

[0080] ② Initialize the height heightRect and width widthRect of the target area to be cropped, calculate the image rotation angle angle according to the disease center coordinates, and calculate the height height and width width of the original image.

[0081] ③ Call the getRotationMatrix2D function in OpenCV to obtain the rotation matrix. This function takes three parameters, namely the rotation center, the rotation angle, and the third parameter 1 indicating equal-proportion scaling. Finally, the function returns a rotation matrix rotateMat.

[0082] In order to be able to perform rotation transformation at any position, the present invention adopts:

[0083]

[0084] ④ Calculate the new height heightNew and width widthNew of the rotated image according to the rotation angle, and update the value of the rotation matrix rotateMat using the original and new image heights and widths.

[0085] ⑤ Call the warpAffine() affine transformation function in OpenCV to rotate the image. This function takes four parameters, namely the original image (in array form), the rotation matrix rotateMat, the size of the output image (widthNew, heightNew), and the border filling value borderValue. Finally, the function returns a cropped image in array form.

[0086] ⑥ Calculate the four vertex coordinates of the disease rectangle according to new_center and the initially set height heightRect and width widthRect of the target area to be cropped. Crop the original image array according to the vertex coordinates, and call the imwrite function to write the cropped result and save it in the new path newImagePath.

[0087] Example 4

[0088] Embodiment 4 is a further improvement of Embodiment 1. The data augmentation uses the Mosaic data augmentation method. The main idea of the Mosaic data augmentation method is to randomly crop and scale four pictures, and then randomly arrange and splice them to form a picture. In addition to Mosaic data augmentation, the following methods can also be used for data augmentation:

[0089] 1. Mixup: Overlay the picture of another calligraphy and painting on a picture of a calligraphy and painting to enhance the data;

[0090] 2. Cutout: Fill a certain area in the picture with a certain color;

[0091] 3. CutMix: Cut off a certain area of the picture, and then fill the cut area with another image. The effects of the three data augmentation methods are as Figure 3 shown.

[0092] Figure 4 、 5 、6, and 7 are schematic diagrams of identifying diseases using the model of the present invention.

[0093] The above embodiments only represent the specific implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the protection scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the technical solution of the present application, several deformations and improvements can still be made, and these all belong to the protection scope of the present application.

Claims

1. A method for identifying diseases of calligraphy and painting cultural relics based on object detection technology, characterized in that, it includes the following steps, Data preprocessing: Obtain the dataset images of the calligraphy and painting cultural relics to be detected, perform manual preprocessing on the obtained dataset images of calligraphy and painting cultural relics, remove invalid images, and perform data augmentation on the processed dataset images; Model training: Establish a calligraphy and painting cultural relics model based on self-adversarial training, CmBN strategy, Dropblock regularization and loss function, and use the dataset images after data augmentation to train the calligraphy and painting cultural relics model; The self-adversarial training includes the FGSM algorithm, and the loss function of the FGSM algorithm is: where J(x) represents the loss function, θ represents the parameters of model training, x represents the input, y represents the true label; α represents the weighting coefficient, and ε represents the amplitude of the perturbation; Picture detection and positioning cropping: Input the dataset images of the calligraphy and painting cultural relics to be detected into the trained model, and perform disease detection in a sliding window manner. The dataset images are analyzed layer by layer to output the disease position coordinates. The layer-by-layer analysis of the dataset images includes at least convolution operation, feature extraction, pooling operation and upsampling operation; Locate and crop the identified virus according to the output position coordinates.

2. A method for identifying diseases of calligraphy and painting cultural relics based on object detection technology according to claim 1, characterized in that, the specific steps of the sliding window method are as follows: S1: Sliding window initialization and scanning preparation: According to the shape of the calligraphy and painting photo, uniformly fix the scanning starting from the top or right side of the calligraphy and painting, and establish the starting position S of the sliding window; And set the initial length of the window to L according to the ratio, then the currently scanned area is L+S; S2: Identification and marking of diseases within the sliding window: Mark and identify diseases in the area within the current window, use the trained model as an algorithm to mark diseases, and save the marking results; S3: Adjust the size and position of the sliding window according to the integrity and effect of disease identification, and update the identification system.

3. A method for identifying diseases of calligraphy and painting cultural relics based on object detection technology according to claim 2, characterized in that, step S3 further includes the following steps: S3.1: For diseases near the front end position L+S of the window, if the disease identification area is incomplete, increase the window size by a step size of ε, that is, mark the new window area [S+ε, L+S+ε]; S3.2: Determine the incomplete disease area in the current window, calculate the farthest distance ε' from the boundary L+S, and at this time move the boundary S of the window to S+ε'; S3.3: When moving to the farthest end position of the calligraphy and painting image, stop the movement and growth of the window.

4. A method for identifying diseases of calligraphy and painting cultural relics based on object detection technology according to claim 1, characterized in that, The convolution operation uses a matrix with a small shape, inputs an image, and transforms the image according to the values of the filter. The calculation formula for the feature map is as follows: Among them, f represents the input image, h represents the kernel of the model, and m and n represent the row and column indices of the result matrix; for the feature extraction: the image is divided into small connected regions, the direction histogram of the gradient or edge of each pixel point is collected, and the histograms are combined into a feature descriptor; for the pooling operation: the original data is compressed; for the upsampling operation: the image features are restored to the original image size.

5. The method for identifying diseases of calligraphy and painting cultural relics based on target detection technology according to claim 1 or 4, characterized in that the specific method of positioning and cropping is: the center coordinates of each disease are obtained according to the output position index in the detection process, and the disease is cropped as a whole and saved.

6. The method for identifying diseases of calligraphy and painting cultural relics based on target detection technology according to claim 1, characterized in that the iterative formula of the CmBN strategy is: Among them, represents the output feature of the convolutional layer of a certain neural network for the i-th sample in the t-th mini-batch; and represent a smooth activation that follows a normal distribution; ε is a small integer used to ensure numerical stability.

7. The method for identifying diseases of calligraphy and painting cultural relics based on target detection technology according to claim 1, characterized in that the formula of the Dropblock regularization is: Among them, keep_prob is the probability that a unit is retained in traditional dropout, and (feat_size - block_size + 1) 2 represents the size of the valid seed region, feat_size represents the size of the feature map, and block_size is a constant.

8. The method for identifying diseases of calligraphy and painting cultural relics based on target detection technology according to claim 1, characterized in that the loss function adopts the CIOU_Loss function, and comprehensively calculates the overlapping area, the distance between the center points and the aspect ratio of the target box and the predicted box.

9. The method for identifying diseases of calligraphy and painting cultural relics based on target detection technology according to claim 1, characterized in that the data augmentation adopts the Mosaic data augmentation method, and the Mosaic data augmentation method is used to randomly crop and scale four pictures, and then randomly arrange and splice them to form a picture.

10. The method for identifying diseases of calligraphy and painting cultural relics based on target detection technology according to claim 1, characterized in that the dataset pictures are pictures of calligraphy and painting cultural relics taken from different angles with authorization.

Images