A calligraphy and painting correlation comparison and analysis system based on AR glasses

Through the AR glasses-based calligraphy and painting correlation analysis system, using algorithms such as Yolov4 and VGG16 to detect and compare seals in calligraphy and painting works, the problems of low accuracy and cumbersome work in the existing technology are solved, and high Accuracy and low-latency seal detection and correlation analysis simplify the research process of humanities workers.

CN114581651BActive Publication Date: 2024-11-05ZHEJIANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210230929.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-10
Publication Date
2024-11-05
Estimated Expiration
2042-03-10

AI Technical Summary

Technical Problem

The existing technology lacks a relevant technical research foundation in the field of comparison and correlation mining of calligraphy and painting works. The work is cumbersome, prone to errors, and the accuracy is not high, and it has not been widely used. In particular, the use of seal information is insufficient, resulting in ancient calligraphy and painting. Difficulty in researching works.

Method used

Using the method and system for calligraphy and painting association comparison based on AR glasses, the seal detection module and seal search module are used to realize automatic detection, feature extraction and comparison of seals, combined with Siamese network to achieve similarity. Degree measurement, integrated on Android mobile terminal, provides an interactive retrieval system.

Benefits of technology

Achieve high accuracy and low delay seal detection and correlation analysis of calligraphy and painting works, improve the accuracy and robustness of the system, simplify the research process of humanities workers, and initially assist in the comparison and correlation analysis of calligraphy and painting works. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114581651B_ABST
    Figure CN114581651B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for comparing and analyzing the association of calligraphy and painting based on AR glasses, including a seal detection module, a seal retrieval module and a user interaction module, wherein the seal detection module includes a calligraphy and painting work image module, the calligraphy and painting work image module is connected to a calligraphy and painting work data set module, the calligraphy and painting work data set module is connected to a seal detection module module, the seal retrieval module includes a seal image module, the seal image module is connected to a seal image preprocessing and correction module, the seal image preprocessing and correction module is connected to a retrieval model module, and the user interaction module includes an input image module, the input image module and the seal detection model module are connected to a detection result module. The present invention proposes a relatively complete interactive calligraphy and painting retrieval system, realizes seal detection in calligraphy and painting works, and efficiently retrieves related calligraphy and painting works in combination with the characteristics of the seal.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of calligraphy and painting works, and specifically to a calligraphy and painting correlation comparison and analysis method and system based on AR glasses. Background Art

[0002] Ancient calligraphy and painting works are an important way to study the development of ancient art history and the humanities history of different dynasties. At present, the research on ancient Chinese calligraphy and painting works at home and abroad mainly analyzes the structural information and main content information of calligraphy and painting works, but ignores the important characteristic information contained in the seals stamped by the authors and collectors. The existing calligraphy and painting works are mixed with true and false, and the number is huge. The storage is extremely scattered. The lack of a complete database and a knowledge network containing the relationship between works has caused some difficulties for the research of humanities workers. Considering that the information in the seal is extremely important, this technology combines the characteristics of the seal image in the calligraphy and painting works to realize the retrieval of related calligraphy and painting works, and transplants the system to the Android mobile terminal, using AR glasses technology to better assist humanities workers in comparing calligraphy and painting works and exploring the relationship between different calligraphy and painting works.

[0003] There is a lack of relevant technical research foundation in the field of calligraphy and painting work comparison and correlation mining, and the work is cumbersome and error-prone; the existing technologies related to calligraphy and painting works are mainly applied to single scenarios, such as seal identification, calligraphy and painting work generation, style migration and other fields, lacking unified methods and data; and the existing technologies are not accurate enough and the calculation is slow, and have not yet been widely used. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for comparing and analyzing the association between calligraphy and painting based on AR glasses.

[0005] The present invention is achieved in that:

[0006] A method and system for comparing and analyzing the association of calligraphy and painting based on AR glasses, comprising a seal detection module, a seal retrieval module and a user interaction module, wherein the seal detection module comprises a calligraphy and painting work image module, the calligraphy and painting work image module is connected to a calligraphy and painting work data set module, the calligraphy and painting work data set module is connected to a seal detection model module, the seal retrieval module comprises a seal image module, the seal image module is connected to a seal image preprocessing and correction module, the seal image preprocessing and correction module is connected to a retrieval model module, the user interaction module comprises an input image module, the input image module and the seal detection model module are connected to a detection result module, the detection result module is connected to an input seal image module, the input seal image module is connected to an input end of the seal image preprocessing and correction module, the user interaction module is further provided with a retrieval result module, and the retrieval model module is connected to an input end of the retrieval result module. The user interaction module performs information interaction through a mobile phone APP. The seal detection module and the seal retrieval module are integrated in a server, and the server is wirelessly connected to a user interaction module via the Internet. The user interaction module is a smart phone or a tablet computer, and is connected to AR glasses via the user interaction module.

[0007] Furthermore, the calligraphy and painting correlation comparison and analysis method based on AR glasses is specifically performed in the following steps:

[0008] S1: First, 831 clear front images of calligraphy and painting works are collected from the Internet and stored in the calligraphy and painting work dataset module in the seal detection module. The Labelme tool is used to mark the seals in the calligraphy and painting images with rectangles, and the coordinates of the upper left corner and lower right corner of the seal are used to describe the position of the seal in the original calligraphy and painting works. The results are saved for training, and the seal detection model is obtained after convergence.

[0009] S2: Through the seal detection module, the seal is automatically detected and framed from the calligraphy and painting images using the object detection algorithm Yolov4;

[0010] The algorithm mainly uses a single CNN model to achieve end-to-end object detection. First, an input image is resized to 448×448 and input into the CNN network. Finally, the predicted result of the CNN network is processed to obtain the detected object.

[0011] Specifically, after the Yolo algorithm resizes the input image and inputs it into the CNN network, the network divides the input image into an S×S grid. Each grid unit is responsible for detecting the targets whose center points fall within the grid unit and predicting B bounding boxes and the confidence of the bounding boxes, that is, the possibility that the bounding box contains the target and the accuracy of the bounding box. The possibility that the bounding box contains the target is recorded as P(object). If the bounding box does not contain the target, P(object) = 0, and if the bounding box contains the target, P(object) = 1. The accuracy of the bounding box is represented by the IOU between the predicted box and the true box, recorded as

[0012] The confidence of the bounding box can be defined as The size and position of the bounding box are represented by four values ​​(x, y, w, h), where (x, y) is the center coordinate of the bounding box, and w and h are the width and height of the bounding box respectively. The predicted value of the bounding box is represented by five values ​​(x, y, w, h, c), where (x, y) is the offset value relative to the coordinate of the upper left corner of the grid cell, w and h are the ratio of the width and height of the entire image respectively, and c is the confidence of the bounding box.

[0013] In addition, the Yolo algorithm will eventually output the predicted C category probability values

[48] Each grid cell is responsible for predicting the probability value of the target belonging to each category. These probability values ​​are conditional probabilities under the confidence level of each bounding box and can be expressed as P(class i |object), from which the category confidence of each bounding box can be calculated, as shown in formula (0)

[0014]

[0015] Category confidence is used to measure how well the bounding box matches the target within it and the likelihood that the target belongs to each category. Therefore, the network's prediction box is filtered according to the category confidence to obtain the final prediction result.

[0016] Assume that the input image is divided into S×S grid cells, there are C categories in total, and each grid cell is responsible for predicting B bounding boxes. Then each grid cell will be responsible for predicting (B×5+C) values. The final prediction value P output by the network is a two-dimensional tensor of size [batch, S×S×(B×5+C)]. The prediction result of the bounding box can be obtained by slicing.

[0017] S3: By extracting the features of the input seal and comparing it with the seal features in the seal database, the seal in the seal database that is most similar to the original input seal image is retrieved. Specifically, the convolutional layer and the first fully connected layer structure of VGG16 are used to extract the 16-dimensional feature f of the seal.i ,f j ∈R 16 To represent the characteristics of a seal, and by calculating the angle cosine of the 16-dimensional feature vectors of different seals The size of is used to characterize the similarity of different seals. The retrieval model module in the seal retrieval module uses VGGNet as the network backbone of the retrieval part and uses Siamese network to measure the seal similarity.

[0018] The main idea of ​​the Siamese network is to transform the input image through a function G W (X) is mapped to the target space, where the parameter is W, and then the similarity is compared in the target space using methods such as Euclidean distance.

[0019] In the training phase, the Siamese network uses Contrasive Loss as the loss function, minimizing the loss function value of samples from the same category and maximizing the loss function value of samples from different categories. That is, given a set of mapping functions G W (X), in the training process, we look for a set of parameters W so that when X1 and X2 belong to the same category, the similarity measure E W (X1,X2)=||G W (X1)-G W (X2)|| is minimized, and when X1 and X2 belong to different categories, the similarity measure E W (X1,X2)=||G W (X1)-G W The value of (X2)|| is maximized.

[0020] For the Siamese network, its structure is symmetrical. The two Convolutional Networks in the figure use VGG16 in this technology. Each input of the Siamese network is a pair of two samples, and the two samples use the same mapping function G W (X), share the same parameter W. For these two samples, there is no longer the concept of the original category label, but they are given a label of 1 or 0 based on whether they come from the same category.

[0021] If the trends of the two eigenvectors are closer, the value of cosθ is closer to 1, and the two seal images are more similar; if the trends of the two eigenvectors are more different, the value of cosθ is closer to 0, and the two seal images are less similar.

[0022] Further, in step S1, during the training process,

[0023] S 1.1First, the collected 831 clear front images of calligraphy and painting works are rotated 30°, and the coordinates of the upper left corner and lower right corner of each seal after rotation are calculated. The width of the original front image is marked as w, and the height is h. The coordinates of the upper left corner of the seal in the original front image are (x min ,y min ), the coordinates of the lower right vertex are (x max ,y max ), the coordinates of the upper left corner of the seal after rotating 30° are shown in formula (1):

[0024]

[0025] The coordinates of the lower right vertex are shown in formula (2);

[0026]

[0027] S 1.2 : Make the annotated bounding box contain the entire seal image, and the offset of the x-axis is as shown in formula (3);

[0028]

[0029] The offset of the y-axis is as shown in formula (4);

[0030]

[0031] The coordinates of the upper left corner vertex of the bounding box marked after rotation are as shown in formula (5);

[0032]

[0033] The coordinates of the vertex at the lower right corner of the bounding box are as shown in formula (6);

[0034]

[0035] Further, in step S 1.1 In the process, before training, the foreground and background of the seal are collected, the seal is extracted as the foreground, and the cropped calligraphy and painting images are used as the background to expand each type of seal samples.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] This technology has made the following contributions. A relatively complete interactive calligraphy and painting retrieval system was proposed, which realized the seal detection in calligraphy and painting works, and retrieved related calligraphy and painting works based on the characteristics of the seal. This technology used different network parameters and data sets for testing and correction, and selected the optimal solution. The network structure used by this system is relatively simple, but the performance is good, and it can be effectively applied to online calligraphy and painting work comparison and correlation analysis scenarios. (2) An innovative data expansion method was proposed. This method preprocesses the cropped seals, selects the best solution by adjusting parameters, and uses background expansion to expand the seal data set, which improves the accuracy and robustness of the system to a certain extent. (3) An Android mobile terminal calligraphy and painting comparison and correlation analysis application was developed, and combined with AR glasses and touchpad, it is mobile and portable, highly interactive, and has high accuracy and low latency in seal detection and calligraphy and painting work retrieval. It can initially assist humanities workers in the comparison and correlation analysis of calligraphy and painting works.

[0038] The frontal data set of 831 frontal images and the comprehensive data set rotated 30° were used for training, and the obtained model was used to detect the test data set rotated 30°. The seal detection rate was greatly improved to almost 100%, and the recognition rate of a single seal was also greatly improved compared to before. In addition, the overall seal detection accuracy was high when the calligraphy and painting works captured in real scenes or calligraphy and painting works with a large number of seals were tested. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a system structure diagram of the present invention;

[0041] Figure 2 It is a flow chart of seal detection and recognition of the present invention;

[0042] Figure 3 It is a seal extraction sample diagram of the present invention;

[0043] Figure 4 is a search result display diagram of the present invention;

[0044] Figure 5 It is a sample diagram of seal detection of the present invention;

[0045] Figure 6is a Siamese network structure diagram of the present invention;

[0046] Figure 7 It is a network structure diagram of VGG16 of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0048] See also Figure 1-7 A calligraphy and painting association comparison and analysis system based on AR glasses, including a seal detection module, a seal retrieval module and a user interaction module, the seal detection module includes a calligraphy and painting work image module, the calligraphy and painting work image module is connected to a calligraphy and painting work data set module, the calligraphy and painting work data set module is connected to a seal detection model module, the seal retrieval module includes a seal image module, the seal image module is connected to a seal image preprocessing and correction module, the seal image preprocessing and correction module is connected to a retrieval model module, the user interaction module includes an input image module, the input image module and the seal detection model module are connected to a detection result module, the detection result module is connected to an input seal image module, the input seal image module is connected to an input end of the seal image preprocessing and correction module, the user interaction module is also provided with a retrieval result module, and the retrieval model module is connected to an input end of the retrieval result module. The user interaction module performs information interaction through a mobile phone APP. The seal detection module and the seal retrieval module are integrated in a server, and the server is wirelessly connected to a user interaction module via the Internet. The user interaction module is a smart phone or a tablet computer, and is connected to AR glasses via the user interaction module.

[0049] In this embodiment, the calligraphy and painting correlation comparison and analysis method based on AR glasses is specifically performed according to the following steps:

[0050] S1: First, 831 clear front images of calligraphy and painting works are collected from the Internet and stored in the calligraphy and painting work dataset module in the seal detection module. The Labelme tool is used to mark the seals in the calligraphy and painting images with rectangles, and the coordinates of the upper left corner and lower right corner of the seal are used to describe the position of the seal in the original calligraphy and painting works. The results are saved for training, and the seal detection model is obtained after convergence.

[0051] S2: Through the seal detection module, the seal is automatically detected and framed from the calligraphy and painting images using the object detection algorithm Yolov4;

[0052] The algorithm mainly uses a single CNN model to achieve end-to-end object detection. First, an input image is resized to 448×448 and input into the CNN network. Finally, the predicted result of the CNN network is processed to obtain the detected object.

[0053] Specifically, after the Yolo algorithm resizes the input image and inputs it into the CNN network, the network divides the input image into an S×S grid. Each grid unit is responsible for detecting the targets whose center points fall within the grid unit and predicting B bounding boxes and the confidence of the bounding boxes, that is, the possibility that the bounding box contains the target and the accuracy of the bounding box. The possibility that the bounding box contains the target is recorded as P(object). If the bounding box does not contain the target, P(object) = 0, and if the bounding box contains the target, P(object) = 1. The accuracy of the bounding box is represented by the IOU between the predicted box and the true box, recorded as

[0054] The confidence of the bounding box can be defined as The size and position of the bounding box are represented by four values ​​(x, y, w, h), where (x, y) is the center coordinate of the bounding box, and w and h are the width and height of the bounding box respectively. The predicted value of the bounding box is represented by five values ​​(x, y, w, h, c), where (x, y) is the offset value relative to the coordinate of the upper left corner of the grid cell, w and h are the ratio of the width and height of the entire image respectively, and c is the confidence of the bounding box.

[0055] In addition, the Yolo algorithm will eventually output the predicted C category probability values. Each grid cell is responsible for predicting the probability value of the target belonging to each category. These probability values ​​are conditional probabilities under the confidence level of each bounding box, which can be expressed as P(class i |object), from which the category confidence of each bounding box can be calculated, as shown in formula (0)

[0056]

[0057] Category confidence is used to measure how well the bounding box matches the target within it and the likelihood that the target belongs to each category. Therefore, the network's prediction box is filtered according to the category confidence to obtain the final prediction result.

[0058] Assume that the input image is divided into S×S grid cells, there are C categories in total, and each grid cell is responsible for predicting B bounding boxes. Then each grid cell will be responsible for predicting (B×5+C) values. The final prediction value P output by the network is a two-dimensional tensor of size [batch, S×S×(B×5+C)]. The prediction result of the bounding box can be obtained by slicing.

[0059] S3: By extracting the features of the input seal and comparing it with the seal features in the seal database, the seal in the seal database that is most similar to the original input seal image is retrieved. Specifically, the convolutional layer and the first fully connected layer structure of VGG16 are used to extract the 16-dimensional feature f of the seal. i ,f j ∈R 16 To represent the characteristics of a seal, and by calculating the angle cosine of the 16-dimensional feature vectors of different seals The size of is used to characterize the similarity of different seals. The retrieval model module in the seal retrieval module uses VGGNet as the network backbone of the retrieval part and uses Siamese network to measure the seal similarity.

[0060] The main idea of ​​the Siamese network is to transform the input image through a function G W (X) is mapped to the target space, where the parameter is W, and then the similarity is compared in the target space using methods such as Euclidean distance.

[0061] In the training phase, the Siamese network uses Contrasive Loss as the loss function, minimizing the loss function value of samples from the same category and maximizing the loss function value of samples from different categories. That is, given a set of mapping functions G W (X), in the training process, we look for a set of parameters W so that when X1 and X2 belong to the same category, the similarity measure E W (X1,X2)=||G W (X1)-G W (X2)|| is minimized, and when X1 and X2 belong to different categories, the similarity measure E W (X1,X2)=||G W (X1)-G W The value of (X2)|| is maximized.

[0062] For Siamese network, its structure is symmetrical, such as Figure 6As shown in the figure, the two Convolutional Networks in this technology select VGG16. The input of the Siamese network each time is two paired samples, and the two samples use the same mapping function G W (X), share the same parameter W. For these two samples, there is no longer the concept of the original category label, but they are given a label of 1 or 0 based on whether they come from the same category. Figure 6 shown.

[0063] If the trends of the two eigenvectors are closer, the value of cosθ is closer to 1, and the two seal images are more similar; if the trends of the two eigenvectors are more different, the value of cosθ is closer to 0, and the two seal images are less similar. The network structure of VGG16 is as follows: Figure 7 shown.

[0064] In this embodiment, in step S1, during the training process,

[0065] S 1.1 First, the collected 831 clear front images of calligraphy and painting works are rotated 30°, and the coordinates of the upper left corner and lower right corner of each seal after rotation are calculated. The width of the original front image is marked as w, and the height is h. The coordinates of the upper left corner of the seal in the original front image are (x min ,y min ), the coordinates of the lower right vertex are (x max ,y max ), the coordinates of the upper left corner of the seal after rotating 30° are shown in formula (1):

[0066]

[0067] The coordinates of the lower right vertex are shown in formula (2);

[0068]

[0069] S 1.2 : Make the annotated bounding box contain the entire seal image, and the offset of the x-axis is as shown in formula (3);

[0070]

[0071] The offset of the y-axis is as shown in formula (4);

[0072]

[0073] The coordinates of the upper left corner vertex of the bounding box marked after rotation are as shown in formula (5);

[0074]

[0075] The coordinates of the vertex at the lower right corner of the bounding box are as shown in formula (6);

[0076]

[0077] In this embodiment, in step S 1.1 In the process, before training, the foreground and background of the seal are collected, the seal is extracted as the foreground, and the cropped calligraphy and painting images are used as the background to expand each type of seal samples.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A calligraphy and painting correlation comparison and analysis system based on AR glasses, Features: It comprises a seal detection module, a seal retrieval module and a user interaction module, wherein the seal detection module comprises a calligraphy and painting work image module, the calligraphy and painting work image module is connected to a calligraphy and painting work data set module, the calligraphy and painting work data set module is connected to a seal detection model module, the seal retrieval module comprises a seal image module, the seal image module is connected to a seal image preprocessing and correction module, the seal image preprocessing and correction module is connected to a retrieval model module, the user interaction module comprises an input image module, the input image module and the seal detection model module are connected to a detection result module, the detection result module is connected to an input seal image module, the input seal image module is connected to an input end of the seal image preprocessing and correction module, the user interaction module is further provided with a retrieval result module, and the retrieval model module is connected to an input end of the retrieval result module; The user interaction module performs information interaction via a mobile phone APP; The seal detection module and the seal retrieval module are integrated in a server, the server is wirelessly connected to the user interaction module via the Internet, and the user interaction module is a smart phone or a tablet computer, which is connected to the AR glasses via the user interaction module; The calligraphy and painting correlation comparison and analysis system based on AR glasses adopts a calligraphy and painting correlation comparison and analysis method based on AR glasses, which is specifically performed in the following steps: S 1 :First, we collect multiple clear front images of calligraphy and painting works from the Internet and store them in the calligraphy and painting works dataset module in the seal detection module. We use the Labelme tool to mark the seals in the calligraphy and painting works images with rectangles, and use the coordinates of the upper left corner and lower right corner of the seal to describe the position of the seal in the original calligraphy and painting works. We save the results for training and converge to obtain the seal detection model. S 2 : Through the seal detection module, the seal is automatically detected and framed from the calligraphy and painting images using the target detection algorithm Yolov4; S 3 : By extracting the features of the input seal and comparing it with the seal features in the seal database, the seal in the seal database that is most similar to the original input seal image is retrieved; The retrieval model module in the seal retrieval module uses VGGNet as the network backbone of the retrieval part, and uses Siamese network to measure the seal similarity; In step S 1 During the training process, S 1.1 First, the collected clear front images of calligraphy and painting works are rotated 30°, and the coordinates of the upper left corner and lower right corner vertices of each seal after rotation are calculated. The width of the original front image is marked as w, and the height is h. The coordinates of the upper left corner vertex of the seal in the original front image are (x min ,y min ), the coordinates of the lower right vertex are (x max ,y max ), the coordinates of the upper left corner of the seal after rotating 30° are shown in formula (1): The coordinates of the lower right vertex are shown in formula (2); S 1.2 : Make the annotated bounding box contain the entire seal image, and the offset of the x-axis is as shown in formula (3); The offset of the y-axis is as shown in formula (4); The coordinates of the upper left corner vertex of the bounding box marked after rotation are as shown in formula (5); The coordinates of the vertex at the lower right corner of the bounding box are as shown in formula (6); 2. According to the AR glasses-based calligraphy and painting correlation comparison and analysis system according to claim 1, It is characterized in that In step S 1.1 In the process, before training, the foreground and background of the seal are collected, the seal is extracted as the foreground, and the cropped calligraphy and painting images are used as the background to expand each type of seal samples.

3. According to claim 2, a calligraphy and painting correlation comparison and analysis system based on AR glasses, It is characterized in that In step S 3 In the above example, the convolutional layer and the first fully connected layer structure of VGG16 are used to extract the 16-dimensional feature f of the seal. i ,f j ∈R 16 To represent the characteristics of a seal, and by calculating the angle cosine of the 16-dimensional feature vectors of different seals The size of is used to characterize the similarity of different seals.