A real-time calligraphy copybook scoring method based on deep learning technology

Abstract

The present invention discloses a real-time calligraphy copy scoring method based on deep learning technology. In order to correct the problem of calligraphy copy deformation caused by different lighting and angles, the method first performs position transformation on the picture, then uses the deep learning method for font positioning detection, crops the detected text using coordinates, then uses the trained deep learning model to recognize the text, and at the same time uses the convolutional neural network in the model to extract feature vectors of the recognized text, extracts feature vectors of the corresponding standard Chinese characters, calculates the cosine distance between the two, and finally completes the scoring.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and more specifically, to a real-time calligraphy copy scoring method based on deep learning technology. Background Art

[0002] Traditional calligraphy copy scoring algorithms include: 1) a method based on traditional calligraphy copy segmentation by color and segmentation ratio and simple comparison with template characters (hereinafter simply referred to as Method 1); 2) a method of extracting features through a neural network and comparing characters (hereinafter simply referred to as Method 2). Among them, the algorithm of Method 1 has a relatively single scenario. It only locates the text box through edge detection to extract the text box and color threshold, and uses traditional algorithms to perform similarity comparison on the segmented characters to complete the scoring. However, due to the large limitations of traditional algorithms such as edge detection, it cannot perform accurate scoring on calligraphy copy pictures taken at different angles and under different lighting conditions.

[0003] With the rise of deep learning technology, a method has been proposed to extract features based on a neural network, correct the position of the extracted text box, and use high-level semantic information for font similarity comparison (Method 2). However, since Method 2 still uses traditional algorithms for text box positioning in the positioning part of the text box, and for calligraphy copies deformed at different angles and under different lighting conditions during shooting, the text box positioning is inaccurate, affecting the subsequent feature extraction of the neural network, resulting in inaccurate scoring. Summary of the Invention

[0004] To solve the above problems, the present invention provides a real-time calligraphy copy scoring method based on deep learning technology. By using deep learning technology methods and performing position transformation for deformations at different angles and under different lighting conditions, it overcomes the influence of deformations and color changes caused by objective factors such as different photographing angles on calligraphy copies, and improves the accuracy of recognition and scoring.

[0005] To achieve the above object, the present invention provides a real-time calligraphy copy scoring method based on deep learning technology, which includes:

[0006] Step S1: Collect training and test data to construct an image dataset, specifically including collecting calligraphy copies with different handwriting styles written by different students, and forming a set with image data obtained by taking pictures under different angles and different lighting conditions using devices with different pixels;

[0007] Step S2: Construct a position transformation module, and perform position transformation on the images in the image dataset through the Hough line detection and clustering algorithm to correct the deformation caused by shooting at different angles and under different lighting conditions for the entire picture;

[0008] Step S3: Construct a text box localization module. Preprocess the set of images output by the position transformation module and divide them into a training set, a validation set, and a test set according to a preset ratio. Then, use a deep learning-based text detection model to perform localization training on all the characters in the copybook;

[0009] Step S4: Construct a text recognition module. Divide the images localized by the text box localization module into a training set, a validation set, and a test set according to a preset ratio. Then, perform recognition training on the characters in the text box through a deep learning-based text recognition model. The deep learning-based text recognition model includes a convolutional neural network for extracting character feature vectors;

[0010] Step S5: Construct a text similarity comparison module. Save the character feature vectors extracted by the convolutional neural network in Step S4. Input the characters in the standard regular script character library corresponding to the above character feature vectors into the convolutional neural network in Step S4 to extract the corresponding standard feature vectors. Compare the character feature vectors with the corresponding standard feature vectors to obtain a similarity value;

[0011] Step S6: Introduce a ranking mechanism to sort the similarity values of all the same recognized characters, and calculate the final score of each character by adding the weights to the base score; and

[0012] Input the copybook picture to be scored into this copybook scoring system for real-time scoring. Specifically:

[0013] Step S7: Input the copybook picture to be scored into this copybook scoring system. Perform position transformation through the position transformation module constructed in Step S2, locate the text through the text localization module constructed in Step S3, recognize the located text through the text recognition module constructed in Step S4, then calculate the similarity value through the text similarity comparison module constructed in Step S5, and finally give the final score through the ranking mechanism introduced in Step S6 and output the system.

[0014] In an embodiment of the present invention, Step S2 is specifically as follows:

[0015] Step S21: Screen the data in the image dataset constructed in Step S1, and detect the entire contour of the corresponding copybook in each image data through Hough line detection;

[0016] Step S22: Use a clustering algorithm and set the number of centroids k = 4 to obtain four straight lines of the corresponding copybook contour. Calculate the four intersection points (x1, y1), (x2, y2), (x3, y3), (x4, y4) where these four straight lines intersect pairwise. Here, x1, x2, x3, and x4 respectively represent the x-axis coordinate values of the corresponding intersection points, and y1, y2, y3, and y4 respectively represent the y-axis coordinate values of the corresponding intersection points;

[0017] Step S23: Perform perspective transformation on the copybook using the coordinates of the obtained 4 intersection points to obtain the corrected copybook picture. Among them, the transformation matrix M for performing perspective transformation is:

[0018]

[0019] Among them, the matrix M includes 4 parts, which are respectively: Used to represent linear transformation; (a31, a32), used for translation; Used to generate perspective transformation; a33 is represented as a proton and is set to a constant 1.

[0020] In an embodiment of the present invention, among them, the specific training process in step S3 is:

[0021] Step S31: Preprocess the images in the set of images output by the position transformation module, and adjust each image to a fixed size;

[0022] Step S32: Divide the preprocessed set of images into a training set, a validation set, and a test set according to a preset ratio, and perform manual annotation on the training set and the validation set;

[0023] Step S33: Input the manually annotated training set into a text detection model based on deep learning for training;

[0024] Step S34: Input the manually annotated validation set and test set into the trained text detection model for validation and testing respectively. When the accuracy of the validation set and the test set reaches more than 98%, the training is completed; otherwise, repeat step S33 to continue training.

[0025] In an embodiment of the present invention, among them, the specific training process in step S4 is:

[0026] Step S41: Crop the image located by the text box positioning module, and adjust the cropped image to a fixed size;

[0027] Step S42: Divide the adjusted image into a training set, a validation set, and a test set according to a preset ratio, and perform manual annotation on the training set and the validation set;

[0028] Step S43: Input the manually annotated training set into a text recognition model based on deep learning for training. Among them, the end of the text recognition model outputs the index of the recognized text through the Softmax function. The formula of the Softmax function is specifically:

[0029]

[0030] In the formula, y jis the probability of predicting the j-th word in the word dictionary, x i is the value of the i-th node, x j is the value of the j-th node, where both i and j are integers greater than 0;

[0031] Step S44: Input the manually labeled validation set and test set into the trained text recognition model for validation and testing respectively. When the accuracies of the validation set and the test set reach more than 98%, the training is completed; otherwise, repeat Step S43 to continue the training.

[0032] In an embodiment of the present invention, specifically, Step S5 is as follows:

[0033] Step S51: Extract and save the text feature vectors extracted by the convolutional neural network in Step S4;

[0034] Step S52: Find the index of the corresponding text in Step S43 according to the text feature vectors;

[0035] Step S53: Download the Chinese standard regular script character library and input it into the convolutional neural network in Step S4 for feature extraction to obtain standard feature vectors. Find the corresponding index through the Softmax function in Step S43, and store the standard feature vectors in the database with the corresponding index as the keyword;

[0036] Step S54: According to the index of each text in Step S52, search for the keyword in the database and obtain the corresponding standard feature vector;

[0037] Step S55: Calculate the cosine distance between the text feature vectors and the standard feature vectors with the same index through the following formula, and take the cosine distance as the similarity value,

[0038]

[0039] In the formula, A is the text feature vector, B is the standard feature vector, n is the dimension of the feature vector, and θ is the angle value between the text feature vector and the standard feature vector in the same dimension.

[0040] In an embodiment of the present invention, the division ratio of the training set, the validation set, and the test set in Step S3 and Step S4 is 3:1:1.

[0041] The real-time copybook scoring method based on deep learning technology provided by the present invention has at least the following advantages compared with the prior art:

[0042] 1) By performing position transformation on the copybook, it is possible to overcome the influence on text recognition caused by the deformation of the copybook image and the color change due to different photographing angles and other objective factors;

[0043] 2) It solves the problem that the traditional algorithm has inaccurate positioning due to partial deformation, which ultimately leads to inaccurate scoring. By using deep learning technology for text positioning, the accuracy of scoring is greatly improved;

[0044] 3) The text recognition module also uses deep learning technology, which greatly improves the accuracy of text recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0046] Figure 1 It is a schematic flowchart of an embodiment of the present invention.

[0047] Description of the reference numerals: S1 to S7 - steps. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0049] Figure 1 It is a schematic flowchart of an embodiment of the present invention. As Figure 1 shown, this embodiment provides a real-time calligraphy scoring method based on deep learning technology, which includes:

[0050] Construct a real-time calligraphy scoring system based on deep learning technology, specifically including:

[0051] Step S1: Collect training and test data to construct an image data set, specifically including collecting calligraphy with different handwriting styles of different students, and the image data obtained by taking pictures under different angles and different lighting conditions using devices with different pixels to form a set;

[0052] Step S2: Construct a position transformation module, and perform position transformation on the images in the image data set through the Hough line detection and clustering algorithm to correct the deformation caused by shooting the entire picture under different angles and different lighting conditions;

[0053] In this embodiment, specifically, step S2 is:

[0054] Step S21: Screen the data in the image dataset constructed in step S1, and detect the entire outline of the corresponding copybook in each image data through Hough line detection; among them, the Hough line detection can be implemented through the corresponding function in OpenCV;

[0055] Step S22: Use the clustering algorithm and set the number of centroids k = 4 to obtain four straight lines corresponding to the outline of the copybook, and calculate the 4 intersection points (x1, y1), (x2, y2), (x3, y3), (x4, y4) where these four straight lines intersect pairwise. Among them, x1, x2, x3, and x4 respectively represent the x-axis coordinate values of the corresponding intersection points, and y1, y2, y3, and y4 respectively represent the y-axis coordinate values of the corresponding intersection points;

[0056] Step S23: Perform perspective transformation on the copybook through the coordinates of the obtained 4 intersection points to obtain the corrected copybook picture. Among them, the transformation matrix M for performing perspective transformation is:

[0057]

[0058] Among them, the matrix M includes 4 parts, which are respectively: Used to represent linear transformation; (a31, a32), used for translation; Used to generate perspective transformation; a33 is expressed as a proton and is set to a constant 1.

[0059] Step S3: Construct a text box positioning module, preprocess the set of images output by the position transformation module, and divide them into a training set, a validation set, and a test set according to a preset ratio respectively, and use a deep learning-based text detection model to perform positioning training on all the characters in the copybook;

[0060] In this embodiment, the division ratio of the training set, the validation set, and the test set is 3:1:1. In other embodiments, it can also be divided into other ratio values according to requirements.

[0061] In this embodiment, the specific training process in step S3 is as follows:

[0062] Step S31: Preprocess the images in the set of images output by the position transformation module, and adjust each image to a fixed size; for subsequent processing, all the images are adjusted to a fixed size here, and this fixed size can be preset according to requirements, and this embodiment does not limit it;

[0063] Step S32: Divide the preprocessed image set into a training set, a validation set, and a test set according to a preset ratio respectively, and perform manual annotation on the training set and the validation set; among them, the ratio of positive and negative samples in the training set, the validation set, and the test set can be set to 3:1, or can be set to other values according to requirements, and this embodiment does not limit it;

[0064] Step S33: Input the training set after manual annotation into a deep learning-based text detection model for training. Among them, due to the particularity of the copybook grid, the text detection model can use any deep learning-based text detection model in the prior art or an improved model in the prior art. This embodiment does not limit it.

[0065] Step S34: Input the validation set and the test set after manual annotation into the trained text detection model for validation and testing respectively. When the accuracy of the validation set and the test set reaches more than 98%, the training is completed; otherwise, repeat Step S33 to continue training.

[0066] Step S4: Construct a text recognition module, divide the image located by the text box positioning module into a training set, a validation set, and a test set respectively according to a preset ratio, and perform recognition training on the text in the text box through a deep learning-based text recognition model. The deep learning-based text recognition model includes a convolutional neural network for extracting text feature vectors.

[0067] In this embodiment, the specific training process in Step S4 is as follows:

[0068] Step S41: Crop the image located by the text box positioning module and adjust the cropped image to a fixed size. For subsequent processing, all images are cropped and adjusted to a fixed size here. This fixed size can be preset according to requirements, and this embodiment does not limit it.

[0069] Step S42: Divide the adjusted image into a training set, a validation set, and a test set respectively according to a preset ratio, and perform manual annotation on the training set and the validation set. Among them, the ratio of positive and negative samples in the training set, the validation set, and the test set can be set to 3:1, or can be set to other values according to requirements. This embodiment does not limit it.

[0070] Step S43: Input the training set after manual annotation into a deep learning-based text recognition model for training. Among them, the text recognition model finally outputs the index of the recognized text through the Softmax function. The formula of the Softmax function is specifically:

[0071]

[0072] In the formula, y j is the probability of predicting the jth word in the word dictionary, x i is the value of the ith node, x jis the value of the j-th node, where both i and j are integers greater than 0; among them, the text recognition model can be an improvement of any existing deep learning-based text recognition model, that is, the last layer of the existing deep learning-based text recognition model is set to the above-mentioned Softmax function, and this embodiment does not limit other layers of the text recognition model itself;

[0073] Step S44: Input the manually annotated validation set and test set into the trained text recognition model for verification and testing respectively. When the accuracy of the validation set and the test set reaches more than 98%, the training is completed; otherwise, repeat step S43 to continue training.

[0074] Step S5: Construct a text similarity comparison module, save the text feature vectors extracted by the convolutional neural network in step S4, input the Chinese characters in the standard regular script character library corresponding to the above-mentioned text feature vectors into the convolutional neural network in step S4 to extract the corresponding standard feature vectors, and compare the text feature vectors with the corresponding standard feature vectors to obtain a similarity value;

[0075] In this embodiment, specifically, step S5 is as follows:

[0076] Step S51: Extract and save the text feature vectors extracted by the convolutional neural network in step S4;

[0077] Step S52: Find the index of the corresponding text in step S43 according to the text feature vectors;

[0078] Step S53: Download the Chinese standard regular script character library and input it into the convolutional neural network in step S4 for feature extraction to obtain standard feature vectors. Find the corresponding index through the Softmax function in step S43, and store the standard feature vectors in the database with the corresponding index as the keyword;

[0079] Step S54: According to the index of each text in step S52, search for the keyword in the database and obtain the corresponding standard feature vector;

[0080] Step S55: Calculate the cosine distance between the text feature vectors and the standard feature vectors with the same index through the following formula, and take the cosine distance as the similarity value,

[0081]

[0082] In the formula, A is the text feature vector, B is the standard feature vector, n is the dimension of the feature vector, and θ is the angle value between the text feature vector and the standard feature vector in the same dimension.

[0083] Step S6: Introduce a ranking mechanism to sort the similarity values of all the identified identical Chinese characters, and calculate the final score of each Chinese character by adding the weights to the basic score. In this embodiment, the basic score is set to 60 points. In other embodiments, it can also be set to other values according to requirements, and the present invention does not limit it.

[0084] Input the calligraphy copybook picture to be scored into the calligraphy copybook scoring system for real-time scoring, specifically:

[0085] Step S7: Input the calligraphy copybook picture to be scored into the calligraphy copybook scoring system, perform position transformation through the position transformation module constructed in step S2, locate the text through the text positioning module constructed in step S3, identify the located text through the character recognition module constructed in step S4, then calculate the similarity value through the character similarity comparison module constructed in step S5, and finally give the final score through the ranking mechanism introduced in step S6 and output the system.

[0086] The real-time calligraphy copybook scoring method based on deep learning technology provided by the present invention, in order to correct the problem of deformation of the calligraphy copybook caused by different illuminations and different angles, first performs position transformation on the picture, then uses the deep learning method for font positioning detection, and crops the detected text using coordinates, and then uses the trained deep learning model to recognize the text. At the same time, the convolutional neural network in the model is used to extract the feature vectors of the recognized text, extract the feature vectors of the corresponding standard Chinese characters, and calculate the cosine distance between the two, and finally complete the scoring.

[0087] In this embodiment, the division ratio of the training set, validation set, and test set in steps S3 and S4 is 3:1:1. In other embodiments, it can also be divided into other ratio values according to requirements.

[0088] The real-time calligraphy copybook scoring method based on deep learning technology provided by the present invention has at least the following advantages compared with the prior art:

[0089] 1) By performing position transformation on the calligraphy copybook, it is possible to overcome the influence of objective factors such as different photographing angles on the deformation of the calligraphy copybook image and the color change on character recognition.

[0090] 2) Solve the problem that the traditional algorithm has inaccurate positioning due to partial deformation, resulting in inaccurate scoring in the end. By using deep learning technology for text positioning, the accuracy of scoring is greatly improved.

[0091] 3) The character recognition module also uses deep learning technology, which greatly improves the accuracy of character recognition.

[0092] Those of ordinary skill in the art will understand that the attached drawings are only schematic diagrams of an embodiment, and the modules or processes in the attached drawings are not necessarily essential for implementing the present invention.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time calligraphy copybook scoring method based on deep learning technology, characterized in that, include: Step S1: Collect training and test data to construct an image data set, specifically including collecting stickers of words written in different handwritings by different students, and using devices with different pixels to take pictures under different angles and different lighting conditions to form a set of image data; Step S2: construct a position transformation module, and transform the images in the image data set by using the Hough line detection and clustering algorithm to correct the deformation of the entire image caused by shooting at different angles and different lighting conditions; Step S3: construct a text box positioning module, pre-process the set of images output by the position transformation module and divide them into a training set, a validation set and a test set according to a preset ratio, and use a text detection model based on deep learning to perform positioning training on all the characters in the copybook; Step S4: constructing a text recognition module, dividing the image located by the text box positioning module into a training set, a validation set and a test set according to a preset ratio, and performing recognition training on the text in the text box through a text recognition model based on deep learning, wherein the text recognition model based on deep learning includes a convolutional neural network for extracting text feature vectors; Step S5: construct a text similarity comparison module, save the text feature vector extracted by the convolutional neural network in step S4, input the Chinese characters in the standard regular script Chinese character library corresponding to the text feature vector into the convolutional neural network of step S4 to extract the corresponding standard feature vector, and perform a similarity comparison between the text feature vector and the corresponding standard feature vector to obtain a similarity value, wherein step S5 is specifically as follows: Step S51: extracting and saving the text feature vector extracted by the convolutional neural network in step S4; Step S52: Find the index of the corresponding character in step S43 according to the character feature vector; Step S53: downloading a Chinese standard Kaiti Chinese character library, and inputting it into the convolutional neural network in step S4 for feature extraction to obtain a standard feature vector, finding the corresponding index through the Softmax function, and storing the standard feature vector in the database with the corresponding index as the keyword; Step S54: according to the index of each word in step S52, searching the keyword from the database and obtaining the standard feature vector corresponding thereto; Step S55: Calculate the cosine distance between the text feature vector with the same index and the standard feature vector by the following formula, and use the cosine distance as the similarity value: In the formula, A is the text feature vector, B is the standard feature vector, n is the dimension of the feature vector, and θ is the angle between the text feature vector and the standard feature vector of the same dimension; Step S6: Introduce a ranking mechanism to sort the similarity values ​​of all the identified identical Chinese characters, and calculate the final score of each Chinese character by weighted addition with the basic score; as well as Input the copybook image to be rated into the copybook rating system for real-time rating, specifically: Step S7: Input the copybook picture to be scored into the copybook scoring system. Perform position transformation through the position transformation module constructed in Step S2, locate the text through the text localization module constructed in Step S3, recognize the located text through the character recognition module constructed in Step S4, then calculate the similarity value through the character similarity comparison module constructed in Step S5, and finally give the final score through the ranking mechanism introduced in Step S6 and output the system.

2. The real-time calligraphy copybook scoring method based on deep learning technology according to claim 1, wherein Specifically, Step S2 is as follows: Step S21: Screen the data in the image dataset constructed in Step S1, and detect the entire contour of the corresponding copybook in each image data through Hough line detection; Step S22: Use the clustering algorithm and set the number of centroids k = 4 to obtain four straight lines of the corresponding copybook contour, and calculate the four intersection points (x1, y1), (x2, y2), (x3, y3), (x4, y4) of these four straight lines intersecting pairwise, where x1, x2, x3, and x4 respectively represent the x-axis coordinate values of the corresponding intersection points, and y1, y2, y3, and y4 respectively represent the y-axis coordinate values of the corresponding intersection points; Step S23: Perform perspective transformation on the copybook through the coordinates of the obtained four intersection points to obtain the corrected copybook picture. Among them, the transformation matrix M for performing perspective transformation is: Among them, matrix M includes 4 parts, namely: used to represent a linear transformation; (a31, a32), used for translation; used to generate a perspective transformation; a33 is represented as a proton and is set to the constant 1.

3. The real-time copybook scoring method based on deep learning technology according to claim 1, characterized in that, The specific training process in Step S3 is as follows: Step S31: Preprocess the images in the set of images output by the position transformation module, and adjust each image to a fixed size; Step S32: Divide the preprocessed image set into a training set, a validation set, and a test set according to a preset ratio, and perform manual annotation on the training set and the validation set; Step S33: Input the manually annotated training set into a deep learning-based text detection model for training; Step S34: Input the manually annotated validation set and test set into the trained text detection model for validation and testing respectively. When the accuracy of the validation set and the test set reaches more than 98%, the training is completed; otherwise, repeat Step S33 to continue training.

4. The real-time copybook scoring method based on deep learning technology according to claim 1, wherein The specific training process in Step S4 is as follows: Step S41: Crop the image located by the text box localization module, and adjust the cropped image to a fixed size; Step S42: Divide the adjusted images into a training set, a validation set, and a test set according to a preset ratio, and perform manual annotation on the training set and the validation set; Step S43: Input the manually annotated training set into a deep learning-based character recognition model for training. Among them, the character recognition model finally outputs the index of the recognized character through the Softmax function. The formula of the Softmax function is specifically: where y j is the probability of predicting the j-th word in the word dictionary, x i is the value of the i-th node, x j is the value of the j-th node, and both i and j are integers greater than 0; Step S44: Input the manually annotated validation set and test set into the trained character recognition model for validation and testing respectively. When the accuracy of the validation set and the test set reaches more than 98%, the training is completed; otherwise, repeat Step S43 to continue training.

5. The real-time copybook scoring method based on deep learning technology according to claim 4, characterized in that, The Softmax function used in Step S53 is the Softmax function in Step S43.

6. The real-time calligraphy copybook scoring method based on deep learning technology according to claim 1, wherein The division ratio of the training set, the validation set, and the test set in Step S3 and Step S4 is 3:1:1.

Images