Copybook chapter evaluation method and device based on deep learning

Abstract

The embodiment of the invention provides a copybook text evaluation method and device based on deep learning, and the method comprises the steps: sequentially carrying out the copybook preprocessing, copybook straight line detection and copybook rotation correction of an input copybook image, and obtaining a corrected copybook image; then, a pre-constructed and trained multi-task model is used for carrying out handwritten character detection to obtain the position and the category of each handwritten character and the position information of the character grid where the handwritten character is located, then alignment of the character grid of the handwritten character and the character grid of the standard character is carried out, and copybook character row and column information is restored to obtain the aligned handwritten character and the restored character row and column information; and finally, calculating the deviation degree of each dimension to obtain a detailed evaluation result. And the writing layout of the whole copybook chapters can be restored, so that the whole copybook is subjected to multi-dimensional evaluation, and corresponding guidance comments are given according to calculation results of all dimensions.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and more specifically, to a method for evaluating calligraphy chapters based on deep learning. Background Art

[0002] With the development of deep learning technology, the quality assessment of calligraphy works has gradually shifted from traditional expert review to artificial intelligence-based methods. In the existing technology, two specially trained deep learning models are used to locate and recognize handwritten fonts, and the feature vectors generated by these models are further used to compare the similarities between individual Chinese characters, thereby giving a quality evaluation for each character.

[0003] However, although this method can effectively provide detailed analysis results for individual characters, it ignores an important aspect - the integrity and coordination of the entire calligraphy work. A complete calligraphy piece requires not only the proper writing of each individual Chinese character, but more importantly, it needs to consider the echo, fluency and overall layout of the relationship between characters and lines. This means that existing technical means have limitations in evaluating the works of calligraphy practitioners and cannot fully reflect the comprehensive ability of practitioners in the entire calligraphy creation process, especially the ability to handle the interaction between different elements to achieve the best visual effect.

[0004] Therefore, it is necessary to develop a new evaluation system or improve the existing method so that it can not only accurately judge the quality of individual characters, but also effectively measure the artistic value and practicality of the entire calligraphy work, thereby promoting calligraphy enthusiasts to improve their skills on a broader level. Summary of the invention

[0005] The present application provides a method and device for evaluating a copybook chapter based on deep learning, which can restore the writing layout of the entire copybook chapter, thereby performing a multi-dimensional evaluation of the copybook as a whole and giving corresponding guidance comments based on the calculation results of each dimension.

[0006] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for evaluating a copybook chapter based on deep learning, the method comprising:

[0008] The input copybook image is sequentially subjected to copybook preprocessing, copybook straight line detection and copybook rotation correction to obtain a corrected copybook image;

[0009] Use the pre-built and trained multi-task model to detect handwritten characters on the corrected copybook image to obtain the position, category, and position information of each handwritten character;

[0010] Based on the position, category and position information of each handwritten character, the handwritten character grid is aligned with the standard character grid and the character row and column information of the copybook is restored to obtain the aligned handwritten characters and the restored character row and column information;

[0011] The degree of deviation of each dimension is calculated based on the aligned handwritten characters and the restored text row and column information to obtain detailed evaluation results, wherein the dimensions include uniform size, smooth writing and uniform spacing.

[0012] In a possible implementation, the steps of sequentially performing copybook preprocessing, copybook straight line detection, and copybook rotation correction on the input copybook image to obtain a corrected copybook image include:

[0013] The input copybook image is cropped, scaled and grayed in sequence to obtain a grayed copybook image;

[0014] The Canny edge detection algorithm is used to process the grayscale copybook image to obtain the boundary contours of the copybook row and column grids. Based on the boundary contours, the Hough transform algorithm is used to determine all straight lines and record the direction angle of each line.

[0015] Based on the direction angle, the straight lines whose absolute values ​​of the direction angles are less than the preset angle are screened from all the straight lines to obtain the screened straight lines, the median of the angles of the screened straight lines is selected as the optimal rotation angle, and a reverse rotation operation is performed according to the optimal rotation angle to obtain the corrected copybook image.

[0016] In a possible implementation, the multi-task model is a YOLOv5 target detection model improved based on key point regression, including a backbone network, a neck network, and a head network;

[0017] The backbone network is used to extract multi-level features from the input image and generate feature maps of different sizes. The backbone network includes convolutional layers, batch normalization layers, and activation function layers. The convolutional layers are used to extract image features, the batch normalization layers are used to accelerate model training and improve the generalization ability of the model, and the activation function layers are used to increase the nonlinear expression ability of the model.

[0018] The neck network is used to fuse feature maps of different scales to enrich semantic information and positioning information. The neck network adopts FPN combined with PAN structure and adds a spatial attention module. The spatial attention module includes 3x3 convolution layer, average pooling, maximum pooling and normalization function sigmoid.

[0019] The head network predicts the feature maps of each scale and performs classification and localization tasks.

[0020] In a possible implementation, the steps of training the YOLOv5 target detection model improved based on key point regression include:

[0021] For the predicted box of the target, encode and decode based on the size and position of the anchor box, and predict the deviation relative to the anchor box;

[0022] The loss functions of the YOLOv5 target detection model improved based on key point regression include corner regression loss function, classification loss function, confidence loss function, and positioning loss function. i represents the corner point number, d represents the Euclidean distance between the predicted corner point coordinates and the true coordinates, S represents the scale factor, σ represents the weight of the current corner point, and δ represents whether the current corner point is visible;

[0023] The overall loss function of the model is L total =α1L cls +α2L box +α3L obj +α4L kpt , where L cls is the classification loss function, L obj is the confidence loss function, L box is the positioning loss function, L kpt is the corner regression loss function, and α1, α2, α3, and α4 are the corresponding weights respectively.

[0024] In a possible implementation, the step of aligning the handwritten character grid with the standard character grid to obtain the aligned handwritten character based on the position, category and position information of each handwritten character grid, including:

[0025] According to the name of the character, search for the corresponding standard character grid size and standard character width and height in a pre-set database;

[0026] The side length of the character grid is determined according to the coordinates of the corner points of the handwritten character grid, the character grid is scaled to be consistent with the size of the standard character grid, and the size of the outer rectangular frame of the handwritten character is scaled according to the character grid scaling ratio to obtain the aligned handwritten character.

[0027] In a possible implementation, the step of restoring the copybook character row and column information based on the position, category and character grid position information of each handwritten character to obtain restored character row and column information includes:

[0028] Sort all handwritten characters from small to large according to the coordinates of the upper left corner of the character grid;

[0029] Set a threshold, group the handwritten words with a vertical coordinate smaller than the threshold into one row, and arrange the words in the same row from left to right according to the size of the horizontal coordinate;

[0030] The handwritten characters whose horizontal coordinates are smaller than the threshold value are grouped into one column, and the characters in the same column are arranged from top to bottom according to the size of the vertical coordinates to obtain the restored character row and column information.

[0031] In a possible implementation, the step of calculating the degree of deviation of uniform size based on the aligned handwritten characters and the restored text row and column information to obtain detailed evaluation results includes:

[0032] Calculate the area of ​​a single handwritten character and a standard character based on the aligned handwritten characters and the restored text row and column information;

[0033] Calculate the area difference based on the area of ​​a single handwritten character and a standard character;

[0034] Calculate the average area difference of all handwritten characters based on the area difference;

[0035] Calculate the degree of deviation for each handwritten word;

[0036] The handwritten characters are screened by a preset first deviation threshold, and the degree of deviation is recorded to obtain detailed evaluation results.

[0037] In a possible implementation, the step of calculating the deviation degree of writing smoothness based on the aligned handwritten characters and the restored character row and column information to obtain detailed evaluation results includes:

[0038] Calculate the center coordinates of the handwritten characters based on the aligned handwritten characters and the restored text row and column information;

[0039] Calculate the mean of the ordinate based on the center coordinates of the handwritten characters;

[0040] The deviation degree of each handwritten character is calculated based on the mean value of the ordinate, the handwritten characters are screened through a preset second deviation threshold, and the deviation degree is recorded to obtain detailed evaluation results.

[0041] In a possible implementation, the step of calculating the degree of deviation of uniform spacing based on the aligned handwritten characters and the restored text row and column information to obtain detailed evaluation results includes:

[0042] Calculate the mean vertical coordinate of each row based on the aligned handwritten characters and restored text row and column information;

[0043] Calculate the spacing between adjacent rows based on the mean of the ordinate of each row;

[0044] Calculate the mean spacing based on the spacing between adjacent rows;

[0045] Calculate the degree of spacing deviation based on the spacing mean;

[0046] Based on the degree of spacing deviation, adjacent rows are screened out through a preset third deviation threshold, and the degree of deviation is recorded.

[0047] In the second aspect, the present application also proposes a copybook chapter evaluation device based on deep learning, the device comprising:

[0048] A copybook correction module is used to perform copybook preprocessing, copybook straight line detection and copybook rotation correction on the input copybook image in sequence to obtain a corrected copybook image;

[0049] The text localization module is used to detect handwritten characters on the corrected copybook image using a pre-built and trained multi-task model to obtain the position, category, and position information of each handwritten character;

[0050] An information integration module is used to align the handwritten character grid with the standard character grid and restore the character row and column information of the copybook based on the position, category and position information of each handwritten character and the character grid where the handwritten character is located to obtain the aligned handwritten characters and restored character row and column information;

[0051] The passage evaluation module is used to calculate the degree of deviation of each dimension based on the aligned handwritten characters and the restored text row and column information to obtain detailed evaluation results. The dimensions include uniform size, smooth writing and uniform spacing.

[0052] The above-mentioned main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application; and in the present application, (non-conflicting options) options and other options can also be freely combined. After understanding the scheme of the present application, those skilled in the art can understand that there are multiple combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0053] Compared with the prior art, the embodiment of the present application provides a method and device for evaluating a copybook chapter based on deep learning. First, the input copybook image is subjected to copybook preprocessing, copybook straight line detection, and copybook rotation correction in sequence to obtain a corrected copybook image. Then, the pre-built and trained multi-task model is used to detect handwritten characters to obtain the position, category, and grid position information of each handwritten character. Then, the handwritten character grid is aligned with the standard character grid and the copybook text row and column information is restored to obtain the aligned handwritten characters and restored text row and column information. Finally, the degree of deviation in each dimension is calculated to obtain detailed evaluation results. The writing layout of the entire copybook chapter can be restored, thereby performing a multi-dimensional evaluation of the copybook as a whole and giving corresponding guidance comments based on the calculation results of each dimension. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 A flow chart of a method for evaluating a copybook chapter based on deep learning proposed in an embodiment of the present application is shown.

[0056] Figure 2 The structure diagram of the YOLOv5 target detection model based on key point regression improvement proposed in an embodiment of the present application is shown.

[0057] Figure 3 A schematic diagram of a neck network proposed in an embodiment of the present application is shown.

[0058] Figure 4 A schematic diagram of a decoupling detection head proposed in an embodiment of the present application is shown.

[0059] Figure 5 A schematic diagram of the model output results of the character grid corner points proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0061] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0062] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0063] In the prior art, two specially trained deep learning models are used to realize the positioning and recognition of handwritten fonts, and the feature vectors generated by these models are further used to compare the similarities between individual Chinese characters, thereby giving a quality evaluation for each character.

[0064] However, although this method can effectively provide detailed analysis results for individual characters, it ignores an important aspect - the integrity and coordination of the entire calligraphy work. A complete calligraphy piece requires not only the proper writing of each individual Chinese character, but more importantly, it needs to consider the echo, fluency and overall layout of the relationship between characters and lines. This means that existing technical means have limitations in evaluating the works of calligraphy practitioners and cannot fully reflect the comprehensive ability of practitioners in the entire calligraphy creation process, especially the ability to handle the interaction between different elements to achieve the best visual effect.

[0065] Therefore, in order to solve the above-mentioned technical problems, the embodiment of the present application proposes a method and device for copybook chapter evaluation based on deep learning, which takes copybook images as input, and outputs multi-dimensional evaluation information and comments through copybook correction, text positioning, information integration, and chapter evaluation, so as to achieve accurate evaluation of copybook chapters.

[0066] Please refer to Figure 1 , Figure 1 A flowchart of a method for evaluating a copybook chapter based on deep learning proposed in an embodiment of the present application is shown, and the method includes the following steps:

[0067] Step S100: performing copybook preprocessing, copybook straight line detection and copybook rotation correction on the input copybook image in sequence to obtain a corrected copybook image.

[0068] Considering that the handwritten copybook is being photographed and the calligraphy test paper is being scanned and uploaded, due to various factors, the copybook image will inevitably be tilted or crooked. Step S100 quickly and effectively rotates the input copybook image, which is a key prerequisite to ensure that the copybook chapter evaluation can be carried out accurately and reliably. Copybook preprocessing involves operations such as removing noise and enhancing contrast, with the aim of laying the foundation for more accurate line recognition in the next step.

[0069] The copybook straight line detection automatically detects the straight line features in the image through algorithms. These straight lines can be the edges of the copybook paper or the horizontal lines formed when writing. This is the key basis for determining the angle that needs to be adjusted. Based on the information obtained in the previous step, the angle that needs to be rotated is calculated and applied to the entire image to ensure that the final output copybook image is positive and has no obvious tilt.

[0070] Step S100 includes:

[0071] The input copybook image is cropped, scaled and grayed in sequence to obtain a grayed copybook image;

[0072] The Canny edge detection algorithm is used to process the grayscale copybook image to obtain the boundary contours of the copybook row and column grids. Based on the boundary contours, the Hough transform algorithm is used to determine all straight lines and record the direction angle of each line.

[0073] Based on the direction angle, the straight lines whose absolute values ​​of the direction angles are less than the preset angle are screened from all the straight lines to obtain the screened straight lines, the median of the angles of the screened straight lines is selected as the optimal rotation angle, and a reverse rotation operation is performed according to the optimal rotation angle to obtain the corrected copybook image.

[0074] To speed up the processing, we first cropped the middle 1 / 2 width and height portion from the original image, then scaled the area to 320 pixels in height, and adjusted the width to the original image ratio. This can reduce the amount of calculation while maintaining key information. Next, we grayscaled the color image and converted it into a grayscale image, which reduced the amount of data for subsequent processing (from RGB three channels to a single channel) and highlighted the brightness information.

[0075] The Canny edge detection algorithm is used to identify the outlines of character boundaries and other important lines on the copybook. Based on the edge information obtained, the Hough transform algorithm is used to further extract the straight line features in the image and record the direction angles of these straight lines.

[0076] By setting an angle threshold (e.g. 45 degrees), we can filter out line segments that may be horizontal or vertical. We calculate the median of the inclination angles of these filtered lines as the final angle that needs to be adjusted. Selecting the median can effectively avoid the influence of extreme values ​​on the results. Finally, we perform a reverse rotation based on this optimal rotation angle to obtain the corrected copybook image.

[0077] Step S100 utilizes the powerful functions provided by the OpenCV library to quickly and accurately complete the image processing task.

[0078] Step S200: Detect handwritten characters on the corrected copybook image using a pre-built and trained multi-task model to obtain the position, category, and character grid position information of each handwritten character.

[0079] Correct the copybook image to ensure that all contents are in the best viewing angle for easy analysis. Use the improved YOLOv5 model (multi-task model) based on key point regression technology to locate and classify handwritten text. Key points refer to the key coordinates used to define the boundaries of each character or character grid. The model output includes but is not limited to the coordinates of the bounding rectangle of each handwritten font detected (the location of the handwritten word), the specific name of the word (the category of the handwritten word), and the exact coordinate values ​​of the four corner points of the character grid where the word is located. Based on the data generated by the model, further analyze the information of each handwritten word and its accurate spatial layout.

[0080] The multi-task model is an improved YOLOv5 target detection model based on key point regression, including the backbone network, neck network and head network;

[0081] The backbone network is used to extract multi-level features from the input image and generate feature maps of different sizes. The backbone network includes convolutional layers, batch normalization layers, and activation function layers. The convolutional layers are used to extract image features, the batch normalization layers are used to accelerate model training and improve the generalization ability of the model, and the activation function layers are used to increase the nonlinear expression ability of the model.

[0082] Figure 2 The structure diagram of the improved YOLOv5 target detection model based on key point regression proposed in the embodiment of the present application is shown. The backbone network extracts the features of the image. The spatial attention module processes the features at different scales (80x80, 40x40, 20x20) to enhance the model's attention to key information. The decoupled detection head performs detection at each scale, outputs the position, category, and position information of the handwritten characters, and outputs the character category, text box, confidence, and character corner point.

[0083] The backbone network is responsible for extracting useful features from the raw pixel values ​​to form representations at different levels of abstraction. By applying learnable filters to local areas, the convolutional layer can capture spatial correlations and extract specific visual features. Batch Normalization (BN): Standardizing each small batch of data during training helps reduce internal covariate shift problems, speeds up convergence, and may improve the expressiveness of the final model. Activation Function Layer: Introducing nonlinear elements into the model allows the neural network to learn more complex decision boundaries.

[0084] The backbone network generates multi-scale feature maps at different depths to accommodate objects or patterns of various sizes. The feature map height, width, and number of channels are (80, 80, 256), (40, 40, 512), and (20, 20, 1024), respectively. 80x80x256 is a shallow layer that retains more details and is suitable for capturing smaller objects; 40x40x512 is an intermediate layer that can balance resolution and semantic richness; 20x20x1024 is a deep layer that has a higher semantic level but lower spatial resolution.

[0085] The neck network is used to fuse feature maps of different scales to enrich semantic information and positioning information;

[0086] The neck network adopts the FPN combined with PAN structure and adds a spatial attention module, which includes a 3x3 convolutional layer, average pooling, maximum pooling, and normalization function sigmoid.

[0087] The neck network combines FPN (Feature Pyramid Networks) and PAN (Path Aggregation Network), and introduces a spatial attention mechanism to further optimize feature maps. FPN uses a top-down approach to pass semantic information from high-level (deep) feature maps to low-level (shallow) feature maps, so that low-resolution feature maps can also obtain rich semantic information. PAN takes the opposite approach, passing fine spatial location information from the bottom layer to the top layer, helping high-level feature maps retain more detailed information. The use of FPN combined with the PAN structure can effectively propagate information between different levels, allowing the model to maintain good positioning accuracy while enhancing the recognition ability of small objects or distant objects.

[0088] In order to further improve the model's attention to specific tasks (detection of corner points around text), the scheme adds a spatial attention module during the transmission of large-scale feature maps between FPN and PAN, aiming to automatically learn which areas are most critical to the final output and give these areas higher weights.

[0089] Figure 3The schematic diagram of the neck network proposed in the embodiment of the present application is shown. First, the input feature map is subjected to average pooling and maximum pooling respectively, both of which act on the channel dimension, reducing the data dimension while retaining the most important information. Then the two pooling results are spliced ​​together along the channel direction to form a new weight map, and the weight map is processed using a convolution kernel of size 3x3 to achieve information fusion from two sources and determine the importance score of each pixel point. Then the Sigmoid function is applied to perform a normalization operation on the above-mentioned result to ensure that all values ​​are between 0 and 1 and the sum is 1, which is equivalent to calculating a probability distribution map reflecting the relative importance of each position, and finally multiplying the generated attention coefficient with the original input feature map, so as to achieve the effect of emphasizing the key parts and weakening the insignificant areas. In this way, the adjusted feature map will be more focused on the key parts that help solve the current problem. In this way, not only the sensitivity of the model to small changes or specific details in complex scenes is enhanced, but also the robustness and accuracy of the overall system are improved.

[0090] The head network predicts the feature maps of each scale and performs classification and localization tasks.

[0091] Figure 4 The schematic diagram of the decoupled detection head proposed in the embodiment of the present application is shown. The head network includes a detection head that predicts each scale feature map. In order to reduce the number of parameters, the basic YOLOv5 model uses a coupled detection head to complete the classification and positioning tasks of a single scale in different channels of the same convolutional layer. In order to improve the accuracy of model prediction under multiple tasks, the detection head of the head part is decoupled. For the four tasks of text category, text box, confidence and character corner point, four 3x3 convolution branches are built to correspond to the prediction outputs of the four tasks. Figure 5 A schematic diagram of the model output results of the character grid corner points proposed in an embodiment of the present application is shown.

[0092] The training steps of the YOLOv5 target detection model improved based on key point regression include:

[0093] For the predicted box of the target, encode and decode based on the size and position of the anchor box, and predict the deviation relative to the anchor box;

[0094] The loss functions of the YOLOv5 target detection model improved based on key point regression include corner regression loss function, classification loss function, confidence loss function, and positioning loss function. i represents the corner point number, d represents the Euclidean distance between the predicted coordinates and the true coordinates of the corner point, S represents the scale factor, the value is the area of ​​the text rectangle, σ represents the weight of the current corner point (set to 1), and δ represents whether the current corner point is visible, 1 for visible and 0 for invisible.

[0095] The overall loss function of the model is L total =α1L cls +α2L box +α3L obj +α4L kpt , where L cls is the classification loss function, L obj is the confidence loss function, L box is the positioning loss function, L kpt is the corner regression loss function, and α1, α2, α3, and α4 are the corresponding weights respectively.

[0096] The model encodes and decodes the predicted box of the target based on the size and position of the anchor box, and then predicts its deviation from the anchor box. Compared with directly predicting the coordinate value, predicting the deviation value will reduce the model output range and improve the accuracy of the model prediction. For the newly added character corner regression branch, a similar approach is used to construct its encoding and decoding method: x = 4*(2σ(d x )-1)+c x , y=4*(2σ(d y )-1)+c y , where σ represents the sigmoid function, with a domain value of (0, 1), and d x and d y Indicates the deviation between the x-coordinate and the y-coordinate, c x and c y Represents the center coordinates of the anchor box.

[0097] Through encoding and decoding, the prediction range of the character corner regression is limited to the 4 grids around the center of the anchor box, so that the corner coordinates can be regressed more accurately. Correspondingly, a new corner regression loss function needs to be added in the loss function design of the model. Classification loss function L cls And the confidence loss function L obj Using the cross entropy loss function, the positioning loss function L boxThe C-IOU loss function is used. The overall loss function of the model is obtained by weighting the loss functions of the four branches. The principle of setting the weight is to keep the order of magnitude of each loss function basically consistent. The data set used for model training consists of a certain number of copybook images. The annotation tool is used to complete the annotation of all text categories, text boxes, and character grid corners corresponding to the text in the copybook image. The data set is divided into a training set and a validation set in a ratio of 7:3. The training set is used for model training and optimization, and the validation set is used to evaluate model performance indicators. During the model training process, the input image size is uniformly scaled to 640x640 and normalized. The batch size is set to 256, the optimization algorithm uses SGD, the initial learning rate is set to 0.01, and the number of data set iterations is 200. During the entire training process, the average accuracy of the model will be calculated on the validation set after each iteration, and the model weight with the highest average accuracy will be saved. In actual use, only the model weight needs to be loaded to complete the model reasoning.

[0098] Step S300: align the handwritten character grid with the standard character grid and restore the copybook text row and column information based on the position, category and position information of each handwritten character to obtain aligned handwritten characters and restored text row and column information.

[0099] The step of aligning the handwritten character grid with the standard character grid based on the position and category of each handwritten character and the character grid position information of the handwritten character to obtain the aligned handwritten character includes:

[0100] According to the name of the character, search for the corresponding standard character grid size and standard character width and height in a pre-set database;

[0101] The side length of the character grid is determined according to the coordinates of the corner points of the handwritten character grid, the character grid is scaled to be consistent with the size of the standard character grid, and the size of the outer rectangular frame of the handwritten character is scaled according to the character grid scaling ratio to obtain the aligned handwritten character.

[0102] According to the name of the text, the corresponding standard character grid size (width and height) is found in the pre-set database. The actual side length of the handwritten character grid is determined by analyzing the coordinates of the corner points of the handwritten character. The handwritten character grid is scaled according to the size of the standard character grid to make the two sizes consistent. According to the scaling ratio of the character grid, the size of the outer rectangular frame of the handwritten character is adjusted accordingly to ensure the consistency of the overall layout.

[0103] The step of restoring the text row and column information of the copybook based on the position, category and position information of each handwritten character in the character grid where the handwritten character is located to obtain the restored text row and column information includes:

[0104] Sort all handwritten characters from small to large according to the coordinates of the upper left corner of the character grid;

[0105] Set a threshold, group the handwritten words with a vertical coordinate smaller than the threshold into one row, and arrange the words in the same row from left to right according to the size of the horizontal coordinate;

[0106] The handwritten characters with abscissas smaller than a threshold value are grouped into one column, and the characters in the same column are arranged from top to bottom according to the size of the ordinates to obtain the restored character row and column information.

[0107] First, collect the coordinates of the upper left corner of all handwritten character grids, and arrange all handwritten characters in ascending order according to these coordinate values ​​(first according to the ordinate y, then the abscissa x), and calculate a threshold based on the average side length of all character grids, taking 20% ​​of the average side length as the standard. This threshold is used to determine which handwritten characters belong to the same row or column. Select the first unprocessed handwritten character as the starting point. Find all handwritten characters whose ordinate difference is less than the set threshold, and consider them to belong to the same row. For the characters identified as the same row, re-sort them according to the size of their respective abscissas so that the characters in the row are arranged from left to right.

[0108] For each row of handwritten characters whose order has been determined, check whether the horizontal distance between them meets the above threshold conditions. Classify the handwritten characters that meet the conditions (the horizontal coordinate difference is less than the threshold) into the same column, and ensure that the characters in each column are correctly sorted from top to bottom. Integrate the processed text information to form an orderly row and column structure, which is convenient for subsequent higher-level analysis such as paragraph evaluation.

[0109] Step S400: Calculate the degree of deviation of each dimension based on the aligned handwritten characters and the restored text row and column information to obtain detailed evaluation results, including uniform size, smooth writing and uniform spacing.

[0110] The steps of calculating the degree of deviation of uniform size based on the aligned handwritten characters and the restored text row and column information to obtain detailed evaluation results include:

[0111] Calculate the area of ​​a single handwritten character and a standard character based on the aligned handwritten characters and the restored text row and column information;

[0112] Calculate the area difference based on the area of ​​a single handwritten character and a standard character;

[0113] Calculate the average area difference of all handwritten characters based on the area difference;

[0114] Calculate the degree of deviation for each handwritten word;

[0115] The handwritten characters are screened by a preset first deviation threshold, and the degree of deviation is recorded to obtain detailed evaluation results.

[0116] The uniform size dimension measures the degree of difference between the area of ​​the bounding rectangle of all handwritten characters and the corresponding standard characters.

[0117] For each handwritten character and its corresponding standard character, first calculate the area of ​​their circumscribed rectangles, and calculate the ratio d of the area difference between the single handwritten character and the standard character. s =(S hw / S gt -1)*100%, S represents area, hw represents handwritten characters, gt represents standard characters, and the average value of the area difference ratio between all handwritten characters and standard characters is obtained.

[0118] The area difference of each handwritten character is compared with its average value to determine whether the handwritten character is larger or smaller than the overall average. A preset first deviation threshold is used to screen out handwritten characters that are obviously larger or smaller, and the specific degree of deviation is given. The first deviation threshold is usually set by calligraphy experts based on actual experience after analyzing a large amount of sample data.

[0119] The steps of calculating the deviation degree of writing smoothness based on the aligned handwritten characters and the restored text row and column information to obtain detailed evaluation results include:

[0120] Calculate the center coordinates of the handwritten characters based on the aligned handwritten characters and the restored text row and column information;

[0121] Calculate the mean of the ordinate based on the center coordinates of the handwritten characters;

[0122] The deviation degree of each handwritten character is calculated based on the mean value of the ordinate, the handwritten characters are screened through a preset second deviation threshold, and the deviation degree is recorded to obtain detailed evaluation results.

[0123] Based on the dimension of writing flatness, the consistency of the vertical (row) or horizontal (column) positions of all handwritten characters in the same row or column is evaluated. First, find the center coordinates (x, y) of each handwritten character in each row. Calculate the average value of the center vertical coordinates yi of all handwritten characters in the row. For each handwritten word, find the difference d between it and the mean of the vertical coordinate of the row y And perform standardization: Where l is the side length of the character grid. A pre-set second deviation threshold is used to identify which handwritten characters deviate far from the normal range, that is, too high or too low, and these handwritten characters and their degree of deviation are recorded. For the case of columns, it is only necessary to replace the ordinate in the above steps with the abscissa.

[0124] The steps of calculating the degree of deviation of uniform spacing based on the aligned handwritten characters and the restored text row and column information to obtain detailed evaluation results include:

[0125] Calculate the mean vertical coordinate of each row based on the aligned handwritten characters and restored text row and column information;

[0126] Calculate the spacing between adjacent rows based on the mean of the ordinate of each row;

[0127] Calculate the mean spacing based on the spacing between adjacent rows;

[0128] Calculate the degree of spacing deviation based on the spacing mean;

[0129] Based on the degree of spacing deviation, adjacent rows are screened out through a preset third deviation threshold, and the degree of deviation is recorded.

[0130] The dimension based on uniform spacing is used to evaluate the consistency of spacing between different rows or columns. The average value of the vertical coordinates of the center of each row of handwritten characters is used as the position of the row. The row spacing is estimated by the difference in the vertical distance between two adjacent rows. The average value of all row spacing is calculated. For any two adjacent rows, calculate the actual distance m between them i With average spacing The percentage deviation of the ratio Use the pre-set deviation threshold to find adjacent rows with abnormal spacing (too large or too small) and mark them out.

[0131] Finally, the detailed evaluation results are output, and the preset comments on each dimension are output according to the deviation degree of each dimension to accurately evaluate the handwritten calligraphy chapter. The output results include the overall deviation degree of each dimension, and the location of the text with large deviation in each dimension. The comments are manually set by professional calligraphy teachers for different degrees of quality in different dimensions.

[0132] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0133] First, the calligraphy copybook character grid detection is transformed into the regression of character grid corner points, and an improved YOLOv5 model based on key point regression is proposed. The spatial attention module is introduced into the model to improve the model feature fusion capability. At the same time, a decoupled detection head is set, and a corner point regression branch is added. The three major tasks of text positioning, recognition, and corresponding character grid corner point regression are output by a single model, which reduces the time consumption caused by the serialization of a single-task model and greatly improves the efficiency of calligraphy copybook evaluation.

[0134] Second, through copybook correction, text positioning, information integration and chapter evaluation, the writing layout of the entire copybook chapter is restored, so as to conduct a multi-dimensional evaluation of the copybook as a whole and give corresponding guidance comments based on the calculation results of each dimension. Calligraphy learners can learn the shortcomings of the current copybook writing through the evaluation method, which provides effective and accurate help for their subsequent calligraphy learning.

[0135] A possible implementation of a copybook chapter evaluation device based on deep learning is given below, which is used to execute the various execution steps and corresponding technical effects of the copybook chapter evaluation method shown in the above embodiment and possible implementation. The device includes:

[0136] A copybook correction module is used to perform copybook preprocessing, copybook straight line detection and copybook rotation correction on the input copybook image in sequence to obtain a corrected copybook image;

[0137] The text localization module is used to detect handwritten characters on the corrected copybook image using a pre-built and trained multi-task model to obtain the position, category, and position information of each handwritten character;

[0138] An information integration module is used to align the handwritten character grid with the standard character grid and restore the character row and column information of the copybook based on the position, category and position information of each handwritten character and the character grid where the handwritten character is located to obtain the aligned handwritten characters and restored character row and column information;

[0139] The passage evaluation module is used to calculate the degree of deviation of each dimension based on the aligned handwritten characters and the restored text row and column information to obtain detailed evaluation results. The dimensions include uniform size, smooth writing and uniform spacing.

[0140] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for evaluating calligraphy chapters based on deep learning, characterized in that: The method comprises: The input copybook image is sequentially subjected to copybook preprocessing, copybook straight line detection and copybook rotation correction to obtain a corrected copybook image; Use the pre-built and trained multi-task model to detect handwritten characters on the corrected copybook image to obtain the position, category, and position information of each handwritten character; Based on the position, category and position information of each handwritten character, the handwritten character grid is aligned with the standard character grid and the character row and column information of the copybook is restored to obtain the aligned handwritten characters and the restored character row and column information; The degree of deviation of each dimension is calculated based on the aligned handwritten characters and the restored text row and column information to obtain detailed evaluation results, wherein the dimensions include uniform size, smooth writing and uniform spacing.

2. The copybook chapter evaluation method according to claim 1, characterized in that: The steps of sequentially performing copybook preprocessing, copybook straight line detection and copybook rotation correction on the input copybook image to obtain a corrected copybook image include: The input copybook image is cropped, scaled and grayed in sequence to obtain a grayed copybook image; The Canny edge detection algorithm is used to process the grayscale copybook image to obtain the boundary contours of the copybook row and column grids. Based on the boundary contours, the Hough transform algorithm is used to determine all straight lines and record the direction angle of each line. Based on the direction angle, the straight lines whose absolute values ​​of the direction angles are less than the preset angle are screened from all the straight lines to obtain the screened straight lines, the median of the angles of the screened straight lines is selected as the optimal rotation angle, and a reverse rotation operation is performed according to the optimal rotation angle to obtain the corrected copybook image.

3. The copybook chapter evaluation method according to claim 1, characterized in that: The multi-task model is an improved YOLOv5 target detection model based on key point regression, including the backbone network, neck network and head network; The backbone network is used to extract multi-level features from the input image and generate feature maps of different sizes. The backbone network includes convolutional layers, batch normalization layers, and activation function layers. The convolutional layers are used to extract image features, the batch normalization layers are used to accelerate model training and improve the generalization ability of the model, and the activation function layers are used to increase the nonlinear expression ability of the model. The neck network is used to fuse feature maps of different scales to enrich semantic information and positioning information. The neck network adopts FPN combined with PAN structure and adds a spatial attention module. The spatial attention module includes 3x3 convolution layer, average pooling, maximum pooling and normalization function sigmoid. The head network predicts the feature maps of each scale and performs classification and localization tasks.

4. The copybook chapter evaluation method according to claim 3 is characterized in that: The training steps of the YOLOv5 target detection model improved based on key point regression include: For the predicted box of the target, encode and decode based on the size and position of the anchor box, and predict the deviation relative to the anchor box; The loss functions of the YOLOv5 target detection model improved based on key point regression include corner regression loss function, classification loss function, confidence loss function, and positioning loss function. i represents the corner point number, d represents the Euclidean distance between the predicted corner point coordinates and the true coordinates, S represents the scale factor, σ represents the weight of the current corner point, and δ represents whether the current corner point is visible; The overall loss function of the model is L total =α1L cls +α2L box +α3L obj +α4L kpt , where L cls is the classification loss function, L obj is the confidence loss function, L box is the positioning loss function, L kpt is the corner regression loss function, and α1, α2, α3, and α4 are the corresponding weights respectively.

5. The copybook chapter evaluation method according to claim 1, characterized in that: The step of aligning the handwritten character grid with the standard character grid based on the position and category of each handwritten character and the character grid position information of the handwritten character to obtain the aligned handwritten character includes: According to the name of the character, search for the corresponding standard character grid size and standard character width and height in a pre-set database; The side length of the character grid is determined according to the coordinates of the corner points of the handwritten character grid, the character grid is scaled to be consistent with the size of the standard character grid, and the size of the outer rectangular frame of the handwritten character is scaled according to the character grid scaling ratio to obtain the aligned handwritten character.

6. The copybook chapter evaluation method according to claim 1, characterized in that: The step of restoring the text row and column information of the copybook based on the position, category and position information of each handwritten character in the character grid where the handwritten character is located to obtain the restored text row and column information includes: Sort all handwritten characters from small to large according to the coordinates of the upper left corner of the character grid; Set a threshold, group the handwritten words with a vertical coordinate smaller than the threshold into one row, and arrange the words in the same row from left to right according to the size of the horizontal coordinate; The handwritten characters with abscissas smaller than a threshold value are grouped into one column, and the characters in the same column are arranged from top to bottom according to the size of the ordinates to obtain the restored character row and column information.

7. The copybook chapter evaluation method according to claim 1, characterized in that: The steps of calculating the degree of deviation of uniform size based on the aligned handwritten characters and the restored text row and column information to obtain detailed evaluation results include: Calculate the area of ​​a single handwritten character and a standard character based on the aligned handwritten characters and the restored text row and column information; Calculate the area difference based on the area of ​​a single handwritten character and a standard character; Calculate the average area difference of all handwritten characters based on the area difference; Calculate the degree of deviation for each handwritten word; The handwritten characters are screened by a preset first deviation threshold, and the degree of deviation is recorded to obtain detailed evaluation results.

8. The copybook chapter evaluation method according to claim 1, characterized in that: The steps of calculating the deviation degree of writing smoothness based on the aligned handwritten characters and the restored text row and column information to obtain detailed evaluation results include: Calculate the center coordinates of the handwritten characters based on the aligned handwritten characters and the restored text row and column information; Calculate the mean of the ordinate based on the center coordinates of the handwritten characters; The deviation degree of each handwritten character is calculated based on the mean value of the ordinate, the handwritten characters are screened through a preset second deviation threshold, and the deviation degree is recorded to obtain detailed evaluation results.

9. The copybook chapter evaluation method according to claim 1, characterized in that: The steps of calculating the degree of deviation of uniform spacing based on the aligned handwritten characters and the restored text row and column information to obtain detailed evaluation results include: Calculate the mean vertical coordinate of each row based on the aligned handwritten characters and restored text row and column information; Calculate the spacing between adjacent rows based on the mean of the ordinate of each row; Calculate the mean spacing based on the spacing between adjacent rows; Calculate the degree of spacing deviation based on the spacing mean; Based on the degree of spacing deviation, adjacent rows are screened out through a preset third deviation threshold, and the degree of deviation is recorded.

10. A calligraphy chapter evaluation device based on deep learning, characterized in that: The device comprises: A copybook correction module is used to perform copybook preprocessing, copybook straight line detection and copybook rotation correction on the input copybook image in sequence to obtain a corrected copybook image; The text localization module is used to detect handwritten characters on the corrected copybook image using a pre-built and trained multi-task model to obtain the position, category, and position information of each handwritten character; An information integration module is used to align the handwritten character grid with the standard character grid and restore the character row and column information of the copybook based on the position, category and position information of each handwritten character and the character grid where the handwritten character is located to obtain the aligned handwritten characters and restored character row and column information; The passage evaluation module is used to calculate the degree of deviation of each dimension based on the aligned handwritten characters and the restored text row and column information to obtain detailed evaluation results. The dimensions include uniform size, smooth writing and uniform spacing.

Images