Method, apparatus and medium for scoring neatness of handwritten text

Abstract

This disclosure relates to a method, apparatus, device, and medium for scoring the neatness of handwritten text. The method involves acquiring an image to be scored, which includes a handwritten piece of paper with at least one line of handwritten text; detecting the image to be scored using a pre-trained detection model and generating detection results, including the vertex positions of the handwritten paper and the positions of each line of handwritten text; determining a first score for the handwritten text in the image to be scored based on the vertex positions of the handwritten paper and the positions of each line of handwritten text; recognizing each line of handwritten text in the image to be scored using a pre-trained recognition model based on the position of each line of handwritten text, and generating a second score based on the recognition results of each line of handwritten text; generating a third score based on the detection results and the recognition results of each line of handwritten text using a pre-trained scoring model; and accurately scoring the neatness of the handwritten text in the image to be scored using the multiple scores obtained above.

Description

Technical Field

[0001] This disclosure relates to the field of education, and more particularly to a method, apparatus, device, and medium for scoring the neatness of handwritten text. Background Technology

[0002] With the rapid development of artificial intelligence technology, AI-based intelligent question-judging methods are gradually being applied in the education field. For example, users can take a picture of the paper after writing their answers using a terminal device, upload the image to a relevant application for question-judging, and then the application sends the image to a question type judgment model to judge the correctness of the question content in the image and output the judgment result.

[0003] However, for any written content, in addition to accuracy, neatness is also a focus of attention. Neatness affects the writer's writing habits and attitude. Therefore, it is particularly important to accurately evaluate the neatness of written text in images. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides a method, apparatus, device, and medium for scoring the neatness of handwritten text, which can quickly and accurately score the neatness of handwritten text in images.

[0005] According to one aspect of this disclosure, a method for scoring the neatness of handwritten text is provided, comprising:

[0006] Obtain an image to be scored, the image to be scored comprising a handwritten piece of paper with at least one line of handwritten text written on it;

[0007] The image to be scored is detected based on a pre-trained detection model, and detection results are generated. The detection results include the vertex position of the handwritten paper and the position of each line of handwritten text in the at least one line of handwritten text.

[0008] The first score of the handwritten text in the image to be scored is determined based on the vertex position of the handwritten paper and the position of each line of handwritten text in the at least one line of handwritten text;

[0009] Based on the position of each line of handwritten text in the at least one line of handwritten text, a pre-trained recognition model is used to recognize each line of handwritten text in the image to be scored, and a second score is generated based on the recognition result of each line of handwritten text.

[0010] Based on a pre-trained scoring model, a third score is generated according to the detection results and the recognition results of each line of handwritten text;

[0011] The neatness score of the handwritten text is obtained based on the first score, the second score, and the third score.

[0012] According to another aspect of this disclosure, a scoring device for the neatness of handwritten text is provided, comprising:

[0013] An acquisition unit is used to acquire an image to be scored, the image to be scored including a handwritten paper with at least one line of handwritten text written on it;

[0014] The detection unit is used to detect the image to be scored based on a pre-trained detection model and generate detection results, the detection results including the vertex position of the handwritten paper and the position of each line of handwritten text in the at least one line of handwritten text;

[0015] The first scoring unit is used to determine the first score of the handwritten text in the image to be scored based on the vertex position of the handwritten paper and the position of each line of handwritten text in the at least one line of handwritten text;

[0016] The second scoring unit is used to identify each line of handwritten text in the image to be scored based on the position of each line of handwritten text in the at least one line of handwritten text, using a pre-trained recognition model, and to generate a second score based on the recognition result of each line of handwritten text.

[0017] The third scoring unit is used to generate a third score based on the detection results and the recognition results of each line of handwritten text, using a pre-trained scoring model.

[0018] The fourth scoring unit is used to obtain a neatness score for the handwritten text based on the first score, the second score, and the third score.

[0019] According to another aspect of this disclosure, an electronic device is provided, the electronic device comprising: a processor; and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the scoring method for neatness of handwritten text as described above.

[0020] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to execute a scoring method based on the neatness of handwritten text.

[0021] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method for scoring the neatness of handwritten text.

[0022] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0023] The method disclosed herein includes: acquiring an image to be scored, the image to be scored including a handwritten paper with at least one line of handwritten text; detecting the image to be scored based on a pre-trained detection model and generating detection results, the detection results including the vertex positions of the handwritten paper and the positions of each line of handwritten text; determining a first score for the handwritten text in the image to be scored based on the vertex positions of the handwritten paper and the positions of each line of handwritten text; recognizing each line of handwritten text in the image to be scored based on the position of each line of handwritten text using a pre-trained recognition model, and generating a second score based on the recognition results of each line of handwritten text; generating a third score based on the detection results and the recognition results of each line of handwritten text using a pre-trained scoring model; and accurately scoring the neatness of the handwritten text in the image to be scored based on the multiple scores obtained above. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0025] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram illustrating an application scenario provided by an embodiment of this disclosure;

[0027] Figure 2 A flowchart of a model training method provided in this embodiment of the disclosure;

[0028] Figure 3 A schematic diagram of a writing sample image provided in an embodiment of this disclosure;

[0029] Figure 4 A network structure diagram of a model provided in an embodiment of this disclosure;

[0030] Figure 5 A flowchart of a model training method provided in this embodiment of the disclosure;

[0031] Figure 6 A flowchart illustrating a method for scoring the neatness of handwritten text, provided as an embodiment of this disclosure;

[0032] Figure 7 A schematic diagram of the structure of a scoring device for the neatness of handwritten text provided in an embodiment of this disclosure;

[0033] Figure 8This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0034] To better understand the above-described objects, features, and advantages of this disclosure, embodiments of the disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of the disclosure are shown in the drawings, it should be understood that the disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0035] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0036] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0037] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0038] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0039] Currently, AI-based intelligent judgment methods are suitable for logically correctable questions, such as elementary school math mental arithmetic problems. These methods have achieved relatively good grading results. For questions that cannot be logically corrected, such as multiple-choice questions, word problems, diagram problems, and matching problems, grading can be achieved through question bank comparison or natural language processing methods, with good results already. This has enabled comprehensive grading, effectively reducing the grading burden on parents and teachers. However, using question bank comparison requires building a dedicated question bank, which is costly. Using natural language processing methods is also difficult to implement. Secondly, for any assignment, judging right or wrong is a primary aspect to help students consolidate their knowledge. However, the neatness of handwriting also needs attention. Neat handwriting is crucial for cultivating good habits and learning attitudes. Therefore, accurately judging the neatness of users' handwriting and providing corresponding suggestions can further help users establish good handwriting habits.

[0040] To address the aforementioned technical problems, this disclosure provides a method for scoring the neatness of handwritten text, which will be described in detail through one or more of the following embodiments.

[0041] Specifically, the scoring method for the neatness of handwritten text can be executed by a terminal or a server. Specifically, the terminal or server can score the neatness of handwritten text in the image to be scored using pre-trained detection, recognition, classification, and scoring models. The entity executing the training methods for the detection, recognition, classification, and scoring models can be the same as or different from the entity executing the scoring method for the neatness of handwritten text.

[0042] For example, in one application scenario, such as Figure 1 As shown, Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this disclosure. Figure 1 Server 12 trains the detection model, recognition model, classification model, and scoring model. Terminal 11 obtains the trained detection model, recognition model, classification model, and scoring model from server 12. Terminal 11 uses these trained models to score the neatness of the handwritten text in the image to be scored. The image to be scored can be captured by terminal 11. Alternatively, the image to be scored can be obtained by terminal 11 from other devices. Or, the image to be scored can be obtained by terminal 11 after processing a preset image, which can be captured by terminal 11 or obtained by terminal 11 from other devices. Here, no specific limitation is made to the other devices.

[0043] In another application scenario, server 12 trains detection, recognition, classification, and scoring models. Further, server 12 uses the trained detection, recognition, classification, and scoring models to score the neatness of the handwritten text in the image to be scored. The method by which server 12 acquires the image to be scored can be similar to the method by which terminal 11 acquires the image to be scored, as described above, and will not be repeated here.

[0044] In another application scenario, terminal 11 trains detection, recognition, classification, and scoring models. Furthermore, terminal 11 uses the trained detection, recognition, classification, and scoring models to score the neatness of the written text in the image to be scored.

[0045] It is understood that the model training method and the handwritten text neatness scoring method provided in this disclosure are not limited to the possible scenarios described above. Since the trained recognition model can be applied to the handwritten text neatness scoring method, the model training method can be introduced below before introducing the handwritten text neatness scoring method.

[0046] The following example, using server 12 to train detection, recognition, classification, and scoring models, illustrates a model training method, specifically the training process for these models. It is understood that this model training method is also applicable to the scenario where terminal 11 trains a recognition model.

[0047] Figure 2 This is a flowchart of a model training method provided in an embodiment of the present disclosure. The detection model, the recognition model, the classification model, and the scoring model are obtained through the following training methods, specifically including: Figure 2 The following steps S210 to S220 are shown:

[0048] S210. Construct a training dataset, wherein the training dataset includes handwritten sample images and the annotation information corresponding to the handwritten sample images.

[0049] Understandably, the method provided in this disclosure scores the neatness of handwritten text from aspects such as whether the handwritten text is neat and square, whether single-line text is curved or slanted, and whether the written text has smudges or scratches, providing an objective and standardized neatness scoring system. Specifically, a large number of handwritten text images are collected. Users write text on handwriting paper, and the handwriting paper can be photographed to generate handwritten text images. The handwriting paper can be homework paper, answer sheet, or other types of paper. For example, the handwritten text image can be an image of a completed homework assignment. After collecting a large number of handwritten sample images, each handwritten sample image is labeled to obtain a large amount of labeling information corresponding to the handwritten sample images. Then, a training dataset is constructed based on the handwritten sample images and the corresponding labeling information.

[0050] Optionally, the training dataset may include multiple subsets of training data.

[0051] Optionally, the construction of training data in S210 above specifically includes the following steps S211 to S215:

[0052] S211. Collect a large number of handwritten sample images, wherein the handwritten sample images include multiple lines of handwritten text.

[0053] S212. The overall neatness of the handwritten text in the handwritten sample image is scored and labeled to generate first labeling information, and the first labeling information constitutes a first training data subset.

[0054] S213. Mark the positions of the four vertices of the writing paper in the handwritten sample image, generate second annotation information, and form a second training data subset from the second annotation information.

[0055] S214. Use angled rectangles to annotate each line of handwritten text in the handwritten sample image to generate third annotation information, and use the third annotation information to form a third training data subset.

[0056] S215. In the first training data subset, a target handwritten sample image with a neatness score greater than a preset threshold is determined, and multiple cropped images are generated by cropping the target handwritten sample image according to the third annotation information of the target handwritten sample image. The handwritten text in the multiple cropped images is annotated to generate fourth annotation information, and the fourth annotation information constitutes a fourth training data subset, wherein each cropped image includes a line of handwritten text.

[0057] Understandably, a large number of handwritten sample images are collected. Each handwritten sample image includes at least a portion of a handwritten paper, and the handwritten paper includes at least one line of handwritten text. Each line of handwritten text consists of at least one character. The following example uses 100 handwritten sample images as an illustration. The overall neatness of the handwritten text in each of the 100 collected handwritten sample images is scored and labeled to generate first labeling information. That is, the neatness of the overall handwritten text in the handwritten sample image is scored. For example, a 5-point scale is used, and the score value is a discrete number, such as 1, 2, 3, 4, and 5. Multiple score values ​​can also be preset. The overall neatness of the handwritten sample image is scored within multiple preset score value ranges. If the handwritten sample image looks relatively neat overall, without smears, scratches, or tilting, the handwritten sample image can be labeled as 5; otherwise, the handwritten sample image can be labeled as 1. Each handwritten sample image has a corresponding first labeling information. Then, the first training data subset is formed based on 100 sets of first labeling information. The first step involves labeling the positions of the four vertices of the writing paper in each of the 100 collected handwritten sample images. Each vertex's position can be understood as the coordinates of a pixel in the image. This generates the second annotation information. Each handwritten sample image has corresponding second annotation information, including the coordinates of the four vertices. If the handwritten paper occupies the entire image, then the four vertices of the handwritten sample image are the four vertices of the handwritten paper. Alternatively, if the four vertices of the handwritten paper are all within the handwritten sample image, then their positions are the pixel positions in the handwritten sample image. These 100 sets of second annotation information constitute the second training data subset. The second step involves labeling each line of handwritten text in the handwritten sample images using angled rectangles, generating the third annotation information. This involves using angled rectangles to select each line of text in the handwritten sample images. Each handwritten sample image has a corresponding set of third annotation information, including the coordinates of multiple angled rectangles. These 100 sets of third annotation information constitute the third training data subset.After constructing the first and third training data subsets, target handwritten sample images with a neatness score greater than a preset threshold are determined in the first training data subset. For example, the preset threshold can be 4 or 5 points. That is, handwritten sample images with a score greater than 4 points in the first training data subset are used as target handwritten sample images. Then, the target handwritten sample images are cropped to generate multiple cropped images according to the third annotation information corresponding to the target handwritten sample images. That is, the target handwritten sample images are cropped according to the box selection position of each line of handwritten text in the third training data subset. Multiple cropped images containing a line of handwritten text are obtained. The handwritten text in the multiple cropped images is annotated to generate fourth annotation information. That is, each character in the cropped image containing a line of handwritten text is annotated. The resulting multiple character sequences are used as the fourth annotation information corresponding to the target handwritten text image. Then, the fourth training data subset is constructed based on 100 sets of fourth annotation information. Finally, 100 original images of handwritten text were used as the fifth training data subset, and the training dataset was constructed based on the first, second, third, fourth, and fifth training data subsets.

[0058] For example, see Figure 3 , Figure 3 This is a schematic diagram of a writing sample image provided in an embodiment of the present disclosure. In the writing sample image 310, the handwritten paper occupies the entire image. In this case, the four vertices of the handwritten paper are the four vertices of the writing sample image 310. In the writing sample image 320, the handwritten paper occupies part of the image. In this case, the four vertices of the handwritten paper are shown as 321 to 324 in the writing sample image 320. Among them, vertices 323 and 324 are the intersection points of the handwritten paper and the edge of the image.

[0059] S220. The detection model, the recognition model, the classification model, and the scoring model are trained using the training dataset.

[0060] Understandably, based on the above S210, after the training dataset is constructed, the pre-built detection model, recognition model, classification model and scoring model are trained using the training dataset.

[0061] For example, see Figure 4 , Figure 4 This is a network structure diagram of a model provided in an embodiment of the present disclosure. Figure 4 This describes the flow between different models during use, specifically... Figure 4It includes a detection model 410, a recognition model 420, a classification model 430, and a scoring model 440. During use, that is, after the model training is completed, the input of the detection model 410, the recognition model 420, and the classification model 430 is the handwritten image, and the input of the scoring model 440 is the output of the detection model 410 and the recognition model 420. The detection model 410 is built based on a CenterNet network. The detection model 410 includes a first feature extraction layer 411, a first convolutional layer 412, and a second convolutional layer 413. The output of the first feature extraction layer 411 serves as the input to the first convolutional layer 412 and the second convolutional layer 413. The first convolutional layer 412 and the second convolutional layer 413 are connected in parallel. The outputs of the first convolutional layer 412 and the second convolutional layer 413 are the outputs of the detection model 410. The first feature extraction layer 411 uses a residual network as its backbone network. The residual network can be a ResNet18. The residual network contains four convolutional blocks, denoted as blocks, and each convolutional block includes several convolutional layers. The feature information of the handwritten sample image is obtained by adjusting the stride length of the convolution operation in different convolution blocks. The feature information is a set of multi-channel feature maps, which can be referred to as the first feature map. The first convolutional layer 412 includes multiple convolutional layers and multiple deconvolutional layers. Specifically, the first convolutional layer 412 includes two convolutional layers and three deconvolutional layers connected in sequence. The first convolutional layer 412 takes the output of the first feature extraction layer 411 as input and outputs a set of four-channel feature maps. The feature map of each channel is a vertex score map. The vertex score map reflects the position of the vertex of the handwritten paper in the handwritten sample image. The four-channel feature map represents the position of the top, bottom, left, and right vertices of the handwritten paper.The second convolutional layer 413 includes multiple convolutional layers and multiple deconvolutional layers. Specifically, the second convolutional layer 413 includes six convolutional layers and three deconvolutional layers connected in sequence. The output of each deconvolutional layer serves as the input to the next deconvolutional layer and then passes through another convolutional layer to become one of the outputs of the detection model 410. The three deconvolutional layers in the second convolutional layer 413 output three sets of feature maps, which can be seen as the outputs of three branches. The feature map output by the second convolutional layer 413 is denoted as the third feature map. The input of the first deconvolutional layer is the output of the last convolutional layer among the six convolutional layers. The output of the first deconvolutional layer passes through a convolutional layer... The first branch outputs a set of one-channel feature maps, which is the center point score map of the single-line handwritten text. The input of the second deconvolution layer out of the three deconvolution layers is the output of the first deconvolution layer. The output of the second deconvolution layer is passed through a convolution layer and outputs a set of two-channel feature maps (the output of the second branch). The output of the second branch represents the height and width of the single-line handwritten text. The third deconvolution layer out of the three deconvolution layers outputs the output of the second deconvolution layer. The output of the third deconvolution layer is passed through a convolution layer and outputs a set of one-channel feature maps (the output of the third branch). The output of the third branch represents the tilt angle of the single-line handwritten text.Understandably, the first convolutional layer 412 outputs the positions of the four vertices of the handwritten paper, corresponding to the second training data subset mentioned above. The second convolutional layer 413 outputs the position of the rectangular frame with an angle for each single line of handwritten text in the handwritten sample image, corresponding to the third training data subset mentioned above. A recognition model 420 is constructed based on a convolutional recurrent neural network (CRNN). The recognition model 420 includes a second feature extraction layer 421 and two layers of bidirectional long short-term memory network (LSTM) 422. The input of the recognition module 420 is multiple images containing single lines of handwritten text obtained after cropping the handwritten sample images. The recognition model 420 is used to recognize the text in the images containing single lines of handwritten text and outputs the recognition result and recognition confidence of each image containing single lines of handwritten text, corresponding to the fourth training data subset. The recognition confidence is used to calculate the score, and the recognition result refers to the recognition result of each character in the image. Classification model 430 is built based on a residual network. Its input is a handwritten sample image, and its output is the overall neatness score of the handwritten sample image, corresponding to the first training data subset. Scoring model 440 is built based on the encoder part of an attention mechanism (Transformer) model. Unlike the Transformer encoder, the input of scoring model 440 is not a single character but all characters of a complete single-line handwritten text. That is, during use, the encoder input of scoring model 440 is the recognition result output by recognition model 430, and during training, the input of scoring model 440 is the fourth training data subset. Secondly, the input of the position encoder in scoring model 440 is the position of the single-line handwritten text output by detection model 410 during use, and during training, it is the third training data subset. Based on the position of the single-line handwritten text and the recognition result of the current line, an output is obtained. This output then passes through two fully connected layers to output the score of the handwritten sample image. The second fully connected layer has 5 nodes, corresponding to the 5-ary scoring of the overall neatness of the handwritten sample image in the first training dataset. ;

[0062] Understandably, the convolutional and deconvolutional network parameters involved in the various models constructed above are different, and the convolutional and deconvolutional layers are different in different models.

[0063] This disclosure provides a model training method that generates multiple training data subsets by annotating a large number of handwriting sample images in different directions. Pre-built detection, classification, recognition, and scoring models are then trained using these subsets. By acquiring annotation information from handwriting sample images at different angles and training multiple different models using this annotation information, the method maximizes the fairness and objectivity of handwriting neatness scoring. It also demonstrates the scoring criteria for handwriting neatness, facilitating subsequent rapid and accurate objective judgment of handwriting neatness by combining the functions of the scoring, classification, detection, and recognition models, along with added geometric constraints.

[0064] Based on the above embodiments, Figure 5 This is a flowchart of a model training method provided in an embodiment of the present disclosure. Optionally, the detection model, the recognition model, the classification model, and the scoring model are trained using the training dataset, specifically including, as follows: Figure 5 The following steps S510 to S540 are shown:

[0065] S510. The handwritten sample image is used as the input to the detection model, and the detection model is trained based on the second training data subset, the third training data subset, and the output of the detection model.

[0066] Optionally, the detection model includes a first feature extraction layer, a first convolutional layer, and a second convolutional layer.

[0067] Optionally, the step S510 above, which uses the handwritten sample image as input to the detection model and trains the detection model based on the second training data subset, the third training data subset, and the output of the detection model, specifically includes the following steps S511 to S514:

[0068] S511. Use the first feature extraction layer to extract features from the handwritten sample image to obtain a set of multi-channel first feature maps.

[0069] S512. Convolve the first feature map using the first convolutional layer to obtain a set of four-channel second feature maps, wherein each channel of the second feature map represents the vertex score map of one of the four vertices of the writing paper in the handwritten sample image, and the vertex score map is used to determine the position of the vertex.

[0070] S513. Convolve the first feature map using the second convolutional layer to obtain multiple sets of third feature maps. The second convolutional layer includes multiple deconvolutional layers connected in sequence, and each deconvolutional layer outputs a set of third feature maps. The first set of third feature maps in the multiple sets of third feature maps represents the center point score map of each line of handwritten text. The second set of third feature maps in the multiple sets of third feature maps represents the width and height of each line of handwritten text. The third set of third feature maps in the multiple sets of third feature maps represents the tilt angle of each line of handwritten text.

[0071] S514. The detection model is trained based on the second training data subset, the third training data subset, the second feature map, and the third feature map.

[0072] Understandably, after constructing the training dataset, handwritten sample images are used as input to the pre-built detection model. The first feature extraction layer in the detection model extracts feature information from the handwritten sample images to obtain the first feature map. This first feature map is then input into the first and second convolutional layers. The second convolutional layer detects the position of each line of handwritten text in the handwritten sample image based on the first feature map and outputs the predicted position of the handwritten text (multiple sets of third feature maps). The first convolutional layer detects the positions of the four vertices of the handwritten paper in the image based on the first feature map and outputs the predicted positions of the four vertices of the handwritten paper (second feature map). Subsequently, the FocalLoss loss function is used to... The first loss value is calculated using the positions of the four vertices of the accurately labeled handwritten paper in the second training data subset and the predicted positions of the four vertices of the handwritten paper output by the first convolutional layer. The angled rectangles in the third annotation information of the third training data subset include the center point position of each line of written text, the width and height of each line of written text, and the tilt angle of each line of written text. The output of the second convolutional layer in the detection model includes three branches, each outputting a set of feature maps. Specifically, the second loss value is calculated using the FocalLoss function based on the center point position of each line of written text in the third training data subset and the predicted first set of third feature maps; the third loss value is calculated using the L1 Loss function based on the width and height of each line of written text in the third training data subset and the predicted second set of third feature maps; the fourth loss value is calculated using the L1 Loss function based on the tilt angle of each line of written text in the third training data subset and the predicted third set of third feature maps; the network parameters of the detection model are updated based on the sum of the first, second, third, and fourth loss values, completing the training of the detection model.

[0073] S520. The handwritten sample image is used as the input to the classification model, and the classification model is trained based on the first training data subset and the output of the classification model.

[0074] Understandably, after obtaining the training dataset, the handwritten sample images in the training dataset are input into the pre-built classification model. The category output by the classification model is the overall neatness score of the handwritten text in the handwritten sample image. Taking a 5-point scale, the category output by the classification model is 5 integers from 1 to 5. For example, the output 1 means that the overall neatness score of the handwritten text in the handwritten sample image is 1. Then, the multi-class cross-entropy loss function is used to calculate the loss value based on the category output by the classification model and the score labeled on the handwritten sample image in the first training data subset. The network parameters of the classification model are updated based on the loss value to complete the training of the classification model.

[0075] S530. The multiple cropped images corresponding to the handwritten sample image are used as inputs to the recognition model, and the recognition model is trained based on the fourth training data subset and the output of the recognition model.

[0076] Understandably, after obtaining the training dataset, the handwritten sample images are cropped according to the rectangular boxes annotating each line of handwritten text in the third training data subset, resulting in multiple cropped images. Each cropped image includes one line of handwritten text, and each handwritten sample image is cropped based on the number of lines of handwritten text it contains. After obtaining multiple cropped images corresponding to the handwritten sample images, these multiple cropped images are used as input to the recognition model. The recognition model identifies the handwritten text in the cropped images and generates recognition results, with each cropped image corresponding to one recognition result. Subsequently, the CTC loss function is used to calculate the loss value based on the character sequence annotated in the fourth training dataset and the recognition result of the cropped image output by the recognition model. The network parameters of the recognition model are updated based on this loss value to obtain the trained recognition model.

[0077] S540. The fourth training data subset and the third training data subset are used as inputs to the scoring model, and the scoring model is trained based on the first training data subset and the output of the scoring model.

[0078] Understandably, the fourth and third training data subsets in the training dataset are used as inputs to the pre-built scoring model. The scoring model can be understood as a fine-grained scoring model for neatness. The inputs to the scoring model are the accurate text recognition results labeled in the fourth training data subset and the position of the written text in the written sample images labeled in the third training data subset. The output is the overall predicted neatness score for the written sample images. Subsequently, a multi-classification loss function is used to calculate the loss value based on the neatness score of the written sample images labeled in the first training data subset and the predicted neatness score output by the scoring model. The network parameters of the scoring model are updated according to the loss value to obtain the trained scoring model.

[0079] This disclosure provides a model training method that constructs different training data subsets and uses multiple training data subsets individually or in combination to train models constructed from different perspectives. It also proposes an objective standard for scoring the neatness of written text. According to the model training method provided by this disclosure, it is possible to accurately score neatness from different perspectives using different models.

[0080] Based on the above embodiments, Figure 6 A flowchart of a method for scoring the neatness of handwritten text provided in this disclosure embodiment specifically includes, as follows: Figure 6 The following steps S610 to S660 are shown:

[0081] S610. Obtain the image to be scored, wherein the image to be scored includes a handwritten paper with at least one line of handwritten text.

[0082] Understandably, after the above model is trained, images to be divided are obtained. Specifically, multiple images to be divided are obtained and their neatness is scored sequentially. The images to be divided include at least a portion of handwritten paper, and each portion of handwritten paper includes at least one line of handwritten text, with each line of handwritten text including at least one handwritten character.

[0083] S620. Detect the image to be scored based on a pre-trained detection model and generate detection results, the detection results including the vertex positions of the handwritten paper and the positions of each line of handwritten text in the at least one line of handwritten text.

[0084] Understandably, based on the above S610, the first feature extraction layer in the pre-trained detection model extracts the feature information of the image to be scored, and obtains a set of multi-channel feature maps. Then, the first convolutional layer and the second convolutional layer in the detection model simultaneously perform detection on the multi-channel feature maps to generate detection results. The first convolutional layer outputs the positions of the four vertices of the handwritten paper in the image to be scored, and the second convolutional layer outputs the position of each line of handwritten text included in the image to be scored. The position of each line of handwritten text can be understood as the position of the rectangle that selects the handwritten text with a tilted rectangle.

[0085] S630. Determine the first score of the handwritten text in the image to be scored based on the vertex position of the handwritten paper and the position of each line of handwritten text in the at least one line of handwritten text.

[0086] Optionally, the above S630 specifically includes the following steps S631 to S633:

[0087] S631. Perform an affine transformation based on the vertex positions of the handwritten paper and the preset vertex positions to obtain a homography matrix.

[0088] S632. Based on the homography matrix, transform the position of each line of handwritten text, and determine the tilt value of each line of handwritten text for the transformed position of each line of handwritten text.

[0089] S633. Determine the first score of the handwritten text in the image to be scored based on the tilt value of each line of handwritten text.

[0090] Understandably, based on the above S620, after the detection model outputs the positions of the four vertices of the handwritten paper and the position of each line of handwritten text, an affine transformation is performed based on the four vertices of the handwritten paper and preset vertex positions to obtain a homography matrix. The preset vertex positions can be four pre-set annotation points. The four annotation points can form a regular rectangle relative to the image to be scored, which is used to determine whether the handwritten paper is tilted in the image to be scored. When the handwritten paper is photographed to generate the image to be scored, the handwritten paper may be tilted. Subsequently, the position of each line of handwritten text is transformed based on the homography matrix, that is, it is determined whether the text line is tilted. For the transformed position of each line of handwritten text, the tilt value of each line of handwritten text is determined. The tilt value is used to represent the degree of tilt of the written text on the writing paper, that is, to determine whether the user's writing is slanted. The first score of the handwritten text in the image to be scored is determined based on the tilt value of each line of handwritten text. The first score is scored from the perspective of the tilt of the handwritten text. For example, if the image to be scored includes 10 lines of handwritten text, and each line of handwritten text has a tilt value, and a tilt threshold is preset, if 4 out of the 10 tilt values ​​are greater than or equal to the preset tilt threshold, then the first score of the image to be scored is 3. That is, 6 tilt values ​​are less than the preset tilt threshold, which is 0.6. In a 5-point system, the product of 0.6 and 5 is 3, so the first score is 3.

[0091] S640. Based on the position of each line of handwritten text in the at least one line of handwritten text, each line of handwritten text in the image to be scored is identified by a pre-trained recognition model, and a second score is generated based on the recognition result of each line of handwritten text.

[0092] Optionally, S640 specifically includes the following steps S641 to S643:

[0093] S641. The image to be scored is cropped according to the position of each line of handwritten text in the at least one line of handwritten text to obtain at least one handwritten text image, wherein each handwritten text image includes a line of handwritten text.

[0094] S642. Based on the pre-trained recognition model, the handwritten text in each handwritten text image is recognized, and the confidence level corresponding to each handwritten text image is obtained.

[0095] S643. Generate a second score based on the confidence level corresponding to each handwritten text image.

[0096] Understandably, based on the above S630, the image to be scored is cropped according to the position of each line of handwritten text in at least one line of handwritten text. The corresponding handwritten text image can be obtained based on the number of lines of handwritten text included, where each handwritten text image includes one line of handwritten text. Then, based on a pre-trained recognition model, the handwritten text in each handwritten text image is recognized, obtaining the recognition result and confidence level corresponding to the handwritten text image. The recognition result is the character recognition result in the handwritten text image, and the confidence level of the handwritten text image ranges from 0 to 1. Subsequently, a second score is generated based on the confidence level corresponding to each handwritten text image. Based on the above example, the image to be scored will be cropped to obtain 10 handwritten text images, and 10 confidence levels will be obtained through the recognition model. The 10 confidence levels are added together, averaged, and multiplied by 5 to obtain the second score, where 5 refers to a 5-point scale. Understandably, the calculation of each score is based on the same scoring system.

[0097] S650. Based on the pre-trained scoring model, a third score is generated according to the detection results and the recognition results of each line of handwritten text.

[0098] Understandably, based on the above S640, a third score is generated based on the pre-trained scoring model, according to the position of each line of handwritten text in the detection results output by the detection model and the recognition results of each line of handwritten text output by the recognition model. The third score is to score the overall neatness of the image to be scored in terms of writing details in the image to be scored. Details can be the accuracy of writing, that is, judging whether there are any characters with writing errors. The third score is also out of 5 points.

[0099] S660. Obtain the neatness score of the handwritten text based on the first score, the second score, and the third score.

[0100] Optionally, the image to be scored is classified based on a pre-trained classification model, and a fourth score is generated based on the classification result.

[0101] Understandably, before determining the neatness score of the image to be scored, the image to be scored can be input into a pre-trained classification model to score the neatness of the image from an overall perspective and output a fourth score, which is also out of 5 points.

[0102] Optionally, S660 specifically includes: obtaining a neatness score for the handwritten text by performing a weighted average of the first score, the second score, the third score, and the fourth score.

[0103] Understandably, based on the above S650, the first, second, third, and fourth scores are weighted and averaged to obtain the neatness score of the handwritten text. The weights can be determined according to the needs. Understandably, the first to fourth scores used for weighted averaging to calculate the neatness score can be determined by the user. That is, the user can choose which of the four scores to use to calculate the neatness score of the image to be scored. For example, the neatness score of the image to be scored can be calculated based on the first, second, and third scores, or the user can choose to use the first, second, and fourth scores to calculate the neatness score of the image to be scored. There is no limitation here.

[0104] Understandably, the first score is calculated based on whether the single line of handwritten text is slanted in the image. A first threshold can be preset to issue a slant reminder to the user based on the slant of the current line of handwritten text. For example, if there are 10 lines of handwritten text in the image to be scored, and the slant value of the second line of handwritten text is greater than the first threshold, it means that the second line of handwritten text was slanted when written. Therefore, the user can be notified that the second line of handwritten text is slanted to avoid slanting in subsequent writing. The second score is based on the handwriting. If the handwriting is neat and legible, the confidence level of the recognition model is higher; conversely, if the handwriting is messy, the confidence level of the recognition model is lower. Based on the second score and the preset second threshold, a reminder can be issued to the user regarding whether the handwriting is messy. The third score is based on the details of the handwritten text, further scoring whether there are scratches on the written characters and whether the writing is clear. Based on the third score and the preset third threshold, a reminder can be issued to the user regarding the details of the writing. The fourth score is based on the overall neatness of the image, which can reflect the overall writing situation, such as whether the writing is clean and tidy. Then, based on the preset fourth threshold and the fourth score, a reminder can be sent to the user to indicate whether the writing is clean and tidy.

[0105] This disclosure provides a method for scoring the neatness of handwritten text. The method scores the neatness of handwritten text in an image based on factors such as whether the written text is neat and square, whether each line of text is curved or slanted, and whether the handwritten text has smudges or scratches. It proposes an objective and standardized neatness scoring system that can quickly and accurately score the neatness of handwritten text in an image. Furthermore, it can calculate scores based on different aspects and issue corresponding writing reminders to users, facilitating correction and helping users develop better writing habits.

[0106] Figure 7 This is a schematic diagram of the structure of the handwritten text neatness scoring device provided in this embodiment of the disclosure. The handwritten text neatness scoring device provided in this embodiment of the disclosure can execute the processing flow provided in the handwritten text neatness scoring method embodiment, such as... Figure 7 As shown, the handwritten text neatness scoring device 700 includes:

[0107] The acquisition unit 710 is used to acquire an image to be scored, the image to be scored including a handwritten paper with at least one line of handwritten text written on it.

[0108] The detection unit 720 is used to detect the image to be scored based on a pre-trained detection model and generate detection results, the detection results including the vertex position of the handwritten paper and the position of each line of handwritten text in the at least one line of handwritten text;

[0109] The first scoring unit 730 is used to determine the first score of the handwritten text in the image to be scored based on the vertex position of the handwritten paper and the position of each line of handwritten text in the at least one line of handwritten text;

[0110] The second scoring unit 740 is used to identify each line of handwritten text in the image to be scored based on the position of each line of handwritten text in the at least one line of handwritten text, using a pre-trained recognition model, and to generate a second score based on the recognition result of each line of handwritten text.

[0111] The third scoring unit 750 is used to generate a third score based on the detection results and the recognition results of each line of handwritten text, using a pre-trained scoring model.

[0112] The fourth scoring unit 760 is used to obtain a neatness score of the handwritten text based on the first score, the second score and the third score.

[0113] Optionally, the first scoring unit 730 determines the first score of the handwritten text in the image to be scored based on the vertex position of the handwritten paper and the position of each line of handwritten text in the at least one line of handwritten text, specifically for:

[0114] An affine transformation is performed based on the vertex positions of the handwritten paper and the preset vertex positions to obtain the homography matrix;

[0115] The position of each line of handwritten text is transformed based on the homography matrix, and the tilt value of each line of handwritten text is determined for the transformed position of each line of handwritten text.

[0116] The first score of the handwritten text in the image to be scored is determined based on the tilt value of each line of handwritten text.

[0117] Optionally, the second scoring unit 740, based on the position of each line of handwritten text in the at least one line of handwritten text, identifies each line of handwritten text in the image to be scored using a pre-trained recognition model, and generates a second score based on the recognition result of each line of handwritten text, specifically for:

[0118] The image to be scored is cropped according to the position of each line of handwritten text in the at least one line of handwritten text to obtain at least one handwritten text image, wherein each handwritten text image includes a line of handwritten text;

[0119] Based on a pre-trained recognition model, the handwritten text in each handwritten text image is recognized, and the confidence level corresponding to each handwritten text image is obtained.

[0120] A second score is generated based on the confidence level corresponding to each handwritten text image.

[0121] Optionally, the device 700 further includes a fifth scoring unit, which is specifically used for:

[0122] The image to be scored is classified based on a pre-trained classification model, and a fourth score is generated based on the classification results.

[0123] Optionally, the fourth scoring unit 760, which obtains the neatness score of the handwritten text based on the first score, the second score, and the third score, is specifically used for:

[0124] The neatness score of the handwritten text is obtained by weighting the first score, the second score, the third score, and the fourth score.

[0125] Optionally, the device 700 further includes a training unit, which is used to obtain the detection model, the recognition model, the classification model, and the scoring model through the following training methods:

[0126] Construct a training dataset, wherein the training dataset includes handwritten sample images and the corresponding annotation information of the handwritten sample images;

[0127] The detection model, the recognition model, the classification model, and the scoring model are trained using the training dataset.

[0128] Optionally, the training dataset in the training unit may include multiple subsets of training data.

[0129] Optionally, the construction of the training dataset in the training unit is specifically used for:

[0130] Collect a large number of handwritten sample images, which include multiple lines of handwritten text;

[0131] The overall neatness of handwritten text in the handwritten sample images is scored and labeled to generate first labeling information, and the first labeling information constitutes a first training data subset;

[0132] The positions of the four vertices of the writing paper in the handwritten sample image are marked to generate second annotation information, and the second annotation information constitutes a second training data subset;

[0133] Each line of handwritten text in the handwritten sample image is labeled using angled rectangles to generate third labeling information, and the third labeling information constitutes a third training data subset.

[0134] In the first training data subset, target handwritten sample images with a neatness score greater than a preset threshold are determined. Based on the third annotation information of the target handwritten sample images, the target handwritten sample images are cropped to generate multiple cropped images. The handwritten text in the multiple cropped images is annotated to generate fourth annotation information. The fourth annotation information constitutes the fourth training data subset, wherein each cropped image includes a line of handwritten text.

[0135] Optionally, the training unit's use of the training dataset to train the detection model, the recognition model, the classification model, and the scoring model is specifically used for:

[0136] The handwritten sample image is used as the input to the detection model, and the detection model is trained based on the second training data subset, the third training data subset, and the output of the detection model.

[0137] The handwritten sample image is used as the input to the classification model, and the classification model is trained based on the first training data subset and the output of the classification model.

[0138] The recognition model is trained by using multiple cropped images corresponding to the handwritten sample images as inputs and by training the recognition model based on the fourth training data subset and the output of the recognition model.

[0139] The fourth and third training data subsets are used as inputs to the scoring model, and the scoring model is trained based on the first training data subset and the output of the scoring model.

[0140] Optionally, the detection model in the training unit includes a first feature extraction layer, a first convolutional layer, and a second convolutional layer.

[0141] Optionally, the training unit uses the handwritten sample image as input to the detection model and trains the detection model based on the second training data subset, the third training data subset, and the output of the detection model, specifically for:

[0142] The first feature extraction layer is used to extract features from the handwritten sample image to obtain a set of multi-channel first feature maps;

[0143] The first feature map is convolved using the first convolutional layer to obtain a set of four-channel second feature maps, wherein each channel of the second feature map represents the vertex score map of one of the four vertices of the writing paper in the handwritten sample image, and the vertex score map is used to determine the position of the vertex.

[0144] The first feature map is convolved by the second convolutional layer to obtain multiple sets of third feature maps. The multiple deconvolutional layers included in the second convolutional layer are connected in sequence, and each deconvolutional layer outputs a set of third feature maps. The first set of third feature maps in the multiple sets of third feature maps represents the center point score map of each line of handwritten text. The second set of third feature maps in the multiple sets of third feature maps represents the width and height of each line of handwritten text. The third set of third feature maps in the multiple sets of third feature maps represents the tilt angle of each line of handwritten text.

[0145] The detection model is trained based on the second training data subset, the third training data subset, the second feature map, and the third feature map.

[0146] The device provided in this embodiment has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0147] Exemplary embodiments of this disclosure also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this disclosure.

[0148] Exemplary embodiments of this disclosure also provide a computer program product, including a computer program, wherein, when executed by a processor of a computer, the computer program is used to cause the computer to perform a method according to an embodiment of this disclosure.

[0149] refer to Figure 8The present invention describes a structural block diagram of an electronic device 800 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0150] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0151] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, output unit 807, storage unit 808, and communication unit 809. Input unit 806 can be any type of device capable of inputting information to electronic device 800. Input unit 806 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 807 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 808 may include, but is not limited to, disks and optical discs. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0152] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above. For example, in some embodiments, the method for scoring the neatness of handwritten text can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. In some embodiments, the computing unit 801 can be configured to perform the method for scoring the neatness of handwritten text by any other suitable means (e.g., by means of firmware).

[0153] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0154] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0155] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0156] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0157] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0158] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0159] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for scoring the neatness of handwritten text, characterized in that, The method includes: Obtain an image to be scored, the image to be scored comprising a handwritten piece of paper with at least one line of handwritten text written on it; The image to be scored is detected based on a pre-trained detection model, and detection results are generated. The detection results include the vertex position of the handwritten paper and the position of each line of handwritten text in the at least one line of handwritten text. The first score of the handwritten text in the image to be scored is determined based on the vertex position of the handwritten paper and the position of each line of handwritten text in the at least one line of handwritten text; Based on the position of each line of handwritten text in the at least one line of handwritten text, a pre-trained recognition model is used to recognize each line of handwritten text in the image to be scored, and a second score is generated based on the recognition result of each line of handwritten text. Based on a pre-trained scoring model, a third score is generated according to the detection results and the recognition results of each line of handwritten text; The neatness score of the handwritten text is obtained based on the first score, the second score, and the third score; The detection model includes a first feature extraction layer, a first convolutional layer, and a second convolutional layer. The first feature extraction layer is used to extract features from the input handwritten sample image to obtain a first feature map; The first convolutional layer is used to convolve the first feature map to obtain the second feature map; the second feature map is associated with the positions of the four vertices of the writing paper in the handwritten sample image; The second convolutional layer is used to convolve the first feature map to obtain a third feature map; the third feature map is associated with the center point position of each line of handwritten text in the handwritten sample image, the width and height of each line of handwritten text, and the tilt angle of each line of handwritten text. The outputs of the first convolutional layer and the second convolutional layer are the outputs of the detection model.

2. The method according to claim 1, characterized in that, The step of determining the first score of the handwritten text in the image to be scored based on the vertex position of the handwritten paper and the position of each line of handwritten text in the at least one line of handwritten text includes: An affine transformation is performed based on the vertex positions of the handwritten paper and the preset vertex positions to obtain the homography matrix; The position of each line of handwritten text is transformed based on the homography matrix, and the tilt value of each line of handwritten text is determined for the transformed position of each line of handwritten text. The first score of the handwritten text in the image to be scored is determined based on the tilt value of each line of handwritten text.

3. The method according to claim 1, characterized in that, The step of recognizing each line of handwritten text in the image to be scored using a pre-trained recognition model based on the position of each line of handwritten text in the at least one line of handwritten text, and generating a second score based on the recognition result of each line of handwritten text, includes: The image to be scored is cropped according to the position of each line of handwritten text in the at least one line of handwritten text to obtain at least one handwritten text image, wherein each handwritten text image includes a line of handwritten text; Based on a pre-trained recognition model, the handwritten text in each handwritten text image is recognized, and the confidence level corresponding to each handwritten text image is obtained. A second score is generated based on the confidence level corresponding to each handwritten text image.

4. The method according to claim 1, characterized in that, The method further includes: The image to be scored is classified based on a pre-trained classification model, and a fourth score is generated based on the classification results. The step of obtaining the neatness score of the handwritten text based on the first score, the second score, and the third score includes: The neatness score of the handwritten text is obtained by weighting the first score, the second score, the third score, and the fourth score.

5. The method according to claim 4, characterized in that, The detection model, the recognition model, the classification model, and the scoring model are obtained through the following training methods: Construct a training dataset, wherein the training dataset includes handwritten sample images and the corresponding annotation information of the handwritten sample images; The detection model, the recognition model, the classification model, and the scoring model are trained using the training dataset.

6. The method according to claim 5, characterized in that, The training dataset includes multiple subsets of training data; The construction of the training dataset includes: Collect a large number of handwritten sample images, which include multiple lines of handwritten text; The overall neatness of handwritten text in the handwritten sample images is scored and labeled to generate first labeling information, and the first labeling information constitutes a first training data subset; The positions of the four vertices of the writing paper in the handwritten sample image are marked to generate second annotation information, and the second annotation information constitutes a second training data subset; Each line of handwritten text in the handwritten sample image is labeled using angled rectangles to generate third labeling information, and the third labeling information constitutes a third training data subset. In the first training data subset, target handwritten sample images with a neatness score greater than a preset threshold are determined. Based on the third annotation information of the target handwritten sample images, the target handwritten sample images are cropped to generate multiple cropped images. The handwritten text in the multiple cropped images is annotated to generate fourth annotation information. The fourth annotation information constitutes the fourth training data subset, wherein each cropped image includes a line of handwritten text.

7. The method according to claim 6, characterized in that, The step of training the detection model, the recognition model, the classification model, and the scoring model using the training dataset includes: The handwritten sample image is used as the input to the detection model, and the detection model is trained based on the second training data subset, the third training data subset, and the output of the detection model. The handwritten sample image is used as the input to the classification model, and the classification model is trained based on the first training data subset and the output of the classification model. The recognition model is trained by using multiple cropped images corresponding to the handwritten sample images as inputs and by training the recognition model based on the fourth training data subset and the output of the recognition model. The fourth and third training data subsets are used as inputs to the scoring model, and the scoring model is trained based on the first training data subset and the output of the scoring model.

8. The method according to claim 7, characterized in that, The step of using the handwritten sample image as input to the detection model and training the detection model based on the second training data subset, the third training data subset, and the output of the detection model includes: The first feature extraction layer is used to extract features from the handwritten sample image to obtain a set of multi-channel first feature maps; The first feature map is convolved using the first convolutional layer to obtain a set of four-channel second feature maps, wherein each channel of the second feature map represents the vertex score map of one of the four vertices of the writing paper in the handwritten sample image, and the vertex score map is used to determine the position of the vertex. The first feature map is convolved by the second convolutional layer to obtain multiple sets of third feature maps. The multiple deconvolutional layers included in the second convolutional layer are connected in sequence, and each deconvolutional layer outputs a set of third feature maps. The first set of third feature maps in the multiple sets of third feature maps represents the center point score map of each line of handwritten text. The second set of third feature maps in the multiple sets of third feature maps represents the width and height of each line of handwritten text. The third set of third feature maps in the multiple sets of third feature maps represents the tilt angle of each line of handwritten text. The detection model is trained based on the second training data subset, the third training data subset, the second feature map, and the third feature map.

9. A scoring device for the neatness of handwritten text, characterized in that, include: An acquisition unit is used to acquire an image to be scored, the image to be scored including a handwritten paper with at least one line of handwritten text written on it; The detection unit is used to detect the image to be scored based on a pre-trained detection model and generate detection results, the detection results including the vertex position of the handwritten paper and the position of each line of handwritten text in the at least one line of handwritten text; The first scoring unit is used to determine the first score of the handwritten text in the image to be scored based on the vertex position of the handwritten paper and the position of each line of handwritten text in the at least one line of handwritten text; The second scoring unit is used to identify each line of handwritten text in the image to be scored based on the position of each line of handwritten text in the at least one line of handwritten text, using a pre-trained recognition model, and to generate a second score based on the recognition result of each line of handwritten text. The third scoring unit is used to generate a third score based on the detection results and the recognition results of each line of handwritten text, using a pre-trained scoring model. The fourth scoring unit is used to obtain a neatness score for the handwritten text based on the first score, the second score, and the third score. The detection model includes a first feature extraction layer, a first convolutional layer, and a second convolutional layer. The first feature extraction layer is used to extract features from the input handwritten sample image to obtain a first feature map; The first convolutional layer is used to convolve the first feature map to obtain the second feature map; the second feature map is associated with the positions of the four vertices of the writing paper in the handwritten sample image; The second convolutional layer is used to convolve the first feature map to obtain a third feature map; the third feature map is associated with the center point position of each line of handwritten text in the handwritten sample image, the width and height of each line of handwritten text, and the tilt angle of each line of handwritten text. The outputs of the first convolutional layer and the second convolutional layer are the outputs of the detection model.

10. An electronic device, characterized in that, The electronic device includes: Processor; and Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the scoring method for handwritten text neatness according to any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the scoring method for handwritten text neatness according to any one of claims 1 to 8.

Images