A method for recognizing abnormal handwritten scripts and a handwritten recognition system

Abstract

A method for identifying abnormal handwritten strokes and a handwritten recognition system, comprising the following steps: establishing a first training data set and a second training data set, extracting features from the preprocessed data of the first training data set through a first convolutional network, and extracting features from the preprocessed data of the second training data set through a second convolutional network, and outputting two groups of feature maps; calculating the difference magnitude between these two groups of features in a high-dimensional space and inputting it into a decision model; performing model training, and finding an optimal cluster center data set through a clustering algorithm; the preprocessed test data enters a handwritten recognition model for handwritten recognition to obtain a first recognition result, and the first recognition result passes through the cluster center data set and then enters the decision model and the weight decision formula in sequence to obtain a final second recognition result. The present invention can achieve the purpose of accurately identifying abnormal handwritten strokes, cooperate with the handwritten recognition model, improve the recognition accuracy and speed of handwritten strokes, and has wide applicability.

Description

Technical Field

[0001] The present invention relates to the technical field of handwriting recognition, and particularly to a method for recognizing abnormal handwriting and a handwriting recognition system. Background Art

[0002] At present, the handwriting recognition technologies on the market are mainly divided into online handwriting recognition and offline handwriting recognition. The online handwritten Chinese character recognition processes the handwritten characters obtained by the writer through a physical device (such as a digital pen, a digital handwriting board or a touch screen) for online writing, and the writing trajectory is instantaneously input into the computer through regular sampling. While the offline handwritten character recognition processes the two-dimensional pictures of handwritten characters collected by an image capture device such as a scanner or a camera.

[0003] There are often some abnormal handwritings in the data collection of handwritten handwriting, such as pure meaningless doodles, handwritten handwriting that does not meet the specified requirements, etc., which do not belong to the scope of recognition. Therefore, in order to improve the recognition accuracy and speed, it is necessary to identify and eliminate these abnormal handwritings. For the definition of abnormal handwriting, different scenarios will have different meanings. For example, if the valid answer in a specified filling area is uppercase or lowercase English letters, and if the filler fills in Chinese characters or draws a five-pointed star in this area, then the "Chinese characters" or "five-pointed star" at this time will be regarded as abnormal handwriting; for example, if a student fills in the wrong position when filling in the answer sheet and fills the answer of the true or false question in the area of the multiple-choice question, then the answer of the true or false question in the multiple-choice question area will be defined as "abnormal".

[0004] Therefore, the abnormal handwriting recognition technology is of great significance to handwriting recognition. How to organically apply the abnormal handwriting recognition technology in the handwriting recognition technology to improve the accuracy and efficiency of handwriting recognition is an important research content in the field of handwriting recognition technology. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned disadvantages of the prior art, and provide a method for recognizing abnormal handwritten handwriting that can improve the accuracy and efficiency of abnormal recognition, and a handwriting recognition system using the method for recognizing abnormal handwritten handwriting.

[0006] The present invention is realized by the following technical solutions:

[0007] A method for recognizing abnormal handwritten handwriting includes the following steps:

[0008] S1. Establish two training data sets from the handwritten handwriting data, namely training data set one and training data set two. Among them, training data set one is the normal handwritten handwriting data within the recognizable range, and training data set two is a combination of the normal handwritten handwriting data within the recognizable range and the abnormal handwritten handwriting data outside the recognizable range;

[0009] S2. Preprocess the data in training dataset 1 and training dataset 2 respectively;

[0010] S3. The preprocessed data of training dataset 1 is subjected to feature extraction through convolutional network 1, and the preprocessed data of training dataset 2 is subjected to feature extraction through convolutional network 2, and two sets of feature maps are output;

[0011] S4. By calculating the two sets of feature maps output in step S3 in a high-dimensional space, the difference magnitude between these two sets of feature maps is obtained and input into the decision model;

[0012] S5. Train convolutional network 1, convolutional network 2 and the decision model until the model converges, find a high-dimensional space such that the individual differences of samples belonging to the same category mapped to this space are as small as possible, and the individual differences of samples not belonging to the same category mapped to this space are as large as possible, and save the model parameters;

[0013] S6. Use the clustering algorithm to find the optimal cluster center dataset, so that samples belonging to the same category are aggregated and the individual differences within the category are as small as possible, and the individual differences of the data distributions of clusters belonging to different categories are as large as possible;

[0014] S7. Repeat steps S5 to S6 until the obtained convolutional network 1, convolutional network 2, decision model and optimal cluster center dataset meet the requirements; that is, the recall rate and precision rate on the test set reach a certain threshold.

[0015] S8. The test data, after being preprocessed, respectively enters the convolutional network 1 and the handwritten recognition model. After entering the handwritten recognition model for handwritten recognition, recognition result 1 is obtained. Recognition result 1 passes through the optimal cluster center dataset in step S7, calculates the specific value of the individual difference magnitude of the sample in the high-dimensional space, and obtains the specific values of the difference magnitudes between the sample and all cluster centers of the category to which recognition result 1 belongs in the high-dimensional space;

[0016] S9. The specific values of the difference magnitudes in the high-dimensional space obtained in step S8 sequentially pass through the decision model and the decision strategy in step S9 to obtain the final recognition result 2.

[0017] Further, the difference magnitude is measured by distance. Let the distance of a certain sample from the j-th cluster center be d j , j ∈ [0, k], k + 1 is the number of cluster centers of the category corresponding to this sample's recognition result 1, and denote d j The value obtained after passing through the decision model in step S9 is p j , f(x) represents the decision strategy in step S9, Then the decision strategy in step S9 is the weight decision formula, and its expression is:

[0018]

[0019] Among them, B is the second recognition result, which is a value belonging to [0, 1]; x = α n (C n , R n ) is the weight function, representing the weight parameter of the nth cluster determined based on the data and sample distribution; C n is the cluster center of the nth class, and R n is the class radius of the nth class.

[0020] Furthermore, the data preprocessing method in the S2 or / and S8 step sequentially includes the following steps:

[0021] S2-1. Denoising through a denoising model;

[0022] S2-2. Through standardization and normalization steps;

[0023] S2-3. Binary encoding;

[0024] S2-4. Data augmentation.

[0025] Furthermore, the denoising model in the S2-1 step performs denoising through the stroke length and the spatial information of the stroke compared to other strokes, and it includes the following steps:

[0026] If the number of trajectory points forming the stroke is lower than the set number of points, then this stroke is considered a noise stroke and is deleted;

[0027] Calculate the length of the stroke. If the stroke length is much lower than the set stroke length, then this stroke is considered a noise stroke and is deleted.

[0028] Furthermore, the data augmentation in the S2-4 step includes data augmentation based on trajectory points, data augmentation based on strokes, and data augmentation based on the overall text. Among them, the data augmentation process based on the overall text is after the data augmentation based on trajectory points and the data augmentation based on strokes.

[0029] Furthermore, the first convolutional network or / and the second convolutional network is ResNet or MobileNet;

[0030] The decision model is a multi-layer perceptron MLP or SVM, and other neural network structures that can complete binary classification tasks and output confidence;

[0031] The difference magnitude in the S4 or / and S8 step is distance, norm, or similarity. The distance is Euclidean distance or Minkowski distance, the similarity is cosine similarity, and the norm is a custom norm.

[0032] Further, the method for finding the optimal cluster center data set through the clustering algorithm in step S6 is as follows: Determine the number of classes k + 1 through feature extraction and the clustering algorithm; Let the cluster center of the i-th class be C i , where i ∈ [0, k]. Then, in the high-dimensional space, based on the intra-class distribution of all samples in this class and the distribution of samples in other classes, set an appropriate class radius for the i-th class, denoted as R i , such that all samples belonging to this class can fall within a circle with C i as the center and R i as the radius, and all samples not belonging to this class can fall outside this circle.

[0033] Further, the training objective of the decision model in step S7 is as follows: When the samples from training data set one and the samples from training data set two have the same class and are not abnormal samples, make the output result as close to 1 as possible; When the samples from training data set one and the samples from training data set two have different classes, make the output result as close to 0 as possible.

[0034] Further, the handwriting recognition model in step S8 includes a deep neural network model;

[0035] The method for the test data to enter the handwriting recognition model and obtain the first recognition result in step S8 is as follows:

[0036] The handwritten stroke data is processed by a cutting algorithm to obtain several possible single-character stroke combinations, denoted as trajectory point segmentation combinations Combs, and the whole formed by the stroke points in one Comb in Combs is defined as a pseudo-character Traj;

[0037] According to the generated trajectory point segmentation combinations Combs, project the pseudo-characters Traj 3-g in each combination onto a two-dimensional plane to generate a binary matrix, denoted as M 3-g , and input the matrix M 3-g into the trained deep neural network model to obtain the first recognition result of each pseudo-character.

[0038] A handwriting recognition system includes a dot matrix paper, a dot matrix pen, and a recognition module. The recognition module uses the above-mentioned handwritten stroke anomaly recognition method, and the recognition module is arranged on a local server, a cloud server, or a smart terminal;

[0039] Several input boxes are set on the dot matrix paper. Specific dots are preset on the input boxes to provide coordinate information for generating dots by the dot matrix pen; The generation method of the dot matrix paper is as follows: Encode the position information and paper information of the input boxes on the paper surface, then draw the corresponding matrix points based on the algorithm, and then print or print the matrix points with specific information on a paper surface of a specific size.

[0040] The dot matrix pen includes a writing module, a nib pressure sensing module, a camera module, a data conversion module, and a data transmission module. The writing module includes a nib for writing on the dot matrix paper. The camera module includes a camera, which is disposed near the nib and is used to capture the handwriting written by the nib. The nib pressure sensing module is respectively connected to the nib and the camera module, and is used to sense the pressure when the nib contacts the dot matrix paper. When the pressure is greater than a set value, the camera module is triggered to take a picture. The data conversion module is connected to the camera module and is used to convert the captured image into an array containing coordinate points based on the dots of the dot matrix paper. The data transmission module is used to wirelessly transmit the data converted by the data conversion module to the recognition module of the cloud server, the local server, or the intelligent terminal.

[0041] The abnormal handwriting recognition method of the present invention forms two different training data sets by dividing the training data into two categories, extracts features respectively, and then calculates the difference between the two. The difference can be measured by distance, norm, or similarity. Then, through model training and searching for the optimal cluster center set, the construction of the abnormal recognition model is completed. During recognition, the test data is preprocessed and recognized by the handwriting recognition model, and then undergoes abnormal recognition processes such as calculating the distance, norm, or similarity to the cluster center in the high-dimensional space, the decision model, and the weight decision formula, so as to achieve the weighing and recognition of abnormal handwriting, and can achieve the purpose of accurately recognizing abnormal handwriting. It cooperates with the handwriting recognition model to improve the recognition accuracy and speed of handwriting. The abnormal recognition method of the present invention can be applied to the recognition of various different types of abnormal handwritings, such as Chinese characters, letters, graffiti, paintings, etc. Just replace the corresponding training data set, and it has wide applicability.

[0042] The handwriting recognition system of the present invention applies the abnormal recognition method and uses the dot matrix technology to convert the handwriting into dot matrix coordinate data, and then the background recognition module recognizes and abnormally recognizes the coordinate data. During the writing process, the writer does not need to pay attention to the recognition result. The dot matrix paper and the dot matrix pen are similar to the writing forms of ordinary pens and papers. The user can follow the normal writing habits of pens and papers, and there is no need to check the recognition result during the writing process. The operation is simple. The user does not need to adapt to complex writing operation habits, nor does he need to hold an intelligent device, and it can also avoid the influence of the intelligent device on the learning and testing of the writer, etc., and can be applied to student teaching. Description of the Drawings

[0043] Figure 1 It is a flowchart of model training in the embodiment of the abnormal recognition method for handwriting of the present invention.

[0044] Figure 2 It is a flowchart of the embodiment of the abnormal recognition method for handwriting of the present invention.

[0045] Figure 3 Schematic diagrams of pseudo characters and Combs in the embodiment of the handwritten handwriting abnormal recognition method of the present invention.

[0046] Figure 4 Block diagram of the structure of the embodiment of the handwritten recognition system of the present invention.

[0047] Figure 5 Schematic diagram of the dot matrix paper in the embodiment of the handwritten recognition system of the present invention.

[0048] Figure 6 Schematic diagram of the structure of the handwritten terminal in the embodiment of the handwritten recognition system of the present invention.

[0049] Figure 7 Schematic diagram of the structure of the handwritten terminal from another perspective in the embodiment of the handwritten recognition system of the present invention.

[0050] Figure 8 Schematic diagram of the answer sheet in the embodiment of the handwritten recognition system of the present invention.

[0051] Reference signs: 1 - pen barrel; 2 - pen tip; 3 - camera; 4 - indicator light; 5 - switch button; 6 - holding part. Detailed implementation manners

[0052] Abnormal handwriting generally refers to handwritten handwriting that is not within the recognition range and has no recognition significance, such as non-text contents like graffiti and blackening. Of course, for different handwriting recognition systems, the definition of abnormal handwriting will be different. For example, in an answer area where only capital English letters "A / B / C / D / E / F / G / H..." are allowed to be filled, lowercase English letters, Chinese characters, graffiti, etc. are regarded as abnormal handwriting; in an answer area where only "√×" are allowed to be filled, handwriting such as letters, Chinese characters, and graffiti are regarded as abnormal handwriting; if it is stipulated that the way of correction can only be represented by a diagonal line covering the text to be deleted, then ways such as "drawing circles", "drawing broken lines", "blackening", or "covering the existing text area with another text" on the text are regarded as abnormal handwriting, and so on.

[0053] The handwritten handwriting abnormal recognition method of the present invention is applicable to the above various situations, and the difference lies in the types of data collected in the training data set. A handwritten handwriting abnormal recognition method of the present invention, as Figure 1 shown, includes the following steps:

[0054] S1. Establish two training data sets from the handwritten handwriting data, namely training data set one and training data set two. Among them, training data set one is the normal handwritten handwriting data within the recognizable range, and training data set two is a combination of the normal handwritten handwriting data within the recognizable range and the abnormal handwritten handwriting data outside the recognizable range.

[0055] Normal handwriting data and abnormal handwriting data should be collected based on the allowed identification content and should be as representative and extensive as possible.

[0056] S2. Preprocess the data in training data set 1 and training data set 2 respectively.

[0057] The data preprocessing method includes the following steps in sequence:

[0058] S2-1, denoising is performed through a denoising model.

[0059] The denoising model performs denoising through the stroke length and the spatial information of the stroke compared to other strokes, which includes the following steps: if the number of trajectory points that make up the stroke is lower than the set number of points, the stroke is considered to be a noise stroke and is deleted; the length of the stroke is calculated, and if the stroke length is far lower than the set stroke length, the stroke is considered to be a noise stroke and is deleted.

[0060] S2-2, after standardization and normalization steps.

[0061] Normalization can be performed using standard processing steps in the field and general data processing methods. Generally speaking, image normalization is the process of centralizing data by removing the mean. For example, in a coordinate system with the upper left corner as the origin, the coordinates of the four vertices of a square are (100,100), (101,100), (101,101), (100,101). After normalization, the coordinates of the four vertices are (0,0), (1,0), (1,1), (0,1). The normalization here can also be performed using standard and general processing methods in the field.

[0062] S2-3. Perform binary encoding.

[0063] S2-4. Perform data enhancement.

[0064] The purpose of data enhancement is to improve the accuracy and applicability of data recognition. It can also be applied to handwriting that is not standardized, illegible, severely altered, or has a high frequency of graffiti, so as to achieve the purpose of small sample training and hardware adaptation. All existing data enhancement methods can be used. As one of the embodiments, data enhancement includes data enhancement based on trajectory points, data enhancement based on strokes, and data enhancement based on the entire text. The data enhancement is performed in units of trajectory points, strokes, and the entire text, respectively. The data enhancement process based on the entire text is located after the data enhancement based on trajectory points and the data enhancement based on strokes.

[0065] Data augmentation based on trajectory points may include the following steps: normalizing and standardizing the trajectory points of a sample; loading a single stroke of a single sample, and counting the number of trajectory points, the trajectory point density, and the stroke length of the stroke as basic features; using an augmented mutation algorithm, a deletion mutation algorithm, and an offset mutation algorithm to perform data augmentation on the trajectory points of the stroke; for the newly generated points, according to the specific conditions of the handwriting device, the user, and the basic features, setting appropriate mutation probabilities for the "starting point", "ending point", and "intermediate points" of the stroke, setting appropriate mutation probabilities for the algorithms in the augmented mutation, deletion mutation, and offset mutation, and updating the original trajectory points to the new trajectory points; loading the next stroke and repeating the above steps.

[0066] Data augmentation based on strokes includes: normalizing and standardizing the trajectory points of a sample; loading single strokes of a single sample one by one, and counting the number of trajectory points of the strokes and the font, the trajectory point density, the stroke density, and the geometric features of the font as basic features; rotating, offsetting, and stretching the stroke, and for the newly generated stroke, with a certain probability, updating the original stroke to the newly generated stroke.

[0067] Data augmentation based on the whole text includes: normalizing and standardizing the trajectory points of a sample; loading single samples one by one, and calculating features such as the glyph ratio, trajectory point density, number of strokes, and number of trajectory points of the sample; performing affine transformation on all strokes of the sample, and for the newly generated strokes, with a certain probability, updating the trajectory points of the original sample to the newly generated trajectory points.

[0068] S3. The preprocessed data of training dataset one is subjected to feature extraction through convolutional network one, and the preprocessed data of training dataset two is subjected to feature extraction through convolutional network two, and two groups of feature maps are output;

[0069] Convolutional network one corresponds to training dataset one, and convolutional network two corresponds to training dataset two. Convolutional network one and convolutional network two can adopt reference convolutional network structures such as ResNet, MobileNet, etc. The two can adopt the same network structure or different ones. The structures and parameters of the two can be shared or independent of each other. The training parameters can be set and adjusted according to the specific situation of the task and the performance of the model during the training process.

[0070] S4. By calculating the two groups of feature maps output in step S3 in a high-dimensional space, the difference magnitude between the two groups of feature maps is obtained and input into the decision model.

[0071] The decision model can select existing multi-layer perceptron MLP or SVM, as well as other neural network structures that can complete binary classification tasks and output confidence levels, and its parameters are set according to the actual situation and task complexity.

[0072] The difference magnitude between the two groups of feature maps can be "distance", "norm", or "similarity". This "distance" can be the ordinary Euclidean distance or the Minkowski distance. The "similarity" can be the cosine similarity. The "norm" can be a custom spatial norm. Here, it is selected and designed according to the actual situation.

[0073] S5. Train the first convolutional network, the second convolutional network, and the decision model until the model converges, find a high-dimensional space such that the individual differences of samples belonging to the same category mapped to this space are as small as possible, and the individual differences of samples not belonging to the same category mapped to this space are as large as possible, and save the model parameters. If the individual differences are measured by distance, then the distances of samples belonging to the same category mapped to this space are as close as possible, and the distances of samples not belonging to the same category mapped to this space are as far as possible.

[0074] S6. Use the clustering algorithm to find the optimal cluster center data set, so that the samples belonging to the same category are aggregated and the individual differences within the category are as small as possible (the distance is close enough), and the individual differences of the data distributions of clusters belonging to different categories are as large as possible (the distance is far enough).

[0075] The role of finding the optimal cluster center set is: 1) Specify the strong scope of the model, making the model more focused on the discrimination task of specific categories; 2) Make the recognition model more strict and reasonably limit the association ability of the model.

[0076] The role of the decision model in finding the optimal cluster center set and the deployment inference process is to calculate the specific values of the distances, norms, or similarities between the sample and all cluster centers of the category to which the recognition result one (the recognition result of the handwritten recognition model) belongs in the high-dimensional space, and judge whether this sample strictly belongs to this category through these specific values. Here, the definition of "strict" is that we assume that for any character, the number of classifications of its handwritten form is limited. After feature extraction and the clustering algorithm, determine the number of these classifications k + 1; Let the cluster center of the i-th category be C i , i ∈ [0, k], then in the high-dimensional space, based on the intra-class distribution of all samples of this category and the distribution of samples of other categories, set an appropriate class radius for the i-th category, denoted as R i , such that all samples belonging to this category can fall within the circle with C i as the center and R i as the radius, and all samples not belonging to this category can fall outside this circle.

[0077] For example, for the English capital letter "Z", assume that according to all the handwritten data in the data set Figure 1Perform feature extraction and clustering according to the process of i (any publicly available clustering algorithm can be selected according to the actual data distribution), and finally determine that its handwritten fonts can be divided into 6 categories. Let the cluster center of the i-th category be C i , where i ∈ [0, 5]. In this high-dimensional space, based on the intra-class distribution of all samples in this category and the distribution of samples in other categories, set an appropriate class radius for the i-th category, denoted as R i , such that all samples belonging to this category can fall within the circle centered at C i with radius R

[0078] During the inference process of the decision model, that is, when it is deployed to the formal recognition environment for users to use, it will combine Figure 1 Find the optimal cluster center data set, select an appropriate threshold, weight and sum the distances from each cluster center to the projection of the sample into the high-dimensional space respectively, and the value of the sum result is in the range of [0, 1].

[0079] S7. Repeat steps S5 to S6 until the obtained Convolutional Network 1, Convolutional Network 2, decision model, and optimal class cluster center data set meet the requirements, that is, the recall rate and precision rate on the test set reach a certain threshold.

[0080] The training objective of the decision model is: when the samples from Training Data Set 1 and the samples from Training Data Set 2 have the same category and are not abnormal samples, make the output result as close to 1 as possible; when the samples from Training Data Set 1 and the samples from Training Data Set 2 have different categories, make the output result as close to 0 as possible.

[0081] The purpose of combining the three models of Convolutional Network 1, Convolutional Network 2, and the decision model is to find a high-dimensional space such that the coordinate distances of samples belonging to the same category mapped to this space are as close as possible, and the coordinate distances of samples not belonging to the same category mapped to this space are as far as possible. The step of "finding the optimal class cluster center set" is carried out after each round of "training the model". The purpose is to determine that after mapping various handwritten samples to this corresponding high-dimensional space, 1) the sample coordinates belonging to the same category are aggregated and the intra-class distance is close enough; 2) the distance between the data distributions of clusters belonging to different categories (centered at C i with radius R i ) is far enough. If both 1) and 2) are satisfied, the process of "training the model" ends, and the modeling of Convolutional Network 1, Convolutional Network 2, and the decision model is completed, and it enters the step of "finding the optimal class cluster center set". Otherwise, the next round of "training the model" step needs to be continued.

[0082] Generally, during the training process, when the train loss and val loss (the values calculated by the loss function in each round) of the model tend to be stable and perform better than the preset threshold on the test set, the training will stop. However, the high-dimensional space corresponding to the model structure at this time may not meet the conditions for finding the optimal cluster centers. These conditions are to be satisfied simultaneously: 1) The coordinates of samples of the same class in this space are as close as possible; 2) The coordinates of samples of different classes in this space are as far as possible. And according to the strictness of the task, the setting of the specific values for this closeness and farness is also related.

[0083] S8. As Figure 2 shown, after being preprocessed, the test data respectively enter the first convolutional network and the handwritten recognition model. After entering the handwritten recognition model for handwritten recognition, the first recognition result is obtained. The first recognition result passes through the optimal cluster center data set in step S7 to calculate the magnitude of the individual differences (distance, norm, or similarity) of the samples in the high-dimensional space, and the specific values of the magnitudes of the differences (distance, norm, or similarity) between the samples in the high-dimensional space and all the cluster centers of the class to which the first recognition result belongs are obtained.

[0084] The first convolutional network in step S8 is the same first convolutional network in the training of the above-mentioned anomaly recognition model. After the training process ends, the first convolutional network is directly retained for feature extraction. The preprocessing method in step S8 can adopt the data processing method in the above-mentioned step S2, including steps such as data denoising, standard normalization, binary encoding, and data augmentation.

[0085] The handwritten recognition model in step S8 may include a deep neural network model. The method for the test data to enter the handwritten recognition model in step S8 and obtain the first recognition result is as follows:

[0086] The handwritten stroke data is obtained by a cutting algorithm to get several possible single-character stroke combinations, denoted as the trajectory point segmentation combination Combs, and the whole formed by the stroke points in one Comb in Combs is defined as a pseudo-character Traj.

[0087] As Figure 3 (a)-(c), the cutting algorithm will, based on features, divide all the stroke points within an answer box into several "pseudo-characters", and these pseudo-characters form a Comb (combination). Usually, for the stroke points within an answer box, the segmentation algorithm will generate multiple Combs (multiple Comb) according to the actual situation of the feature data. These Combs contain the correctly divided trajectory point combinations (correct division means that in this Comb, the stroke points in each "pseudo-character" belong to the same handwritten character, and all the stroke points of this handwritten character are divided into this "pseudo-character").

[0088] AsFigure 3 (a)-(c), the trajectory points belonging to a certain character separated by the cutting algorithm are collectively defined as pseudo-characters. Since this segmentation result may not be correct, the word "pseudo-" is added for distinction. And in the next processing stage, it is treated as an independent single character. However, it may not be composed of the handwriting points of a complete character. It may be part of the handwriting points of a complete character (such as Figure 3 (a)) or the handwriting points of multiple characters (such as Figure 3 (b)), or it may be abnormal handwriting, such as Figure 3 "pseudo-character 8" in (a), Figure 3 "pseudo-character 5" in (b), Figure 3 the handwriting stroke "!" corresponding to "pseudo-character 6" in (c), which does not belong to the handwriting within the recognition range. Therefore, these pseudo-characters need to be processed, such as being recognized through various recognition models and algorithms, and so on.

[0089] In the present invention, according to the generated trajectory point segmentation combination Combs, the pseudo-characters Traj within each combination are respectively 3-g projected onto a two-dimensional plane to generate a binary matrix, denoted as M 3-g , and the matrix M 3-g is input into the trained deep neural network model to obtain the recognition result one of each pseudo-character.

[0090] The specific values of the difference magnitudes (distance, norm, or similarity) obtained in steps S9 and S8 in the high-dimensional space are successively passed through the decision model and decision strategy to obtain the final recognition result two.

[0091] The difference magnitude is measured by distance. Let the distance of a certain sample from the j-th cluster center be d j , j ∈ [0, k], k + 1 is the number of cluster centers of the class corresponding to the recognition result one of this sample, and denote d j the value obtained after passing through the decision model in step S9 as p j , f(x) represents the decision strategy in step S9,

[0092] Then the decision strategy in step S9 is a weight decision formula, and its expression is:

[0093]

[0094] where B is the recognition result two, which is a value belonging to [0, 1]; x = α n (C n , R n ) is the weight function, representing the weight parameter of the n-th cluster determined based on the data and sample distribution; C n is the cluster center of the n-th class, Rn is the class radius of the nth class.

[0095] This strategy consists of a threshold, a conditional decision, and a system of conditional inequalities, all of which need to be determined according to the requirements of the recognition recall rate of abnormal text, the recognition accuracy of normal text, and the task complexity in the application scenario. For example, in an application scenario with high requirements for recognition accuracy, the model can be made more stringent by adjusting the threshold, such as by increasing the recognition rate of non-prescribed deletion methods.

[0096] A handwriting recognition system, such as Figure 4 shown, includes a dot matrix paper, a dot matrix pen, and a recognition module. The recognition module adopts the above-mentioned handwriting anomaly recognition method and is deployed on a local server, a cloud server, or a smart terminal.

[0097] Such as Figure 5 , a number of input boxes are provided on the dot matrix paper, and dots with specific information are preset on the input boxes. The dots can provide coordinate information for the dot matrix pen to generate dots.

[0098] The medium of the dot matrix paper can be ordinary paper fiber paper. The generation method of the dot matrix paper is: encode the position information and paper information of the input boxes on the paper surface, then draw the corresponding matrix points based on an algorithm, and then print or print the matrix points with specific information on the paper surface of a specific size.

[0099] Such as Figure 6 , Figure 7 shown, the dot matrix pen can be any dot matrix pen device using dot matrix technology and can be set in any shape such as rod-shaped, plate-shaped, or block-shaped. In this embodiment, such as Figure 6 , Figure 7 , the dot matrix pen has a structure of a rod-shaped pen, which includes a pen shaft 1 and a writing module, a nib 2 pressure sensing module, a camera module, a data conversion module, a data transmission module, an indicator light 4, and a switch button 5 provided on the pen shaft 1.

[0100] The writing module includes a nib 2 for writing on the dot matrix paper. The nib 2 is provided at the bottom of the pen shaft 1, and the nib 2 is connected to an ink supply component. The ink supply component is provided in the pen shaft 1 and can be in the form of an ordinary water-based pen refill or a combination structure of the nib 2 and an ink cartridge of an ordinary fountain pen.

[0101] The camera module includes a camera 3. The camera 3 is provided near the nib 2 and is used to capture the handwriting written by the nib 2 and can be provided below the handgrip 6 of the pen shaft 1. To prevent the camera from being blocked, the appearance of the dot matrix pen can be specially designed. For example, a longer slot is provided at the position of the camera 3 to prevent the loss of handwriting points due to the pen-holding posture, and a special pen shape and pen slot can be set to prevent incorrect pen-holding postures.

[0102] The pressure sensing module of the pen tip 2 is respectively connected to the pen tip 2 and the camera module, and is used to sense the pressure when the pen tip 2 contacts the dot matrix paper. When the pressure is greater than the set value, the camera module is triggered to take a picture. By analyzing the dot matrix codes in the two pictures, the corresponding x and y coordinates of the two pictures are obtained and recorded in chronological order.

[0103] The data conversion module is connected to the camera module and is used to convert the captured image into an array containing coordinate points based on the dot matrix of the dot matrix paper. The input of the data conversion part is the picture taken by the camera 3, and the output is a string of coordinate points. The coordinate data is obtained based on the dot matrix. The dot matrix coordinates in the picture will be converted into a string of codes, and by identifying the specific digital arrangement in the codes, the coordinate information of the handwriting points is parsed. For example, use "123123" to locate the start and end of the "x" coordinate, and use "321321" to locate the start and end of the "y" coordinate. Then, if a string of codes is parsed from a picture as follows "46576912312335123123321321100321321", the coordinates corresponding to this picture can be obtained as (35, 100).

[0104] The data transmission module is used to wirelessly transmit the data converted by the data conversion module to the recognition module of the cloud server, local server or smart terminal.

[0105] When the dot matrix pen wirelessly transmits the data to the recognition module on the cloud server or local server, the recognition module recognizes according to the coordinate point array and wirelessly transmits the recognition result to the smart terminal. A display and viewing module is set on the smart terminal for displaying and viewing the recognition result;

[0106] When the dot matrix pen wirelessly transmits the data to the recognition module on the smart terminal, the recognition module recognizes according to the coordinate point array and can view the recognition result in real time on the display screen of the smart terminal.

[0107] The smart terminal is an embedded device, mobile phone terminal or PC terminal, and corresponding application programs are pre-installed on these terminals for operators to operate and view.

[0108] In this embodiment, an indicator light 4 and a switch button 5 are further arranged on the dot matrix pen, both of which are arranged at the upper position of the pen shaft 1. The indicator light 4 is used to indicate whether the dot matrix pen is working normally. When the indicator light 4 is on, it means it is in a normal working state; when the indicator light 4 is off, it means it is in a closed, power-deficient or faulty state. The switch button 5 is used to turn on and off the dot matrix pen. A battery module is arranged on the dot matrix pen, configured with ordinary batteries or lithium-ion rechargeable batteries, and equipped with a connector for charging the rechargeable battery. These configurations can all adopt existing technologies.

[0109] One application scenario of the handwriting recognition system of the present invention is classroom teaching. For example, Figure 8 As shown, the answer sheet is a dot matrix paper. Matrix points are set on the corresponding answer boxes of the answer sheet. The matrix points contain position information and paper information. During a classroom quiz, students hold a dot matrix pen and write answers by hand in the input box of the answer sheet. When the pressure of the pen tip 2 is greater than a preset threshold, the action of triggering the camera 3 to take a picture is triggered. The taken picture is converted by a data conversion module into an array containing coordinate points, and taking strokes as units, it is continuously transmitted to the intelligent terminal device through the Bluetooth module at intervals of a certain duration. The intelligent terminal device recognizes the written characters on each answer sheet. The teacher can view the answer content on each student's answer sheet through the intelligent terminal device, and can achieve automatic marking and obtain the answer scores of each student in real time when the teacher has previously entered the correct answers into the intelligent terminal.

[0110] Recognition effect of the present invention:

[0111] Applying the recognition system and the abnormal recognition method of the present invention to classroom teaching, when only using a 4-core CPU server with a CPU MHz of 2199.998 and without using a GPU, it can easily support the real-time recognition of the high-frequency answering handwriting of 50 students, and the recognition results are displayed in real time on the intelligent terminal. The recognition accuracy is relatively high and can meet the requirements of examinations, tests, and assignments.

[0112] The above detailed description is a specific description of the feasible embodiments of the present invention. This embodiment is not intended to limit the patent scope of the present invention. Any equivalent implementation or modification made without departing from the present invention shall be included in the patent scope of this case.

Claims

1. A method for recognizing abnormal handwriting, characterized in that, It includes the following steps: S1. Establish two training data sets from the handwritten data, namely training data set one and training data set two. Among them, training data set one is the normal handwritten data within the recognizable range, and training data set two is a combination of normal handwritten data within the recognizable range and abnormal handwritten data outside the recognizable range; S2. Preprocess the data in training data set one and training data set two respectively; S3. The preprocessed data of training data set one is subjected to feature extraction through convolutional network one, and the preprocessed data of training data set two is subjected to feature extraction through convolutional network two, and two groups of feature maps are output; S4. By calculating the two groups of feature maps output in step S3 in the high-dimensional space, obtain the difference magnitude between these two groups of feature maps, and input it into the decision model; S5. Train convolutional network one, convolutional network two and the decision model until the model converges, find a high-dimensional space, so that the individual differences of samples belonging to the same category mapped to this space are as small as possible, and the individual differences of samples not belonging to the same category mapped to this space are as large as possible, and save the model parameters; S6. Use the clustering algorithm to find the optimal cluster center data set, so that the samples belonging to the same category are aggregated and the individual differences within the category are as small as possible, and the individual differences of the data distributions of the clusters belonging to different categories are as large as possible; S7. Repeat steps S5 to S6 until the obtained convolutional network one, convolutional network two, decision model and optimal cluster center data set meet the requirements; S8. The test data is preprocessed and then enters the convolutional network one and the handwritten recognition model respectively. After entering the handwritten recognition model for handwritten recognition, recognition result one is obtained. Recognition result one passes through the optimal cluster center data set in step S7, calculates the specific value of the individual difference magnitude of the sample in the high-dimensional space, and obtains the specific value of the difference magnitude between the sample and all cluster centers of the category to which recognition result one belongs in the high-dimensional space; S9. The specific values of the difference magnitude in the high-dimensional space obtained in step S8 pass through the decision model and the decision-making strategy in turn to obtain the final recognition result two.

2. The method for recognizing abnormal handwriting according to claim 1, wherein The magnitude of the difference is measured by distance. Let the distance of a certain sample from the center of the j-th cluster be d j , where j ∈ [0, k], and k + 1 is the number of cluster centers of the class corresponding to the first recognition result of the sample. Denote d j The value obtained after passing through the decision model in step S9 is p j , and f(x) represents the decision strategy in step S9 Then the decision-making strategy in step S9 is a weight decision formula, and its expression is: Among them, B is the second recognition result, which is a value belonging to [0, 1]; x = α n (C n , R n ) is the weight function, representing the weight parameter of the nth cluster determined based on the data and sample distribution; C n is the cluster center of the nth class, and R n is the class radius of the nth class.

3. A method for identifying abnormal handwriting according to claim 1, characterized in that The method of data preprocessing in step S2 or / and S8 includes the following steps in turn: S2-1. Denoise through the denoising model; S2-2. Pass through the steps of standardization and normalization; S2-3. Perform binary encoding; S2-4. Perform data augmentation.

4. A method for identifying abnormal handwriting according to claim 3, characterized in that, The denoising model in step S2-1 performs denoising through the stroke length and the spatial information of the stroke compared with other strokes, and it includes the following steps: If the number of trajectory points forming the stroke is lower than the set number of points, then this stroke is considered a noise stroke and is deleted; Calculate the length of the stroke. If the stroke length is much lower than the set stroke length, then this stroke is considered a noise stroke and is deleted.

5. A method for identifying abnormal handwritten strokes according to claim 3, characterized in that, The data augmentation in step S2-4 includes data augmentation based on trajectory points, data augmentation based on strokes, and data augmentation based on the whole text. Among them, the process of data augmentation based on the whole text is after the data augmentation based on trajectory points and the data augmentation based on strokes.

6. The method for abnormal recognition of handwritten handwriting according to claim 1, wherein The convolutional network one or / and convolutional network two is ResNet or MobileNet; The decision-making model is a multi-layer perceptron MLP or SVM; The difference magnitude in step S4 or / and S8 is distance, norm or similarity. The distance is Euclidean distance or Minkowski distance. The similarity is cosine similarity. The norm is a custom-defined norm.

7. A method for identifying abnormal handwriting according to claim 1, characterized in that The method for finding the optimal cluster center data set through the clustering algorithm in step S6 is as follows: Determine the number of classes k + 1 through feature extraction and the clustering algorithm; Let the cluster center of the i-th class be C i , where i ∈ [0, k]. Then, in the high-dimensional space, based on the within-class distribution of all samples in this class and the distribution of samples in other classes, set an appropriate class radius for the i-th class, denoted as R i , such that all samples belonging to this class can fall within the circle with C i as the center and R i as the radius, and all samples not belonging to this class can fall outside this circle.

8. A method for identifying abnormal handwriting according to claim 1, characterized in that, The training objective of the decision-making model in step S7 is as follows: when the samples from the first training data set and the samples from the second training data set have the same category and are not abnormal samples, make the output result as close to 1 as possible; when the samples from the first training data set and the samples from the second training data set have different categories, make the output result as close to 0 as possible.

9. A method for identifying abnormal handwritten handwriting according to claim 1, characterized in that The handwriting recognition model in step S8 includes a deep neural network model. The method for the test data to enter the handwriting recognition model and obtain the first recognition result in step S8 is as follows: The handwritten stroke data is processed by a cutting algorithm to obtain several possible single-character stroke combinations, denoted as the trajectory point segmentation combination Combs. And the whole composed of the handwriting points in one Comb in Combs is defined as a pseudo-character Traj. Segment and combine the Combs according to the generated trajectory points, and respectively project the pseudo-characters Traj within each combination 3-g onto a two-dimensional plane to generate a binary matrix, denoted as M 3-g , and input the matrix M 3-g into the trained deep neural network model to obtain the recognition result one of each pseudo-character.

10. A handwritten recognition system, characterized in that, It includes a dot matrix paper, a dot matrix pen and a recognition module. The recognition module adopts the handwritten stroke anomaly recognition method as described in any one of claims 1 to 9. The recognition module is arranged on a local server, a cloud server or a smart terminal. Several input boxes are set on the dot matrix paper. Specific dots are preset on the input boxes to provide coordinate information for generating dots by the dot matrix pen. The generation method of the dot matrix paper is: encode the position information and paper information of the input boxes on the paper surface, then draw the corresponding matrix points based on the algorithm, and then print or print the matrix points with specific information on the paper surface of a specific size. The dot matrix pen includes a writing module, a nib pressure sensing module, a camera module, a data conversion module and a data transmission module. The writing module includes a nib for writing on the dot matrix paper. The camera module includes a camera. The camera is arranged near the nib and is used to capture the handwriting written by the nib. The nib pressure sensing module is connected to the nib and the camera module respectively and is used to sense the pressure when the nib contacts the dot matrix paper. When the pressure is greater than the set value, it triggers the camera module to take a picture. The data conversion module is connected to the camera module and is used to convert the captured image into an array containing coordinate points based on the dots of the dot matrix paper. The data transmission module is used to wirelessly transmit the data converted by the data conversion module to the recognition module.

Images