Handwritten Chinese character recognition and correction method

Abstract

The invention discloses a handwritten Chinese character recognition and correction method, which comprises the following steps: acquiring a handwritten Chinese character image, and carrying out noise reduction processing on the handwritten Chinese character image; performing image segmentation on the handwritten Chinese character image after noise reduction processing, and extracting a to-be-recognized image with Chinese characters; establishing a training data set and a first target handwritten Chinese character recognition model; detecting the to-be-recognized image based on the trained target handwritten Chinese character first recognition model to obtain a first recognition result of handwritten Chinese characters in the to-be-recognized image; correcting scribble fonts in the first recognition result based on a hypergraph neural network; according to the method, the image is recognized by adopting the OfficientDet dual network, the feature pyramid model is fused, the scribble and non-scribble handwritten Chinese characters are subjected to feature processing, and the adversarial network is used for generating massive adversarial sample data to improve the recognition effect of the scribble Chinese characters, so that the recognition efficiency of the scribble Chinese characters is improved. And meanwhile, the hypergraph neural network is adopted to perform accurate scribble type recognition on the recognized scribble Chinese characters, so that the speed of recognizing the scribble Chinese characters is high, and the accuracy of correcting the scribble Chinese characters is high.

Description

Technical Field

[0001] The present invention relates to the technical field of handwritten Chinese characters, and in particular to a method for recognizing and correcting handwritten Chinese characters. Background Art

[0002] As an important information carrier, Chinese characters play an important role in the fields of large-scale handwritten material input, document recognition, and teaching assistance. Handwritten Chinese character recognition technology is an important research direction in the field of computer vision and pattern recognition. It involves converting handwritten Chinese character images into editable and recognizable text information. With the rapid development of deep learning technology, the field of handwritten Chinese character recognition has ushered in new opportunities.

[0003] Chinese patent application CN109800763A discloses a method for handwritten Chinese character recognition based on deep learning. The method improves the performance of handwritten Chinese character recognition by using P2DMN normalization, NCFE feature extraction, ADBN coarse classification and MQDF fine classification, combined with deep learning technology. However, the above method is not very accurate in recognizing untrained handwritten Chinese characters, the generalization ability of the model is not strong, and a large data set is required for training to prevent the model from overfitting. It is impossible to provide a large amount of training data, and it also fails to recognize sloppy handwritten Chinese characters.

[0004] In the prior art, the recognition of handwritten Chinese characters relies on a single neural network model, which easily leads to low accuracy of handwritten Chinese character recognition and cannot provide the large data set required for training. In addition, when handwritten Chinese characters encounter sloppy fonts, they need to be quickly recognized and accurately corrected through a neural network. Summary of the invention

[0005] The purpose of the present invention is to provide a handwritten Chinese character recognition and correction method to solve the above-mentioned problems existing in the prior art.

[0006] The specific application is as follows: A method for recognizing and correcting handwritten Chinese characters comprises the following steps: S1: Obtain handwritten Chinese character images and perform noise reduction on them; S2: performing image segmentation on the handwritten Chinese character image after noise reduction processing to extract the image to be recognized with Chinese characters; S3: Obtain the bounding box of a single Chinese character in the image to be recognized based on template matching; S4: establishing a training data set and a first recognition model for target handwritten Chinese characters, and training the first recognition model for target handwritten Chinese characters based on the training data set; S5: detecting the image to be recognized based on the trained first recognition model of the target handwritten Chinese characters, and obtaining a first recognition result of the handwritten Chinese characters in the image to be recognized, wherein the first recognition result includes cursive fonts and non-cursive fonts; S6: Correct the sloppy handwriting in the first recognition result based on the hypergraph neural network.

[0007] Further, in step S1, the noise reduction process is a Gaussian filter process, specifically: ,

[0008] In the formula, g(x,y) is the handwritten Chinese character image, (x,y) is the handwritten Chinese character image after Gaussian filtering, u(i,j) is the weight, and D(x,y) is the size range of (N+1)×(N+1) centered at (x,y); The weights are specifically:

[0009] In the formula, is the spatial channel proximity weight value, Spatial channel similarity weight value, is the input random coefficient.

[0010] Furthermore, in step S2, the image segmentation of the handwritten Chinese character image after the noise reduction processing is specifically: the image segmentation of the handwritten Chinese character image after the noise reduction processing is performed based on finding the target minimum circumscribed rectangle, which specifically includes the following steps: S21: Binarizing the handwritten Chinese character image after the noise reduction process to obtain a binary image; S22: Based on the contour detection algorithm, search and draw the contour of the binary image, and then fit the drawn target contour to find the minimum circumscribed square of the contour; S23: based on the center point of the smallest circumscribed square being at 2 / 4 to 3 / 4 of the handwritten Chinese character image, searching for the Chinese character at the center, and intercepting a segmented image with the central Chinese character; S24: Correct the segmented image to obtain an image to be recognized.

[0011] Furthermore, in step S24, the segmented image is corrected to obtain the image to be recognized, specifically: S251: Take the four corner points of the minimum circumscribed square and the four corners of the image to make a correspondence, and then obtain the perspective transformation matrix based on the fitgeotrans function in MATLAB according to these four pairs of corresponding corner points; S252: Based on the perspective transformation matrix, call the function imwarp in MATLAB to transform the segmented image into a corrected view to obtain an image to be recognized; S253: Output the image to be recognized.

[0012] Furthermore, the S3 specifically includes the following steps: S31: constructing a first image template and a second image template, wherein the first image template is a standard image of Chinese characters, and the second image template is a standard inner frame image of Chinese characters; S32: Slide the first image template and the second image template on the image to be recognized respectively to find the position with the largest best matching value, that is, to obtain the inner and outer frames of the Chinese characters in the image to be recognized.

[0013] Furthermore, in step S4, the specific implementation process of establishing the training data set and the first recognition model of the target handwritten Chinese character is as follows: S41: Obtaining sloppy and non-sloppy Chinese character image data through handwritten Chinese characters, a part of which is used for a test set, and a part of which is used for a training set, and two EfficientDet network models are trained through the training set, wherein the feature acquisition network of the EfficientDet network model is ResNeXt, and the two network models obtained after training are respectively recorded as ResNeXt-A and ResNeXt-B, and the Chinese character image data is manually labeled and defined, including being defined as two labels of sloppy and non-sloppy; S42: Divide the EfficientDet network model of S41 into a feature extraction network ResNeXt, a feature pyramid and a sloppy classification regression network, use the feature extraction network ResNeXt to extract features of sloppy and non-sloppy images, connect the feature pyramid in series to the feature extraction network ResNeXt, perform convolution on each layer of the feature pyramid, and perform classification and regression through the sloppy classification regression network; S43: Use the test set to perform network iteration test on the feature extraction network ResNeXt, and output the recognition result. When the recognition accuracy reaches the set threshold, stop the model training. The recognition result is used to determine whether the image is sloppy.

[0014] Furthermore, the specific steps of building the feature extraction network ResNeXt are: S411: judging the image of the training set data according to the two labels of sloppy and non-sloppy, if the label of the image is sloppy, using ResNeXt-A for processing, if the label of the image is non-sloppy, using ResNeXt-B for processing, if the current input training set data image is judged to be sloppy, constructing a deep convolutional generative adversarial network for sloppy images; S412: Generate scribbled image training set samples for the scribbled images using a deep convolutional generative adversarial network; The processing flow for sloppy and non-sloppy features is as follows: The feature acquisition network adopts the network framework of the feature extraction network ResNeXt to extract the features of scribbled and non-scribbled Chinese characters respectively. First, the feature maps of the scribbled and non-scribbled convolutional layers in the second layer are superimposed based on channels, and then dimensionality reduction processing is performed through 2×2 convolution. The same operation is executed on the third convolutional layer, the fourth convolutional layer, the fifth convolutional layer, and the sixth convolutional layer; The specific process of using the deep convolutional generative adversarial network to generate scribbled image training set samples from the scribbled images is as follows: According to the original scribbled image given in the current training, adversarial scribbled image samples are generated. The specifications of the adversarial scribbled image samples are: diverse font size features, scribbled stroke features, and overall scribbled font features. Input 1 original scribbled image, and through the deep convolutional generative adversarial network, a perturbation factor is added to generate 200 scribbled sample images. The scribbled stroke features at least include the feature of non-fixed stroke length and the feature of deviation of the relative position of the strokes. The overall scribbled font features at least include the feature of deviation of the overall font width-to-height ratio and the feature of deviation of the overall font center of gravity position.

[0015] Further, the specific process of connecting the feature pyramid to the feature extraction network ResNeXt is as follows: Sample the feature map of the third convolutional layer, that is, the scribbled and non-scribbled fused feature map, and then superimpose it on the scribbled and non-scribbled fused feature maps of the second convolutional layer to obtain the first layer of the pyramid. Continue to execute this step on the fourth convolutional layer, the fifth convolutional layer, and the sixth convolutional layer. Channel-wise superposition of every two adjacent layers of feature maps can obtain one layer of the pyramid. Finally, a total of four layers of feature pyramids are obtained. The specific processing process of the four layers of the pyramid is as follows: The first layer of the pyramid is used to process the first image resolution, and the brightness of the image belongs to the first-level brightness; The second layer of the pyramid is used to process the second image resolution, and the brightness of the image belongs to the second-level brightness; The third layer of the pyramid is used to process the third image resolution, and the brightness of the image belongs to the third-level brightness; The fourth layer of the pyramid is used to process the fourth image resolution, and the brightness of the image belongs to the fourth-level brightness; The resolution levels of the images are specifically divided as follows: The first image resolution: w < 160, h < 160; The second image resolution: 160 < w ≤ 320, 160 < h ≤ 320; The third image resolution: 320 < w ≤ 1080, 320 < h ≤ 1080; The fourth image resolution: 1080 < w, 1080 < h, where w represents the width of the image and h represents the height of the image; The brightness of the image is calculated by using the histogram to calculate the average brightness of the image, which is specifically divided into: The first average brightness: 0 ≤ L ≤ 60; The second average brightness: 60 < L ≤ 100; The third average brightness: 100 < L ≤ 160; The fourth average brightness: 160 < L ≤ 255, where L represents the average brightness value of the image; Two branch networks are added after the feature map of each fusion layer of the pyramid. One branch is used for classification and the other branch is used for regression. And each branch first performs 4 convolutions on the feature map to enhance the image features of scribbled and non-scribbled respectively, and the convolution kernel size is 2×2 and the number is 128.

[0016] Further, the specific implementation process of S6 is as follows: Input the stroke scribble feature and the overall font scribble feature of the scribbled image to be processed into the first hypergraph neural network layer of the preset Chinese character correction model to obtain the first scribble feature vector; wherein, the first scribble feature vector represents the association relationship between the stroke scribble feature and the overall font scribble feature of the scribbled image to be processed; the preset Chinese character correction model is a model obtained based on the hypergraph neural network model; Input the sub-features of the stroke scribble feature and the sub-features of the overall font scribble feature of the scribbled image to be processed into the second hypergraph neural network layer of the preset Chinese character correction model to obtain the second scribble feature vector; wherein, the second feature vector represents the association relationship between the sub-features of the stroke scribble feature and the sub-features of the overall font scribble feature of the scribbled image to be processed; Connect the first scribble feature vector and the second scribble feature vector to obtain the feature vector to be processed; Based on the preset Chinese character correction model, identify the feature vector to be processed to obtain the scribble type of the scribbled image, and the scribble type at least includes stroke length scribble, stroke intersection scribble, stroke inclination scribble, pen gesture scribble, whole character position scribble, whole character width-to-height ratio scribble, and whole character center-of-gravity scribble; For the scribble types in the preset handwritten Chinese character correction library, calculate the target similarity between the scribble type of the scribbled image to be processed and the scribble types in the preset handwritten Chinese character correction library through the cosine similarity algorithm, and preset a similarity threshold. If the target similarity is greater than the preset similarity threshold, find the corresponding scribble type in the handwritten Chinese character correction library and then correct the Chinese characters in the scribbled image to generate standard Chinese characters. If the target similarity is less than or equal to the preset similarity threshold, find the scribble type in the handwritten Chinese character correction library with the closest similarity to correct the Chinese characters in the scribbled image.

[0017] Further, the loss function of the preset Chinese character correction model is:

[0018] In the formula, is the loss function, k is the balance factor, is the incidence matrix aggregation parameter, is the target probability of the scribbled type to be classified.

[0019] Compared with the prior art, the embodiments of the present invention achieve the following beneficial effects: The embodiment of the present invention provides a method for obtaining a handwritten Chinese character image and performing noise reduction processing on the handwritten Chinese character image after the noise reduction processing, performing image segmentation on the handwritten Chinese character image to be recognized with Chinese characters, obtaining a border of a single Chinese character in the image to be recognized based on template matching, establishing a training data set and a first recognition model for a target handwritten Chinese character, and training the first recognition model for the target handwritten Chinese character based on the training data set, detecting the image to be recognized based on the trained first recognition model for the target handwritten Chinese character to obtain a first recognition result of the handwritten Chinese character in the image to be recognized, wherein the first recognition result includes a sloppy font and a non-sloppy font, and correcting the sloppy font in the first recognition result based on a hypergraph neural network. The present invention recognizes the image by using an EfficientDet dual network, integrates a feature pyramid model, performs feature processing on sloppy and non-sloppy handwritten Chinese characters, uses an adversarial network to generate a large amount of adversarial sloppy sample data to improve the recognition effect of sloppy Chinese characters, and simultaneously uses a hypergraph neural network to accurately recognize the sloppy type of the recognized sloppy Chinese characters, thereby recognizing sloppy Chinese characters quickly and correcting sloppy Chinese characters with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of a handwritten Chinese character recognition and correction method provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of the operation process of a feature extraction network ResNeXt of a handwritten Chinese character recognition and correction method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0022] Example 1

[0023] The embodiment of the present invention provides a method for recognizing and correcting handwritten Chinese characters. Figure 1 As shown, the following steps are included: S1: Obtain handwritten Chinese character images and perform noise reduction on them; S2: performing image segmentation on the handwritten Chinese character image after noise reduction processing to extract the image to be recognized with Chinese characters; S3: Obtain the bounding box of a single Chinese character in the image to be recognized based on template matching; S4: establishing a training data set and a first recognition model for target handwritten Chinese characters, and training the first recognition model for target handwritten Chinese characters based on the training data set; S5: detecting the image to be recognized based on the trained first recognition model of the target handwritten Chinese characters, and obtaining a first recognition result of the handwritten Chinese characters in the image to be recognized, wherein the first recognition result includes cursive fonts and non-cursive fonts; S6: Correct the sloppy handwriting in the first recognition result based on the hypergraph neural network.

[0024] Further, in step S1, the noise reduction process is a Gaussian filter process, specifically: , In the formula, g(x,y) is the handwritten Chinese character image, (x,y) is the handwritten Chinese character image after Gaussian filtering, u(i,j) is the weight, and D(x,y) is the size range of (N+1)×(N+1) centered at (x,y); The weights are specifically:

[0025] In the formula, is the spatial channel proximity weight value, Spatial channel similarity weight value, is the input random coefficient.

[0026] Furthermore, in step S2, the image segmentation of the handwritten Chinese character image after the noise reduction processing is specifically: the image segmentation of the handwritten Chinese character image after the noise reduction processing is performed based on finding the target minimum circumscribed rectangle, which specifically includes the following steps: S21: Binarizing the handwritten Chinese character image after the noise reduction process to obtain a binary image; S22: Based on the contour detection algorithm, search and draw the contour of the binary image, and then fit the drawn target contour to find the minimum circumscribed square of the contour; S23: based on the center point of the smallest circumscribed square being at 2 / 4 to 3 / 4 of the handwritten Chinese character image, searching for the Chinese character at the center, and intercepting a segmented image with the central Chinese character; S24: Correct the segmented image to obtain an image to be recognized.

[0027] Furthermore, in step S24, the segmented image is corrected to obtain the image to be recognized, specifically: S251: Take the four corner points of the minimum circumscribed square and the four corners of the image to make a correspondence, and then obtain the perspective transformation matrix based on the fitgeotrans function in MATLAB according to these four pairs of corresponding corner points; S252: Based on the perspective transformation matrix, call the function imwarp in MATLAB to transform the segmented image into a corrected view to obtain an image to be recognized; S253: Output the image to be recognized.

[0028] Furthermore, the S3 specifically includes the following steps: S31: constructing a first image template and a second image template, wherein the first image template is a standard image of Chinese characters, and the second image template is a standard inner frame image of Chinese characters; S32: Slide the first image template and the second image template on the image to be recognized respectively to find the position with the largest best matching value, that is, to obtain the inner and outer frames of the Chinese characters in the image to be recognized.

[0029] Furthermore, in step S4, the specific implementation process of establishing the training data set and the first recognition model of the target handwritten Chinese character is as follows: S41: Obtain sloppy and non-sloppy Chinese character image data through handwritten Chinese characters, one part of which is used as a test set and the other part is used as a training set. Two EfficientDet network models are trained through the training set. The feature acquisition network of the EfficientDet network model is ResNeXt, such as Figure 2 As shown, the two network models obtained after training are respectively recorded as ResNeXt-A and ResNeXt-B, and the Chinese character image data is manually labeled and defined, including being defined as two labels: sloppy and non-sloppy; S42: Divide the EfficientDet network model of S41 into a feature extraction network ResNeXt, a feature pyramid and a sloppy classification regression network, use the feature extraction network ResNeXt to extract features of sloppy and non-sloppy images, connect the feature pyramid in series to the feature extraction network ResNeXt, perform convolution on each layer of the feature pyramid, and perform classification and regression through the sloppy classification regression network; S43: Use the test set to perform network iteration test on the feature extraction network ResNeXt, and output the recognition result. When the recognition accuracy reaches the set threshold, stop the model training. The recognition result is used to determine whether the image is sloppy.

[0030] Furthermore, the specific steps of building the feature extraction network ResNeXt are: S411: judging the image of the training set data according to the two labels of sloppy and non-sloppy, if the label of the image is sloppy, using ResNeXt-A for processing, if the label of the image is non-sloppy, using ResNeXt-B for processing, if the current input training set data image is judged to be sloppy, constructing a deep convolutional generative adversarial network for sloppy images; S412: Generate scribbled image training set samples for the scribbled images using a deep convolutional generative adversarial network; The processing flow for sloppy and non-sloppy features is as follows: The feature acquisition network adopts the network framework of the feature extraction network ResNeXt to extract the features of the sloppy and non-sloppy Chinese characters respectively, and first performs channel-based superposition on the feature maps of the second sloppy and non-sloppy convolution layers, and then performs dimensionality reduction processing through 2×2 convolution, and the same operation is performed on the third convolution layer, the fourth convolution layer, the fifth convolution layer and the sixth convolution layer; The specific process of using the deep convolutional generative adversarial network to generate scribbled image training set samples for the scribbled images is as follows: according to the scribbled image original image given by the current training, adversarial scribbled image samples are generated, and the specifications of the adversarial scribbled image samples are: font size diversity features, stroke scribbled features and overall font scribbled features. One given scribbled image original image is input, and a perturbation factor is added through the deep convolutional generative adversarial network to generate 200 scribbled sample images. The stroke scribbled features at least include the feature of non-fixed stroke length and the feature of relative stroke position deviation, and the overall font scribbled features at least include the feature of overall font width-to-height ratio deviation and the feature of overall font center of gravity position deviation.

[0031] Specifically, the sloppy image training set samples may also include a data enhancement algorithm based on DropSegment. The DropSegment method is divided into three steps: first, corner point detection is performed, then the Chinese character is segmented into a group of shorter stroke segments according to the detected corner points, and finally some stroke segments are discarded by random selection. Each handwritten Chinese character contains at least one stroke, and each stroke contains multiple stroke segments connected by corner points. After each Chinese character passes through DropSegment, the order of the remaining stroke segments in the Chinese character remains unchanged, and a new Chinese character with writing features is synthesized. Corner points can be defined by the defined segmentation method.

[0032] It should be noted that when a handwritten Chinese character contains 4 strokes, and each stroke contains 3 stroke segments, the total number of Chinese characters enhanced by the DropSegment algorithm is 32,768. Therefore, by randomly deleting some stroke segments from Chinese characters using the DropSegment algorithm, the generated new characters can greatly expand the sample set of scribbled image training, generating a large number of scribbled image training samples to improve the recognition effect of scribbled Chinese characters.

[0033] Further, the specific process of concatenating the feature pyramid to the feature extraction network ResNeXt is as follows: Sample the feature map of the third convolutional layer, that is, the fused feature map of scribbled and non-scribbled, and then superimpose it on the fused feature map of scribbled and non-scribbled of the second convolutional layer to obtain the first layer of the pyramid. Continue to execute this step for the fourth convolutional layer, the fifth convolutional layer, and the sixth convolutional layer. Superimpose the feature maps of every two adjacent layers in the channel dimension to obtain one layer of the pyramid. Eventually, a total of four layers of feature pyramids are obtained. The specific processing process of the four layers of the pyramid is as follows: The first layer of the pyramid is used to process the first image resolution, and the brightness of the image belongs to the first level of brightness; The second layer of the pyramid is used to process the second image resolution, and the brightness of the image belongs to the second level of brightness; The third layer of the pyramid is used to process the third image resolution, and the brightness of the image belongs to the third level of brightness; The fourth layer of the pyramid is used to process the fourth image resolution, and the brightness of the image belongs to the fourth level of brightness; The resolution levels of the images are specifically divided as follows: The first image resolution: w < 160, h < 160; The second image resolution: 160 < w ≤ 320, 160 < h ≤ 320; The third image resolution: 320 < w ≤ 1080, 320 < h ≤ 1080; The fourth image resolution: 1080 < w, 1080 < h, where w represents the width of the image and h represents the height of the image; The average brightness of the image is calculated using the histogram method, and is specifically divided as follows: The first average brightness: 0 ≤ L ≤ 60; The second average brightness: 60 < L ≤ 100; The third average brightness: 100 < L ≤ 160; The fourth average brightness: 160 < L ≤ 255, where L represents the average brightness value of the image; A two-branch network is added behind each fusion layer feature map of the pyramid, one branch is used for classification and the other branch is used for regression, and each branch first performs 4 convolutions on the feature map to enhance the sloppy and non-sloppy image features respectively, and the convolution kernel size is 2×2, and the number is 128.

[0034] Furthermore, the specific implementation process of S6 is as follows: Inputting the stroke scribbling features and the overall font scribbling features of the scribbled image to be processed into a first hypergraph neural network layer of a preset Chinese character correction model to obtain a first scribbled feature vector; wherein the first scribbled feature vector represents the correlation between the stroke scribbling features and the overall font scribbling features of the scribbled image to be processed; and the preset Chinese character correction model is a model obtained based on a hypergraph neural network model; Inputting the sub-features of the stroke scribbling features and the sub-features of the overall font scribbling features of the scribbled image to be processed into a second hypergraph neural network layer of a preset Chinese character correction model to obtain a second scribbled feature vector; wherein the second feature vector represents the correlation between the sub-features of the stroke scribbling features and the sub-features of the overall font scribbling features of the scribbled image to be processed; Connecting the first sloppy feature vector and the second sloppy feature vector to obtain a feature vector to be processed; Based on the preset Chinese character correction model, the feature vector to be processed is identified to obtain the scribbling type of the scribbling image, wherein the scribbling type at least includes scribbling of stroke length, scribbling of stroke intersection, scribbling of stroke tilt, scribbling of stroke gesture, scribbling of whole character position, scribbling of whole character aspect ratio and scribbling of whole character center of gravity; The scribbling types in the preset handwritten Chinese character correction library are calculated by the cosine similarity algorithm. The target similarity between the scribbling type of the scribbled image to be processed and the scribbling type in the preset handwritten Chinese character correction library is preset. If the target similarity is greater than the preset similarity threshold, the scribbling type corresponding to the handwritten Chinese character correction library is found to correct the Chinese characters in the scribbled image and generate standard Chinese characters. If the target similarity is less than or equal to the preset similarity threshold, the scribbling type of the handwritten Chinese character correction library with the nearest similarity is found to correct the Chinese characters in the scribbled image.

[0035] It should be noted that the method for extracting the cursive Chinese character features extracts some geometric features of Chinese characters, such as the endpoints, bifurcation points, concave and convex parts, and line segments in horizontal, vertical, inclined and other directions, closed loops, etc. of the characters, and performs logical combination and judgment based on the positions and mutual relationships of these features to obtain the cursive Chinese character feature information. For handwritten fonts, the Chinese character feature information is relatively complex. Therefore, the embodiment of the present invention effectively improves the recognition accuracy of the cursive type of handwritten Chinese characters by performing overall and local feature extraction on handwritten fonts.

[0036] Specifically, the cosine similarity algorithm:

[0037] in, represents the scribble type feature of the scribble image to be processed, Preset sloppy type features in the handwritten Chinese character correction library; The cosine similarity algorithm can be used to calculate the target similarity of the scribbled image to be processed. The larger the value, the more similar the target similarity is to the preset similarity threshold, and vice versa; If the target similarity is less than or equal to the preset similarity threshold, the similarity values ​​are used for sorting, and the closest sort value is set as the standard Chinese character corresponding to the sloppy type in the handwritten Chinese character correction library found, and the sloppy Chinese character is corrected to generate a standard Chinese character.

[0038] Furthermore, the loss function of the preset Chinese character correction model is:

[0039] In the formula, is the loss function, k is the balance factor, is the incidence matrix aggregation parameter, is the target probability of the scribbled type to be classified.

[0040] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0041] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0042] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0043] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into a plurality of submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device so disclosed may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0044] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0045] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the apparatus according to an embodiment of the present invention. The present invention may also be implemented as a device or device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

Claims

1. A method for recognizing and correcting handwritten Chinese characters, characterized in that: The following steps are involved: S1: Obtain handwritten Chinese character images and perform noise reduction on them; S2: performing image segmentation on the handwritten Chinese character image after noise reduction processing to extract the image to be recognized with Chinese characters; S3: Obtain the bounding box of a single Chinese character in the image to be recognized based on template matching; S4: establishing a training data set and a first recognition model for target handwritten Chinese characters, and training the first recognition model for target handwritten Chinese characters based on the training data set; S5: detecting the image to be recognized based on the trained first recognition model of the target handwritten Chinese characters, and obtaining a first recognition result of the handwritten Chinese characters in the image to be recognized, wherein the first recognition result includes cursive fonts and non-cursive fonts; S6: Correct the sloppy handwriting in the first recognition result based on the hypergraph neural network.

2. A handwritten Chinese character recognition and correction method according to claim 1, characterized in that: In step S1, the noise reduction process is a Gaussian filter process, specifically: ， In the formula, g(x,y) is the handwritten Chinese character image, (x,y) is the handwritten Chinese character image after Gaussian filtering, u(i,j) is the weight, and D(x,y) is the size range of (N+1)×(N+1) centered at (x,y); The weights are specifically: In the formula, is the spatial channel proximity weight value, Spatial channel similarity weight value, is the input random coefficient.

3. A handwritten Chinese character recognition and correction method according to claim 2, characterized in that: In step S2, the image segmentation of the handwritten Chinese character image after the noise reduction processing is specifically: the image segmentation of the handwritten Chinese character image after the noise reduction processing is performed based on finding the target minimum circumscribed rectangle, which specifically includes the following steps: S21: Binarizing the handwritten Chinese character image after the noise reduction process to obtain a binary image; S22: Based on the contour detection algorithm, search and draw the contour of the binary image, and then fit the drawn target contour to find the minimum circumscribed square of the contour; S23: based on the center point of the smallest circumscribed square being at 2 / 4 to 3 / 4 of the handwritten Chinese character image, searching for the Chinese character at the center, and intercepting a segmented image with the central Chinese character; S24: Correct the segmented image to obtain an image to be recognized.

4. A handwritten Chinese character recognition and correction method according to claim 3, characterized in that: In step S24, the segmented image is corrected to obtain the image to be recognized, specifically: S251: Take the four corner points of the minimum circumscribed square and the four corners of the image to make a correspondence, and then obtain the perspective transformation matrix based on the fitgeotrans function in MATLAB according to these four pairs of corresponding corner points; S252: Based on the perspective transformation matrix, call the function imwarp in MATLAB to transform the segmented image into a corrected view to obtain an image to be recognized; S253: Output the image to be recognized.

5. A handwritten Chinese character recognition and correction method according to claim 4, characterized in that: The S3 specifically includes the following steps: S31: constructing a first image template and a second image template, wherein the first image template is a standard image of Chinese characters, and the second image template is a standard inner frame image of Chinese characters; S32: Slide the first image template and the second image template on the image to be recognized respectively to find the position with the largest best matching value, that is, to obtain the inner and outer frames of the Chinese characters in the image to be recognized.

6. A handwritten Chinese character recognition and correction method according to claim 5, characterized in that: In step S4, the specific implementation process of establishing the training data set and the first recognition model of the target handwritten Chinese character is as follows: S41: Obtaining sloppy and non-sloppy Chinese character image data through handwritten Chinese characters, a part of which is used for a test set, and a part of which is used for a training set, and two EfficientDet network models are trained through the training set, wherein the feature acquisition network of the EfficientDet network model is ResNeXt, and the two network models obtained after training are respectively recorded as ResNeXt-A and ResNeXt-B, and the Chinese character image data is manually labeled and defined, including being defined as two labels of sloppy and non-sloppy; S42: Divide the EfficientDet network model of S41 into a feature extraction network ResNeXt, a feature pyramid and a sloppy classification regression network, use the feature extraction network ResNeXt to extract features of sloppy and non-sloppy images, connect the feature pyramid in series to the feature extraction network ResNeXt, perform convolution on each layer of the feature pyramid, and perform classification and regression through the sloppy classification regression network; S43: Use the test set to perform network iteration test on the feature extraction network ResNeXt, and output the recognition result. When the recognition accuracy reaches the set threshold, stop the model training. The recognition result is used to determine whether the image is sloppy.

7. A handwritten Chinese character recognition and correction method according to claim 6, characterized in that: The specific steps of building the feature extraction network ResNeXt are: S411: judging the image of the training set data according to the two labels of sloppy and non-sloppy, if the label of the image is sloppy, using ResNeXt-A for processing, if the label of the image is non-sloppy, using ResNeXt-B for processing, if the current input training set data image is judged to be sloppy, constructing a deep convolutional generative adversarial network for sloppy images; S412: Generate scribbled image training set samples for the scribbled images using a deep convolutional generative adversarial network; The processing flow for sloppy and non-sloppy features is as follows: The feature acquisition network adopts the network framework of the feature extraction network ResNeXt to extract the features of the sloppy and non-sloppy Chinese characters respectively, and first performs channel-based superposition on the feature maps of the second sloppy and non-sloppy convolution layers, and then performs dimensionality reduction processing through 2×2 convolution, and the same operation is performed on the third convolution layer, the fourth convolution layer, the fifth convolution layer and the sixth convolution layer; The specific process of using the deep convolutional generative adversarial network to generate scribbled image training set samples for the scribbled images is as follows: according to the scribbled image original image given by the current training, adversarial scribbled image samples are generated, and the specifications of the adversarial scribbled image samples are: font size diversity features, stroke scribbled features and overall font scribbled features. One given scribbled image original image is input, and a perturbation factor is added through the deep convolutional generative adversarial network to generate 200 scribbled sample images. The stroke scribbled features at least include the feature of non-fixed stroke length and the feature of relative stroke position deviation, and the overall font scribbled features at least include the feature of overall font width-to-height ratio deviation and the feature of overall font center of gravity position deviation.

8. A handwritten Chinese character recognition and correction method according to claim 7, characterized in that: The specific process of concatenating the feature pyramid to the feature extraction network ResNeXt is as follows: For the third convolutional layer, that is, the feature map of the fusion of scribbled and non-scribbled, sampling is performed, and then it is superimposed with the feature map of the fusion of scribbled and non-scribbled in the second convolutional layer to obtain the first layer of the pyramid. Continuing this step for the fourth convolutional layer, the fifth convolutional layer, and the sixth convolutional layer, and performing channel-wise superposition of every two adjacent feature maps, one layer of the pyramid can be obtained. Eventually, a total of four feature pyramids are obtained. The specific processing process of the four layers of the pyramid is as follows: The first layer of the pyramid is used to process the first image resolution, and the brightness of the image belongs to the first-level brightness; The second layer of the pyramid is used to process the second image resolution, and the brightness of the image belongs to the second-level brightness; The third layer of the pyramid is used to process the third image resolution, and the brightness of the image belongs to the third-level brightness; The fourth layer of the pyramid is used to process the fourth image resolution, and the brightness of the image belongs to the fourth-level brightness; The resolution levels of the images are specifically divided as follows: The first image resolution: w < 160, h < 160; The second image resolution: 160 < w ≤ 320, 160 < h ≤ 320; The third image resolution: 320 < w ≤ 1080, 320 < h ≤ 1080; The fourth image resolution: 1080 < w, 1080 < h, where w represents the width of the image and h represents the height of the image; The average brightness of the image is calculated by using the histogram method, and is specifically divided as follows: The first average brightness: 0 ≤ L ≤ 60; The second average brightness: 60 < L ≤ 100; The third average brightness: 100 < L ≤ 160; The fourth average brightness: 160 < L ≤ 255, where L represents the average brightness value of the image; Two branch networks are added after the feature map of each fusion layer of the pyramid. One branch is used for classification, and the other branch is used for regression. And for each branch, the feature map is first convolved 6 times to enhance the image features of scribbled and non-scribbled respectively, and the convolution kernel size is 3×3 and the number is 64.

9. A handwritten Chinese character recognition and correction method according to claim 8, characterized in that: The specific implementation process of S6 is as follows: Input the stroke scribble feature and the overall font scribble feature of the scribbled image to be processed into the first hypergraph neural network layer of the preset Chinese character correction model to obtain the first scribble feature vector; wherein, the first scribble feature vector represents the correlation relationship between the stroke scribble feature and the overall font scribble feature of the scribbled image to be processed; the preset Chinese character correction model is a model obtained based on the hypergraph neural network model; Input the sub-feature of the stroke scribble feature and the sub-feature of the overall font scribble feature of the scribbled image to be processed into the second hypergraph neural network layer of the preset Chinese character correction model to obtain the second scribble feature vector; wherein, the second feature vector represents the correlation relationship between the sub-feature of the stroke scribble feature and the sub-feature of the overall font scribble feature of the scribbled image to be processed; Connect the first scribble feature vector and the second scribble feature vector to obtain the feature vector to be processed; Based on the preset Chinese character correction model, the feature vector to be processed is identified to obtain the scribbling type of the scribbling image, wherein the scribbling type at least includes scribbling of stroke length, scribbling of stroke intersection, scribbling of stroke tilt, scribbling of stroke gesture, scribbling of whole character position, scribbling of whole character aspect ratio and scribbling of whole character center of gravity; The scribbling types in the preset handwritten Chinese character correction library are calculated by the cosine similarity algorithm. The target similarity between the scribbling type of the scribbled image to be processed and the scribbling type in the preset handwritten Chinese character correction library is preset. If the target similarity is greater than the preset similarity threshold, the scribbling type corresponding to the handwritten Chinese character correction library is found to correct the Chinese characters in the scribbled image and generate standard Chinese characters. If the target similarity is less than or equal to the preset similarity threshold, the scribbling type of the handwritten Chinese character correction library with the nearest similarity is found to correct the Chinese characters in the scribbled image.

10. A handwritten Chinese character recognition and correction method according to claim 9, characterized in that: The loss function of the preset Chinese character correction model is: In the formula, is the loss function, k is the balance factor, is the incidence matrix aggregation parameter, is the target probability of the scribbled type to be classified.

Images