Handwritten form identification method and system based on metadata

Abstract

The invention provides a handwritten table recognition method and system based on metadata, and relates to the field of optical character recognition in the artificial intelligence technology, and the method comprises the steps: based on a standard picture corresponding to an original record table in a production environment, defining metadata which is used for describing table information in the picture, comprising organization structure information of the table and information of each cell; restoring the to-be-recognized handwritten table picture into a standard table picture by adopting a table restoration method based on a neural network; according to the information of each cell in the metadata, carrying out cell segmentation on the restored handwritten table picture; performing recognition and error correction on characters in each unit lattice graph, and assembling the corrected texts into a table file in a set format according to organization structure information in metadata; according to the method, the table picture is restored and segmented by means of the metadata, and character recognition and error correction are performed on the segmented cells, so that the speed, precision and reusability of handwritten table recognition are improved.

Description

Technical Field

[0001] The present invention belongs to the field of optical character recognition in artificial intelligence technology, and in particular relates to a metadata-based handwritten form recognition method and system. Background Art

[0002] At present, in the manufacturing industry, it is necessary to test the pass rate of products; inspectors need to conduct on-site sampling measurements of products, manually fill in the test results in the standard original record form, scan or take a photo to convert it into an electronic version of the original record picture; then manually identify the indicator data related to product quality in the picture, use relevant formulas to calculate, and fill in the calculation results in the quality inspection report.

[0003] Since there are a large number of original record images to be identified and there are also many indicators to be identified in each image, manual identification will take a lot of time. Therefore, it is hoped that there will be an end-to-end OCR (optical character recognition) module that can batch recognize image tables, convert them into corresponding EXCEL tables, and recognize the text in the image table and correspond it to the corresponding position in EXCEL.

[0004] Recognition of handwritten tables requires two core technologies, including table parsing technology and target recognition technology: table parsing is used to identify the structure of tables in images and effectively restore them; target recognition can identify text within an area. For table recognition, commonly used solutions include line detection, box detection, segmentation, multimodal fusion, GCN, and img2seq; for text recognition, algorithms such as CRNN and SVTR are often used for recognition.

[0005] In the production environment, both table recognition and text recognition have corresponding pain points; the table recognition algorithm has a high accuracy rate for axis-aligned printed images, while the actual production environment tables have more content and are more complex; and the text position in the table is not fixed, the content cannot be axis-aligned, and it is difficult to train; these two situations lead to low table recognition accuracy and easy to go wrong. In actual production, it is often necessary to customize multiple models according to the scenario and stack them to achieve the expected effect; text recognition is easily affected by table lines in the actual production environment, and the recognition accuracy is related to the backbone neural network model used by the algorithm; large models have high accuracy, but many parameters and slow recognition speed, and can only be deployed in servers with GPUs; small models have poor accuracy, but the model is smaller, has fewer parameters, and is fast, which is convenient for deployment on edge or mobile devices; for table recognition scenarios, they are generally deployed on ordinary PC terminals, so models with high model accuracy and relatively small models are required.

[0006] Therefore, for some handwritten forms of original inspection orders in the manufacturing industry, the existing recognition solutions are insufficient in speed, accuracy and reusability. Summary of the invention

[0007] In order to overcome the shortcomings of table recognition and text recognition in the prior art in a production environment, the present invention provides a metadata-based handwritten table recognition method and system, which uses metadata describing the table information in the image to restore and segment the table image, and performs text recognition and error correction on the segmented cells, thereby improving the speed, accuracy and reusability of handwritten table recognition.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] A first aspect of the present invention provides a metadata-based handwritten form recognition method.

[0010] A metadata-based handwritten form recognition method, comprising:

[0011] Based on the standard image corresponding to the original record table in the production environment, define metadata to describe the table information in the image, including the organizational structure information of the table and the information of each cell;

[0012] A table restoration method based on a neural network is used to restore the handwritten table image to be recognized into a standard table image;

[0013] According to the information of each cell in the metadata, the restored handwritten table image is segmented into cells to obtain several cell sub-images;

[0014] The text in each cell sub-image is identified and corrected, and the corrected text is assembled into a table file in a set format according to the organizational structure information in the metadata.

[0015] Furthermore, the organizational structure information of the table includes two tags, tr and td, where tr represents a row and td represents a column; td includes two attributes, including colspan and rowspan, where colspan represents the number of rows a cell occupies and rowspan represents the number of columns a cell occupies.

[0016] Furthermore, the cell information includes the coordinates and rules of the cell;

[0017] The coordinates of the cell are used for cell segmentation;

[0018] The rule of the cell is used to correct errors in the recognized text.

[0019] Furthermore, the table restoration method based on neural network is specifically as follows:

[0020] Use neural networks to find straight lines in handwritten table images;

[0021] Use a line segment-based distortion correction algorithm to correct the angle of handwritten table images;

[0022] Based on the four corner points of the table in the corrected image, the corrected image is transformed into a standard table image.

[0023] Furthermore, the neural network uses the trained U-Net model to find the straight lines in the handwritten table image, and each straight line is represented by the coordinates of both ends.

[0024] Furthermore, the line segment-based distortion correction algorithm is to define a polynomial for correction, which is used to convert points on the surface into corrected corresponding points;

[0025] The polynomial is defined based on the identified straight line in the table, and the correction angle is calculated through the line segments therein, and the cost function is constructed, and the parameters of the polynomial are obtained through the LM algorithm.

[0026] Furthermore, the standard table image is obtained based on the corrected image, and the specific steps are as follows:

[0027] Based on the position characteristics of the four corner points, the coordinates of the four corner points of the table in the corrected image are obtained;

[0028] Taking the four corner points of the table in the corrected image as source points and the four corner points of the table in the standard table image as target points, the projection center and projection angle are determined, thereby transforming the corrected image into a standard table image.

[0029] A second aspect of the present invention provides a metadata-based handwritten form recognition system.

[0030] A metadata-based handwritten form recognition system includes a definition module, a restoration module, a segmentation module and a recognition module:

[0031] The definition module is configured to: define metadata based on a standard image corresponding to an original record table in a production environment, for describing table information in the image, including organizational structure information of the table and information of each cell;

[0032] The restoration module is configured to: restore the handwritten table image to be recognized into a standard table image using a table restoration method based on a neural network;

[0033] The segmentation module is configured to: perform cell segmentation on the restored handwritten table image according to the information of each cell in the metadata to obtain a plurality of cell sub-images;

[0034] The recognition module is configured to: recognize and correct the text in each cell sub-image, and assemble the corrected text into a table file of a set format according to the organizational structure information in the metadata.

[0035] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the metadata-based handwritten form recognition method as described in the first aspect of the present invention.

[0036] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the metadata-based handwritten form recognition method as described in the first aspect of the present invention are implemented.

[0037] One or more of the above technical solutions have the following beneficial effects:

[0038] The present invention uses metadata that describes table information in images to recognize handwritten tables, thereby improving recognition speed, accuracy and reusability. For a manufacturing industry, metadata templates are relatively fixed and limited in types, and the process of making metadata is relatively simple, with little workload. The entire end-to-end recognition only relies on metadata, and only one more type of original record table needs to be added, without modifying the product itself.

[0039] Since the accuracy of handwritten text recognition is much lower than that of printed text, the data recognized by the machine cannot be used directly even after training. The present invention adds some error correction mechanisms, annotates the value ranges of some cells in the metadata, and marks the values ​​outside the range, so that users only need to pay attention to the marked text, thereby saving users' workload while ensuring the accuracy of artificial intelligence recognition.

[0040] Based on metadata, the present invention restores and segments the table image, and performs text recognition and error correction on the segmented cells. In particular, a table restoration method based on a neural network is adopted to perform image correction and perspective transformation based on the identified table straight lines to obtain a standard image. The standard image has strong robustness and can adapt to various scenarios of machine scanning and mobile phone shooting.

[0041] The present invention reduces the size of the CRNN model while ensuring accuracy through a distillation algorithm, allowing the model to run on terminals with different performances.

[0042] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0044] Figure 1 This is a flow chart of the method of the first embodiment.

[0045] Figure 2 This is a flowchart of metadata definition for the first embodiment.

[0046] Figure 3 It is a structural diagram of the U-NET model of the first embodiment.

[0047] Figure 4 This is a comparison chart of the straight line recognition effect of the first embodiment.

[0048] Figure 5 This is a model diagram of the distortion correction algorithm of the first embodiment.

[0049] Figure 6 Schematic diagram of perspective transformation of the first embodiment.

[0050] Figure 7 FIG. 1 is a diagram of the DML distillation process of the first embodiment.

[0051] Figure 8 This is a CS side page diagram of the first embodiment.

[0052] FIG. 9 is a diagram showing the recognition effect of the first embodiment. DETAILED DESCRIPTION

[0053] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0054] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0055] Embodiment 1

[0056] In one embodiment of the present disclosure, a metadata-based handwritten table recognition method is provided, which is used to generate a table file of a set format for a handwritten table image. In this embodiment, the table file of the set format is an Excel file, such as Figure 1 As shown, the following steps are included:

[0057] Step S1: Based on the standard image corresponding to the original record table in the production environment, metadata is defined to describe the table information in the image, including the organizational structure information of the table and the information of each cell.

[0058] Furthermore, the organizational structure information of the table includes two tags, tr and td, where tr represents a row and td represents a column; td includes two attributes, including colspan and rowspan, where colspan represents the number of rows a cell occupies and rowspan represents the number of columns a cell occupies.

[0059] Furthermore, the cell information includes the coordinates and rules of the cell;

[0060] The coordinates of the cell are used for cell segmentation;

[0061] The rule of the cell is used to correct errors in the recognized text.

[0062] Specifically, metadata is used to describe the standard image, including id, structure, cells, width, and height; id refers to the unique identifier of the template, structure refers to the structure of the template table, that is, the organizational structure information, cells refers to the information of each cell, including coordinates and rules, width refers to the length of the standard image, and height is the width of the standard image.

[0063] The organizational structure information structure contains two tags, tr and td. tr represents a row and td represents a column. td contains two attributes, including colspan and rowspan. Colspan represents the number of rows a cell occupies, and rowspan represents the number of columns a cell occupies. In this way, structure can describe any table.

[0064] The cell information cells includes coordinates and rules. The coordinates are represented by bbox, which includes the coordinates of the four corners of the cell, represented as [x1, y1, x2, y2, x3, y3, x4, y4], and is used for cell segmentation in step S3; the rules are represented by rules, including the value range and value type, such as "less than or equal to", "greater than or equal to", "is a number", "is a floating point type", and "is a string", which are uniformly represented as [≤; ≥; isNumber; isFloat; Str], and are used to correct the recognized text in step S4.

[0065] The process of defining metadata, such as Figure 2 As shown, specifically:

[0066] First use the labelme tool Figure 2 The standard image shown in (1) is marked in the order from top to bottom and from left to right. The standard image refers to a minimized image containing all information generated by scanning with a scanner. Minimization means removing the surrounding blank areas and only including the table area. The image is clear, square and wrinkle-free. The marking method is to use the rectangular frame in the lebelme tool to frame the area where the table cell is located. The area must be within the borders around the cell. When framing the cell, you need to enter a label. If the cell contains rules, the label is written as rules = xxx, where xxx is the specific content of the rule. The visualization effect after framing is as follows: Figure 2 As shown in (2).

[0067] right Figure 2 The EXCEL table shown in (3) scans the area containing the number 1 and obtains the structure of the table as follows Figure 2 As shown in (4), the json file generated by labelme is converted to generate Figure 2 The file shown in (5) and Figure 2 The content in (4) is merged into the structure part of the file to generate the final metadata file, as shown below:

[0068] {

[0069] 'id':'001',

[0070] 'structure':'['','',...]',

[0071] 'cells':'[{'bbox':[x1,y1,x2,y2,x3,y3,x4,y4]},'rules':['isNumber','≤70','≥20'],...],

[0072] 'width':'670',

[0073] 'height':'884'

[0074] }

[0075] Step S2: Using a table restoration method based on a neural network to restore the handwritten table image to be recognized into a standard table image, specifically:

[0076] Step S201: using a neural network to find a table straight line in a handwritten table image;

[0077] Furthermore, the neural network uses the trained U-Net model to find the straight lines in the handwritten table image, and each straight line is represented by the coordinates of both ends.

[0078] Use the deep learning method of neural network to identify the table lines in the picture. The deep learning method is based on image semantic segmentation, and the corresponding table line detection model is trained to realize the automatic detection of the table line area in the picture; image semantic segmentation is to segment the image into regional blocks with certain semantic meanings through a certain method, and identify the semantic category of each regional block, to realize the semantic reasoning process from the bottom to the top, and finally obtain a segmented image with pixel-by-pixel semantic annotation.

[0079] In this embodiment, the U-NET model is used as the semantic segmentation model. The main execution process of the U-NET model is similar to other semantic segmentation models, and it can adapt to a very small training set. The structure of the U-NET model is as follows: Figure 3 As shown in the figure, it consists of two parts. The first part is the feature extraction part on the left, and the second part is the upsampling part on the right, which is the encoder-decoder architecture. The input of the U-NET model is a handwritten table image, and the output is the lboxes value of all lines, expressed as (x begin ,y begin ,x end ,y end ).

[0080] Use labeled data to train a U-Net model that can recognize table lines, specifically:

[0081] First, select training data; analyze the image containing the table, use a line segment of preset width to mark each table line, and mark the pixels on the line segment; the unmarked part is regarded as the background part;

[0082] Secondly, use the training data to build a data set; for any data set, randomly split it into a training set and a test set;

[0083] Then, the U-Net model is trained using the training set. The training is stopped when the model prediction performance reaches the preset value after being tested on the test set, and the image table line extraction model is constructed.

[0084] The U-Net model is trained by inputting photos of linear tables in various scenarios, so that the model can still accurately find the table lines when the light is dim and the picture structure is more complex. Therefore, the algorithm is more robust in discovering table lines.

[0085] The U-Net model is better than traditional methods in recognizing table lines, such as Figure 4As shown in the figure, the improved Huffman algorithm is on the left and the U-Net model is on the right. The upper edge line of the method table on the left is affected by the number in the upper right corner and is not fully recognized and disconnected from the point in the upper right corner, while the Unet model has higher robustness.

[0086] Step S202: using a line segment-based distortion correction algorithm to correct the angle of the handwritten table image;

[0087] Furthermore, the line segment-based distortion correction algorithm is to define a polynomial for correction, which is used to convert points on the surface into corresponding points after correction;

[0088] The polynomial is defined based on the identified straight line in the table, and the correction angle is calculated through the line segments therein, and the cost function is constructed, and the parameters of the polynomial are obtained through the LM algorithm.

[0089] The model of the distortion correction algorithm is as follows Figure 5 As shown in the figure, camera frame refers to the camera's three-dimensional coordinate system, which describes the position of the camera when taking a photo; document frame is the document's coordinate system, which is the coordinate system in the world obtained by shooting the document (image plane) with a camera, called the world coordinate system. Both the world coordinate system and the camera's three-dimensional coordinate system are three-dimensional coordinate systems, which can be converted to each other through translation and rotation; image plane is the image's two-dimensional coordinate system, which is the plane in the camera coordinate system whose distance from the camera's optical center is f, denoted by (α, β), where f is called the focal length of the camera.

[0090] Then, according to Figure 5 As shown, the document coordinate system is rotated and translated to obtain the camera coordinate system, and then the image two-dimensional coordinate system is obtained by scaling, and finally the corrected pixel two-dimensional coordinate system is obtained by pixel transformation.

[0091] The mapping model of document 3D point (document frame) -> camera 3D point (camera frame) -> image 2D coordinate system (image plane) is:

[0092]

[0093] Where f is the focal length of the camera, (c x ,c y ) is the image center, (s, R, t) represents scaling, rotation, and translation. Since the parameters s and t are not related to the correction but to the image scale and resolution, we set s = 1 and t = [0, 0, f] during the correction process. T , then the points of the image two-dimensional coordinate system can be expressed as:

[0094]

[0095] The points of the image two-dimensional coordinate system are converted into plane document coordinates (u, v), that is, the corrected two-dimensional coordinate system.

[0096] Through the polynomial fitting quantization process, the image coordinate system is converted to the pixel coordinate system. The specific conversion formula is:

[0097]

[0098] Among them, g(x) can be expressed using a 4th-order polynomial, which is as follows:

[0099]

[0100] The reason why the 4th-order polynomial is used is that it is found through experiments that polynomial equations with orders lower than 4 cannot represent page curl well, and when the order is greater than 4, overfitting will occur, and the 4th-order polynomial is the best choice; by combining the above three formulas, the points of the image's two-dimensional coordinate system can be converted to corresponding points on the corrected document image.

[0101] The Levenberg-Marquardt (LM) algorithm is used to obtain the parameters of the polynomial. The Levenberg-Marquardt (LM) algorithm is an optimization algorithm for nonlinear least squares fitting. It belongs to the trust region method. It finds the local minimum of the function by adjusting the parameters within the trust region. The LM algorithm estimates the parameters through the cost function. Since the original record table has many line segments, which are horizontally or vertically aligned in the well-corrected image, this feature is encoded into the proposed cost function. Since the line segments in the original table are horizontal or vertical, the cost function is set to Since the lboxes of all lines were obtained in the previous step, the angle θ in the corrected image can be obtained as arctan2(y end -y begin ,x end -x begin ), so that the polynomial parameters can be obtained through the LM algorithm, and finally the transformation from the distorted image to the target image is obtained.

[0102] Step S203: Based on the four corner points of the table in the corrected image, the corrected image is perspective transformed to obtain a standard table image. The specific steps are as follows:

[0103] (1) Based on the position characteristics of the four corner points, the coordinates of the four corner points of the table in the corrected image are obtained.

[0104] After correction, the U-Net model is used again to identify all the lboxes of the line, and then all the lboxes are traversed to find the point in the upper right corner and the point in the lower left corner of the table, and to find the horizontal line containing the upper right point; a straight line can be represented as [x s ,y s ,x e ,y e ], where x s ,y s Indicates the horizontal and vertical coordinates of the starting node of the line, x e ,y e is the horizontal and vertical coordinates of the end node. Since the point at the top right of the table has the smallest y value and the largest x value, the point at the top right is found by finding the point with the smallest yx value; the point at the bottom left corner has the largest y value and the smallest x value, so the point at the bottom left is found by finding the point with the largest yx value.

[0105] Similarly, the point in the upper left corner of the table has the smallest y value and the smallest x value, so the point in the upper left corner is found by finding the point with the smallest y+x; the point in the lower right corner has the largest y value and the largest x value, so the point in the lower right corner is found by finding the point with the largest y+x; the four corner points of the table are represented clockwise from the upper left corner as (x1, y1), (x2, y2), (x3, y3), and (x4, y4).

[0106] (2) Taking the four corner points of the table in the corrected image as source points and the four corner points of the table in the standard table image as target points, determine the projection center and projection angle, that is, the transformation matrix, so as to transform the perspective of the corrected image into the standard table image.

[0107] The length and width of the standard image defined in the metadata are w and h, the four original coordinate points srcPoints are [[x'1, y'1], [x'2, y'2], [x'3, y'3], [x'4, y'4]], and the four target coordinate points dstPoints are [[0, 0], [w, 0], [0, h], [w, h]]. By corresponding the original points to the target points, the transformation matrix required for the projection transformation can be obtained, which is expressed as follows:

[0108] M=getPerspectiveTransform(srcPoints,dstPoints)

[0109] Among them, M is the transformation matrix, and getPerspectiveTransform is a method to obtain the transformation matrix in Python.

[0110] Through the transformation matrix, each coordinate in the original image is transformed and mapped into the coordinate in the target image. The mapping process is as follows Figure 6As shown, the following transformation process is used to transform the original image plane into a new image plane:

[0111]

[0112] Among them, [u,v,w] are the original coordinates, [x',y',w'] are the transformed coordinates, is the transformation matrix M.

[0113] Step S3: According to the information of each cell in the metadata, the restored handwritten table image is segmented into cells to obtain a plurality of cell sub-images.

[0114] The cell information includes the coordinates of the cell. The handwritten table image is segmented according to the coordinates, and the cell borders are removed to obtain the internal area of ​​the cell as the cell sub-image.

[0115] Step S4: Identify and correct the text in each cell sub-image, and assemble the corrected text into a table file in a set format according to the organizational structure information in the metadata.

[0116] Based on the Convolutional Recurrent Neural Network (CRNN), text is extracted from the cell sub-graph. In order to reduce the size of the model and improve the recognition speed of CRNN, an attempt is made to use MobileNetV3 as the CRNN backbone for direct training, but the accuracy does not meet the requirements.

[0117] The DML mutual learning distillation method is adopted, that is, two MobileNetV3 models with the same model structure learn from each other. While ensuring accuracy, the model size is reduced from 40MB to 10MB, which improves the recognition speed of the model. In text recognition, CTC LOSS is usually used as the loss function. The loss function calculates the distance between the output value and the true value, but in text recognition, there will be problems such as the output of the network is "-4-.-13-5", and its ground truth is "4.135". There will also be Many-to-One situations, for example: "-ss-tate-" and "--stt-a-tt-e" can both correspond to "state", and CTC LOSS solves these two problems. While using CTC LOSS, DML distillation adds DML LOSS and uses it together as the total loss function. In the process of mutual distillation, it is hoped that the final output distribution of the two networks is expected to be the same, so KL divergence is used to calculate the difference between the two networks.

[0118] DML distillation process Figure 7As shown, the formula corresponding to DMLLOSS is:

[0119]

[0120] Among them, S and T represent the student network and the teacher network, KL(S out ||T out ) represents the divergence comparison between the output results of the two networks.

[0121] The sum of DML LOSS, CTC LOSS of the student network, and CTC LOSS of the teacher network is used as the total loss function as follows:

[0122] Loss total =Loss ctcs +Loss ctct +Loss dml

[0123] In view of the above scheme, based on the current status of the handwritten form of the original inspection form of the manufacturing industry, an interactive and friendly CS end page based on JAVA was developed, such as Figure 8 As shown, the main function is to batch import photos under a folder, and choose whether to correct them (scanned photos do not need to be corrected), then select metadata, and recognize handwritten forms. The program has made many improvements in many aspects. It can recognize pictures in batches. After each picture is recognized, the path to generate EXCEL will be printed on the panel. When the user clicks the panel, the original picture and the generated EXCEL picture will pop up automatically. The user adjusts the position of the two for comparison, corrects the yellow data in EXCEL, and closes the two files. When clicking the next file, the position of the picture and EXCEL file remains unchanged, which is convenient for users to compare. The program will generate a perspective subfolder under the user's photo folder, a subfolder named after the picture name, and a result EXCEL named after the picture name. The perspective subfolder stores the result of restoring the original picture to the template picture, and the subfolder named after the picture name stores the segmented photos. The naming rule starts from 00001 and is accumulated in the order of metadata. If the recognition effect is poor, the user can judge whether it is a problem with the photo restoration or the text recognition model based on the perspective result and segmentation result.

[0124] pass Figure 2 The method shown uses labelme to annotate metadata. If the cell has a rule, the annotation value is the rule, otherwise it is -1; the target EXCEL corresponding to the editing and metadata is automatically generated after program conversion and merging; in the case of prefabricated component original record sheet recognition in the construction industry, three metadata are generated: "concrete rebound.json", "concrete structure.json", and "rebar.json".

[0125] pass Figure 3 The network is used to train the U-Net model for table line recognition. The weight parameter of the model is named "table-line.weights" and its size is 13M.

[0126] Through the ICDAR2015 dataset, use Figure 7 As shown, the DML mutual learning distillation method is used to train the CRNN model, and a standard training set of 10,000 handwritten images is used to train the weight parameter infer.pdiparams, which is 10M in size.

[0127] Execute the main program to identify the photos in the selected folder and generate the result EXCEL; after practice, for the 232 indicators that need to be identified in the original record of concrete rebound, on an ordinary laptop with 4 cores and i7 CPU, it takes 40 seconds to complete each identification, and the average recognition speed is 0.17 seconds per photo.

[0128] The recognition effect is shown in Figure 9, where Figure 9(a), Figure 9(b), and Figure 9(c) are the recognition effect diagrams of the original wrinkle image, the corrected image, and the scanned image, respectively. The original wrinkle image is taken with an ordinary camera and is distorted and wrinkled, while the scanned image is a clear, square, wrinkle-free standard image scanned with a scanner, which can be used to compare the correction effect. It can be seen from the figure that the recognition accuracy of the correction and scanning is high, meeting the requirements of the production environment.

[0129] Embodiment 2

[0130] In one embodiment of the present disclosure, a metadata-based handwritten form recognition system is provided, including a definition module, a restoration module, a segmentation module and a recognition module:

[0131] The definition module is configured to: define metadata based on a standard image corresponding to an original record table in a production environment, for describing table information in the image, including organizational structure information of the table and information of each cell;

[0132] The restoration module is configured to: restore the handwritten table image to be recognized into a standard table image using a table restoration method based on a neural network;

[0133] The segmentation module is configured to: perform cell segmentation on the restored handwritten table image according to the information of each cell in the metadata to obtain a plurality of cell sub-images;

[0134] The recognition module is configured to: recognize and correct the text in each cell sub-image, and assemble the corrected text into a table file of a set format according to the organizational structure information in the metadata.

[0135] Embodiment 3

[0136] The purpose of this embodiment is to provide a computer-readable storage medium.

[0137] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a handwritten form recognition method based on metadata as described in the first embodiment of the present disclosure.

[0138] Embodiment 4

[0139] The purpose of this embodiment is to provide an electronic device.

[0140] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in a metadata-based handwritten form recognition method as described in the first embodiment of the present disclosure are implemented.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A metadata-based handwritten form recognition method, characterized in that: include: Based on the standard image corresponding to the original record table in the production environment, define metadata to describe the table information in the image, including the organizational structure information of the table and the information of each cell; A table restoration method based on a neural network is used to restore the handwritten table image to be recognized into a standard table image; According to the information of each cell in the metadata, the restored handwritten table image is segmented into cells to obtain several cell sub-images; The text in each cell sub-image is identified and corrected, and the corrected text is assembled into a table file in a set format according to the organizational structure information in the metadata.

2. The metadata-based handwritten form recognition method according to claim 1, characterized in that: The organizational structure information of the table includes two tags, tr and td. tr represents a row and td represents a column. td includes two attributes, including colspan and rowspan. colspan represents the number of rows occupied by a cell and rowspan represents the number of columns occupied by a cell.

3. The metadata-based handwritten form recognition method according to claim 1, characterized in that: The information of the cell includes the coordinates and rules of the cell; The coordinates of the cell are used for cell segmentation; The rule of the cell is used to correct errors in the recognized text.

4. The metadata-based handwritten form recognition method according to claim 1, characterized in that: The table restoration method based on neural network is specifically as follows: Use neural networks to find straight lines in handwritten table images; Use a line segment-based distortion correction algorithm to correct the angle of handwritten table images; Based on the four corner points of the table in the corrected image, the corrected image is transformed into a standard table image.

5. The metadata-based handwritten form recognition method according to claim 4, characterized in that: The neural network uses a trained U-Net model to find the straight lines in the handwritten table image, and each straight line is represented by the coordinates of both ends.

6. The metadata-based handwritten form recognition method according to claim 4, characterized in that: The line segment-based distortion correction algorithm is to define a polynomial for correction, which is used to convert points on the surface into corresponding points after correction; The polynomial is defined based on the identified straight line in the table, and the correction angle is calculated through the line segments therein, and the cost function is constructed, and the parameters of the polynomial are obtained through the LM algorithm.

7. The metadata-based handwritten form recognition method according to claim 4, characterized in that: The specific steps of obtaining a standard table image based on the corrected image are as follows: Based on the position characteristics of the four corner points, the coordinates of the four corner points of the table in the corrected image are obtained; Taking the four corner points of the table in the corrected image as source points and the four corner points of the table in the standard table image as target points, the projection center and projection angle are determined, thereby transforming the corrected image into a standard table image.

8. A metadata-based handwritten form recognition system, characterized in that: Including definition module, restoration module, segmentation module and recognition module: The definition module is configured to: define metadata based on a standard image corresponding to an original record table in a production environment, for describing table information in the image, including organizational structure information of the table and information of each cell; The restoration module is configured to: restore the handwritten table image to be recognized into a standard table image using a table restoration method based on a neural network; The segmentation module is configured to: perform cell segmentation on the restored handwritten table image according to the information of each cell in the metadata to obtain a plurality of cell sub-images; The recognition module is configured to: recognize and correct the text in each cell sub-image, and assemble the corrected text into a table file of a set format according to the organizational structure information in the metadata.

9. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; a processor for executing the computer-readable instructions; Wherein, when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is executed.

10. A storage medium, characterized in that: The computer-readable instructions are non-transitorily stored, wherein when the computer-readable instructions are executed by a computer, the method of any one of claims 1 to 7 is performed.

Images