Multi-line text paper form OCR method based on semantic analysis

Abstract

The present invention discloses an OCR method for paper forms with multiple lines of text based on semantic analysis. The method first divides the cells of the paper form image to be digitized, and then recognizes the text content in the form through a neural network. Based on the calculation of semantic fluency, it is judged whether the recognition result is accurate. For the erroneous recognition results that are out of order, the number of rows and columns of the text in the original cell is first extracted by projection method, and then a matrix is ​​constructed, and the erroneous recognition results are filled into the matrix in sequence, and then the matrix is ​​transposed, and then the text is output in sequence to obtain the recognition results in the correct sequence. This method can realize automatic judgment and adjustment of multi-line text recognition in paper forms in the case of disordered row / column recognition. It reduces the workload caused by manual proofreading and improves the intelligence level of OCR.

Description

Technical Field

[0001] The invention belongs to the technical field of pattern recognition, and in particular relates to an OCR method for a multi-line text paper form based on semantic analysis. Background Art

[0002] Paper forms store a lot of important information. With the continuous development of science and technology, the digitization of paper forms in certain fields has become an inevitable trend, such as cadre files, company employee files, etc. Digitizing these form data is beneficial to optimize management. The main method of digitizing paper forms is to use OCR for recognition. OCR stands for optical character recognition, which refers to the process of converting image text input into a computer into editable text. The accuracy of OCR in recognizing paper form information often depends on the complexity of the form frame line. The simpler the form, the higher the accuracy of its content recognition; the more complex the form, the lower the accuracy of its content recognition. At present, the main steps of text recognition in tables include: first, preprocessing the image, that is, grayscale, binarization and other processing; second, extracting each cell in the table according to the coordinates of the intersection of the horizontal and vertical lines of the table; finally, performing text recognition on each cell and outputting the results. However, in the last step, when recognizing multiple lines of text in the table, there will be a problem of disordered recognition of multiple lines of text by row / column, and this problem has not been solved.

[0003] Multi-line text recognition disorder by row / column refers to the problem that when recognizing multi-line text, when the correct recognition order is to recognize text by column, the final output result is the result of recognition by row, resulting in disordered recognition results. Most existing technologies manually compare the recognition results with the table content in the paper document to determine whether the results are disordered, and then manually adjust the disordered parts of the recognition results to the correct order. However, this process takes a lot of time and effort.

[0004] Language models can be used to calculate semantic fluency. By judging whether a sentence conforms to our daily expression habits, we can determine whether the order of the recognition results is correct. Common methods for calculating semantic fluency include N-gram language models and language models based on neural networks. The N-gram model based on machine learning uses the idea of ​​maximum likelihood estimation to calculate the probability of a sentence based on the number of times a word appears in the corpus and the correlation between the word and the previous one or several words. This method is easy to train and has strong interpretability, but it cannot establish connections with words that are far away and has weak generalization ability. The language model based on neural networks sets each word in the sentence in the form of a vector, that is, one vector represents one word; then the word vector is input into the constructed neural network model, and the optimal vector expression form of each word is obtained by the gradient descent method of the loss function according to the relationship between the word and the context. The word vectors obtained by training similar words are closer. This method can avoid the shortcomings of the N-gram language model, has stronger generalization, and more accurate sentence probability calculation results. However, after obtaining the judgment result through the language model, there is no method for automatically adjusting the word order. Summary of the invention

[0005] In view of the problem in the prior art that multiple lines of text in a table are disordered and cannot be automatically adjusted when being recognized by row / column, the present invention proposes an OCR method for paper tables with multiple lines of text based on semantic analysis, which combines semantic coherence calculation and reorders the disordered text through a matrix.

[0006] The OCR method for multi-line text paper forms based on semantic analysis, the specific steps are as follows:

[0007] Step 1: Input the table image to be digitized, extract the cells in the image, use the neural network to recognize the text in each cell in the image through the trained character library, and output the recognition result.

[0008] Step 2: Use the Bengio language model to calculate the semantic fluency of the recognition result output in step 1.

[0009] Step 3: Compare the calculation result of step 2 with the set threshold. If the calculated semantic coherence is greater than the set threshold, the word order of the recognition result is judged to be correct; if the calculated semantic coherence is less than the set threshold, the word order of the recognition result is judged to be incorrect.

[0010] Preferably, the threshold for determining whether the word order of the recognition result is correct is set to 0.1.

[0011] Step 4: If the word order of the recognition result is correct as determined in step 3, the recognition result is directly output; if the word order of the recognition result is wrong as determined in step 3, the process proceeds to step 5.

[0012] Step 5. Use the horizontal projection and vertical projection methods to determine the number of rows and columns of text in the cell extracted in step 1. The horizontal projection method is to add the number of black pixels in each row of the cell. If the sum of several consecutive rows of black pixels is not 0, it constitutes a row of text; if a row of black pixels is 0, it means that there is no text in this row. In this way, the number of rows of text is determined. The vertical projection method is to add the number of black pixels in each column of the cell. If the sum of several consecutive columns of black pixels is not 0, it constitutes a column of text; if a column of black pixels is 0, it means that there is no text in this column. In this way, the number of text columns is determined.

[0013] Step 6: Construct a matrix based on the number of rows and columns of the text obtained in step 5. The number of rows and columns of the matrix is ​​equal to the number of rows and columns of the text in the cell. Then fill the text recognized in step 4 into the rows of the matrix in sequence, and finally output the text in the order of the columns of the matrix to complete the automatic adjustment of the disordered text.

[0014] Beneficial effects of the present invention:

[0015] The present invention has made improvements to the problem of disordered row / column recognition of multiple lines of text in paper forms. By performing semantic analysis on the recognition results and judging the fluency of the sentences, if the recognition results do not conform to the correct logical expression of Chinese, the sentence order is changed to the correct logical order, which can further improve the accuracy of OCR printed text recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Flowchart for text extraction in cells;

[0017] Figure 2 Schematic diagram of cells extracted in the embodiment;

[0018] Figure 3 Schematic diagram of horizontal projection and vertical projection in the embodiment;

[0019] Figure 4 Schematic diagram of the result after matrix order change in the embodiment. DETAILED DESCRIPTION

[0020] The present invention will be further explained below in conjunction with the accompanying drawings;

[0021] This embodiment performs digital processing on the forms in the paper cadre files, and the input image selects the scanned image of the paper form. This type of image is clearer and less prone to noise interference.

[0022] The OCR method for multi-line text paper forms based on semantic analysis, the specific steps are as follows:

[0023] Step 1: Figure 1As shown, the scanned cadre file image is input, and the table part thereof is subjected to image preprocessing, including grayscale and binarization processing.

[0024] Step 2: Extract cells from the table image preprocessed in step 1. First, extract the horizontal and vertical lines of the table through operations such as expansion and corrosion, and then intersect the horizontal and vertical lines to obtain the coordinates of their intersection points; secondly, determine the coordinate information of the four vertices of each cell in the table based on the coordinate points, thereby extracting each cell in the table. The extracted cells are as follows: Figure 2 shown.

[0025] Step 3: Train the Chinese character data set through the CNN+LSTM neural network to obtain a character library; then use the trained character library to recognize the content in each cell, and output the recognition result as "political appearance" or "political governance appearance".

[0026] Step 4: Select the classic Bengio language model to calculate the semantic fluency of the recognition result output in step 3. The Bengio language model includes an input layer, a fully connected layer with a tanh activation function, and an output layer with a softmax activation function.

[0027] ① Let the word ω in the recognition result t Only with the first n-1 wordsω t-1 ,ω t-2 …ω t-n+1 ω t Probability

[0028]

[0029] ②Write the word ω t The word vector C(ω) represented by the first n-1 words t-n+1 )…C(ω t-1 ) is input into the input layer of the neural network to obtain the unnormalized probability y:

[0030] y=W·x+U·tanh(d+H·x)+b

[0031] x=(C(ω t-1 ),C(ω t-2 ),…,C(ω t-n+1 ))

[0032] Among them, x is the input, which represents the selected word ω in the recognition result tThe word vectors represented by the first n-1 words of . W is the input layer weight matrix, which can be set to 0. d and H are the bias vector and weight matrix of the hidden layer respectively, and tanh is the activation function of the hidden layer. U is the weight matrix from the hidden layer to the output layer, and b is the output layer bias vector.

[0033] ③Loss function L:

[0034] θ=(H、U、W、d、b)

[0035]

[0036] Among them, R(θ) prevents overfitting, and θ is the weight parameter and bias in y in step ②.

[0037] ④Use the gradient descent method and the loss function given in step ③ to continuously update the parameters contained in θ according to the following formula.

[0038]

[0039] Among them, η is the number of iterations, also known as the learning rate.

[0040] ⑤Finally, the first n-1 words after the output layer are ω t-n+1 …ω t-1 The next word is ω t The normalized probability of is:

[0041]

[0042] Among them, y ωi The next word before normalization is ω i The probability of is calculated by y in step ②.

[0043] Step 5. Use the probability calculated in the principle of step 4 to represent the fluency of the recognition result. The greater the probability, the more the sentence conforms to the correct word order. In the range of [0,1], select a threshold, compare the fluency of the recognition result with the threshold, and judge whether the word order of the recognition result is correct. In this embodiment, the threshold is set to 0.1. If the calculated semantic fluency is greater than the set threshold, the word order of the recognition result is judged to be correct; if the calculated semantic fluency is less than the set threshold, the word order of the recognition result is judged to be incorrect.

[0044] Step 6: After step 5, it is determined that the word order of the recognition result "political appearance" is correct, and the recognition result is directly output; while the word order of the recognition result "political appearance" is wrong, and step 7 is entered.

[0045] Step 7: Use horizontal projection and vertical projection methods to Figure 2Judge the number of rows and columns of the text within the cell shown. The horizontal projection method adds up the number of black pixel points in each row of the cell. If the sum of the black pixel points in several consecutive rows is not zero, it constitutes a line of text; if a row of black pixel points is zero, it indicates that there is no text in this row. Based on this, judge the number of rows of text. The vertical projection method adds up the number of black pixel points in each column of the cell. If the sum of the black pixel points in several consecutive columns is not zero, it constitutes a column of text; if a column of black pixel points is zero, it indicates that there is no text in this column. Based on this, judge the number of columns of text. The projection result is as Figure 3 shown.

[0046] Step 8: According to the projection result in Step 7, obtain that the number of rows and columns of the text within the cell are 2 rows and 2 columns respectively, and construct a matrix with a size of 2x2. Then fill in the text "政面治貌" recognized in Step 4 into the rows of the matrix in sequence, and finally output the text in the order of the columns of the matrix to complete the automatic adjustment of the scrambled text, as Figure 4 shown.

Claims

1. A multi-line text paper form OCR method based on semantic analysis, characterized by: The specific steps include: Step 1: Input the table image to be digitized, extract the cells in the image, use the neural network to recognize the text in each cell in the image through the trained character library, and output the recognition result; The specific steps include: s1.1, grayscale and binarize the digitized table image; s1.

2. Extract the lines in the preprocessed table image through dilation and erosion operations to obtain the intersection coordinates of the lines, and then determine the coordinates of the four vertices of each cell in the table image according to the intersection coordinates to complete the extraction of the cell; s1.

3. Train the Chinese character data set through a neural network to obtain a character library; then recognize the content in each cell based on the character library and output the recognition result; Step 2: Use the Bengio language model to calculate the semantic fluency of the recognition result output in step 1; assuming that the word ω in the recognition result t Only with the first n-1 wordsω t-1 ,ω t-2 …ω t-n+1 Association; the first n-1 words in the recognition result are calculated by the Bengio language model as ω t-1 ,ω t-2 …ω t-n+1 The next word is ω t The probability of , as the semantic fluency; Step 3: Compare the calculation result of step 2 with the set threshold value. If the calculated semantic coherence is greater than the set threshold value, the word order of the recognition result is judged to be correct; if the calculated semantic coherence is less than the set threshold value, the word order of the recognition result is judged to be incorrect. Step 4: If the word order of the recognition result is correct in step 3, the recognition result is directly output; if the word order of the recognition result is wrong in step 3, the process proceeds to step 5; Step 5, using the horizontal projection and vertical projection methods respectively, to determine the number of rows and columns of the text in the cell extracted in step 1; the horizontal projection method is to add the number of black pixels in each row of the cell, and if the sum of several consecutive rows of black pixels is not 0, a row of text is formed; if a row of black pixels is 0, it indicates that there is no text in this row; accordingly, the number of rows of text is determined; the vertical projection method is to add the number of black pixels in each column of the cell, and if the sum of several consecutive columns of black pixels is not 0, a column of text is formed; if a column of black pixels is 0, it indicates that there is no text in this column; accordingly, the number of columns of text is determined; Step 6. Construct a matrix based on the number of text rows and columns obtained in step 5, where the number of matrix rows and columns are equal to the number of rows and columns of text in the cell respectively; then fill the text recognized in step 4 into the rows of the matrix in sequence, and finally output the text in the order of the columns of the matrix to complete the automatic adjustment of the disordered text.

2. The multi-line text paper form OCR method based on semantic analysis as claimed in claim 1, characterized in that: In step 3, the threshold for judging whether the word order of the recognition result is correct is set to 0.1.

Images