A method, system and storage medium for structuring OCR recognition results

Abstract

The present invention discloses a method, system and storage medium for structuring OCR recognition results. The method first uses a target detection model to frame key-value pairs, performs text detection and recognition, then uses a Bert model to perform binary classification of keys and values, sorts the value texts by position based on relative position differences, uses NLP to semantically sort the value texts, and finally performs weighted summation on the position sorting and semantic sorting results and sorts them from small to large as the value text recognition result. The present invention identifies and cuts out key-value pairs through a target detection model, solves the problem of key-value matching misalignment, and simultaneously combines the position and semantic comprehensive sorting of values ​​to improve the accuracy of value merging, improve the generalization ability of the model, and enhance the accuracy of structuring OCR recognition results.

Description

Technical Field

[0001] The present invention relates to an OCR post-processing technology, and in particular to a method for structuring OCR recognition results. Background Art

[0002] The advent of OCR technology has improved the efficiency and accuracy of text entry within images. Most ID images are in the form of key-value pairs, requiring customized key:value pairs for each document. The current main method involves matching the key content against a library of different ID keys. The value is then assigned to the content near the key. If the value exceeds a single box, the text is merged based on the position of the text box, from top to bottom and from left to right, using a threshold. This method is simple, but has several disadvantages: Setting a threshold for key:value matching can easily lead to mismatched key1:value2 and key2:value1 pairs, resulting in errors in both fields. Setting a threshold for value merging can also lead to limited generalization performance. Different IDs require customized key dictionaries, which is cumbersome. If a character within a key is misrecognized, a field can be easily lost. Consequently, current OCR recognition results are poorly structured and prone to errors. Summary of the Invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a method, system and storage medium for structuring OCR recognition results with high accuracy and strong generalization, so as to avoid errors in key-value pair matching and improve the accuracy of value merging.

[0004] Technical solution: The OCR recognition result structuring method of the present invention comprises the following steps:

[0005] (1) Input the image to be identified, use the target detection model to frame the key-value pair key:value, and crop the sub-image to be identified along the frame;

[0006] (2) Perform text detection and recognition on each sub-image to obtain the text corresponding to each text box;

[0007] (3) Classify the text into key and value;

[0008] (4) Perform position sorting and semantic sorting on the value text. The semantic sorting is to arrange the text into sentences and select the sentence with the smallest perplexity as the semantic sorting result.

[0009] (5) Perform weighted summation on the position sorting results and the semantic sorting results, and sort them in ascending order as the value text recognition result;

[0010] (6) Output structured recognition results in key:value format.

[0011] Furthermore, the method for sorting the position of the value text in step (4) is as follows: the value text boxes are arranged in a row. If the height difference between two adjacent text boxes and the height ratio of the previous text box is less than a threshold, and the previous text box is located on the right side of the two adjacent text boxes, then the two text boxes are swapped; otherwise, they are not swapped.

[0012] Furthermore, the method for performing binary classification in step (2) is as follows: training a Bert binary classification model, inputting the text position and text content into the Bert model, and outputting a result vector and a score vector, wherein the result vector is composed of 1 and 0; in the result vector, if only one value is 1, the text corresponding to the position of 1 is the key, and the rest are values; if there are multiple values ​​1, then among these text box positions with values ​​1, the text box closest to the origin is selected as the key, and the rest are values; if the result vector is all 0, the text corresponding to the largest score in the score vector is selected as the key, and the rest are values.

[0013] The OCR recognition result structuring system of the present invention includes:

[0014] The key-value pair matching module is used to identify the key-value pair positions in the image through the neural network model and crop the sub-image to be identified;

[0015] Text detection and recognition module, used to perform text detection and text recognition on the image of the word to be recognized;

[0016] Key-value classification module, used to perform binary classification of keys and values;

[0017] The position sorting module is used to sort the value text. If the height difference between two adjacent text boxes and the height of the previous text box is less than a threshold, and the previous text box is located to the right of the two adjacent text boxes, the two text boxes are swapped to obtain the position sorting result.

[0018] The semantic sorting module is used to calculate the perplexity of the sentences formed after all the value texts are arranged, and the sentence with the smallest perplexity is used as the semantic sorting result;

[0019] The comprehensive sorting module is used to perform weighted summation of the position sorting results and the semantic sorting results, and sort the summation results from small to large to obtain the value text recognition results.

[0020] The computer-readable storage medium of the present invention stores a computer program thereon, and when the computer program is executed by a processor, the steps of the method for structuring OCR recognition results of the present invention are implemented.

[0021] Beneficial Effects: Compared with the existing technology, the advantages of this invention are as follows: (1) By selecting key-value pairs through a neural network model, the key-value pair matching problem is solved, and the generalization performance is strong; (2) Based on the text box position and text box content, the Bert model is used to perform binary classification of key and value on the text box content, which has stronger generalization performance than the traditional vocabulary matching. (3) The value is merged using text position and text content information, which improves the accuracy of the structured OCR recognition results and enhances the robustness of the model; (4) The value text position is adjusted based on the relative position difference, which has stronger generalization performance than the absolute position. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Flowchart of the method for structuring OCR recognition results of the present invention;

[0023] Figure 2 This is a legend for the marking of a business license in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0025] like Figure 1 As shown, described, including the following steps:

[0026] (1) Key-value pair matching

[0027] To train a YOLO model, you can draw the key-value pairs of different ID cards in a large box. Before training, you need to mark several ID photos, such as Figure 2 Shown is a legend of the business license marking.

[0028] Input the image into the trained YOLO model, and get the positions of all the large frames of the image as: B = (b1, b2, ..., b N ), where b i =(x i,min ,y i,min ,x i,max ,y i,max ), that is, a large box is composed of the coordinates of the upper left corner and the lower right corner. This step solves the key-value pair matching problem. That is, the text within a box is a key-value pair, and key-value pair matching will not be misplaced. In actual applications, other object detection models can also be used.

[0029] According to the above text box position and the original picture, the sub-pictures after cropping of each large box are obtained, which are recorded as:

[0030] P=(p1,p2,......,p N), then we only need to perform OCR on each image to obtain its text and structure it. Since the subsequent processing of each sub-image is the same, the following describes the processing of only one sub-image.

[0031] (2) Text detection and text recognition

[0032] For image p, use text detection algorithms such as DBnet, PSEnet, and Craft to obtain the position information of each text detection box, which is recorded as S = (s1, s2, ..., s n ), where s j =(x 1,j ,y 1,j ,x 2,j ,y 2,j ,x 3,j ,y 3,j ,x 4,j ,y 4,j ) are the coordinate values ​​of the upper left, upper right, lower right, and lower left points of a text detection box.

[0033] According to the coordinates identified above, combined with the image p, the image corresponding to each text detection frame is cropped, binarized, and sent to the text recognition algorithm CTPN to realize the text recognition process. Finally, the text corresponding to each text detection frame is W=(w1,w2,......,w n ).

[0034] (3) Key-value classification

[0035] The next step is to structure the text. This is done in two steps: first, distinguishing the keys and values ​​in the text; and second, merging the values.

[0036] Compared to traditional vocabulary matching methods, we chose to use NLP technology to solve this problem. Based on the BERT pre-trained model, we fine-tuned a classification algorithm to classify the input text into two categories: 1 for keys and 0 for non-keys (i.e., values).

[0037] To fine-tune the BERT model, the data must first be labeled to prepare a training set. Specifically, positive samples (i.e., key samples) can be obtained from previous data, while negative samples can be randomly cropped from previous data. Unlike general text classification, OCR also uses positional information as a feature input into the model. Therefore, during the labeling process, positional information must also be appended to the text to generate training data. This data is then used to fine-tune a BERT classification model.

[0038] Set (W,S)=((w1,s1),(w2,s2),......,(wn ,s n )) Input the above fine-tuned Bert classification model and get the result vector R=(r1,......,r n ), where r i is 0 or 1, and outputs its score vector P=(p1,......,p n ).

[0039] For the classification discussion of the above output results, if the result vector has only one 1, the text corresponding to this position is the key, and the rest are values; if the result vector is all 0, the text corresponding to the position with the largest score in the score vector is found as the key, and the rest are values; for the remaining cases, all text box positions corresponding to 1 are found, and the text corresponding to the position of the text box closest to the upper left corner of the image (that is, the origin of the image) is selected as the key, and the rest are values.

[0040] (4)Value text merging

[0041] The value text needs to be merged. For an image, text can be merged simply based on position. However, the position identified by text detection algorithms is often not precise, and there is a certain threshold fluctuation. Even within the same line of text, the heights of different detection text boxes often vary. Traditional methods generally use a threshold to sort, but this threshold is difficult to control. Therefore, we use two methods to sort: position and text content.

[0042] (41) Position sorting

[0043] For text merging, the general direction of sorting by position is: first from top to bottom, then from left to right, and then fine-tuning based on the x and y values. The specific steps are as follows:

[0044] (411) Sort the text boxes directly from top to bottom and then from left to right based on the position of the upper left corner of each text box. The sorting result of the text detection box is: Subscript i j The position corresponding to the original text detection box.

[0045] (412) Fine-tune the above sorting results. The traditional method is that when the position height pixel difference of two text detection frames is within a certain threshold range, if the x value of the latter text frame is less than the x value of the previous text frame, then they are swapped. Here, in order to improve its generalization performance, we choose to sort according to the x value of the text frame when the ratio of the position height pixel difference of two adjacent text detection frames to the height of the previous text frame is within 0.5, that is, when two texts meet the following inequality, and Swap positions. :

[0046]

[0047] The final position sorting result is From this, the subscript position vector of the original order can be calculated as P S =(I1,......,I n ), where I j S in S2 j The corresponding position.

[0048] (42) Semantic Sorting

[0049] For semantic sorting, the N-Gram language model is mainly used for sorting. The specific steps are as follows:

[0050] (421) Prepare an N-Gram model, either by pre-training it based on your own data or by directly downloading an open source N-Gram model.

[0051] (422) All the obtained texts are fully arranged to form sentences, and the perplexity of each sentence is calculated. Finally, the sentence with the smallest perplexity is selected for sorting, which is the sorting obtained by the language model. The sorting corresponding to the final text detection box is recorded as

[0052] From this, the subscript position vector of the original order can be calculated as P C =(J1,......,J n ), where J i S in S3 i The corresponding position.

[0053] (43) Comprehensive ranking

[0054] For the above two results, a comprehensive ranking is performed. The specific calculation method is as follows:

[0055] P f =α*P s +(1-α)*P C

[0056] Where α represents the proportion of text box position, which is a number between 0 and 1. If it is 0, the final sorting is completely based on semantics, and if it is 1, the final sorting is completely based on text box position. The final sorting score is recorded as Finally, the scores are sorted from small to large, and the sorting of the corresponding subscripts is the final text sorting.

[0057] (5) Result output

[0058] Output structured OCR recognition results in key:value format.

Claims

1. A method for structuring OCR recognition results, characterized in that: The steps include: (1) Input the image to be identified, use the target detection model to frame the key-value pair key:value, and crop the sub-image to be identified along the frame; (2) Perform text detection and recognition on each sub-image to obtain the text corresponding to each text box and the initial order of each text box; (3) Based on the position and content of the text box, the Bert model is used to classify the text into key and value; (4) Sort the value text by position and semantics; The position sorting is: sort the text boxes from top to bottom and then from left to right according to the position of the upper left corner of the text box to obtain the first sorting result , for the Fine-tune to get the second sorting result , identification Each text box in The corresponding sorting position in the text box is formed into a second position vector according to the initial order of the text box. ; The fine-tuning is specifically: If the ratio of the height difference between two adjacent text boxes and the height of the previous text box between the two adjacent text boxes is less than a threshold, and the x-coordinate of the upper left corner of the previous text box is greater than the x-coordinate of the upper left corner of the next text box between the two adjacent text boxes, then the two text boxes are swapped; otherwise, the two text boxes are not swapped, and the height difference is the difference between the y-coordinate of the upper left corner of the next text box minus the y-coordinate of the upper left corner of the previous text box; the height of the previous text box is the difference between the y-coordinate of the lower left corner of the text box minus the y-coordinate of the upper left corner of the text box; The semantic sorting is: sorting all the texts into sentences, selecting the sentences with the smallest perplexity, and obtaining the third sorting result corresponding to the text box. , identification Each text box in The corresponding sorting position in the text box is formed into the first position vector according to the initial order of the text box. ; (5) and Perform weighted summation and sort the elements in the vector obtained by weighted summation from small to large to obtain the value text recognition result; (6) Output structured recognition results in key:value format.

2. The OCR recognition result structuring method according to claim 1, characterized in that: The method for performing binary classification in step (3) is as follows: training a Bert binary classification model, inputting the text box position and text box content into the Bert model, and outputting a result vector and a score vector, wherein the result vector is composed of 1 and 0; in the result vector, if only one value is 1, the text corresponding to the position of 1 is the key, and the rest are values; if there are multiple values ​​1, then among these text box positions with values ​​1, the text corresponding to the text box closest to the origin is selected as the key, and the rest are values; If the result vector is all 0, the text with the largest score in the score vector is selected as the key, and the rest are the values.

3. The OCR recognition result structuring method according to claim 1, characterized in that: In step (4), the sentence perplexity is calculated using the N-Gram model.

4. An OCR recognition result structuring system based on the method according to any one of claims 1 to 3, characterized in that: include: The key-value pair matching module is used to identify the key:value pairs in the image through a neural network model and crop the sub-image to be identified. A text detection and recognition module, which is used to perform text detection and text recognition on the sub-image to be identified; A key-value classification module is used to classify the text identified by the text detection and recognition module into key and value. The key-value classification module is provided with a Bert classification unit. The Bert classification unit is used to input the text box position and text box content into the Bert classification unit to achieve the key and value classification. A position sorting module is used to perform position sorting on the value text to obtain a position sorting result; The semantic sorting module is used to calculate the perplexity of the sentences formed after the value text is fully arranged, and select the sentence with the smallest perplexity to obtain the semantic sorting result; Comprehensive sorting module, used to and Perform weighted summation and sort the elements in the vector obtained by weighted summation from small to large to obtain the value text recognition result.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the OCR recognition result structuring method according to any one of claims 1 to 3 are implemented.

Images