A table recognition method and system

Abstract

The application discloses a table recognition method and system, and relates to the technical field of table recognition. Pictures obtained by shooting a table to be recognized are acquired, a text detection model is used to recognize the pictures to determine text regions in the pictures, a text recognition model is used to recognize the text regions to obtain text recognition results, the text recognition results include vertex coordinates and text contents of each element in the text regions, and the text contents of the table to be recognized can be quickly determined by recognizing the text regions first and then recognizing the texts. Each element in the text regions is positioned according to the text recognition results to determine the positions of each element in the text regions, so that the positions and the text contents of each element in the table to be recognized can be obtained, and the table contents can be accurately and efficiently recognized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of table recognition, in particular to a high-efficiency and high-accuracy table recognition method and system. BACKGROUND

[0002] Paper forms are widely used in data collection processes, and converting the data of paper forms into electronic data is an inevitable trend of informatization. However, the traditional conversion method is a manual data entry method, which requires a special data entry operator to manually enter the data of the paper form into the computer one by one for storage. This method consumes a large amount of manpower and time, and has many disadvantages, such as low efficiency, high error rate, and high cost. With the proliferation of mobile devices equipped with cameras, more and more users use these mobile devices to take pictures of paper forms and upload them, making the demand for extracting information from these pictures containing paper forms more urgent. Usually, a key part of extracting information from pictures is to digitize the data in the pictures. However, due to the diversity of table structures and the variation of graphical elements used to visually separate table components, it is a very challenging problem to extract data from these pictures.

[0003] Therefore, it is an urgent problem to efficiently find a table from a picture and effectively extract the data and structural information in the table to complete table recognition. SUMMARY

[0004] The purpose of the present application is to provide a table recognition method and system that can accurately and efficiently recognize the content of a table.

[0005] To achieve the above purpose, the present application provides the following solutions.

[0006] A table recognition method, the recognition method comprising:

[0007] obtaining a picture obtained by taking a picture of a table to be recognized;

[0008] recognizing the picture using a text detection model to determine a text region in the picture;

[0009] performing text recognition on the text region using a text recognition model to obtain a text recognition result; the text recognition result comprising vertex coordinates and text content of each element in the text region;

[0010] positioning each element in the text region according to the text recognition result to determine the position of each element in the text region, and obtaining the position and text content of each element in the table to be recognized; the position comprising the number of rows and columns.

[0011] In some embodiments, the text detection model is a DB model.

[0012] In some embodiments, before the text recognition on the text region by using the text recognition model, the recognition method further comprises:

[0013] determining whether the text in the text region is in a preset direction by using a direction classifier model;

[0014] if not, performing rotation correction on the text region so that the text in the text region is in the preset direction, obtaining a corrected text region, and taking the corrected text region as a new text region.

[0015] In some embodiments, the text recognition model is a CRNN model.

[0016] In some embodiments, before the position positioning of each element in the text region according to the text recognition result, the recognition method further comprises: correcting the text content of an element in the text recognition result whose text content is a number or a symbol by using a number recognition model, obtaining a corrected recognition result, and taking the corrected recognition result as a new text recognition result.

[0017] In some embodiments, before the position positioning of each element in the text region according to the text recognition result, the recognition method further comprises:

[0018] performing non-table element filtering according to the text recognition result, removing non-table elements in the text region, obtaining a filtered region, and taking the filtered region as a new text region; removing the vertex coordinates and the text content of the non-table elements in the text recognition result, obtaining a filtered recognition result, and taking the filtered recognition result as a new text recognition result.

[0019] In some embodiments, the position positioning of each element in the text region according to the text recognition result specifically comprises:

[0020] fitting the rows and columns in the text region according to the text recognition result to perform row positioning and column positioning on each element in the text region, and determining the position of each element in the text region.

[0021] In some embodiments, the fitting of the rows and columns in the text region according to the text recognition result specifically comprises:

[0022] for each element in the text recognition result, calculating the upper edge midpoint coordinates, the lower edge midpoint coordinates and the center point coordinates of the element according to the vertex coordinates of the element.

[0023] For any two elements in the text recognition result, it is determined whether a distance between a left bottom corner vertex coordinate of a first element and a left top corner vertex coordinate of a second element is less than or equal to a preset value, whether a distance between a right bottom corner vertex coordinate of the first element and a right top corner vertex coordinate of the second element is less than or equal to the preset value, or whether a distance between a lower edge midpoint coordinate of the first element and an upper edge midpoint coordinate of the second element is less than or equal to the preset value; if yes, the first element and the second element belong to a same column, and all elements in the text recognition result are grouped by column to obtain a plurality of column combinations;

[0024] For each column combination, a straight line fitting is performed based on coordinates of each element in the column combination to obtain a fitting straight line corresponding to the column combination;

[0025] The plurality of column combinations are arranged in descending order according to a number of elements contained therein, and column merging is performed based on distances of the fitting straight lines corresponding to each column combination in an order after the descending arrangement to obtain columns in the text region.

[0026] Row scanning is performed based on center point coordinates of all the elements to obtain rows in the text region.

[0027] In some embodiments, the straight line fitting based on the coordinates of each element in the column combination to obtain the fitting straight line corresponding to the column combination specifically includes:

[0028] A first straight line is obtained by performing straight line fitting on left top corner vertex coordinates of each element in the column combination, a second straight line is obtained by performing straight line fitting on upper edge midpoint coordinates of each element in the column combination, and a third straight line is obtained by performing straight line fitting on right top corner vertex coordinates of each element in the column combination.

[0029] A first average distance of the center point coordinates of each element in the column combination to the first straight line is calculated, a second average distance of the center point coordinates of each element in the column combination to the second straight line is calculated, and a third average distance of the center point coordinates of each element in the column combination to the third straight line is calculated.

[0030] A minimum value of the first average distance, the second average distance, and the third average distance is determined, and a straight line corresponding to the minimum value is taken as the fitting straight line corresponding to the column combination.

[0031] A table recognition system, the recognition system comprising:

[0032] A picture acquisition module configured to acquire a picture obtained by photographing a table to be recognized;

[0033] The text area recognition module is configured to recognize the picture by using a character detection model, and determine a text area in the picture.

[0034] The text content recognition module is configured to recognize the text area by using a character recognition model, and obtain a text recognition result.

[0035] The text position positioning module is configured to position each element in the text area according to the text recognition result, determine the position of each element in the text area, and obtain the position and text content of each element in the table to be recognized.

[0036] According to the embodiments of the present application, the following technical effects are provided.

[0037] The present application provides a table recognition method and system, which obtains a picture obtained by photographing a table to be recognized, recognizes the picture by using a character detection model, determines a text area in the picture, recognizes the text area by using a character recognition model, and obtains a text recognition result. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0039] Figure 1 The method flowchart of the recognition method provided by the embodiment 1 of the present application;

[0040] Figure 2 The overall architecture diagram of the test report sheet recognition method provided by the embodiment 1 of the present application;

[0041] Figure 3 The recognition result schematic diagram of the test report sheet displayed in the form of a table provided by the embodiment 1 of the present application;

[0042] Figure 4A schematic diagram of column title guessing provided by Embodiment 1 of the present application;

[0043] Figure 5 A schematic diagram of item name recognition provided by Embodiment 1 of the present application;

[0044] Figure 6 A schematic diagram of dictionary library correction provided by Embodiment 1 of the present application;

[0045] Figure 7 A schematic diagram of item name recognition model provided by Embodiment 1 of the present application;

[0046] Figure 8 A system block diagram of the recognition system provided by Embodiment 2 of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0048] The object of the present application is to provide a table recognition method and system, which can accurately and efficiently recognize the content of a table.

[0049] In order to make the above object, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] Embodiment 1:

[0051] Specifically, the table recognition refers to sending the table in the form of a picture to a computer, automatically extracting various data information in the picture, and applying the table recognition technology to greatly save the time for processing the content of the table and reduce the errors caused by human factors. In addition, the recognized data information can be restacked into a new table to complete the reconstruction of the table.

[0052] In recent years, table recognition has gradually evolved into multiple sub-branch research fields, including: specialized dataset construction, table detection, table structure recognition, table detection and structure recognition, etc. Domestic and foreign experts and scholars have conducted a large number of studies on table recognition problems, and introduced deep learning models such as graph convolutional network (GCN), fast R-CNN (Fast R-CNN), fully convolutional neural network (FCN), conditional generative adversarial network (CGAN) for automatic table recognition. However, due to the existence of a large number of pictures with no table lines, missing lines, rotation, noise (seals and watermarks), etc., the above deep learning models and ordinary OCR (Optical Character Recognition) algorithms cannot accurately and effectively recognize the picture content and complete the table recognition process.

[0053] The purpose of table recognition is to accurately, quickly and truly extract table data information in the picture and minimize the amount of manual work. However, the existing methods cannot achieve the above purpose. Therefore, the present embodiment provides a table recognition method, which is an experience-guided heuristic and two-rule-constrained table recognition method, which can efficiently and accurately complete the table recognition process. As shown in Figure 1 The recognition method comprises:

[0054] S1: obtaining a picture obtained by photographing a table to be recognized;

[0055] The present embodiment can convert a paper table to be recognized into an electronic picture containing the content of the table to be recognized by various methods such as mobile phone shooting and scanner scanning.

[0056] S2: recognizing the picture using a text detection model to determine the text area in the picture;

[0057] The text detection model of the present embodiment can be a DB (Differentiable Binarization) model, which is a segmentation-based text detection algorithm and a general text detection model. Preferably, the DB model of the present embodiment uses the open source version of paddleOCR, which is chinese_text_detection_db_server 1.0.2. The picture is input into the text detection model DB for recognition, and the pixel area occupied by the text in the picture can be obtained, which is the text area in the picture.

[0058] S3: performing character recognition on the text region by using a character recognition model to obtain a text recognition result; the text recognition result includes the vertex coordinates and text content of each element in the text region;

[0059] The embodiment can also determine whether the text region needs to be corrected in rotation by using a direction classifier model. The direction classifier model of the embodiment uses the function provided in the DB model. Specifically, before S3, the recognition method of the embodiment can further include: determining whether the characters in the text region are in a preset direction by using a direction classifier model, the preset direction can be upward, i.e., determining whether the characters in the text region are upward; if yes, the text region does not need to be corrected in rotation, and S3 is directly performed; if no, the text region is corrected in rotation so that the characters in the text region are in the preset direction, i.e., upward, to obtain a corrected text region, and the corrected text region is taken as a new text region to perform S3.

[0060] The character recognition model of the embodiment can be a CRNN model, which is used to recognize the text in the text region obtained by the character detection model. The text can include characters, numbers and symbols. Preferably, the CRNN model of the embodiment uses the open source version chinese_ocr_db_crnn_server 1.1.1 of paddleOCR. The text recognition result can be obtained by inputting the text region into the character recognition model CRNN. In addition to the vertex coordinates and text content of each element in the text region, the text recognition result can also include the recognition confidence of the text content.

[0061] S4: positioning the position of each element in the text region according to the text recognition result to determine the position of each element in the text region, and obtaining the position and text content of each element in the to-be-recognized table; the position includes the number of rows and the number of columns.

[0062] The embodiment can also set a number recognition model, which can use CRNN as the recognition algorithm, and is used to correct numbers and symbols, i.e., the embodiment can send the numbers and symbols in the text recognition result into the number recognition model for correction, such as text with only a symbol “.” or at most one letter, because “0” can be misrecognized as “o” and “2” can be misrecognized as “z”. Therefore, before S4, the recognition method of the embodiment can further include: correcting the text content of the elements with text content of numbers or symbols in the text recognition result by using a number recognition model to obtain a corrected recognition result, and taking the corrected recognition result as a new text recognition result.

[0063] Since the text in the picture can not only include the text in the table to be recognized, but also can include other text, preferably, before S4, the recognition method of the embodiment further includes: performing non-table element filtering according to the text recognition result, removing the non-table element in the text area, obtaining a filtered area, taking the filtered area as a new text area, removing the vertex coordinates and text content of the non-table element in the text recognition result to obtain a filtered recognition result, taking the filtered recognition result as a new text recognition result, and performing S4.

[0064] The filtering process of the non-table element can include: adaptive determination of the margin of the text area according to the text recognition result, which can be an empirical adaptive determination by human. The text area is adaptively dilated by a morphological method, and does not exceed the margin. The key fields of the table are used to locate the width of the table and the last row of the table, the elements on both sides of the width are deleted, the last row is fitted as a straight line, the elements under the straight line are deleted, and a preliminary filtered area is obtained. The preliminary filtered area is dilated, where the dilation is twice the width of the last dilation, so that it exceeds the preliminary filtered area on the left and right and is connected between different rows, and then the elements not in the preliminary filtered area are removed, and the remaining elements are all elements in the table.

[0065] It should be noted that the embodiment can first perform the correction process of the numbers and symbols, and then perform the filtering process of the non-table element; or the filtering process of the non-table element can be performed first, and then the correction process of the numbers and symbols can be performed.

[0066] In S4, the position of each element in the text area is located according to the text recognition result, and the position of each element in the text area can include: fitting the rows and columns in the text area according to the text recognition result to locate the rows and columns of each element in the text area, and determine the position of each element in the text area.

[0067] The embodiment completes positioning of each element of the table by performing adaptive table column search and merging and row scanning. The column search and merging is to find the same column by dynamically calculating the row height based on the text recognition result, using the alignment attribute (left alignment, center alignment or right alignment) between the same columns, and then merging the small column into the large column, i.e., performing descending order sorting according to the element number of each column, dynamically fitting the alignment line of the largest column, and then merging the element of other columns on the line, and constantly switching to the next column until all columns are merged. The row scanning is to locate the first row by using the key field of the table to be recognized, and to fit a multi-segment function to adapt to the curved line and other non-straight lines, to scan down along the last line by using the little change of the trend between the upper and lower lines and between the lines, to fit a new multi-segment function again, and to appropriately smooth when the trend changes greatly, and finally to complete the row scanning of the entire table. Specifically, fitting the rows and columns in the text region according to the text recognition result can include:

[0068] (1) For each element in the text recognition result, the upper edge midpoint coordinate, the lower edge midpoint coordinate and the center point coordinate of the element are calculated according to the vertex coordinates of the element, and the vertex coordinates of the element include the left upper corner vertex coordinate, the right upper corner vertex coordinate, the left lower corner vertex coordinate and the right lower corner vertex coordinate.

[0069] (2) For any two elements in the text recognition result, it is judged whether the distance between the left lower corner vertex coordinate of the first element and the left upper corner vertex coordinate of the second element is less than or equal to a preset value, whether the distance between the right lower corner vertex coordinate of the first element and the right upper corner vertex coordinate of the second element is less than or equal to the preset value, or whether the distance between the lower edge midpoint coordinate of the first element and the upper edge midpoint coordinate of the second element is less than or equal to the preset value; if yes, the first element and the second element belong to the same column, and all elements in the text recognition result are grouped by column to obtain a plurality of column combinations.

[0070] In the embodiment, the element set containing the most elements in the title of the table to be recognized is first found, and the average row height of each element in the element set is taken as the standard height, which is the preset value of the embodiment.

[0071] (3) For each column combination, a straight line is fitted based on the coordinates of each element in the column combination to obtain a fitting straight line corresponding to the column combination.

[0072] The top-left vertex coordinates of each element in the column combination are linearly fitted to obtain a first straight line, the upper edge midpoint coordinates of each element in the column combination are linearly fitted to obtain a second straight line, and the top-right vertex coordinates of each element in the column combination are linearly fitted to obtain a third straight line. The first average distance of the center point coordinates of each element in the column combination to the first straight line is calculated, that is, the distance of the center point coordinates of each element to the first straight line is calculated and averaged to obtain the first average distance, the second average distance of the center point coordinates of each element in the column combination to the second straight line is calculated, and the third average distance of the center point coordinates of each element in the column combination to the third straight line is calculated. The minimum value of the first average distance, the second average distance and the third average distance is determined, and the straight line corresponding to the minimum value is taken as the fitting straight line corresponding to the column combination.

[0073] (4) The plurality of column combinations are arranged in descending order according to the number of elements contained, and the column combinations are merged according to the distance of the fitting straight line corresponding to each column combination in the descending order to obtain the columns in the text region.

[0074] The first column combination with the first ranking is first selected, and it is determined whether the distance between the fitting straight line of the other column combinations and the fitting straight line of the first column combination is less than or equal to a preset value. If yes, the column combination less than or equal to the preset value is merged with the first column combination, and the fitting straight line corresponding to the first column combination is taken as the fitting straight line corresponding to the merged column combination. Then, the first column combination with the first ranking in the other column combinations that have not been merged is selected, and the above process is repeated until all column combinations are merged. The merged column combinations are the columns of the table to be recognized, and the elements in each merged column combination belong to the column.

[0075] (5) The rows in the text region are obtained by scanning the center point coordinates of all elements.

[0076] The title of the table to be recognized is selected, and the center point coordinates of all elements contained in the title are taken as input. A fitting straight line R1 is obtained by using a multi-segment function. The next element that is not divided into a row and is closest to the fitting straight line R1 of the first row is selected. The fitting straight line R1 is moved down until it passes through the center point of the element, and a multi-segment function R2 is obtained. Then, a fitting straight line R2_new is obtained based on the center point coordinates of all elements passing through R2. The fitting straight line R2_new and the multi-segment function R2 are combined to obtain the fitting straight line of the second row of the table to be recognized. Specifically, the combination process can include: for each corresponding vertex of the fitting straight line R2_new and the multi-segment function R2, determining whether the straight line distance between the corresponding vertices is less than 0.2 times a preset value. If yes, the vertex of the fitting straight line R2_new is selected as a new vertex. If no, the vertex of the multi-segment function R2 is selected as a new vertex. If there is a separate vertex in addition to the corresponding vertices, the separate vertex is directly selected as a new vertex. All new vertices are connected to complete the combination process of the fitting straight line R2_new and the multi-segment function R2. The above steps are repeatedly performed until all elements are divided into rows, and the row scanning process is completed to obtain all rows of the table to be recognized.

[0077] Here, the above row and column positioning method is further introduced:

[0078] (1) Let A be a text recognition result, each element in A includes [x1, y1, x2, y2, x3, y3, x4, y4, text], (x1, y1) is the top-left corner coordinate, (x2, y2) is the top-right corner coordinate, (x3, y3) is the bottom-right corner coordinate, (x4, y4) is the bottom-left corner coordinate, and text is the text content. The elements in A are sorted from top to bottom and from left to right based on the top-left corner coordinate (x1, y1).

[0079] (2) Let the columns of the table have an alignment attribute, and the alignment attribute includes left alignment, center alignment, and right alignment.

[0080] (3) Let the title of the table be a finite set B = [b1, b2, b3,... bn].

[0081] Based on the above three premises, the row and column positioning process includes:

[0082] (1) Find the element set C in A whose text is bi, and take the average row height of the element set with the most number of words in C as the standard height hs.

[0083] (2) Calculate the upper mid-point coordinate (tx, ty) and the lower mid-point coordinate (bx, by) of each element in A, at this time each element in A includes [x1, y1, x2, y2, x3, y3, x4, y4, tx, ty, bx, by, text].

[0084] (3) If the distance between the point (x4, y4) of any two elements ai and aj in A and the point (x1, y1) of aj is <= hs, or the distance between the point (bx, by) of ai and the point (tx, ty) of aj is <= hs, or the distance between the point (x3, y3) of ai and the point (x2, y2) of aj is <= hs, then ai and aj are classified into the same column; finally, n column combinations are obtained, denoted as G = [g1, g2,...gn].

[0085] (4) Fit the [x1, y1] points of each element in each column combination in G as a straight line L1, fit the [tx, ty] points of each element in each column combination in G as a straight line L2, and fit the [x2, y2] points of each element in each column combination in G as a straight line L3, wherein the least square method is used when the number of elements in the column combination is >= 5, and the x = c (c = avg(sum(xi))) is used when the number of elements is less than 5, that is, the straight line is fitted by summing first and then averaging. Then the average distance of each element in the column combination to the three straight lines is calculated to obtain d1, d2, d3, and the straight line corresponding to the smallest average distance is taken as the fitting straight line of the column combination.

[0086] (5) Sort the column combinations in G in descending order of the number of elements, then traverse from large groups to small groups, and merge the column combinations with a distance <= hs to the fitting straight line of the large group with the column combination in the large group, so as to determine the columns of the text region.

[0087] (6) Calculate the center point coordinates cx, cy of each element in A, at this time each element in A includes [x1, y1, x2, y2, x3, y3, x4, y4, cx, cy, text].

[0088] (7) Take the element set C obtained in the first step, select the center point coordinates [cx, cy] of each element in the element set, and represent the line of the center point coordinates with a multi-segment function R1, R1 represents the first row.

[0089] (8) Calculate the next point closest to R1 line in the un-divided row, set as temp_x, temp_y, and move R1 down until it passes through the point, obtain a multi-segment function R2, then re-calculate the center points of all elements passing through R2, re-fit the line as R2_new, and combine R2_new with R2, finally obtain a new row R2.

[0090] (9) Repeat (8) to complete all row scans and determine all rows of the text region. Finally, the row and column grouping information of each element in A is obtained, (r, g) represents the row and column, and each element in A includes [x1, y1, x2, y2, x3, y3, x4, y4, r, g, text].

[0091] In view of the problem that the content of the test report sheet cannot be accurately and effectively recognized when a traditional OCR recognition algorithm is used to recognize the test report sheet picture, the embodiment takes the test report sheet as an example of the table to be recognized, and provides an efficient and portable test report sheet recognition method. The OCR technology is used for automatic recognition, extraction and structured conversion, the recognition result can be checked and modified, and the above problems are effectively solved.

[0092] The method for recognizing the text content of the test report sheet picture will be introduced below, so as to recognize and extract three contents of test items, results and units. As shown in Figure 2 The recognition method of the test report sheet includes: inputting the picture into a text detection model DB to obtain a pixel area of the text in the picture (i.e. a text region), judging whether the text region needs to be rotated and corrected by a direction classifier model, if yes, rotating; inputting the text region into a text recognition model CRNN to obtain a text recognition result; inputting the numbers and symbols in the text recognition result into a digital recognition model for correction, such as a text with only symbols or at most one letter, because "0" may be misrecognized as "o" and "2" may be misrecognized as "z"; using a wireless table recognition algorithm, first performing adaptive non-table element filtering, then performing adaptive table column search and merging and row scanning, completing the positioning of each element of the table, and finally performing column title guessing. After knowing the table elements, the items that are incorrectly recognized in the column title can be corrected according to some characteristics; for the final extraction result of the wireless table recognition algorithm, the item name is corrected using an item name recognition model, and then the unit item is corrected and completed using a dictionary library according to the specified unit corresponding to the item name, and finally the three contents of test items, results and units are obtained. The recognition result of the test report sheet is shown in Figure 3 It should be noted that, Figure 2 The two "cls models" in

[0093] The adaptive non-table element filtering includes adaptive determination of the margin based on the text recognition result, which can be artificially determined based on experience. The text region is adaptively dilated by a morphological method without exceeding the margin, and the maximum frame after expansion is the test report form. The last row of the table is located using the test time and other key fields, and a straight line is fitted to delete the elements below the straight line. The width of the test report form can be inferred using the test time and other key fields, and the elements on both sides of the width are deleted. Then, the remaining region is dilated by 2 times the adaptive dilation of the last time, so that it exceeds the form on the left and right and is connected between different rows. Then, the elements outside the maximum frame are removed, and the remaining elements in the test report form table are obtained.

[0094] As shown in Figure 4 , the column title guessing can include correcting the items incorrectly identified as column titles based on some characteristics after knowing the table elements, such as the following characteristics: 1) a--b is definitely a reference range; 2) the item starting with * and with a text length < 20 is definitely a project name; 3) the column to the right of the project name is definitely a result, and the column to the left of the result is definitely a project name; 4) the distance between two project names in the same row is approximately equal to the distance between two results or two units in the same row (when double columns); 5) the left of the reference range is not a result or a unit, and the left of the result is a unit. Based on the above characteristics, the column to the right of the project name is definitely a result, and the numbers in that column are definitely numbers. If the result column identifies English or Chinese characters, it will be corrected to numbers, and for example, the “unit” column should be “%”, “mg / L”, etc. If other units appear, they will also be corrected. As shown in Figure 5 and Figure 6 , they are respectively a schematic diagram of project name identification and dictionary library correction.

[0095] The project name identification model pn_attentionOCR can refer to Image Caption with Attention, use the basic block of Resnet2 as the main backbone network layer, and adjust part of the structure of the basic block. The structure diagram of the basic block is as shown in Figure 7 . The project name identification model also uses Attention as an attention allocation module and uses GRU for indefinite length prediction.

[0096] The dictionary library correction can include correcting and completing the unit according to the corresponding item name. Through the arrangement of the data of 100 training pictures, through some detection sheets from Baidu pictures and some LIS test systems with free use time, the units of the item names are arranged, and when the item names are consistent and meet certain restrictions, the units in the dictionary library are used for completion.

[0097] The character recognition model CRNN adds a multi-scale fusion module, a rotation correction module, and a secondary digital correction module. The rotation correction module is not for correcting the character area, but for correcting the entire picture. The secondary digital correction module is essentially a digital recognition model with a certain filtering mechanism, which does not need to re-identify all the content again. The digital recognition model digit uses user-generated data for training, and the validation set comes from 100 training examples cropped pictures, with a line recognition accuracy of 99.31%.

[0098] The embodiment can select all steps according to the specific use environment, such as removing the adaptive non-table element filtering step and the item name recognition model correcting item name, to speed up the recognition speed and achieve the purpose of real-time use.

[0099] Although the embodiment takes the test report as an example, the model can be adjusted and improved according to the layout characteristics of various paper bills, forms, and certificates, to adapt to different paper bills, forms, and certificates. Similar format report sheets can also be processed by slightly modifying the model.

[0100] The recognition method of the embodiment has many advantages and beneficial effects compared with the prior art, which are embodied in the following aspects:

[0101] (1) It can efficiently and accurately identify the wireless table part of the test report.

[0102] (2) It can detect the test report in real time and output the recognition result, meeting the actual use requirements.

[0103] (3) The model can be slightly modified to process similar format forms.

[0104] (4) In the future, it can combine knowledge graph to give potential disease types and risks, provide solutions for corresponding diseases, act as a doctor's assistant, and work in the fields of patient virtual assistants, precision medicine, and health management.

[0105] Embodiment 2:

[0106] The embodiment is used to provide a table recognition system, as shown in Figure 8 The recognition system includes:

[0107] The picture acquisition module M1 is configured to acquire a picture obtained by photographing the table to be recognized;

[0108] The text region identification module M2 is configured to identify the picture by using a text detection model to determine a text region in the picture.

[0109] The text content identification module M3 is configured to perform text recognition on the text region by using a text recognition model to obtain a text recognition result; the text recognition result includes vertex coordinates and text content of each element in the text region.

[0110] The text position positioning module M4 is configured to position each element in the text region according to the text recognition result to determine the position of each element in the text region and obtain the position and text content of each element in the table to be recognized; the position includes the number of rows and the number of columns.

[0111] For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant part can be referred to the method part.

[0112] The principles and implementation manners of the present application are described by using specific examples in the present application, and the above description of the examples is only used to help understand the method of the present application and its core idea; meanwhile, for the person skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed. In conclusion, the content of the present application should not be understood as the limitation of the present application.

Claims

1. A form recognition method characterized by, The recognition method comprises: obtaining a picture obtained by photographing a table to be recognized; recognizing the picture by using a text detection model to determine a text region in the picture; recognizing the text region by using a text recognition model to obtain a text recognition result; the text recognition result comprises vertex coordinates and text content of each element in the text region; positioning each element in the text region according to the text recognition result to determine the position of each element in the text region, and obtaining the position and text content of each element in the table to be recognized; the position comprises the number of rows and columns; the positioning of each element in the text region according to the text recognition result to determine the position of each element in the text region specifically comprises: fitting the rows and columns in the text region according to the text recognition result to position each element in the text region by row and column, and determine the position of each element in the text region; the fitting of the rows and columns in the text region according to the text recognition result specifically comprises: for each element in the text recognition result, calculating the upper edge midpoint coordinates, lower edge midpoint coordinates and center point coordinates of the element according to the vertex coordinates of the element; for any two elements in the text recognition result, judging whether the distance between the lower left corner vertex coordinates of the first element and the upper left corner vertex coordinates of the second element, the distance between the lower right corner vertex coordinates of the first element and the upper right corner vertex coordinates of the second element, or the distance between the lower edge midpoint coordinates of the first element and the upper edge midpoint coordinates of the second element is less than or equal to a preset value; if yes, the first element and the second element belong to the same column, and all elements in the text recognition result are grouped by column to obtain a plurality of column combinations; for each column combination, performing straight line fitting based on the coordinates of each element in the column combination to obtain a fitting straight line corresponding to the column combination; arranging a plurality of column combinations in descending order according to the number of elements contained, and performing column merging based on the distance of the fitting straight line corresponding to each column combination in the order after the descending arrangement to obtain the columns in the text region; performing row scanning based on the center point coordinates of all elements to obtain the rows in the text region.

2. The identification method according to claim 1, characterized in that, The text detection model is a DB model.

3. The identification method according to claim 1, characterized in that, Before the text recognition of the text region by using the text recognition model, the recognition method further comprises: judging whether the text in the text region is in a preset direction by using a direction classifier model; if not, performing rotation correction on the text region to make the text in the text region in the preset direction to obtain a corrected text region, and taking the corrected text region as a new text region.

4. The identification method according to claim 1, characterized in that, The text recognition model is a CRNN model.

5. The identification method according to claim 1, characterized in that, Before the position of each element in the text region is located according to the text recognition result, the recognition method further comprises: correcting the text content of the element whose text content is a number or a symbol in the text recognition result by using a number recognition model to obtain a corrected recognition result, and taking the corrected recognition result as a new text recognition result.

6. The identification method of claim 1, wherein, Before the position of each element in the text region is located according to the text recognition result, the recognition method further comprises: According to the text recognition result, non-table element filtering is performed to remove non-table elements in the text region to obtain a filtered region, and the filtered region is taken as a new text region; the vertex coordinates and text content of the non-table elements are removed from the text recognition result to obtain a filtered recognition result, and the filtered recognition result is taken as a new text recognition result.

7. The identification method according to claim 1, characterized in that, The straight line fitting based on the coordinates of each element in the column combination comprises: The upper left vertex coordinates of each element in the column combination are subjected to straight line fitting to obtain a first straight line; the upper middle point coordinates of each element in the column combination are subjected to straight line fitting to obtain a second straight line; the upper right vertex coordinates of each element in the column combination are subjected to straight line fitting to obtain a third straight line; The first average distance from the center point coordinates of each element in the column combination to the first straight line is calculated; the second average distance from the center point coordinates of each element in the column combination to the second straight line is calculated; the third average distance from the center point coordinates of each element in the column combination to the third straight line is calculated; The minimum value of the first average distance, the second average distance and the third average distance is determined, and the straight line corresponding to the minimum value is taken as the fitted straight line corresponding to the column combination.

8. A form recognition system, characterized by The recognition system comprises: A picture acquisition module configured to acquire a picture obtained by photographing a table to be recognized; A text region recognition module configured to recognize the picture by using a character detection model to determine a text region in the picture; A text content recognition module configured to recognize the text region by using a character recognition model to obtain a text recognition result; the text recognition result comprises vertex coordinates and text content of each element in the text region; A text position positioning module configured to position each element in the text region according to the text recognition result to determine the position of each element in the text region and obtain the position and text content of each element in the table to be recognized; the position comprises the number of rows and the number of columns; The position of each element in the text region is determined according to the text recognition result, specifically comprising: fitting the rows and columns in the text region according to the text recognition result to position each element in the text region in rows and columns, and determine the position of each element in the text region; The fitting of the rows and columns in the text region according to the text recognition result specifically comprises: For each element in the text recognition result, calculating, according to the vertex coordinates of the element, the upper edge midpoint coordinates, the lower edge midpoint coordinates and the center point coordinates of the element; For any two elements in the text recognition result, judging whether the distance between the left lower corner vertex coordinates of the first element and the left upper corner vertex coordinates of the second element is less than or equal to a preset value, whether the distance between the right lower corner vertex coordinates of the first element and the right upper corner vertex coordinates of the second element is less than or equal to a preset value, or whether the distance between the lower edge midpoint coordinates of the first element and the upper edge midpoint coordinates of the second element is less than or equal to a preset value; if yes, the first element and the second element belong to the same column, and all the elements in the text recognition result are grouped by column to obtain a plurality of column combinations; For each column combination, performing straight line fitting based on the coordinates of each element in the column combination to obtain a fitting straight line corresponding to the column combination; Arranging the plurality of column combinations in descending order according to the number of elements contained, and performing column merging based on the distance of the fitting straight line corresponding to each column combination in the order after the descending arrangement to obtain the columns in the text region; Performing row scanning based on the center point coordinates of all the elements to obtain the rows in the text region.

Images