Method and apparatus for recognizing a table image with a turning field row

Abstract

The application provides a kind of identification method and device with turning field row table image, it is related to artificial intelligence and table identification technical field, method includes: adjusting the table image of user input;Row and column target detection model is output based on row and column rectangular frame information and corresponding category information;Base rectangular frame information calculates row and column split line and cell position;Text positioning detection is carried out, and the position of several field frames is output;Based on field frame position and cell position, the cell to which each field frame belongs is calculated;Recognize the character information in cell and output image recognition table.The application can utilize row and column detection model to carry out more refined row and column segmentation, and the table image with turning field row can have better support.In addition, through column target detection model, column content can be determined in advance, and then the corresponding character recognition model is selected to recognize the character information in cell, so that the accuracy of field recognition is further improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and table recognition technology, and in particular to a method and apparatus for recognizing table images with transitional field rows. Background Technology

[0002] Tables with transition fields refer to tables where, due to paper size limitations during printing, some column content cannot be fully displayed, resulting in some columns appearing on the second line. This situation frequently occurs when printing bank transaction records. Please see [link to other documentation]. Figure 1 , Figure 1 This is a representation of a bank transaction record printout. The dashed box in the image indicates the section where the transition field appears. Figure 1 The content of "Other party's account and name" should have appeared on the right side of the table, that is, "1053100010209XXXX" and "XX City XX Environmental Protection Co., Ltd." should have appeared on the same row as "Salary". However, due to paper limitations, it could not be printed on the right side of the table, so it appeared on the second row, that is, the "Salary" section has a transition field row.

[0003] Current mainstream table image recognition solutions include the following: First, methods based on table lines perform structural analysis, extracting table segmentation lines through image morphology or semantic segmentation. Second, methods based on field projection analysis combine field coordinate information to form projections, then calculate columns according to rules. Third, general graph neural network-based methods describe the table structure recognition problem as a graph problem compatible with graph neural networks. Fourth, methods based on HTML sequences employ multi-task learning to simultaneously predict table structure sequences and regress cell positions, fusing table structure sequences and cell text content to obtain HTML code.

[0004] However, the above methods do not support tables with transitional rows well. The resulting table cannot clearly display the table content, and the recognition result is either similar to the original table or contains recognition errors. In addition, the above methods cannot remove column information of human fields, only judge the row and column position information, and use single full-text recognition. They lack a targeted model for the recognition accuracy of multiple numerical amounts and dates, such as bank statements, which limits the improvement of recognition accuracy. Summary of the Invention

[0005] In view of this, the present invention provides a method and apparatus for recognizing table images with transition field rows to solve at least one of the aforementioned problems.

[0006] To achieve the above objectives, the present invention adopts the following solution:

[0007] According to a first aspect of the present invention, a method for recognizing a table image with a row of turning-off fields is provided. The method includes: adjusting the tilt and orientation of an input table image with a row of turning-off fields; outputting column rectangle information and corresponding category information of normal columns in the adjusted table image based on a column object detection model, and outputting column rectangle information and corresponding category information of the row of turning-off fields, wherein the normal columns are columns that do not contain the row of turning-off fields and the column to which the first row of the row of turning-off fields belongs; outputting row rectangle information and corresponding category information of normal rows in the adjusted table image based on a row object detection model, and outputting row rectangle information and corresponding category information of the row of turning-off fields, wherein the normal rows are the rows to which the first row of the row of turning-off fields belongs; calculating row and column dividing lines and cell positions based on the column rectangle information of normal columns and the row of turning-off fields, and the row rectangle information of normal rows and the row of turning-off fields; performing text localization detection on the table image with a row of turning-off fields, and outputting several field box positions; determining the cell to which each field box belongs based on the field box positions and the cell positions; recognizing the text information in the cell and outputting an image recognition table.

[0008] As an embodiment of the present invention, the above method for adjusting the tilt and orientation of the input table image with turning field rows includes: using a corner detection model trained based on deep learning to detect the coordinate information of the four corner points of the image; calculating the transformation matrix to complete the perspective transformation calculation, thereby correcting the image orientation, perspective tilt and angle rotation.

[0009] As an embodiment of the present invention, the column rectangle information in the above method includes the coordinate information of a pair of diagonal points of the column rectangle, and the category information corresponding to the column rectangle is the column header title.

[0010] As an embodiment of the present invention, the row rectangle information in the above method includes the coordinate information of a pair of diagonal points of the column rectangle, and the category information corresponding to the row rectangle is the header row, the first row of the content row, or the second row of the content row.

[0011] As an embodiment of the present invention, the method described above for determining row and column dividing lines and cell positions based on column rectangle information of normal columns and transition field rows, and row rectangle information of normal rows and transition field rows includes: calculating the first column median line based on the column rectangle information of normal columns, the right line of the left column and the left line of the right column; calculating the second column median line based on the column rectangle information of transition field rows, the right line of the left column and the left line of the right column; calculating the row median line based on the row rectangle information of normal rows and transition field rows, the bottom line of the top row and the top line of the bottom row between two adjacent rows; calculating the first intersection point of the first column median line and the row median line, forming n×m first cells based on the first intersection point, where n is the number of rows in the table content part and m is the number of normal columns; calculating the first intersection point of the second column median line and the row median line, forming n×m' second cells based on the first intersection point, where m' is the number of columns with transition field rows; and outputting the cell matrix of the first cells and the second cells.

[0012] As an embodiment of the present invention, the position elements in the cell matrix in the above method include the coordinate information of a pair of diagonal points of the cell and the category information of the corresponding column. The cell matrix has n rows and m+m' columns.

[0013] As an embodiment of the present invention, the method described above for recognizing text information within a cell includes: recognizing text information within a cell based on category information in the cell and a corresponding text recognition model. The text recognition module includes a date model, a numerical amount model, and a general text recognition model.

[0014] According to a second aspect of the present invention, a recognition device for a table image with a turning-field row is provided. The device includes: an image adjustment unit for adjusting the tilt and directional deviation of an input table image with a turning-field row; a column detection unit for outputting column bounding box information and corresponding category information of normal columns in the adjusted table image based on a column target detection model, and outputting column bounding box information and corresponding category information of turning-field rows, wherein normal columns are columns that do not contain turning-field rows and columns to which the content of the first row of the turning-field row belongs; and a row detection unit for outputting row bounding box information and corresponding category information of normal rows in the adjusted table image based on a row target detection model. The system includes: a column rectangle information and corresponding category information for outputting the row rectangle information of the transition field row, wherein the normal row is the row containing the first row of the transition field row content; a cell calculation unit for determining the row and column dividing lines and cell positions based on the column rectangle information of the normal column and the transition field row, and the row rectangle information of the normal row and the transition field row; a field positioning unit for performing text positioning detection on the table image with the transition field row and outputting several field box positions; a field attribution calculation unit for determining the cell to which each field box belongs based on the field box positions and the cell positions; and a recognition output unit for recognizing the text information within the cells and outputting an image recognition table.

[0015] As an embodiment of the present invention, the preprocessing unit includes: a corner calculation module, used to detect the coordinate information of the four corners of the image using a corner detection model trained based on deep learning methods; and a transformation calculation module, used to calculate the transformation matrix to complete the perspective transformation calculation, thereby correcting the image direction, perspective tilt, and angle rotation.

[0016] As an embodiment of the present invention, the column rectangle information in the above device includes the coordinate information of a pair of diagonal points of the column rectangle, and the category information corresponding to the column rectangle is the column header title.

[0017] As an embodiment of the present invention, the row rectangle information in the above device includes the coordinate information of a pair of diagonal points of the column rectangle, and the category information corresponding to the row rectangle is the header row, the first content row, or the second content row.

[0018] As an embodiment of the present invention, the cell calculation unit in the above-described device includes:

[0019] The first column median line calculation module is used to calculate the first column median line based on the column rectangle information of the normal column, the right line of the left column and the left line of the right column.

[0020] The second column median line calculation module is used to calculate the second column median line of the right line of the left column and the left line of the right column based on the column rectangle information of the turning field row;

[0021] The row midline calculation module is used to calculate the row midline between two adjacent rows based on the row rectangle information of normal rows and transition field rows;

[0022] The first cell generation module is used to calculate the first intersection point of the first column midline and the row midline, and form n×m first cells based on the first intersection point, where n is the number of rows of the table content part and m is the number of columns of the normal column;

[0023] The second cell generation module is used to calculate the first intersection point of the second column median line and the row median line, and to form n×m' second cells based on the first intersection point, where m' is the number of columns with the turning field;

[0024] The output module is used to output the cell matrix of the first cell and the second cell.

[0025] As an embodiment of the present invention, the position elements in the cell matrix of the above device include the coordinate information of a pair of diagonal points of the cell and the category information of the corresponding column. The cell matrix has n rows and m+m' columns.

[0026] As an embodiment of the present invention, the recognition output unit in the above-mentioned device recognizes the text information in the cell by: recognizing the text information in the cell based on the category information in the cell and the corresponding text recognition model. The text recognition module includes a date model, a digital amount model and a general text recognition model.

[0027] According to a third aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0028] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0029] As can be seen from the above technical solution, the recognition method and apparatus for table images with turning-point rows provided by the present invention can utilize a row and column detection model for more refined row and column segmentation, providing better support for table images with turning-point rows. Furthermore, it offers more comprehensive support for table line types, effectively supporting standard boxes, half-boxes, and frameless tables. Additionally, the column object detection model allows for pre-definition of column content, and then the selection of the corresponding text recognition model further improves the accuracy of field recognition. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0031] Figure 1 This application provides an embodiment of a bank transaction record printing representation.

[0032] Figure 2 This is a flowchart illustrating a method for recognizing a table image with a transition field provided in an embodiment of this application;

[0033] Figure 3 This is a schematic diagram of an image preprocessing process provided in an embodiment of this application;

[0034] Figure 4 This is a schematic diagram of the column rectangles output by the column target detection model provided in this application embodiment;

[0035] Figure 5 This is a schematic diagram of the row rectangle output by the row target detection model provided in the embodiments of this application;

[0036] Figure 6 This is a flowchart illustrating the process of determining cell position provided in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram illustrating the determination of row and column dividing lines provided in an embodiment of this application;

[0038] Figure 8 This is a schematic diagram illustrating the determination of the first intersection point provided in an embodiment of this application;

[0039] Figure 9 This is a schematic diagram illustrating the determination of the first cell based on the first intersection point provided in an embodiment of this application;

[0040] Figure 10 This is a schematic diagram illustrating the determination of the second intersection point provided in an embodiment of this application;

[0041] Figure 11 This is a schematic diagram illustrating the determination of the second cell based on the second intersection point, provided in an embodiment of this application.

[0042] Figure 12 This is a schematic diagram of the output image recognition table provided in the embodiments of this application;

[0043] Figure 13 This is a schematic diagram of the structure of a recognition device for a table image with a transition field provided in an embodiment of this application;

[0044] Figure 14 This is a schematic diagram of the structure of the preprocessing unit provided in the embodiments of this application;

[0045] Figure 15 This is a schematic diagram of the structure of the cell calculation unit provided in the embodiments of this application;

[0046] Figure 16 This is a schematic block diagram of the system configuration of the electronic device provided in the embodiments of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0048] like Figure 2 The diagram shown is a flowchart illustrating a method for recognizing a table image with a transition field provided in an embodiment of this application. The method includes the following steps:

[0049] Step S201: Adjust the tilt and orientation of the input table image with the turning field.

[0050] The table image with transition fields entered by the user can come from scanning, photography, or other channels. The image format can be any known unencrypted image or document format, such as JPG, TIF, PNG, DOCX, PDF, etc.

[0051] Since the acquired images may have problems such as tilting and orientation deviation, mainly tilting on the plane and tilting on the z-axis, this application preprocesses the received table images in this step to solve these problems, so that the subsequent table recognition steps can proceed normally without recognition errors caused by image tilting and deviation.

[0052] Preferred, such as Figure 3 As shown, in this embodiment, this step can proceed to include the following sub-steps:

[0053] Step S2011: Use the corner detection model trained based on deep learning methods to detect the coordinate information of the four corner points of the image.

[0054] In this embodiment, the detection of the four corner points of the image adopts a deep learning method. First, the relative coordinates of the top left, top right, bottom right, and bottom left corners of the main part of the transaction flow in the image are labeled in the training data definition as (x1, y1), (x2, y2), (x3, y3), and (x4, y4), respectively. Second, in the network structure definition, a Convolutional Neural Network (CNN) backbone network is selected (this embodiment does not restrict the selection of the backbone network). Convolutional and pooling layers are constructed to extract features, and finally, four fully connected layers are connected to regress the extracted features to obtain the relative coordinate information of the four vertices (xi, yi), where i is from 1 to 4, for a total of 8 values.

[0055] Step S2012: Calculate the transformation matrix to complete the perspective transformation calculation and correct the image orientation, perspective tilt, and angle rotation.

[0056] After processing through steps S2011-S2012, the orientation and angle of the image are corrected, and the image as a whole becomes a document rectangle. This makes the rows and columns required by the table structure restoration method close to a rectangle, so that the row and column detection in the subsequent steps can achieve better detection results.

[0057] Step S202: Based on the column target detection model, output the column rectangle information and corresponding category information of the normal columns in the adjusted table image, and output the column rectangle information and corresponding category information of the turning field rows. The normal columns are columns that do not contain the turning field rows and the columns to which the first row of the turning field rows belongs.

[0058] In this embodiment, to reduce data volume, the column bounding box information output by the column detection model does not need to include the coordinates of all four corner points, but only the coordinates of two opposite corner points. For example, it needs to include the top-left corner information (x_col_left, y_col_up) and bottom-right corner information (x_col_right, y_col_down), or the top-right corner information and the bottom-left corner information. Additionally, the information output by the column target detection model in this embodiment also includes the category information corresponding to the column bounding box. This category information is the column header title, for example... Figure 1 The transaction details include "transaction date", "transaction amount", "amount for this transaction", and "transaction location".

[0059] The column target detection model in this embodiment can be obtained by labeling and training a large number of preprocessed images. These images can cover various formats of real bank statements, including framed, half-framed, and frameless formats, as well as densely packed columns. Transaction statement content rows can have two types of fields: single-line fields and fields with transitional fields. If certain field cells contain line breaks but subsequent rows still belong to the same field column information, this is still considered a single-line field. For example... Figure 1 If the "Transaction Location" field has two lines due to excessive code length, it is still considered a single-line field; a line with a transition field refers to a second line containing new field information, such as... Figure 1 In the summary column, the "1053100010209XXXX" below "salary" does not belong to the summary, but to the other party's account, so it is a transition field.

[0060] By training the column object detection model, the model can identify the transition field row and the table header title corresponding to the second row in the transition field row, for example... Figure 1 In this context, "1053100010209XXXX" belongs to the "recipient account," while "XX City XX Environmental Protection Co., Ltd." belongs to the "recipient name." Therefore, the correct category information corresponding to the column rectangle in the transition field row can be output. Object detection models can utilize commonly used algorithms to achieve these functions, such as YOLO, R-CNN series algorithms, and the Transformer-based detection algorithm DETR, etc.

[0061] like Figure 4 The image shows a schematic diagram of the column rectangles for the normal columns and transition field rows output after this step. The normal columns include columns excluding transition field rows and the column containing the first row of content within the transition field rows. Through this step, the output column rectangle information and corresponding category information can ultimately form a multi-column information list. Assuming the normal columns are from left to right (first column) and right to left (m-th column), the output information is represented as follows: List({x_col_left) i y_col_up i , x_col_right i y_col_down i label i}), where i is 1 to m. If the transition field row consists of columns 1 to m' from left to right, then the additional output information in this embodiment is represented as follows: List({x_col_left_second) i y_col_up_second i x_col_right_second i y_col_down_second i label i}), where i is from 1 to m', and label represents the category information.

[0062] Step S203: Based on the row target detection model, output the row rectangle information and corresponding category information of the normal rows in the adjusted table image, and output the row rectangle information and corresponding category information of the turning field rows, wherein the normal rows are the rows containing the first row of the turning field rows.

[0063] Similarly, the row object detection model in this embodiment can also be trained using the same method as the column object detection model described above. The row rectangle information can include the top-left and bottom-right point information of the column rectangle, while the category information corresponding to the row rectangle can be of three types: header row, first row of content, or second row of content.

[0064] The header row is distinct from the table content, and it can have multiple rows, for example... Figure 1 In the table, rows such as "XX Bank XX Branch Account Transaction Record," "Date: 20**1107-20**0507," "Name, Date," and "Transaction Date, Transaction Amount..." are all header rows. For tables with transition fields, the number of content rows will double. For example, if there were n content rows in the first row, there will now be n more content rows in the second row. Normal rows are the rows containing the content of the first row of transition field rows. Therefore, for the row rectangle of a normal row, the corresponding category information is the content of the first row; for the row rectangle of a transition field row, the corresponding category information is the content of the second row.

[0065] like Figure 5 The diagram shows the row rectangles of the normal rows and transition field rows output after this step. This step outputs the header row information, multiple content first rows, and multiple content second rows. In this embodiment, the header row information output by the row object detection model is represented as follows: {x_row_left} head y_row_up head ,x_row_right head y_row_down head label head Since the columns corresponding to the table content only correspond to the table header titles, this embodiment only outputs the information of the row containing the table header titles. The first row of information for multiple content items is represented as follows: List({x_row_left) i y_row_up i ,x_row_right i y_row_down i `List({x_row_left_second})`, where `i` is from 1 to n, indicating n normal rows. Multiple rows of content are represented as follows: `List({x_row_left_second})`. i y_row_up_second i x_row_right_second i y_row_down_second i, row_content_sencond}), where i is from 1 to n.

[0066] Step S204: Determine the row and column dividers and cell positions based on the column rectangle information of the normal column and the transition field row, and the row rectangle information of the normal row and the transition field row.

[0067] After obtaining the row and column rectangles of the table through steps S202 and S203, the row and column dividing lines can be calculated based on the rectangle information, and the position of the cell can be determined based on the intersection of the row and column dividing lines.

[0068] Preferred, such as Figure 6 As shown, step S204 may specifically include the following sub-steps:

[0069] Step S2041: Based on the column rectangle information of the normal column, calculate the first column median line of the right line of the left column and the left line of the right column.

[0070] For two adjacent columns, the dividing line is set as the midline of the right line of the left column and the left line of the right column. To distinguish the midline of the columns in normal columns from the midline of the column to be transitioned, this embodiment refers to the dividing line of the normal columns as the midline of the first column. As can be seen from the definition of the midline, if a table has m normal columns, it has m-1 dividing lines.

[0071] Step S2042: Based on the column rectangle information of the turning field row, calculate the second column median line of the right line of the left column and the left line of the right column.

[0072] Similarly, for the column rectangle information of the transition field row, its dividing line, i.e. the median line of the second column, can also be calculated according to the above method. For a table with m' transition field rows, it has m'-1 dividing lines.

[0073] like Figure 7 The diagram shown is a schematic of the row and column dividing lines obtained after processing in steps S2041 and S2042 in this embodiment. Column dividing lines can be represented by two points, one above and one below. For a normal column dividing line, the coordinates of the upper vertex of the i-th column dividing line are ((x_col_right)). i +x_col_left i+1 ) / 2, (y_col_up i +y_col_up i+1 ) / 2)), the lower vertex is ((x_col_right) i +x_col_left i+1 ) / 2, (y_col_down i +y_col_down i+1() / 2)), where i is from 1 to m, thus forming a colLineList table. Similarly, for the dividing line of a column with a transition field, a secondColLineList table can also be formed.

[0074] Step S2043: Based on the row rectangle information of normal rows and transition field rows, calculate the row midline between the top row bottom line and the bottom row top line of adjacent rows;

[0075] As mentioned above, for a table with a transition field, its original n rows will become 2n rows, thus generating 2n-1 dividing lines. See also... Figure 7 This step can also create a rowLineList table and a secondrowLineList table.

[0076] Step S2044: Calculate the first intersection point of the first column median line and the row median line, and form n×m first cells based on the first intersection point, where n is the number of rows in the table content part and m is the number of columns in the normal column.

[0077] Through this step, the following can be calculated: Figure 8 The intersection point shown can be used to generate n×m first cells, as follows: Figure 9 As shown, Figure 9 The outer border of a cell can be filled using the coordinates of a grid dividing line or a rectangle.

[0078] Step S2045: Calculate the first intersection point of the second column median line and the row median line, and form n×m' second cells based on the first intersection point, where m' is the number of columns of the row with the turning field.

[0079] Through this step, the following can be calculated: Figure 10 The intersection point shown can be used to generate n×m' second cells. See [link to documentation] for details. Figure 11 .

[0080] Step S2046: Output the cell matrix of the first cell and the second cell.

[0081] For the first and second units mentioned above, the coordinates of each cell are represented as the top-left corner (x_unit_left, y_unit_up) and the bottom-right corner (x_unit_right, y_unit_down). The cell in the i-th row and j-th column of the output cell matrix unitMatrix can be represented as {x_col_left} i,j ,y_col_up i,j ,x_col_right i,jy_col_down i,j label i,j}, where label represents the category information of the corresponding column, and the cell matrix unitMatrix has n rows and m+m' columns.

[0082] Step S205: Perform text positioning detection on the table image with the turning field row, and output the positions of several field boxes.

[0083] It should be noted that this step can be executed synchronously with steps S202-S204. That is, after receiving the user-input table image with the turning field row in step S201, OCR text localization detection can be performed directly to output the positions of several field boxes, thereby speeding up the entire recognition process. This embodiment does not limit the algorithm for text localization detection. For example, common algorithms such as PSENET, DBNet, and East can be used for data training, or mature localization models in the industry can be reused.

[0084] The field box positions output in this step can form a list of bounding boxes (bboxes). Assuming the entire text has p fields (fields are the closely spaced text at one end), the output `fieldList = List({x_field_left)` will be calculated. i ,y_field_up i ,x_field_right i y_field_down i}), where i is from 1 to p.

[0085] Step S206: Determine the cell to which each field box belongs based on the field box position and the cell position.

[0086] If a field is within the range of a single cell (row i, column j), i.e. x_field_left >= x_unit_left i,j And x_field_right<=x_unit_right i,j And y_field_up <= y_unit_up i,j And y_field_down>=y_unit_down i,j If so, then the field belongs to the cell in the i-th row and j-th column.

[0087] If there are cases that span across columns, such as spanning columns j and j+1, i.e., x_field_left <x_unit_right i,j And x_field_right > x_unit_left i,j+1Then, the field is divided into two fields based on the right boundary of the j-th column, with coordinates {x_field_left, y_field_up, x_unit_right}. i,j ,y_field_down} and {x_unit_right i,j+1 y_field_up, x_field_right i y_field_down i The values ​​are assigned to columns j and j+1 respectively. The same rule is used to judge and split the second field.

[0088] If a field spans multiple rows, such as spanning rows i and i+1, i.e., y_field_up > y_unit_down i,j And y_field_down <y_unit_up i+1,j Cells are assigned based on their height percentage, i.e., y_field_up - y_unit_down. i,j >=y_unit_up i+1,j When -y_field_down is used, it belongs to row i; otherwise, it belongs to row i+1.

[0089] Based on the above rules, each cell will ultimately contain a list of field position information, denoted by `subFiledListij`, representing the list of field positions in the cell at row i and column j. Update the cell matrix `unitMatrix`: in addition to position and category information, supplement the field list position information, `{x_col_left`. i,j ,y_col_up i,j ,x_col_right i,j y_col_down i,j label i,j subFilldList i,j}, i represents rows 1 to n, j represents columns 1 to m+m', label represents the category label of the corresponding column, and each element of subFieldList contains the top left and bottom right information of the field.

[0090] Step S207: Recognize the text information within the cell and output an image recognition table.

[0091] Preferably, recognizing the text information within the cell in this step may include: selecting a corresponding text recognition model to recognize the text information within the cell based on the category (label) information in the cell. This text information is the content of the cell, including dates, numbers, and text. The text recognition module includes a date model, a number amount model, and a general text recognition model. The date model, number amount model, and general text recognition model can adopt existing corresponding recognition models. For example, the date model can be a template matching model for various date formats, the number amount model can be a dedicated model trained based on a neural network, and the general text recognition model can be an ORC text recognition model.

[0092] Based on the recognition results, the output image recognition matrix wordResultMatrix is ​​given, where each element is {wordResult...} i,j Let i be rows 1 to n, and j be columns 1 to m+m'. Then, based on this image recognition matrix wordResultMatrix, an image recognition table of size n×(m+m') can be output. For details, please refer to [link to documentation]. Figure 12 .

[0093] As can be seen from the above technical solution, the recognition method for table images with turning-point rows provided by this invention can utilize a row and column detection model for more refined row and column segmentation, providing better support for table images with turning-point rows. Furthermore, it offers more comprehensive support for table line types, effectively supporting standard border, half-border, and borderless tables. Additionally, the column object detection model allows for pre-definition of column content, and then the selection of the corresponding text recognition model further improves the accuracy of field recognition.

[0094] like Figure 13 The diagram shown is a structural schematic of a recognition device for a table image with a turning field provided in an embodiment of this application. The device includes: an image adjustment unit 110, a column detection unit 120, a row detection unit 130, a cell calculation unit 140, a field positioning unit 150, a field attribution calculation unit 160, and a recognition output unit 170. The column detection unit 120 is connected to the image adjustment unit 110 and the cell calculation unit 140, respectively. The row detection unit 130 is also connected to the image adjustment unit 110 and the cell calculation unit 140, respectively. The field positioning unit 150 is connected to the cell calculation unit 140 and the field attribution calculation unit 160, respectively. The recognition output unit 170 is connected to the field attribution calculation unit 160.

[0095] The image adjustment unit 110 is used to adjust the tilt and orientation of the input table image with turning fields.

[0096] The column detection unit 120 is used to output the column rectangle information and corresponding category information of normal columns in the adjusted table image based on the column target detection model, and to output the column rectangle information and corresponding category information of the turning field row. The normal column is the column that does not contain the turning field row and the column to which the first row of the turning field row belongs.

[0097] The row detection unit 130 is used to output the row rectangle information and corresponding category information of normal rows in the adjusted table image based on the row target detection model, and to output the row rectangle information and corresponding category information of the turning field row, wherein the normal row is the row where the first row of the turning field row content is located.

[0098] Cell calculation unit 140 is used to determine row and column dividers and cell positions based on column rectangle information of normal columns and transition field rows, and row rectangle information of normal rows and transition field rows.

[0099] The field positioning unit 150 is used to perform text positioning detection on the table image with the turning field row and output the positions of several field boxes.

[0100] The field attribution calculation unit 160 is used to determine the cell to which each field box belongs based on the field box position and the cell position.

[0101] The recognition output unit 170 is used to recognize the text information in the cell and output an image recognition table.

[0102] Preferred, such as Figure 14 As shown, the image adjustment unit 110 may include: a corner calculation module 111 and a transformation calculation module 112, wherein:

[0103] The corner calculation module 111 is used to detect the coordinate information of the four corners of the image using a corner detection model trained based on deep learning methods.

[0104] The transformation calculation module 112 is used to calculate the transformation matrix to complete the perspective transformation calculation, so as to correct the image orientation, perspective tilt and angle rotation.

[0105] Preferably, the column rectangle information in the above device includes the coordinate information of a pair of diagonal points of the column rectangle, and the category information corresponding to the column rectangle is the column header title.

[0106] Preferably, the row rectangle information in the above device includes the coordinate information of a pair of diagonal points of the column rectangle, and the category information corresponding to the row rectangle is the header row, the first row of the content row, or the second row of the content row.

[0107] Preferred, such as Figure 15 As shown, the cell calculation unit 140 in the above-mentioned device includes:

[0108] The first column median line calculation module 141 is used to calculate the first column median line based on the column rectangle information of the normal column, the right line of the left column and the left line of the right column.

[0109] The second column median line calculation module 142 is used to calculate the second column median line based on the column rectangle information of the turning field row, the right line of the left column and the left line of the right column.

[0110] The row midline calculation module 143 is used to calculate the row midline between the top row bottom line and the bottom row top line based on the row rectangle information of the normal row and the turning field row.

[0111] The first cell generation module 144 is used to calculate the first intersection point of the first column midline and the row midline, and form n×m first cells based on the first intersection point, where n is the number of rows of the table content part and m is the number of columns of the normal column.

[0112] The second cell generation module 145 is used to calculate the first intersection point of the second column median line and the row median line, and to form n×m' second cells based on the first intersection point, where m' is the number of columns with the turning field.

[0113] Output module 146 is used to output the cell matrix of the first cell and the second cell.

[0114] Preferably, the position elements in the cell matrix of the above device include the coordinate information of a pair of diagonal points of the cell and the category information of the corresponding column. The cell matrix has n rows and m+m' columns.

[0115] Preferably, the recognition output unit 170 in the above device recognizes the text information in the cell by: selecting the corresponding text recognition model to recognize the text information in the cell according to the category information in the cell, wherein the text recognition module includes a date model, a numerical amount model and a general text recognition model.

[0116] As can be seen from the above technical solution, the recognition device for table images with turning-point rows provided by the present invention can perform more refined row and column segmentation using a row and column detection model, providing better support for table images with turning-point rows. Furthermore, it offers more comprehensive support for table line types, effectively supporting standard boxes, half-boxes, and frameless tables. Additionally, the column object detection model allows for pre-definition of column content, and then the selection of the corresponding text recognition model further improves the accuracy of field recognition.

[0117] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method.

[0118] This invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described method.

[0119] like Figure 16 As shown, the electronic device 600 may also include: a communication module 110, an input unit 120, an audio processor 130, a display 160, and a power supply 170. It is worth noting that the electronic device 600 does not necessarily need to include these components. Figure 16 All components shown; in addition, the electronic device 600 may also include Figure 16 For components not shown, please refer to existing technologies.

[0120] like Figure 16 As shown, the central processing unit 100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device. The central processing unit 100 receives inputs and controls the operation of various components of the electronic device 600.

[0121] The memory 140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 100 may execute the program stored in the memory 140 to perform information storage or processing, etc.

[0122] Input unit 120 provides input to central processing unit 100. Input unit 120 may be, for example, a keypad or touch input device. Power supply 170 provides power to electronic device 600. Display 160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0123] The memory 140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 140 can also be some other type of device. The memory 140 includes a buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application / function storage unit 142 for storing application programs and function programs or processes for executing the operation of the electronic device 600 via the central processing unit 100.

[0124] The memory 140 may also include a data storage unit 143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 144 of the memory 140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0125] The communication module 110 is a transmitter / receiver 110 that transmits and receives signals via antenna 111. The communication module (transmitter / receiver) 110 is coupled to the central processing unit 100 to provide input signals and receive output signals, which can be the same as in a conventional mobile communication terminal.

[0126] Based on different communication technologies, multiple communication modules 110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) 110 is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide audio output via the speaker 131 and receive audio input from the microphone 132, thereby enabling typical telecommunications functions. The audio processor 130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 130 is coupled to a central processing unit 100, enabling on-device recording via the microphone 132 and on-device playback of stored audio via the speaker 131.

[0127] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0129] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0131] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for recognizing table images with transitional rows, characterized in that, The method includes: The tilt and orientation of the input table image with a turning field are adjusted; wherein, the turning field is the row corresponding to the field cell where there is a line break and the content of the subsequent row does not belong to the same field column information as the content of the first row; The column rectangle information and corresponding category information of normal columns in the adjusted table image output by the column target detection model, as well as the column rectangle information and corresponding category information of the turning field row, are output. The normal column is the column that does not contain the turning field row and the column to which the first row of the turning field row belongs. The column rectangle information includes the coordinate information of a pair of diagonal points of the column rectangle, and the category information corresponding to the column rectangle is the column header title. Based on the row target detection model, the adjusted table image outputs the row rectangle information and corresponding category information of normal rows, as well as the row rectangle information and corresponding category information of the turning field rows. The normal rows are the rows containing the first row of content of the turning field rows. The row rectangle information includes the coordinate information of a pair of diagonal points of the column rectangle, and the category information corresponding to the row rectangle is the header row, the first row of the content row, or the second row of the content row. The row and column dividers and cell positions are determined based on the column rectangle information of the normal column and the transition field row, and the row rectangle information of the normal row and the transition field row. Perform text localization detection on the table image with the transition field and output the positions of several field boxes; The cell to which each field box belongs is determined based on the field box position and the cell position; Recognize the text information within the corresponding cell and output an image recognition table.

2. The method for recognizing table images with transition fields as described in claim 1, characterized in that, The adjustment of the tilt and orientation of the input table image with the transition field includes: The coordinate information of the four corner points of the image is detected using a corner detection model trained by deep learning methods; The transformation matrix is ​​calculated to perform perspective transformation, thereby correcting image orientation, perspective tilt, and angular rotation.

3. The method for recognizing table images with transition fields as described in claim 1, characterized in that, The determination of row and column dividers and cell positions based on the column rectangle information of normal columns and transition field rows, and the row rectangle information of normal rows and transition field rows includes: Based on the column rectangle information of the normal column, calculate the first column median line of the right line of the left column and the left line of the right column; Based on the column rectangle information of the turning field row, calculate the second column median line of the right line of the left column and the left line of the right column; Based on the row rectangle information of normal rows and transition field rows, calculate the row midline between the top row bottom line and the bottom row top line of adjacent rows; Calculate the first intersection point of the first column median and the row median, and form a... The first cell, where This represents the number of rows in the table's content section. This is the column number of the normal column; Calculate the first intersection point of the second column median and the row median, and form a... The second cell, in which The number of columns in rows with transition fields; Output the cell matrix of the first cell and the second cell.

4. The method for recognizing table images with transition fields as described in claim 3, characterized in that, The position elements in the cell matrix include the coordinates of a pair of diagonal points of the cell and the category information of the corresponding column. The cell matrix has OK, List.

5. The method for recognizing table images with transition fields as described in claim 4, characterized in that, The recognition of text information within the cell includes: Based on the category information in the cell, the text information in the cell is recognized using the corresponding text recognition model, which includes a date model, a numerical amount model, and a general text recognition model.

6. A device for recognizing table images with transitional field rows, characterized in that, The device includes: An image adjustment unit is used to adjust the tilt and orientation of an input table image with a turning field row; wherein the turning field row is the row corresponding to a field cell where there is a line break and the content of the subsequent row does not belong to the same field column information as the content of the first row; The column detection unit is used to output the column rectangle information and corresponding category information of normal columns in the adjusted table image based on the column target detection model, and to output the column rectangle information and corresponding category information of the turning field row. The normal column is the column that does not contain the turning field row and the column to which the first row of the turning field row belongs. The column rectangle information includes the coordinate information of a pair of diagonal points of the column rectangle, and the category information corresponding to the column rectangle is the column header title. The row detection unit is used to output the row rectangle information and corresponding category information of normal rows in the adjusted table image based on the row target detection model, and to output the row rectangle information and corresponding category information of the turning field rows, wherein the normal rows are the rows containing the first row of content of the turning field rows; wherein the row rectangle information includes the coordinate information of a pair of diagonal points of the column rectangle, and the category information corresponding to the row rectangle is the header row, the first row of content rows, or the second row of content rows; The cell calculation unit is used to determine the row and column dividers and cell positions based on the column rectangle information of the normal column and the row rectangle information of the normal row and the row rectangle information of the row. The field positioning unit is used to perform text positioning detection on the table image with the turning field row and output the positions of several field boxes. The field attribution calculation unit is used to determine the cell to which each field box belongs based on the field box position and the cell position; The recognition output unit is used to recognize the text information in the cell and output an image recognition table.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images