A method and apparatus for recognizing three-line table images

Abstract

This application provides a method and apparatus for recognizing three-line table images. The method includes: acquiring an image to be recognized, wherein the image to be recognized includes a three-line table; determining key point information of the three-line table, and determining the top line, bottom line, and column line of the three-line table based on the key point information; performing text detection on the three-line table based on the top line, bottom line, and column line to determine the table column information, item column information, and table body information of the three-line table; and generating a table corresponding to the image to be recognized based on the table column information, item column information, and table body information. This method improves the recognition accuracy of three-line table images.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for recognizing three-line table images, a computing device, and a computer-readable storage medium. Background Technology

[0002] Three-line tables are a form of data representation that is widely used in scientific papers, medical articles, and other fields because they clearly convey quantitative and variable information. Three-line tables are simple in form, have distinct functions, and are easy to read. A three-line table usually has only three lines: the top line, the bottom line, and the column lines. The part between the top line and the column lines is called the item column, and the part between the column lines and the bottom line is called the table body.

[0003] In daily applications, three-line tables are often stored as images. When data from a three-line table is needed, it is necessary to recognize and output the data as a corresponding table. However, most existing three-line table image recognition methods are based on image pixel analysis, such as feature point detection. The accuracy of the final detection and recognition results is not high, and the actual conversion effect is not ideal.

[0004] Therefore, improving the recognition accuracy of three-line table images has become an urgent problem for technicians. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method and apparatus for recognizing three-line table images, a computing device, and a computer-readable storage medium to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this application, a method for recognizing a three-line table image is provided, comprising:

[0007] Obtain the image to be identified, wherein the image to be identified includes a three-line table;

[0008] Determine the key point information of the three-line table, and determine the top line, bottom line, and column lines of the three-line table based on the key point information;

[0009] The text is detected by the top line, the bottom line and the column line of the three-line table to determine the table column information, item column information and table body information of the three-line table;

[0010] A table corresponding to the image to be identified is generated based on the table column information, the item column information, and the table body information.

[0011] Optionally, determining the key point information of the three-line table includes:

[0012] The image to be identified is input into a pre-trained keypoint localization network model;

[0013] The key point localization network model determines the key point information of the three-line table based on the input image to be identified.

[0014] Optionally, the keypoint localization network model is trained through the following steps:

[0015] Obtain the training image and the corresponding key point standard information of the training image;

[0016] The training images are input into the keypoint localization network model;

[0017] The keypoint localization network model generates keypoint information of the training image in response to the training image as input;

[0018] Calculate the loss value based on the key point information and the key point standard information;

[0019] The keypoint localization network model is trained by backpropagating the loss value.

[0020] Optionally, text detection is performed on the three-line table based on the top line, the bottom line, and the column line to determine the table column information, item column information, and table body information of the three-line table, including:

[0021] Text detection is performed on the item columns of the three-line table to determine the table column information and item column information of the three-line table, wherein the item column is determined by the top line and the column line;

[0022] Based on the column information of the table, text detection is performed on the body of the three-line table to determine the body information of the three-line table, wherein the body is determined by the column line and the bottom line.

[0023] Optionally, text detection is performed on the item columns of the three-line table to determine the table column information and item column information of the three-line table, including:

[0024] Text detection is performed on the item columns of the three-line table to obtain multiple first regions and the first region information corresponding to each first region;

[0025] The column information of the three-line table is determined based on the information of each of the first regions;

[0026] The item column information of the three-line table is determined based on each of the first regions.

[0027] Optionally, the first region information includes the coordinates of the first vertex and the second vertex of the first region;

[0028] The table column information of the three-line table is determined based on the information of each of the first regions, including:

[0029] The center point coordinates of each of the first regions are determined based on the coordinates of the first vertex and the second vertex of each of the first regions.

[0030] Cluster the center point coordinates of each of the first regions;

[0031] The column information of the three-line table is determined based on the clustering results.

[0032] Optionally, the item column information of the three-line table is determined based on each of the first regions, including:

[0033] Perform text recognition on each of the first regions to obtain the first text information of each of the first regions;

[0034] The item column information of the three-line table is determined based on each of the first text information.

[0035] Optionally, text detection is performed on the body of the three-line table based on the column information to determine the body information of the three-line table, including:

[0036] Text detection is performed on the body of the three-line table to obtain multiple second regions and the second region information corresponding to each second region;

[0037] The location information corresponding to each second region is determined based on the table column information and the information of each second region.

[0038] Perform text recognition on each of the second regions to obtain the second text information of each second region;

[0039] The table body information of the three-line table is determined based on the location information corresponding to each of the second regions and each of the second text information.

[0040] Optionally, the second region information includes the coordinates of the first vertex and the second vertex of the second region;

[0041] Based on the table column information and each second region information, determine the location information corresponding to each second region, including:

[0042] The center point coordinates of each second region are determined based on the coordinates of the first and second vertices of each second region.

[0043] Cluster the center point coordinates of each of the second regions;

[0044] The location information corresponding to each of the second regions is determined based on the clustering results and the information in the table columns.

[0045] According to a second aspect of the embodiments of this application, a three-line table image recognition device is provided, comprising:

[0046] The acquisition module is configured to acquire an image to be recognized, wherein the image to be recognized includes a three-line table;

[0047] The determination module is configured to determine the key point information of the three-line table, and determine the top line, bottom line and column line of the three-line table based on the key point information;

[0048] The detection module is configured to perform text detection on the three-line table based on the top line, the bottom line, and the column line, and determine the table column information, item column information, and table body information of the three-line table;

[0049] The generation module is configured to generate a table corresponding to the image to be recognized based on the table column information, the item column information, and the table body information.

[0050] According to a third aspect of the present application, a computing device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor executes the instructions to implement the steps of the three-line table image recognition method.

[0051] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions which, when executed by a processor, implement the steps of the three-line table image recognition method.

[0052] According to a fifth aspect of the embodiments of this application, a chip is provided that stores computer instructions, which, when executed by the chip, implement the steps of the three-line table image recognition method.

[0053] In this embodiment, by acquiring an image to be recognized including a three-line table, the key point information of the three-line table is determined through a key point localization network model. The recognition accuracy of key points can be greatly improved by using the neural network model. Then, the top line, bottom line, and column line of the three-line table are determined by the key point information, which can further infer a more accurate table. By performing text detection on the three-line table based on the top line, bottom line, and column line, the table column information, item column information, and table body information of the three-line table are determined, which improves the accuracy of image recognition. Finally, the table corresponding to the image to be recognized is generated based on the table column information, item column information, and table body information, resulting in a more accurate table and greatly improving the accuracy of three-line table image recognition. Attached Figure Description

[0054] Figure 1 This is a structural block diagram of the computing device provided in the embodiments of this application;

[0055] Figure 2 This is a flowchart of the three-line table image recognition method provided in the embodiments of this application;

[0056] Figures 3a to 3c This is a schematic diagram of the three-line table image recognition method provided in the embodiments of this application;

[0057] Figure 4 This is a flowchart of a three-line table image recognition method provided in another embodiment of this application;

[0058] Figure 5 This is a schematic diagram of a three-line table image recognition method provided in another embodiment of this application;

[0059] Figure 6 This is a schematic diagram of the structure of the three-line table image recognition device provided in the embodiments of this application. Detailed Implementation

[0060] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0061] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” used in one or more embodiments of this application and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” used in one or more embodiments of this application refers to and includes any or all possible combinations of one or more associated listed items.

[0062] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this application, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "in response to a determination".

[0063] First, the terminology used in one or more embodiments of the present invention will be explained.

[0064] Three-line table: A common table format used in scientific and technological publications, which usually has only three lines: the top line, the column line, and the bottom line.

[0065] Project column: The section between the top line and the column line in a three-line table.

[0066] Table body: In a three-line table, the part below the column lines and above the bottom line is called the table body. It contains most of the information in the table and is the main body of the three-line table.

[0067] Project information: Text information in the project section.

[0068] Watch body information: Text information on the watch body.

[0069] Keypoint localization network model: A model used to identify key points in an image to be identified, which can be a CNN model.

[0070] First area: The text area obtained after text detection in the project column. The first area is a rectangular area.

[0071] First region information: The location information corresponding to the first region, which is at least the coordinates of two vertices on a diagonal.

[0072] First vertex coordinates: The coordinates of a vertex in the text region. The line connecting the first vertex and the second vertex is the diagonal of the text region.

[0073] Second vertex coordinates: The coordinates of another vertex in the text region. The line connecting the first and second vertices forms the diagonal of the text region.

[0074] Clustering: The process of dividing a collection of physical or abstract objects into multiple classes composed of similar objects.

[0075] Table column information: The number of columns in a three-line table, such as a three-line table with 3 columns.

[0076] First text information: The text information in each first text area.

[0077] The second region: The text area obtained after text detection in the table body; the second region is a rectangular area.

[0078] Second region information: The location information corresponding to the second region, which is at least the coordinates of two vertices on a diagonal.

[0079] Second text information: Text information in each second text area.

[0080] This application provides a method and apparatus for recognizing three-line table images, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0081] Figure 1A structural block diagram of a computing device 100 according to an embodiment of this application is shown. The components of the computing device 100 include, but are not limited to, a memory 110 and a processor 120. The processor 120 is connected to the memory 110 via a bus 130, and a database 150 is used to store data.

[0082] The computing device 100 also includes an access device 140, which enables the computing device 100 to communicate via one or more networks 160. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 140 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0083] In one embodiment of this application, the aforementioned components of the computing device 100 and Figure 1 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 1 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0084] The computing device 100 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 100 can also be a mobile or stationary server.

[0085] Among them, processor 120 can execute Figure 2 The steps in the three-line table image recognition method shown are as follows. Figure 2 A flowchart of a three-line table image recognition method according to an embodiment of this application is shown, including steps 202 to 208.

[0086] Step 202: Obtain the image to be identified, wherein the image to be identified includes a three-line table.

[0087] The image to be identified is the image waiting to be identified. In this application, the image to be identified is an image including a three-line table. A three-line table is a common table format used in scientific and technological publications. It usually has only 3 lines, namely the top line, the column line, and the bottom line.

[0088] In the embodiments provided in this application, see Figure 3a , Figure 3a The figure shows a schematic diagram of an image to be identified according to an embodiment of this application. The image to be identified includes a three-line table, as shown in the figure. The three line segments in the table from top to bottom are the top line, the column line, and the bottom line, respectively.

[0089] Step 204: Determine the key point information of the three-line table, and determine the top line, bottom line and column line of the three-line table based on the key point information.

[0090] The key information consists of the left and right endpoints of the top line, bottom line, and column line of a three-line table. A three-line table contains six key information points, which include the coordinates and type of each key point.

[0091] Optionally, determining the key point information of the three-line table includes: inputting the image to be recognized into a pre-trained key point localization network model; the key point localization network model determining the key point information of the three-line table based on the input image to be recognized.

[0092] The keypoint localization network model is a pre-trained neural network model, which can be a CNN network model. The keypoint localization network model is trained through the following steps: acquiring a training image and corresponding keypoint standard information; inputting the training image into the keypoint localization network model; the keypoint localization network model generating keypoint information of the training image in response to the training image as input; calculating a loss value based on the keypoint information and the keypoint standard information; and training the keypoint localization network model by backpropagating the loss value.

[0093] In a specific embodiment provided in this application, following the previous example, the image to be recognized is input into a pre-trained keypoint localization network model based on a CNN network model to obtain the keypoint information corresponding to the three-line table. See [link to relevant documentation]. Figure 3b , Figure 3bA schematic diagram of key points provided in the embodiments of this application is shown. As shown in the figure, the key points of the three-line table are six points A, B, C, D, E, and F. The key point information is the coordinates of the key points: A(X1, Y1), B(X2, Y1), C(X1, Y2), D(X2, Y2), E(X1, Y3), F(X2, Y3) and the type of the key points: A-left endpoint, B-right endpoint, C-left endpoint, D-right endpoint, E-left endpoint, F-right endpoint. Based on the coordinates and type of each key point, line segment AB is determined as the top line of the three-line table, line segment EF as the bottom line of the three-line table, and line segment CD as the column line of the three-line table.

[0094] It is important to note that since the image to be recognized may be offset, in order to improve recognition efficiency, in practical applications, the left and right endpoints can be considered as the same line segment if the height difference is less than a preset percentage of the image height. For example, if the difference in the y-axis between the left endpoint A and the right endpoint B is less than three percent of the image height, then the left endpoint A and the right endpoint B are considered as the same line segment. Only in the most ideal state are the left and right endpoints of the same line segment identical in y-coordinate, x-coordinate, and x-coordinate.

[0095] In another specific embodiment provided in this application, the key point information includes the coordinates of the key points: A(X1, Y1), B(X2, Y2), C(X3, Y3), D(X4, Y3), E(X5, Y4), F(X6, Y5) and the type of the key points: A-left endpoint, B-right endpoint, C-left endpoint, D-right endpoint, E-left endpoint, F-right endpoint. If the difference between the Y coordinates of the left endpoint A and the right endpoint B is less than 3% of the image height, then the left endpoint A and the right endpoint B are considered to be the same line segment. If the Y coordinates of the left endpoint C and the right endpoint D are the same, then the left endpoint C and the right endpoint D are considered to be the same line segment. If the difference between the Y coordinates of the left endpoint E and the right endpoint F is less than 3% of the image height, then the left endpoint E and the right endpoint F are considered to be the same line segment. The difference between the X coordinates of the left endpoints A, C, and E is less than a preset threshold, and the difference between the X coordinates of the right endpoints B, D, and F is less than a preset threshold.

[0096] Step 206: Perform text detection on the three-line table based on the top line, the bottom line, and the column line to determine the table column information, item column information, and table body information of the three-line table.

[0097] Optionally, text detection is performed on the three-line table based on the top line, the bottom line, and the column line to determine the table column information, item column information, and table body information of the three-line table, including S2062 to S2064.

[0098] S2062. Perform text detection on the item columns of the three-line table to determine the table column information and item column information of the three-line table, wherein the item column is determined by the top line and the column line.

[0099] The area between the top line and the column lines of a three-line table is called the item column. Text detection, such as OCR recognition, is performed on the image content in the item column to identify the table column information and item column information. The column information in the item column is the number of columns in the three-line table, such as three columns, four columns, etc., and the item column information is the specific text information in the item column.

[0100] Optionally, text detection is performed on the item columns of the three-line table to determine the table column information and item column information of the three-line table, including: performing text detection on the item columns of the three-line table to obtain multiple first regions and first region information corresponding to each first region; determining the table column information of the three-line table based on the information of each first region; and determining the item column information of the three-line table based on each first region.

[0101] In practical applications, text detection can be performed on the item column of a three-line table to obtain multiple first regions and the first region information corresponding to each first region. The first region refers to the text area obtained after text detection in the item column. The first region is a rectangular area, and the first region information is the position information corresponding to the first region, which includes at least the coordinates of the two vertices on the diagonal of the first region, such as the coordinates of the top left vertex and the bottom right vertex, or the coordinates of the bottom left vertex and the top right vertex.

[0102] The column information of the three-line table is determined based on the information of each first region, and the item column information of the three-line table is determined based on the information of each first region.

[0103] Specifically, the first region information includes the coordinates of the first vertex and the second vertex of the first region; determining the column information of the three-line table based on each first region information includes: determining the center point coordinates of each first region based on the coordinates of the first vertex and the second vertex of each first region; clustering the center point coordinates of each first region; and determining the column information of the three-line table based on the clustering results.

[0104] In practical applications, the first vertex is a vertex corresponding to the first region, and the line connecting the second vertex and the first vertex is the diagonal of the first region. The coordinates of the center point of the corresponding first region are determined based on the coordinates of the first vertex and the second vertex. Clustering is performed on the center point coordinates of each first region, such as K-means clustering algorithm, One-Pass Cluster clustering algorithm, etc. The column information of the three-line table is determined based on the clustering results.

[0105] Optionally, determining the item column information of the three-line table based on each of the first regions includes: performing text recognition on each of the first regions to obtain the first text information of each of the first regions; and determining the item column information of the three-line table based on each of the first text information.

[0106] In the embodiments provided in this application, the above example is used, see [link to example]. Figure 3c , Figure 3c A schematic diagram of the first area in the item bar provided in an embodiment of this application is shown. As shown in the figure, a total of 6 first areas are detected. The first area is the text area detected in the item bar. The first area is a rectangular area, as shown in Figure 3 for the 1-6 first areas. The first area information corresponding to each first area is information indicating the position of the first area, such as the coordinates of the four corners of the first area, the coordinates of the upper left corner and the lower right corner of the first area, the coordinates of the lower left corner and the upper right corner of the first area, etc.

[0107] The center point coordinates of each first region are determined based on the information of the first region. Each center point is then clustered using K-means clustering based on its X-axis coordinates. Clustering is performed based on the X-coordinates of the center points of first regions 1-6. Based on the clustering results, the three-line table is determined to have four columns. The first text information in each first region is then identified using OCR. The first regions of the same column in the item column are grouped together, and the corresponding first text information is saved to a cell. The item column information of the three-line table is then determined based on each piece of first text information.

[0108] like Figure 3c As shown, the first and second first regions are in the same column. The first text information identified in the first and second first regions is saved to the first cell. The third and fourth first regions are in the same column. The first text information identified in the third and fourth first regions is saved to the second cell. The first text identified in the fifth first region is saved to the third cell. The first text identified in the sixth first region is saved to the fourth cell. The item column information of the three-line table is generated based on the first to fourth cells.

[0109] S2064. Perform text detection on the body of the three-line table based on the column information of the table to determine the body information of the three-line table, wherein the body is determined by the column line and the bottom line.

[0110] The area between the column lines and the bottom line of a three-line table is called the table body. Text detection, such as OCR recognition, is performed on the image content in the table body to determine the table body information of the three-line table.

[0111] Optionally, text detection is performed on the body of the three-line table based on the table column information to determine the body information of the three-line table, including: performing text detection on the body of the three-line table to obtain multiple second regions and second region information corresponding to each second region; determining the position information corresponding to each second region based on the table column information and each second region information; performing text recognition on each second region to obtain second text information of each second region; and determining the body information of the three-line table based on the position information corresponding to each second region and each second text information.

[0112] The steps for performing text detection on the body of a three-line table are the same as those for performing text detection on the item columns of a three-line table. For details on performing text detection on the body of a three-line table, please refer to the description of performing text detection on the item columns of a three-line table above, which will not be repeated here.

[0113] Optionally, the second region information includes the coordinates of the first vertex and the second vertex of the second region; determining the position information corresponding to each second region based on the table column information and each second region information includes: determining the center point coordinates of each second region based on the coordinates of the first vertex and the second vertex of each second region; clustering the center point coordinates of each second region; and determining the position information corresponding to each second region based on the clustering results and the table column information.

[0114] Based on the information of each second region, the center point coordinates of each second region are determined. Clustering is performed based on the X-axis coordinates of each center point. Referring to the column count of the three-line table obtained in the previous steps, the number of columns corresponding to each second region is determined. Clustering is also performed based on the Y-axis coordinates of each center point. Based on the clustering results, the number of rows corresponding to each second region is determined, and thus the number of columns and rows corresponding to each second region is determined. The second text information in each second region is identified using ORC. Based on the number of columns, rows, and second text information corresponding to each second region, the table body information of the three-line table is determined.

[0115] Step 208: Generate a table corresponding to the image to be recognized based on the table column information, the item column information, and the table body information.

[0116] Finally, based on the project column information and the table body information, generate and output the table corresponding to the three-line table, and fill in the table column information into the table. The table can be an Excel spreadsheet.

[0117] The three-line table image recognition method provided in this application acquires an image to be recognized, including a three-line table. The three-line table is then processed through a key point localization network model to determine its key point information. The recognition accuracy of key points is greatly improved through the neural network model. The top line, bottom line, and column lines of the three-line table are then determined using the key point information, allowing for further inference of a more accurate table. Text detection is performed on the three-line table based on the top line, bottom line, and column lines to determine the table column information, item column information, and table body information, thus improving the accuracy of image recognition. Finally, the table corresponding to the image to be recognized is generated based on the table column information, item column information, and table body information, resulting in a more accurate final table and improving the accuracy of three-line table image recognition.

[0118] Figure 4 A three-line table image recognition method according to an embodiment of this application is shown, including steps 402 to 414.

[0119] Step 402: Obtain the image to be identified, wherein the image to be identified includes a three-line table.

[0120] In the embodiments provided in this application, see Figure 5 , Figure 5 A schematic diagram of the image to be identified is shown, which includes a three-line table.

[0121] Step 404: Input the image to be identified into a pre-trained keypoint localization network model.

[0122] In the embodiments provided in this application, Figure 5 The image to be identified is input into a key point localization network model, which is a CNN network model.

[0123] Step 406: The key point localization network model determines the key point information of the three-line table based on the input image to be identified.

[0124] In the embodiments provided in this application, the key point localization network model outputs six key point information, namely A(X1, Y1) - left endpoint, B(X2, Y1) - right endpoint, C(X1, Y2) - left endpoint, D(X2, Y2) - right endpoint, E(X1, Y3) - left endpoint, and F(X2, Y3) - right endpoint.

[0125] Step 408: Determine the top line, bottom line, and column lines of the three-line table based on the key point information.

[0126] In the embodiments provided in this application, line segment AB is defined as the top line of the three-line table, line segment CD as the column line of the three-line table, and line segment EF as the bottom line of the three-line table.

[0127] Step 410: Perform text detection on the item columns of the three-line table to determine the table column information and item column information of the three-line table.

[0128] In the embodiments provided in this application, text detection is performed on the item column between line segment AB and line segment CD to determine four first regions. Based on the X coordinates of the center points of the four first regions, the table column information of the item column is determined to be four columns. The item column information of each first region is identified as "January", "February", "March" and "Total".

[0129] Step 412: Perform text detection on the body of the three-line table based on the column information of the table to determine the body information of the three-line table.

[0130] In the embodiments provided in this application, text detection is performed on the table body between line segment CD and line segment EF. The specific steps are as described in 410, and the table body information of the three-line table is obtained as follows: “East”, “7”, “7”, “5”, “19”, “West”, “6”, “4”, “7”, “17”, “South”, “8”, “7”, “9”, “24”, “Total”, “21”, “18”, “21”, “60”.

[0131] The columns are as follows: "East", "West", "South", and "Total" form one column; "7", "6", "8", and "21" form another column, corresponding to "January" in the project column; "7", "4", "7", and "18" form another column, corresponding to "February" in the project column; "5", "7", "9", and "21" form another column, corresponding to "March" in the project column; and "19", "17", "24", and "60" form another column, corresponding to "Total" in the project column.

[0132] Step 414: Generate a table corresponding to the image to be recognized based on the item column information and the table body information.

[0133] In the embodiments provided in this application, a table corresponding to the image to be identified is generated based on the item column information and the table body information.

[0134] The three-line table image recognition method provided in this application acquires an image to be recognized, including a three-line table. The three-line table is then processed through a key point localization network model to determine its key point information. The recognition accuracy of key points is greatly improved through the neural network model. The top line, bottom line, and column lines of the three-line table are then determined using the key point information, allowing for further inference of a more accurate table. Text detection is performed on the three-line table based on the top line, bottom line, and column lines to determine the table column information, item column information, and table body information, thus improving the accuracy of image recognition. Finally, the table corresponding to the image to be recognized is generated based on the table column information, item column information, and table body information, resulting in a more accurate final table and improving the accuracy of three-line table image recognition.

[0135] Corresponding to the above method embodiments, this application also provides an embodiment of a three-line table image recognition device. Figure 6 A schematic diagram of the structure of a three-line table image recognition device according to an embodiment of this application is shown. Figure 6 As shown, the device includes:

[0136] The acquisition module 602 is configured to acquire an image to be recognized, wherein the image to be recognized includes a three-line table;

[0137] The determination module 604 is configured to determine the key point information of the three-line table, and determine the top line, bottom line and column line of the three-line table based on the key point information;

[0138] The detection module 606 is configured to perform text detection on the three-line table based on the top line, the bottom line and the column line, and determine the table column information, item column information and table body information of the three-line table;

[0139] The generation module 608 is configured to generate a table corresponding to the image to be recognized based on the table column information, the item column information, and the table body information.

[0140] Optionally, the determining module 604 is further configured to input the image to be recognized into a pre-trained keypoint localization network model; the keypoint localization network model determines the keypoint information of the three-line table based on the input image to be recognized.

[0141] Optionally, the keypoint localization network model is trained through the following steps:

[0142] Obtain the training image and the corresponding key point standard information of the training image;

[0143] The training images are input into the keypoint localization network model;

[0144] The keypoint localization network model generates keypoint information of the training image in response to the training image as input;

[0145] Calculate the loss value based on the key point information and the key point standard information;

[0146] The keypoint localization network model is trained by backpropagating the loss value.

[0147] Optionally, the detection module 606 includes:

[0148] The first detection subunit is configured to perform text detection on the item column of the three-line table to determine the table column information and item column information of the three-line table, wherein the item column is determined by the top line and the column line;

[0149] The second detection subunit is configured to perform text detection on the body of the three-line table based on the table column information to determine the body information of the three-line table, wherein the body is determined by the column line and the bottom line.

[0150] Optionally, the first detection subunit is further configured to perform text detection on the item column of the three-line table to obtain multiple first regions and first region information corresponding to each first region; determine the table column information of the three-line table based on the information of each first region; and determine the item column information of the three-line table based on each first region.

[0151] Optionally, the first region information includes the coordinates of the first vertex and the second vertex of the first region;

[0152] Optionally, the first detection subunit is further configured to determine the center point coordinates of each first region based on the first vertex coordinates and the second vertex coordinates of each first region; cluster the center point coordinates of each first region; and determine the table column information of the three-line table based on the clustering results.

[0153] Optionally, the first detection subunit is further configured to perform text recognition on each of the first regions to obtain first text information for each of the first regions; and to determine the item column information of the three-line table based on each of the first text information.

[0154] Optionally, the second detection subunit is further configured to perform text detection on the body of the three-line table to obtain multiple second regions and second region information corresponding to each second region; determine the position information corresponding to each second region based on the table column information and the information of each second region; perform text recognition on each second region to obtain second text information of each second region; and determine the body information of the three-line table based on the position information corresponding to each second region and the second text information.

[0155] Optionally, the second region information includes the coordinates of the first vertex and the second vertex of the second region;

[0156] The second detection subunit is further configured to determine the center point coordinates of each second region based on the first vertex coordinates and the second vertex coordinates of each second region; cluster the center point coordinates of each second region; and determine the location information corresponding to each second region based on the clustering results and the table column information.

[0157] The three-line table image recognition device provided in this application acquires an image to be recognized, including a three-line table. It then uses a key point localization network model to determine the key point information of the three-line table. The recognition accuracy of key points is greatly improved through the neural network model. Furthermore, the top line, bottom line, and column lines of the three-line table are determined using the key point information, allowing for a more accurate table deduction. Text detection is performed on the three-line table based on the top line, bottom line, and column lines to determine the table column information, item column information, and table body information, thus improving the accuracy of image recognition. Finally, the table corresponding to the image to be recognized is generated based on the table column information, item column information, and table body information, resulting in a more accurate final table and improving the accuracy of three-line table image recognition.

[0158] In one embodiment of this application, a computing device is also provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor executes the instructions to implement the steps of the three-line table image recognition method.

[0159] An embodiment of this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the three-line table image recognition method described above.

[0160] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the three-line table image recognition method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the three-line table image recognition method described above.

[0161] This application discloses a chip that stores computer instructions, which, when executed by a processor, implement the steps of the three-line table image recognition method described above.

[0162] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0163] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0164] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0165] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0166] The preferred embodiments disclosed above are merely illustrative of this application. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this application. These embodiments are selected and specifically described in this application to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents.

Claims

1. A method for recognizing three-line table images, characterized in that, include: Obtain the image to be identified, wherein the image to be identified includes a three-line table; Determine the key point information of the three-line table, and determine the top line, bottom line, and column lines of the three-line table based on the key point information; The text detection of the three-line table is performed based on the top line, the bottom line, and the column line to determine the table column information, item column information, and table body information of the three-line table. This process includes: performing text detection on the item column and table body of the three-line table respectively to determine the table column information, item column information, and table body information. The item column is determined by the top line and the column line, and the table body is determined by the column line and the bottom line. The table column information is the number of columns in the three-line table, and the item column information is the text information within the item column. A table corresponding to the image to be identified is generated based on the table column information, the item column information, and the table body information.

2. The three-line table image recognition method as described in claim 1, characterized in that, Determine the key point information of the three-line table, including: The image to be identified is input into a pre-trained keypoint localization network model; The key point localization network model determines the key point information of the three-line table based on the input image to be identified.

3. The three-line table image recognition method as described in claim 2, characterized in that, The keypoint localization network model is trained through the following steps: Obtain the training image and the corresponding key point standard information of the training image; The training images are input into the keypoint localization network model; The keypoint localization network model generates keypoint information of the training image in response to the training image as input; Calculate the loss value based on the key point information and the key point standard information; The keypoint localization network model is trained by backpropagating the loss value.

4. The three-line table image recognition method as described in claim 1, characterized in that, Based on the top line, the bottom line, and the column line, text detection is performed on the three-line table to determine the table column information, item column information, and table body information, including: Text detection is performed on the item columns of the three-line table to determine the table column information and item column information of the three-line table, wherein the item column is determined by the top line and the column line; Based on the column information of the table, text detection is performed on the body of the three-line table to determine the body information of the three-line table, wherein the body is determined by the column line and the bottom line.

5. The three-line table image recognition method as described in claim 4, characterized in that, Perform text detection on the item columns of the three-line table to determine the table column information and item column information of the three-line table, including: Text detection is performed on the item columns of the three-line table to obtain multiple first regions and the first region information corresponding to each first region; The column information of the three-line table is determined based on the information of each of the first regions; The item column information of the three-line table is determined based on each of the first regions.

6. The three-line table image recognition method as described in claim 5, characterized in that, The first region information includes the coordinates of the first vertex and the second vertex of the first region; The table column information of the three-line table is determined based on the information of each of the first regions, including: The center point coordinates of each of the first regions are determined based on the coordinates of the first vertex and the second vertex of each of the first regions. Cluster the center point coordinates of each of the first regions; The column information of the three-line table is determined based on the clustering results.

7. The three-line table image recognition method as described in claim 5, characterized in that, The item column information of the three-line table is determined based on each of the first regions, including: Perform text recognition on each of the first regions to obtain the first text information of each of the first regions; The item column information of the three-line table is determined based on each of the first text information.

8. The three-line table image recognition method as described in claim 4, characterized in that, Based on the column information of the table, text detection is performed on the body of the three-line table to determine the body information of the three-line table, including: Text detection is performed on the body of the three-line table to obtain multiple second regions and the second region information corresponding to each second region; The location information corresponding to each second region is determined based on the table column information and the information of each second region. Perform text recognition on each of the second regions to obtain the second text information of each second region; The table body information of the three-line table is determined based on the location information corresponding to each of the second regions and each of the second text information.

9. The three-line table image recognition method as described in claim 8, characterized in that, The second region information includes the coordinates of the first vertex and the second vertex of the second region; Based on the table column information and the information of each second region, determine the location information corresponding to each second region, including: The center point coordinates of each second region are determined based on the coordinates of the first and second vertices of each second region. Cluster the center point coordinates of each of the second regions; The location information corresponding to each of the second regions is determined based on the clustering results and the information in the table columns.

10. A three-line table image recognition device, characterized in that, include: The acquisition module is configured to acquire an image to be recognized, wherein the image to be recognized includes a three-line table; The determination module is configured to determine the key point information of the three-line table, and determine the top line, bottom line and column line of the three-line table based on the key point information; The detection module is configured to perform text detection on the three-line table based on the top line, the bottom line, and the column line to determine the table column information, item column information, and table body information of the three-line table. The step of performing text detection on the three-line table based on the top line, the bottom line, and the column line to determine the table column information, item column information, and table body information includes: performing text detection on the item column and table body of the three-line table respectively to determine the table column information, item column information, and table body information of the three-line table. The item column is determined by the top line and the column line, and the table body is determined by the column line and the bottom line. The table column information is the number of columns in the three-line table, and the item column information is the text information in the item column. The generation module is configured to generate a table corresponding to the image to be recognized based on the table column information, the item column information, and the table body information.

11. A computing device, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the instructions, it implements the steps of the method according to any one of claims 1-9.

12. A computer-readable storage medium storing computer instructions, characterized in that, When executed by the processor, this instruction implements the steps of the method according to any one of claims 1-9.

Images