Table recognition method, device, electronic device and storage medium

Abstract

The present disclosure provides a table recognition method, device, electronic device, and storage medium, wherein the method comprises: acquiring a target document image; performing cell detection on a target table in the target document image to determine the coordinate information of each cell in the target table; and performing text detection on the target document image to determine the text content corresponding to each cell in the target table; determining the arrangement relationship between each cell based on the coordinate information of each cell in the target table; and combining the text content of the corresponding cells according to the arrangement relationship to obtain a table recognition result for the target document image. The present disclosure performs table recognition based on the minimum recognition dimension of cells, which can ensure the integrity of the recognized table. At the same time, since the reconstructed table recognition result is generated according to the arrangement relationship, it is more standardized.

Description

Technical Field

[0001] The present disclosure relates to the field of image recognition technology, and in particular to a table recognition method, device, electronic device, and storage medium. Background Art

[0002] Tables are a common and crucial form of expression in documents. They can segment complex document information into multiple cell components and organize cells into standardized correspondences, facilitating quick information retrieval. Therefore, table recognition is a crucial task in document layout understanding.

[0003] Generally, a table can be processed by manual entry or traditional simple rules. For example, a region of interest at a specific position of a table can be obtained through a pre-configured table template, and then the region of interest can be processed in real time.

[0004] However, in actual applications, the formats and contents of tables vary greatly. Whether it is manual entry or recognition through manual configuration of templates, it will bring a large workload and cannot ensure a high recognition accuracy. Summary of the Invention

[0005] The embodiments of the present disclosure at least provide a table recognition method, device, electronic device, and storage medium.

[0006] In a first aspect, an embodiment of the present disclosure provides a table recognition method, comprising:

[0007] Obtain target document image;

[0008] Performing cell detection on the target table in the target document image to determine coordinate information of each cell in the target table; and performing text detection on the target document image to determine text content corresponding to each cell in the target table;

[0009] Determining an arrangement relationship between the cells based on coordinate information of the cells in the target table;

[0010] The text contents of the corresponding cells are combined according to the arrangement relationship to obtain a table recognition result for the target document image.

[0011] By adopting the above table recognition method, when the target document image is obtained, on the one hand, the target table in the target document image can be spaced by cells, and on the other hand, the target document image can be detected. In this way, when the arrangement relationship between each cell is determined based on the coordinate information of each cell in the target table, the text content of the cell can be combined according to the arrangement relationship, so that the table recognition result can be obtained. The present disclosure performs table recognition based on the minimum recognition dimension of the cell, which can ensure the integrity of the recognized table. At the same time, in the process of reconstructing the table based on the arrangement relationship between each cell, since the arrangement relationship is determined based on the coordinate information of the cell, the accuracy of the arrangement position of each cell can be guaranteed, and the reconstructed table recognition result is generated according to the arrangement relationship, which is more standardized.

[0012] In a possible implementation, before performing cell detection on the target table in the target document image, the method further includes:

[0013] Performing corner point detection based on the target document image to determine image position information of a plurality of corner points included in a target table in the target document image and corner point sequence information between the corner points;

[0014] The target table is corrected based on the image position information of the multiple corner points in the target document image and the corner point sequence information between the corner points to obtain the image position range information of the table image area corresponding to the processed target table in the target document image.

[0015] Here, before performing cell detection on the target table, we can first combine corner detection and correction processing to obtain the image position range information of the table image area corresponding to the target table in the target document image, so as to facilitate the subsequent extraction of the target table from the target document image.

[0016] In a possible implementation, performing cell detection on the target table in the target document image to determine coordinate information of each cell in the target table includes:

[0017] extracting a table image area corresponding to the target table from the target document image based on the image position range information;

[0018] Performing cell detection on the table image area corresponding to the target table to determine image position information of each cell in the table image area;

[0019] The image position information of each cell in the table image area is determined as the coordinate information of each cell in the target table.

[0020] Here, the coordinate information of each cell can be determined by using the cell detection method of the table image area, and the detection accuracy and efficiency are both high.

[0021] In a possible implementation, performing cell detection on the table image area corresponding to the target table to determine image position information of each cell in the table image area includes:

[0022] Performing cell detection on the table image area corresponding to the target table using the trained cell detection neural network to determine image position information of each cell in the table image area;

[0023] The cell detection neural network is trained by image samples and position annotation results obtained by performing position annotation on different cells in the image samples.

[0024] The cell detection here can be implemented based on a trained cell detection neural network. Since the cell detection neural network can dig out deeper image features, it further improves the detection accuracy while ensuring detection efficiency.

[0025] In one possible implementation, the cell detection neural network is trained according to the following steps:

[0026] Get image samples;

[0027] Inputting the image sample into a multi-level feature extraction layer included in a cell detection neural network to be trained, and determining cell feature information output by each level of feature extraction layer; each level of feature extraction layer correspondingly extracts feature information of cells within a size range;

[0028] Inputting the cell feature information into a prediction layer connected to a corresponding hierarchical feature extraction layer, determining a prediction result output by the prediction layer, and matching the prediction result with a position annotation result of the cell corresponding to the corresponding hierarchical feature extraction layer;

[0029] In response to the prediction result not matching the position labeling result of the cell corresponding to the corresponding hierarchical feature extraction layer, the network parameter value of the cell detection neural network is adjusted until the prediction result matches the position labeling result of the cell corresponding to the corresponding hierarchical feature extraction layer.

[0030] The cell detection neural network here can use different levels of feature extraction layers to extract feature information of cells in different size ranges. That is, cells in different size ranges can be assigned to feature extraction layers at different levels for feature extraction. In this way, the extracted features can be more conducive to detecting cells of different sizes, preventing the problem of missed cell detection, and further ensuring the accuracy of detection.

[0031] In a possible implementation, performing corner point detection based on the target document image to determine image position information of a plurality of corner points included in the target table in the target document image includes:

[0032] Performing table detection on the target document image to determine a candidate rectangular area where the target table is located; the candidate rectangular area is a rectangular area formed by a plurality of candidate corner points;

[0033] Performing corner detection on the candidate rectangular area based on a trained offset detection neural network to determine the offset between each corner point in the target table and the corresponding candidate corner point;

[0034] For each corner point among the multiple corner points included in the target table, the image position information of the corner point in the target document image is determined based on the offset between the corner point and the corresponding candidate corner point and the image position information of the candidate corner point in the target document image.

[0035] Here, the table detection and corner point detection can be combined to determine the offset corresponding to each corner point included in the target table. On the one hand, the offset is used to indicate the deviation of the target table relative to the candidate rectangular area, so as to facilitate the determination of the image position information corresponding to the target table based on the image position information corresponding to the candidate rectangular area. On the other hand, it is used to indicate the degree of table correction required. For example, in the case of a relatively large offset, a larger degree of correction is required, which is more convenient for subsequent table correction.

[0036] In a possible implementation, determining the arrangement relationship between the cells based on the coordinate information of the cells in the target table includes:

[0037] For a first cell among the cells, determining a structural relationship between the first cell and other adjacent cells based on coordinate information of the cells in the target table;

[0038] Determining an arrangement position of each of the cells in the target table based on the structural relationships determined for the cells;

[0039] An arrangement relationship between the cells is determined based on an arrangement position of each of the cells in the target table.

[0040] Here, the arrangement position of each cell in the target table can be determined based on the structural relationship between each cell and other adjacent cells. The arrangement position here can be used to fill in the text content of the corresponding cell, which ensures the integrity and standardization of the table recognition results.

[0041] In a possible implementation, the structural relationship includes a horizontal structural relationship and a vertical structural relationship; and determining the structural relationship between the first cell and other adjacent cells based on the coordinate information of each cell in the target table includes:

[0042] determining a horizontal structural relationship between the first cell and other adjacent cells based on a vertical coordinate range included in the coordinate information of each cell in the target table; and

[0043] Based on the horizontal coordinate range included in the coordinate information of each cell in the target table, a vertical structural relationship between the first cell and other adjacent cells is determined.

[0044] In a possible implementation, combining the text contents of corresponding cells according to the arrangement relationship to obtain a table recognition result for the target document image includes:

[0045] For each of the plurality of cells, according to the arrangement position of the cell in the target table indicated by the arrangement relationship, fill the text content corresponding to the cell into the arrangement position;

[0046] When the text content is filled in based on the arrangement positions indicated by the plurality of cells, a table recognition result for the target document image is obtained.

[0047] In a second aspect, an embodiment of the present disclosure further provides a table recognition device, comprising:

[0048] An acquisition module, used for acquiring a target document image;

[0049] a detection module configured to perform cell detection on a target table in the target document image to determine coordinate information of each cell in the target table; and perform text detection on the target document image to determine text content corresponding to each cell in the target table;

[0050] a determination module, configured to determine an arrangement relationship between the cells based on coordinate information of the cells in the target table;

[0051] The recognition module is used to combine the text contents of the corresponding cells according to the arrangement relationship to obtain a table recognition result for the target document image.

[0052] In a third aspect, an embodiment of the present disclosure further provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the table recognition method as described in any one of the first aspect and its various embodiments are performed.

[0053] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the table recognition method as described in the first aspect and any of its various embodiments are executed.

[0054] For a description of the effects of the above-mentioned table recognition device, electronic device, and computer-readable storage medium, please refer to the description of the above-mentioned table recognition method, which will not be repeated here.

[0055] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0057] Figure 1 A flow chart of a table recognition method provided by an embodiment of the present disclosure is shown;

[0058] Figure 2 A schematic diagram of reconstructing a table in the table recognition method provided by an embodiment of the present disclosure is shown;

[0059] Figure 3 A schematic diagram of a table recognition device provided by an embodiment of the present disclosure is shown;

[0060] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0062] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0063] The term "and / or" herein simply describes an association relationship, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, and the existence of B alone. In addition, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0064] Research has found that tables can be processed through manual entry or traditional simple rules. For example, the area of ​​interest at a specific location in the table can be obtained through a pre-configured table template, and then the area of ​​interest can be processed in real time.

[0065] However, in actual applications, the formats and contents of tables vary greatly. Whether it is manual entry or recognition through manual configuration of templates, it will bring a large workload and cannot ensure a high recognition accuracy.

[0066] To address the aforementioned issues, several deep learning-based solutions exist for identifying table structures. These solutions, after identifying all rows and columns in a table, extract the data within the table cells, resulting in a standard reconstructed table. These solutions primarily include the following: First, table structure identification is considered the prediction of row and column separators. In the row dimension, several separators are predicted as row separators, and similarly for columns. However, this approach is inapplicable to tables with real-world tilts or curves. Furthermore, this approach determines the presence of a separator based on whether it passes through text, potentially separating merged empty cells. Second, the table structure is parsed into Hypertext Markup Language (HTML) encoding, directly using sequence encoding and decoding to predict the HTML sequence corresponding to the table image. However, this approach places few constraints on the table structure itself, resulting in unstable predictions and a high risk of incorrect rows and columns. As can be seen, all of the aforementioned methods have significant shortcomings.

[0067] Based on the above research, the present disclosure provides at least one solution for realizing table recognition based on cell detection, and the recognized table has better integrity and is more standardized.

[0068] To facilitate understanding of this embodiment, a table recognition method disclosed in an embodiment of the present disclosure is first introduced in detail. The table recognition method provided in the embodiment of the present disclosure is generally executed by an electronic device with certain computing capabilities, such as a terminal device, a server, or other processing device. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a personal digital assistant (PDA), a handheld device, a computing device, etc. In some possible implementations, the table recognition method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0069] See also Figure 1 FIG. 1 is a flowchart of a table recognition method provided by an embodiment of the present disclosure, and the method includes steps S101 to S104, wherein:

[0070] S101: Acquire a target document image;

[0071] S102: Performing cell detection on the target table in the target document image to determine the coordinate information of each cell in the target table; and performing text detection on the target document image to determine the text content corresponding to each cell in the target table;

[0072] S103: Determine the arrangement relationship between the cells based on the coordinate information of the cells in the target table;

[0073] S104: combining the text contents of corresponding cells according to the arrangement relationship to obtain a table recognition result for the target document image.

[0074] To facilitate understanding of the table recognition method provided by the embodiments of this disclosure, we will first describe its application scenarios. The table recognition method provided by the embodiments of this disclosure can be applied to any application scenario requiring table recognition in an image. In practical applications, it can be widely used in tasks such as recognizing invoices, checks, and contracts in various industries, including government, taxation, insurance, healthcare, finance, and factories and mines.

[0075] The table recognition method provided by the embodiment of the present disclosure can, on the one hand, be used to recognize cells in a table, and on the other hand, can be used to recognize text content corresponding to the cells, and then the entire table can be reconstructed.

[0076] The target document images obtained here vary depending on the recognition task. For example, they may include images of invoices, images of contract documents, or other document images, without any specific limitation. In practical applications, images of bank invoices, contract documents, etc., may be obtained by photographing or scanning them using a camera, scanner, or other device.

[0077] To reconstruct the target table in the target document image, cell recognition can be performed on the target table to determine the coordinate information of each cell in the target table. Furthermore, text detection can be performed on the target document image to determine the text content corresponding to each cell in the target table. Once the arrangement relationship between the cells is determined based on the coordinate information, the text content of the corresponding cells can be combined according to this arrangement relationship to obtain a table recognition result for the entire target table.

[0078] The arrangement relationship may correspond to the row and column positions of each cell in the target table. That is, when the row and column positions of each cell in the target table are determined, the target table may be reconstructed in combination with the text content of the cell.

[0079] In the embodiment of the present disclosure, the arrangement relationship can be determined based on the coordinate information of each cell obtained by cell detection of the target table in the target document image. This is mainly due to the fact that there are certain inclusion and being included relationships between the coordinate information of cells corresponding to different arrangement positions. Utilizing these relationships will help to reconstruct a more standardized and normalized table.

[0080] The coordinate information here may be determined based on cell detection of the target table in the target document image. In practical applications, the coordinate information here may be image position information of each cell determined by cell detection of the table image area corresponding to the target table.

[0081] Before performing cell detection, the disclosed embodiments can first perform table location, that is, can detect the table image area corresponding to the target table from the target document image. The target table can be one or more, and can be any table in the target document image, all tables in the target image, or a specific table in the target document image, without specific limitation.

[0082] During the cell detection process, text detection can also be performed to determine the text content corresponding to each cell. In practical applications, the text content corresponding to the cell can be recognized from the target document image based on optical character recognition (OCR).

[0083] In order to achieve the reorganization between the cell arrangement and the text content in the cell, here we can determine the arrangement relationship between each cell based on the coordinate information of each cell in the target table, and then combine the text content of the corresponding cells according to the arrangement relationship to obtain the table recognition result.

[0084] In the process of combining text content according to the arrangement relationship, the text content of each cell can be added to the corresponding arrangement position, so that a table recognition result including the table content can be obtained, making the table structure more standardized. In addition, the arrangement relationship here is more simplified, which will further improve the efficiency of the relevant table recognition.

[0085] Considering the key role of cell detection in the table recognition method provided by the embodiment of the present disclosure, the following will specifically describe the relevant content related to detecting cells.

[0086] In the embodiment of the present disclosure, the cells in the target table can be detected according to the following steps:

[0087] Step 1: performing corner point detection based on the target document image to determine image position information of multiple corner points included in the target table in the target document image and corner point sequence information between the corner points;

[0088] Step 2: Based on the image position information of the multiple corner points in the target document image and the corner point sequence information between the corner points, the target table is corrected to obtain the image position range information of the table image area corresponding to the processed target table in the target document image;

[0089] Step 3: extracting a table image area corresponding to the target table from the target document image based on the image position range information;

[0090] Step 4: Perform cell detection on the table image area corresponding to the target table to determine the image position information of each cell in the table image area;

[0091] Step 5: Determine the image position information of each cell in the table image area as the coordinate information of each cell in the target table.

[0092] Here, before performing cell detection, table positioning is required, that is, the table image area corresponding to the target table can be extracted from the target document image based on the image position range information of the table image area corresponding to the target table in the target document image, and then cell detection is performed on this table image area, and then the coordinate information of each cell in the target table can be obtained.

[0093] During table location, corner point detection can be combined to determine the image position information of multiple corner points included in the target table in the target document image. Based on this image position information, the position of the target table in the target document image can be preliminarily determined. In the embodiment of the present disclosure, the image position information of multiple corner points included in the target table in the target document image can be determined specifically through the following steps:

[0094] Step 1: Perform table detection on the target document image to determine the candidate rectangular area where the target table is located; the candidate rectangular area is a rectangular area composed of multiple candidate corner points;

[0095] Step 2: Perform corner detection on the candidate rectangular area based on the trained offset detection neural network, and determine the offset between each corner point in the target table and the corresponding candidate corner point;

[0096] Step 3: For each corner point in the target table, determine the image position information of the corner point in the target document image based on the offset between the corner point and the corresponding candidate corner point and the image position information of the candidate corner point in the target document image.

[0097] Here, we can first determine the candidate rectangular area where the target table is located based on table detection, and then use the trained offset detection neural network to perform corner detection on the candidate rectangular area. Finally, we can determine the image position information of multiple corner points included in the target table based on the offset and the image position information of the candidate rectangular area.

[0098] In the process of table positioning, it can be specifically implemented based on the detection algorithm of an arbitrary quadrilateral. First, a candidate area of ​​the target rectangle (i.e., the candidate rectangular area) can be generated. Then, the offset of the four corner points of the target table on the edge of the candidate rectangular area can be predicted. The candidate rectangular area is combined with the coordinate offset to form an arbitrary quadrilateral, and the four vertices of the arbitrary quadrilateral correspond to the four corner points based on the target table. It can be seen that the table positioning method provided by the embodiment of the present application can be used to locate a table at any position.

[0099] It should be noted that the offset detection described above can be determined using a trained offset detection neural network. The offset detection network can be trained using table pattern samples labeled with corner point offsets and corner point order. By comparing the offsets output by the network with the labeled offsets, the offset detection network can be trained multiple times until network convergence is achieved, resulting in a trained offset detection network. In this way, once a candidate rectangular area is determined, the offset between each of the multiple corner points included in the target table and the corresponding candidate corner point can be quickly determined directly based on the trained offset detection network.

[0100] During the table positioning process, it is possible to predict which of the four corner points of the target table is the upper left corner point of the table itself, thereby determining the direction of the table. The table image area can then be corrected through perspective transformation. For example, the table can be rotated according to the determined direction to obtain the corrected target table.

[0101] During the target table correction process, the table image area corresponding to the target table also changes accordingly, and then the image position range information of the corrected target table in the target document image can be determined.

[0102] In the disclosed embodiment, a table image area corresponding to a target table can be extracted from a target document image based on image position range information, and then image position information of each cell in the table image area can be determined based on cell detection.

[0103] In the embodiment of the present disclosure, a trained cell detection neural network may be used to perform cell detection on a table image region corresponding to a target table to determine image position information of each cell in the table image region.

[0104] Among them, cell detection in the embodiment of the present disclosure refers to inputting the table image area corresponding to a single target table after normalization, and outputting the image position information of each cell in the table image area. For dense scenes with many cells, the cell detection neural network in the embodiment of the present disclosure can use a cascade network with a feature pyramid structure to build sufficient preset boxes to cover all cell scenes, and assign cells of different sizes to different feature layers, thereby improving the recall rate of the neural network for cell detection and preventing cells from being missed.

[0105] The cell detection neural network here can be trained as follows:

[0106] Step 1: Obtain image samples;

[0107] Step 2: Input the image sample into the multi-level feature extraction layer included in the cell detection neural network to be trained, and determine the cell feature information output by each level feature extraction layer; each level feature extraction layer correspondingly extracts feature information of cells within a size range;

[0108] Step 3: Input the cell feature information into the prediction layer connected to the corresponding hierarchical feature extraction layer, determine the prediction result output by the prediction layer, and match the prediction result with the position annotation result of the cell corresponding to the corresponding hierarchical feature extraction layer;

[0109] Step 4: In response to the prediction result not matching the position labeling result of the cell corresponding to the corresponding hierarchical feature extraction layer, the network parameter values ​​of the cell detection neural network are adjusted until the prediction result matches the position labeling result of the cell corresponding to the corresponding hierarchical feature extraction layer.

[0110] The cell detection neural network here can include multiple levels of feature extraction layers, and each level of feature extraction layer extracts feature information of cells within a size range. When the cell feature information output by each level of feature extraction layer is determined, the cell feature information can be input into the prediction layer connected to the corresponding level of feature extraction layer to determine the prediction result output by the prediction layer, and a matching verification is performed between the prediction result and the position annotation result of the cell corresponding to the corresponding level of feature extraction layer. If the two results do not match, it can be explained to a certain extent that the detection ability of the corresponding layer is weaker, and the network needs to be trained again. If the two results match, it can be explained to a certain extent that the detection ability of the corresponding layer is stronger. In this case, the network training can be terminated.

[0111] In specific applications, a feature pyramid structure comprising three levels of feature extraction layers can be used to extract features from cells of different size ranges. For example, for a higher-level feature extraction layer, the corresponding feature dimension is smaller, and the feature information that can be extracted is deeper, which can be adapted to feature extraction of larger-sized cells. For another example, for a lower-level feature extraction layer, the corresponding feature dimension is larger, and the feature information that can be extracted is shallower, which can be adapted to feature extraction of smaller-sized cells.

[0112] After determining the coordinate information of each cell in the target table based on cell detection, the arrangement relationship between the cells can be determined. This can be achieved through the following steps:

[0113] Step 1: for a first cell among the cells, determine the structural relationship between the first cell and other adjacent cells based on the coordinate information of each cell in the target table;

[0114] Step 2: Based on the structural relationships determined for each cell, determine the arrangement position of each cell in the target table;

[0115] Step 3: Based on the arrangement position of each cell in the target table, determine the arrangement relationship between the cells.

[0116] The first cell here can be every cell in the target table, any cell in the target table, or a specific cell in the target table. In a specific application, every cell can be selected as the first cell.

[0117] For each cell, a structural relationship between the first cell and other adjacent cells can be determined based on the coordinate information of each cell in the target table. The structural relationship here can include a relationship of contains, is contained by, intersects, is equal, etc. Then, based on the structural relationship determined for each cell, the arrangement position of each cell in the target table can be determined to determine the arrangement relationship based on the arrangement position.

[0118] In the process of determining the structural relationship, the determination can be made in both horizontal and vertical directions. That is, the horizontal structural relationship between a first cell and other adjacent cells can be determined based on the vertical coordinate range included in the coordinate information of each cell in the target table; and the vertical structural relationship between the first cell and other adjacent cells can be determined based on the horizontal coordinate range included in the coordinate information of each cell in the target table.

[0119] When the horizontal and vertical structural relationships of the cells are determined, the table rows and columns can be reconstructed accordingly.

[0120] In order to understand the process of the above table reconstruction, you can combine Figure 2 Provide an example.

[0121] like Figure 2 As shown, for a target table including 2 rows and 4 columns, the target table has 5 cells. When the coordinate information of each cell is determined, the horizontal structural relationship and the vertical structural relationship between each of the 5 cells and the adjacent cells can be determined.

[0122] Taking the construction of horizontal structural relationships as an example, for cell A in the 1st row and 1st column, the adjacent cells of cell A are cell B and cell C. Since the horizontal coordinate information of cell A contains the horizontal coordinate information of cell B and cell C, cell A here contains cell B and cell C. Similarly, it can be determined that cell C and cell D belong to the same structural relationship. By analogy, the structural relationship between each cell and its adjacent cells can be determined.

[0123] In this way, when the minimum unit cell (i.e., a cell that is not contained by any cell) is determined, the above-mentioned structural relationship can be used to construct the arrangement relationship between each cell. The minimum unit cells here are cells B, C, and D. Cell B here is contained in cell A and cell E, cell C is equal to cell D and is contained in cell A, and cell D is contained in cell E. Then the horizontal arrangement position of each cell can be determined.

[0124] According to the above-mentioned method of constructing the horizontal structural relationship, the vertical structural relationship can be constructed in the same way, so as to determine the arrangement relationship between each cell.

[0125] In the embodiment of the present disclosure, based on the above-reconstructed arrangement relationship, for each cell, the text content corresponding to the cell can be filled into the arrangement position according to the arrangement position of the cell in the target table indicated by the arrangement relationship, and then the table recognition result for the target document image can be obtained.

[0126] In specific applications, the table recognition results can be converted into Excel format to facilitate better table editing and other operations, which has better practicality.

[0127] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0128] Based on the same inventive concept, a table recognition device corresponding to the table recognition method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned table recognition method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0129] Reference Figure 3 FIG. 1 is a schematic diagram of a table recognition device provided by an embodiment of the present disclosure, the device comprising: an acquisition module 301, a detection module 302, a determination module 303 and a recognition module 304; wherein,

[0130] An acquisition module 301 is used to acquire a target document image;

[0131] Detection module 302, configured to perform cell detection on the target table in the target document image to determine the coordinate information of each cell in the target table; and perform text detection on the target document image to determine the text content corresponding to each cell in the target table;

[0132] Determining module 303, for determining the arrangement relationship between each cell based on the coordinate information of each cell in the target table;

[0133] The recognition module 304 is configured to combine the text contents of corresponding cells according to the arrangement relationship to obtain a table recognition result for the target document image.

[0134] By adopting the above-mentioned table recognition device, when the target document image is acquired, on the one hand, the target table in the target document image can be spaced by cells, and on the other hand, the target document image can be detected. In this way, when the arrangement relationship between each cell is determined based on the coordinate information of each cell in the target table, the text content of the cell can be combined according to the arrangement relationship, so that the table recognition result can be obtained. The present disclosure performs table recognition based on the minimum recognition dimension of the cell, which can ensure the integrity of the recognized table. At the same time, in the process of reconstructing the table based on the arrangement relationship between each cell, since the arrangement relationship is determined based on the coordinate information of the cell, the accuracy of the arrangement position of each cell can be guaranteed, and the reconstructed table recognition result is generated according to the arrangement relationship, which is more standardized.

[0135] In a possible implementation, the detection module 302 is further configured to:

[0136] Before performing cell detection on the target table in the target document image, performing corner point detection based on the target document image to determine image position information of multiple corner points included in the target table in the target document image and corner point sequence information between the corner points;

[0137] The target table is corrected based on the image position information of multiple corner points in the target document image and the corner point sequence information between each corner point to obtain the image position range information of the table image area corresponding to the processed target table in the target document image.

[0138] In a possible implementation, the detection module 302 is configured to perform cell detection on the target table in the target document image to determine coordinate information of each cell in the target table according to the following steps:

[0139] Based on the image position range information, extracting a table image area corresponding to the target table from the target document image;

[0140] Performing cell detection on the table image area corresponding to the target table to determine the image position information of each cell in the table image area;

[0141] The image position information of each cell in the table image area is determined as the coordinate information of each cell in the target table.

[0142] In a possible implementation, the detection module 302 is configured to perform cell detection on the table image area corresponding to the target table to determine image position information of each cell in the table image area according to the following steps:

[0143] Using the trained cell detection neural network to perform cell detection on the table image area corresponding to the target table, the image position information of each cell in the table image area is determined;

[0144] The cell detection neural network is trained by image samples and position labeling results obtained by labeling different cells in the image samples.

[0145] In a possible implementation, the above device further includes:

[0146] The training module is used to train the cell detection neural network according to the following steps:

[0147] Get image samples;

[0148] Input the image sample into the multi-level feature extraction layer included in the cell detection neural network to be trained, and determine the cell feature information output by each level feature extraction layer; each level feature extraction layer correspondingly extracts feature information of cells within a size range;

[0149] Input the cell feature information into the prediction layer connected to the feature extraction layer of the corresponding level, determine the prediction result output by the prediction layer, and match the prediction result with the position annotation result of the cell corresponding to the feature extraction layer of the corresponding level;

[0150] In response to the prediction result not matching the position labeling result of the cell corresponding to the corresponding hierarchical feature extraction layer, the network parameter values ​​of the cell detection neural network are adjusted until the prediction result matches the position labeling result of the cell corresponding to the corresponding hierarchical feature extraction layer.

[0151] In one possible implementation, the detection module 302 is configured to perform corner point detection based on the target document image according to the following steps to determine image position information of multiple corner points included in the target table in the target document image:

[0152] Perform table detection on the target document image to determine the candidate rectangular area where the target table is located; the candidate rectangular area is a rectangular area composed of multiple candidate corner points;

[0153] Performing corner detection on the candidate rectangular area based on the trained offset detection neural network to determine the offset between each corner point in the target table and the corresponding candidate corner point;

[0154] For each of the multiple corner points included in the target table, the image position information of the corner point in the target document image is determined based on the offset between the corner point and the corresponding candidate corner point and the image position information of the candidate corner point in the target document image.

[0155] In a possible implementation, the determination module 303 is configured to determine the arrangement relationship between the cells based on the coordinate information of the cells in the target table according to the following steps:

[0156] For a first cell among the cells, determining a structural relationship between the first cell and other adjacent cells based on coordinate information of the cells in the target table;

[0157] Determining an arrangement position of each of the cells in the target table based on the structural relationships determined for each of the cells;

[0158] Based on the arrangement position of each of the cells in the target table, an arrangement relationship between the cells is determined.

[0159] In a possible implementation, the structural relationship includes a horizontal structural relationship and a vertical structural relationship; the determination module 303 is configured to determine the structural relationship between the first cell and other adjacent cells according to the following method:

[0160] Determining a horizontal structural relationship between a first cell and other adjacent cells based on a vertical coordinate range included in the coordinate information of each cell in the target table; and

[0161] Based on the horizontal coordinate range included in the coordinate information of each cell in the target table, a vertical structural relationship between the first cell and other adjacent cells is determined.

[0162] In a possible implementation, the recognition module 304 is configured to combine the text contents of corresponding cells according to the arrangement relationship in accordance with the following steps to obtain a table recognition result for the target document image:

[0163] For each of the plurality of cells, according to the arrangement position of the cell in the target table indicated by the arrangement relationship, fill the text content corresponding to the cell into the arrangement position;

[0164] When the text content is filled based on the arrangement positions indicated by the plurality of cells, a table recognition result for the target document image is obtained.

[0165] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0166] The present disclosure also provides an electronic device, such as Figure 4 FIG. 4 is a schematic diagram of the electronic device structure provided by an embodiment of the present disclosure, including: a processor 401, a memory 402, and a bus 403. The memory 402 stores machine-readable instructions (e.g., Figure 3 When the electronic device is running, the processor 401 communicates with the memory 402 via the bus 403. When the machine-readable instructions are executed by the processor 401, the following processing is performed:

[0167] Obtain target document image;

[0168] Performing cell detection on the target table in the target document image to determine the coordinate information of each cell in the target table; and performing text detection on the target document image to determine the text content corresponding to each cell in the target table;

[0169] Determine the arrangement relationship between the cells based on the coordinate information of the cells in the target table;

[0170] The text contents of the corresponding cells are combined according to the arrangement relationship to obtain the table recognition result for the target document image.

[0171] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program executes the steps of the table recognition method described in the above method embodiment. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0172] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the table recognition method described in the above method embodiment. For details, please refer to the above method embodiment and will not be repeated here.

[0173] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0174] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0175] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0176] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0177] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0178] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. A table recognition method, characterized in that: include: Obtain target document image; Performing cell detection on a target table in the target document image to determine coordinate information of each cell in the target table; and performing text detection on the target document image to determine the text content corresponding to each cell in the target table; Determining an arrangement relationship between the cells based on coordinate information of the cells in the target table; Combining the text contents of the corresponding cells according to the arrangement relationship to obtain a table recognition result for the target document image; Wherein, determining the arrangement relationship between the cells based on the coordinate information of the cells in the target table includes: For a first cell among the cells, determining a structural relationship between the first cell and other adjacent cells based on coordinate information of the cells in the target table; wherein the structural relationship includes at least one of contains, is contained by, intersects, and is equal; Determining an arrangement position of each of the cells in the target table based on the structural relationships determined for the cells; An arrangement relationship between the cells is determined based on an arrangement position of each of the cells in the target table.

2. The method according to claim 1, characterized in that Before performing cell detection on the target table in the target document image, the method further includes: Performing corner point detection based on the target document image to determine image position information of a plurality of corner points included in a target table in the target document image and corner point sequence information between the corner points; The target table is corrected based on the image position information of the multiple corner points in the target document image and the corner point sequence information between the corner points to obtain the image position range information of the table image area corresponding to the processed target table in the target document image.

3. The method according to claim 2, characterized in that The performing cell detection on the target table in the target document image to determine coordinate information of each cell in the target table includes: extracting a table image area corresponding to the target table from the target document image based on the image position range information; Performing cell detection on the table image area corresponding to the target table to determine image position information of each cell in the table image area; The image position information of each cell in the table image area is determined as the coordinate information of each cell in the target table.

4. The method according to claim 3, characterized in that The performing cell detection on the table image area corresponding to the target table to determine the image position information of each cell in the table image area includes: Performing cell detection on the table image area corresponding to the target table using the trained cell detection neural network to determine image position information of each cell in the table image area; The cell detection neural network is trained by image samples and position annotation results obtained by performing position annotation on different cells in the image samples.

5. The method according to claim 4, characterized in that Train the cell detection neural network as follows: Get image samples; Inputting the image sample into a multi-level feature extraction layer included in a cell detection neural network to be trained, and determining cell feature information output by each level of feature extraction layer; each level of feature extraction layer correspondingly extracts feature information of cells within a size range; Inputting the cell feature information into a prediction layer connected to a corresponding hierarchical feature extraction layer, determining a prediction result output by the prediction layer, and matching the prediction result with a position annotation result of the cell corresponding to the corresponding hierarchical feature extraction layer; In response to the prediction result not matching the position labeling result of the cell corresponding to the corresponding hierarchical feature extraction layer, the network parameter value of the cell detection neural network is adjusted until the prediction result matches the position labeling result of the cell corresponding to the corresponding hierarchical feature extraction layer.

6. The method according to any one of claims 2 to 5, characterized in that: The performing corner point detection based on the target document image to determine image position information of a plurality of corner points included in the target table in the target document image includes: Performing table detection on the target document image to determine a candidate rectangular area where the target table is located; the candidate rectangular area is a rectangular area formed by a plurality of candidate corner points; Performing corner detection on the candidate rectangular area based on a trained offset detection neural network to determine the offset between each corner point in the target table and the corresponding candidate corner point; For each corner point among the multiple corner points included in the target table, the image position information of the corner point in the target document image is determined based on the offset between the corner point and the corresponding candidate corner point and the image position information of the candidate corner point in the target document image.

7. The method according to claim 1, characterized in that The structural relationship includes a horizontal structural relationship and a vertical structural relationship; and determining the structural relationship between the first cell and other adjacent cells based on the coordinate information of each cell in the target table includes: determining a horizontal structural relationship between the first cell and other adjacent cells based on a vertical coordinate range included in the coordinate information of each cell in the target table; and Based on the horizontal coordinate range included in the coordinate information of each cell in the target table, a vertical structural relationship between the first cell and other adjacent cells is determined.

8. The method according to claim 1 or 7, characterized in that Combining the text contents of the corresponding cells according to the arrangement relationship to obtain a table recognition result for the target document image includes: For each of the plurality of cells, according to the arrangement position of the cell in the target table indicated by the arrangement relationship, fill the text content corresponding to the cell into the arrangement position; When the text content is filled in based on the arrangement positions indicated by the plurality of cells, a table recognition result for the target document image is obtained.

9. A table recognition device, characterized in that: include: An acquisition module, used for acquiring a target document image; A detection module, configured to perform cell detection on a target table in the target document image and determine coordinate information of each cell in the target table; and performing text detection on the target document image to determine the text content corresponding to each cell in the target table; a determination module, configured to determine an arrangement relationship between the cells based on coordinate information of the cells in the target table; a recognition module, configured to combine the text contents of the corresponding cells according to the arrangement relationship to obtain a table recognition result for the target document image; The determining module is specifically configured to: For a first cell among the cells, determining a structural relationship between the first cell and other adjacent cells based on coordinate information of the cells in the target table; wherein the structural relationship includes at least one of contains, is contained by, intersects, and is equal; Determining an arrangement position of each of the cells in the target table based on the structural relationships determined for the cells; An arrangement relationship between the cells is determined based on an arrangement position of each of the cells in the target table.

10. An electronic device, characterized in that: include: A processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the steps of the table recognition method according to any one of claims 1 to 8 are performed.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the table recognition method according to any one of claims 1 to 8.

Images