Table reconstruction method and device, electronic equipment and storage medium

Abstract

The invention provides a table reconstruction method and device, electronic equipment and a storage medium, and relates to the technical field of table processing, and the method comprises the steps: carrying out table detection on at least one target picture, and determining a table region image contained in the target picture; for each table region image, performing table structure identification on the table region image to obtain at least one table structure object contained in the to-be-reconstructed table; table structure building is carried out based on the at least one table structure object, table structure information of the to-be-reconstructed table is determined, and the table structure information at least comprises cell coordinates of at least one target cell contained in the to-be-reconstructed table; performing text recognition on the target picture to obtain at least one text content and a text coordinate corresponding to the text content; and generating a target reconstruction table contained in the target picture based on the table structure information, the text coordinates and the text content. According to the method, the universality of the table reconstruction method can be guaranteed while the reconstruction effect is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of table processing, and in particular to a table reconstruction method, device, electronic device and storage medium. Background Art

[0002] Tables are a visual communication mode and a means of organizing and collating data. In communication, scientific research, and data analysis activities, table sizes, table types, and table formats are complex and diverse, such as tables with different numbers of rows and columns, wireless tables, system tables, and card-line tables. In some scenarios, users have the need to reconstruct the table in the image. Based on the complex and diverse table structure, how to improve the reconstruction effect on various tables is an urgent problem to be solved. Summary of the invention

[0003] The present disclosure provides a table reconstruction method, device, electronic device and storage medium. The technical solution is as follows:

[0004] According to one aspect of the present disclosure, a table reconstruction method is provided, the method comprising:

[0005] Performing table detection on at least one target image to determine a table area image contained in the target image, wherein the table area image contains a single table to be reconstructed;

[0006] For each of the table area images, perform table structure recognition on the table area image to obtain at least one table structure object included in the table to be reconstructed;

[0007] Building a table structure based on the at least one table structure object, determining table structure information of the table to be rebuilt, wherein the table structure information at least includes cell coordinates of at least one target cell included in the table to be rebuilt;

[0008] Performing text recognition on the target image to obtain at least one text content and text coordinates corresponding to the text content;

[0009] Based on the table structure information, the text coordinates and the text content, a target reconstruction table included in the target image is generated.

[0010] According to another aspect of the present disclosure, a table reconstruction device is provided, the device comprising:

[0011] A table detection module, used to perform table detection on at least one target image, and determine a table area image contained in the target image, wherein the table area image contains a single table to be reconstructed;

[0012] A structure recognition module, used for performing table structure recognition on each of the table area images to obtain at least one table structure object contained in the table to be reconstructed;

[0013] A structure building module, used to build a table structure based on the at least one table structure object, and determine the table structure information of the table to be rebuilt, wherein the table structure information at least includes the cell coordinates of at least one target cell included in the table to be rebuilt;

[0014] A text recognition module, used to perform text recognition on the target image to obtain at least one text content and text coordinates corresponding to the text content;

[0015] A table reconstruction module is used to generate a target reconstruction table contained in the target image based on the table structure information, the text coordinates and the text content.

[0016] According to one aspect of the present disclosure, an electronic device is provided, including: a processor and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the table reconstruction method as described above.

[0017] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the table reconstruction method as described above.

[0018] According to another aspect of the present disclosure, a computer program product is provided, the computer program product comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above table reconstruction method.

[0019] The beneficial effects brought by the technical solution provided by the embodiments of the present disclosure include at least:

[0020] The disclosed embodiment provides a table reconstruction method: by performing table structure recognition on the table area image in the target image, at least one table structure object for building the table structure is obtained, and the table structure information is reconstructed; then, by performing text recognition on the target image, the text content required to be filled in the table is obtained; and then based on the table structure information and text content, a target reconstructed table is generated. By splitting and identifying the table structure object and building the structure of the table to be reconstructed, a universal table reconstruction method that does not depend on cell lines is proposed, and is applicable to different types of tables such as wired tables, wireless tables, and three-line tables, while improving the reconstruction effect, the universality of the table reconstruction method is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A flowchart of a table reconstruction method according to an exemplary embodiment of the present disclosure is shown;

[0023] Figure 2 is a schematic diagram of a table reconstruction process provided by an exemplary embodiment of the present disclosure;

[0024] Figure 3 A flowchart of another table reconstruction method according to an exemplary embodiment of the present disclosure is shown;

[0025] Figure 4 is a schematic diagram of a table structure object provided by an exemplary embodiment of the present disclosure;

[0026] Figure 5 A flowchart of another table reconstruction method according to an exemplary embodiment of the present disclosure is shown;

[0027] Figure 6 is a structural schematic diagram of a table reconstruction device provided by an embodiment of the present disclosure;

[0028] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0029] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0030] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0031] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative and not restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more". The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are for illustrative purposes only, and are not used to limit the scope of these messages or information.

[0032] The solution of the present invention is described below with reference to the accompanying drawings, and the technical solution provided by the embodiment of the present invention is explained in detail through specific embodiments and their application scenarios.

[0033] Please refer to Figure 1 , which shows a flow chart of a table reconstruction method according to an exemplary embodiment of the present disclosure. The method is applied to an electronic device as an example for explanation. Figure 1 As shown, the method includes:

[0034] Step 101 : Perform table detection on at least one target image to determine a table area image contained in the target image, wherein the table area image contains a single table to be reconstructed.

[0035] Among them, the table reconstruction task generally refers to the process of reconstructing a complete table from a picture containing a table object. Since there are other types of objects in the picture besides the table object, such as text, images, etc., in order to avoid interference of other types of objects in the subsequent table reconstruction, firstly, table detection is performed on at least one input target picture to locate the coordinates of the position of the table object in the target picture; and in order to avoid interference between multiple table objects in the same target picture during reconstruction, it is also necessary to crop the target picture based on the coordinates of the located table object to obtain at least one table area image, and each table area image contains a single table to be reconstructed.

[0036] Optionally, the target image can be input into the target detection model for table detection to obtain at least one target detection frame output by the target detection model, and the position coordinates of the target detection frame are the position coordinates of the table object in the target image. The training data used by the target detection model during training are the sample image and the annotated frame of the table area in the sample image. The sample image is input into the target detection model to obtain the predicted detection frame output by the target detection model, and then the table area detection loss is determined by comparing the position difference between the predicted detection frame and the annotated frame to train the target detection model; until the table area detection loss is less than the preset value, it is determined that the target detection model training is completed; the trained target detection model can be used to perform table detection on the input target image.

[0037] Step 102: for each table area image, perform table structure recognition on the table area image to obtain at least one table structure object included in the table to be reconstructed.

[0038] When reconstructing a table, it is first necessary to extract the structural information of the table from the image, such as the number of rows and columns, whether there is a column header, the number of column headers, whether there is a projected row header, the number of projected row headers, whether there is a row header, whether there is a cross-cell and the cross-cell position, etc.; then reconstruct the table based on the acquired structural information; and no matter whether the table is a three-line table, a wired table, a wireless table or a system table, it contains specific table structure objects. Therefore, in a possible implementation, for each table area image, the table area image is identified for the table structure to obtain at least one table structure object contained in the table to be reconstructed.

[0039] Exemplarily, the table structure object may include at least one of a table row object, a table column object, a column header object, a projected row header object, a cross-cell object, a row header object, and a table header object.

[0040] Optionally, a table structure recognition model is pre-trained, and the table area image is input into the table structure recognition model to perform table structure recognition to obtain at least one table structure object included in the table to be reconstructed.

[0041] Step 103 : constructing a table structure based on at least one table structure object, and determining table structure information of the table to be reconstructed, wherein the table structure information at least includes the cell coordinates of at least one target cell included in the table to be reconstructed.

[0042] After obtaining at least one table structure object extracted from the table area image, a complete table structure can be constructed according to the preset table construction logic based on at least one table structure object, and then the table structure information of the table to be reconstructed can be obtained; since the table is composed of cells, the table structure information at least includes the cell coordinates of at least one target cell contained in the table to be reconstructed.

[0043] The cell coordinates may be the position coordinates of the target cell in the table to be reconstructed; or the cell coordinates may be converted into the position coordinates of the target cell in the entire target image. The conversion process is: firstly determine the first position coordinates of the target cell in the table to be reconstructed, and obtain the second position coordinates of the table to be reconstructed in the target image, and then determine the position coordinates of the target cell in the target image based on the first position coordinates and the second position coordinates.

[0044] Step 104: Perform text recognition on the target image to obtain at least one text content and text coordinates corresponding to the text content.

[0045] In the process of rebuilding a table, in addition to rebuilding the table structure, the table content also needs to be rebuilt. Correspondingly, text recognition needs to be performed on the target image to obtain at least one text content. After obtaining the text content, it is also necessary to determine whether the text content belongs to the table content, and if it belongs to the table content, which target cell is the filled content. Correspondingly, the text coordinates corresponding to the text content need to be identified during text recognition. And the text coordinates are the position coordinates of the text content in the target image.

[0046] Optionally, when performing text recognition, in order to avoid interference from other non-table text content, text recognition can be performed only on the table area image to obtain at least one text content and text coordinates corresponding to the text content, which are the position coordinates of the text content in the table area image.

[0047] Step 105 , generating a target reconstruction table contained in the target image based on the table structure information, text coordinates and text content.

[0048] Furthermore, after obtaining the table structure information and the table content that may be filled in the table, the text content can be filled in the reconstructed table structure based on the table structure information, text coordinates and text content to reconstruct the target reconstructed table contained in the target image.

[0049] Please refer to Figure 2 , which is a schematic diagram of the principle of a table reconstruction process provided by an exemplary embodiment of the present disclosure. Figure 2As shown, a table detection is performed on the input target image 201 to obtain at least one table area image 202; a table structure recognition is performed on each table area image 202 to obtain at least one table structure object 203, and a table structure is constructed for the table structure object 203 to obtain table structure information 204; in addition, text recognition is performed on the target image 201 to obtain text content and text coordinates 205; based on the table structure information 204 and text content + text coordinates 205, table filling and table reconstruction are performed to obtain a reconstruction result 206 (i.e., a target reconstructed table).

[0050] In summary, the disclosed embodiment provides a table reconstruction method: by performing table structure recognition on the table area image in the target image, at least one table structure object for building the table structure is obtained, and the table structure information is reconstructed; then, by performing text recognition on the target image, the text content required to be filled in the table is obtained; and then based on the table structure information and text content, a target reconstructed table is generated. By splitting and identifying the table structure object and building the structure of the table to be reconstructed, a universal table reconstruction method that does not rely on cell lines is proposed, and is applicable to different types of tables such as wired tables, wireless tables, and three-line tables, while improving the reconstruction effect, the universality of the table reconstruction method is guaranteed.

[0051] There are differences in the categories of table structure objects contained in different types of tables. For example, some tables have cross-cells while some do not; some tables have projected row headers while some do not; in order to improve the accuracy of table structure recognition, a table structure recognition model with multiple output units is pre-trained to recognize different categories of table structure objects.

[0052] Please refer to Figure 3 , which shows a flow chart of another table reconstruction method according to an exemplary embodiment of the present disclosure. The method is applied to an electronic device as an example for explanation. Figure 3 As shown, the method includes:

[0053] Step 301 : Perform table detection on at least one target image to determine a table area image contained in the target image, wherein the table area image contains a single table to be reconstructed.

[0054] The implementation of step 301 may refer to step 101 , and this embodiment will not be described in detail here.

[0055] Step 302: for each table area image, input the table area image into a table structure recognition model to obtain an object recognition result of at least one type of object output by the table structure recognition model.

[0056] Since there are differences in the table structure objects contained in different types of tables, in order to adapt to the table structure recognition of different types of tables, the table structure recognition model is provided with multiple output units, and different output units of the table structure recognition model are used to output object recognition results of different classes of objects. In other words, the table structure recognition task is converted into a target recognition problem of multiple classes of objects by using the table structure recognition model. In a corresponding possible implementation, the table area image is input into the table structure recognition model to obtain the object recognition result of at least one class of objects output by the table structure recognition model.

[0057] Exemplarily, the object recognition result indicates whether there is an object of this type in the table area image, and if there is an object of this type, the location of the object of this type.

[0058] Optionally, by analyzing different types of table structures, the table structure recognition problem can be converted into 6 types of table structure objects, namely, table overall object recognition, table row object recognition, table column object recognition, column header object recognition, projected row header object recognition, cross-cell object recognition, etc. The corresponding table structure recognition model can include six output units, which are respectively used to output the object recognition results of the above 6 types of objects.

[0059] Further, step 302 may include step 302A.

[0060] Step 302A, input the table area image into the table structure recognition model, and obtain the first recognition result of the table overall object output by the first output unit, the second recognition result of the table row object output by the second output unit, the third recognition result of the table column object output by the third output unit, the fourth recognition result of the column header object output by the fourth output unit, the fifth recognition result of the projected row header object output by the fifth output unit, and the sixth recognition result of the cross-cell object output by the sixth output unit.

[0061] When the table structure recognition model has six output units, after the table area image is input into the table structure recognition model, the object recognition results output by the six output units can be obtained respectively, that is, the first recognition result of the table overall object output by the first output unit, the second recognition result of the table row object output by the second output unit, the third recognition result of the table column object output by the third output unit, the fourth recognition result of the column header object output by the fourth output unit, the fifth recognition result of the projection row header object output by the fifth output unit and the sixth recognition result of the cross-cell object output by the sixth output unit.

[0062] Exemplarily, the first recognition result is used to indicate whether there is a table object as a whole in the table area image, that is, whether there is a table to be reconstructed, and the position coordinates of the table object as a whole when there is one; the second recognition result is used to indicate whether there is at least one table row object in the table to be reconstructed in the table area image, and the position coordinates of at least one table row object when there is one; the third recognition result is used to indicate whether there is at least one table column object in the table to be reconstructed in the table area image, and the position coordinates of at least one table column object when there is one; the fourth recognition result is used to indicate whether there is at least one column header object in the table to be reconstructed in the table area image, and the position coordinates of at least one column header object when there is one; the fifth recognition result is used to indicate whether there is at least one projected row header object in the table to be reconstructed in the table area image, and the position coordinates of at least one projected row header object when there is one; the sixth recognition result is used to indicate whether there is at least one cross-cell object in the table to be reconstructed in the table area image, and the position coordinates of at least one cross-cell object when there is one.

[0063] Please refer to Figure 4 , which is a schematic diagram of a table structure object provided by an exemplary embodiment of the present disclosure. Figure 4 As shown, the first row in the table is a column header object 401 , the second row in the table is a projection row header object 402 , each row in the table is a table row object 403 , and each column in the table is a table column object 404 .

[0064] It should be noted that the purpose of repeatedly identifying the entire table object in the table structure recognition stage includes two aspects: on the one hand, it is to detect again whether there is a table to be reconstructed in the table area image to avoid subsequent invalid table reconstruction steps; on the other hand, the position coordinates of the entire table object can be further accurately located.

[0065] Optionally, some table headers also have special settings, and the table structure recognition model can also set a seventh output unit to output a seventh recognition result of the header object. Among them, the special settings refer to the text format of the text content filled in the cell corresponding to the header. There may be some special settings, such as text bold, text alignment, text color, or other pre-set special format requirements. Therefore, it is necessary to identify the cell corresponding to the header object in advance through the table structure recognition model, and add identification information to the cell, so that when the table is subsequently rebuilt, the required special settings can be set for the text content filled in the cell.

[0066] Step 303: Determine at least one table structure object included in the table to be reconstructed based on the object recognition result of at least one type of object.

[0067] Further, after obtaining the object recognition result of at least one type of object output by the table structure recognition model, if the object recognition result indicates that the type of object exists in the table area image, the type of object is determined as at least one table structure object included in the table to be reconstructed.

[0068] Since the first recognition result is used to re-identify whether there is a table object in the table area image, only when there is a precursor of the table object is there a need to reconstruct the table. Therefore, after obtaining the object recognition result output by the table structure recognition model, first determine whether the first recognition result indicates the existence of the table object. If the first recognition result indicates the existence of the table object, then determine at least one table structure object contained in the table to be reconstructed based on the second recognition result, the third recognition result, the fourth recognition result, the fifth recognition result and the sixth recognition result; conversely, if the first recognition result indicates the absence of the table object, stop the table reconstruction task.

[0069] Exemplarily, if the object recognition result indicates that there are 3 table row objects and 5 table column objects, the 3 table row objects and 5 table column objects are determined as table structure objects included in the table to be reconstructed.

[0070] Optionally, taking the example that the object recognition result indicates the existence of 6 types of table structure objects, that is, if the first recognition result indicates the existence of a table object as a whole, the second recognition result indicates the existence of a table row object, the third recognition result indicates the existence of a table column object, the fourth recognition result indicates the existence of a column header object, the fifth recognition result indicates the existence of a projected row header object, and the sixth recognition result indicates the existence of a cross-cell object, then at least one table row object indicated by the second recognition result, at least one table column object indicated by the third recognition result, at least one column header object indicated by the fourth recognition result, at least one projected row header object indicated by the fifth recognition result, and at least one cross-cell object indicated by the sixth recognition result can be determined as at least one table structure object contained in the table to be reconstructed; that is, the table to be reconstructed exists, and the table to be reconstructed is composed of at least one row object, at least one column object, at least one column header object, at least one projected row header object, and at least one cross-cell object.

[0071] It should be noted that if the first recognition result indicates that the entire table object does not exist, then regardless of whether the second, third, fourth, fifth and sixth recognition results indicate that an object of the corresponding category exists, the subsequent table reconstruction steps are stopped.

[0072] Step 304 : constructing a table structure based on at least one table structure object, and determining table structure information of the table to be reconstructed, wherein the table structure information at least includes the cell coordinates of at least one target cell included in the table to be reconstructed.

[0073] When building a table structure based on at least one identified table structure object, a certain building order needs to be followed. The table structure building process corresponding to the identification of 6 types of table structure objects as an example may include steps 304A to 304F, that is, step 304 may include steps 304A to 304F.

[0074] Step 304A: construct a first table structure of the table to be reconstructed based on the second recognition result.

[0075] The construction order is: table row object - table column object - cross-cell object - column header object - projected row header object. First, based on the coordinate chronological relationship of the position coordinates of at least one table row object indicated by the second recognition result, the table row objects are arranged to construct a first table structure of the table to be reconstructed.

[0076] Step 304B: update the first table structure based on the third recognition result to obtain a second table structure of the table to be reconstructed.

[0077] Furthermore, based on the coordinate context of the position coordinates of at least one table column object indicated by the third recognition result, the table column objects are arranged on the basis of the first table structure to obtain an updated second table structure.

[0078] After the table row objects and table column objects are constructed, a table with N*M cells can be obtained, where N is the number of table row objects indicated by the second recognition result, and M is the number of table column objects indicated by the third recognition result.

[0079] Step 304C: update the second table structure based on the sixth recognition result to obtain a third table structure of the table to be reconstructed.

[0080] After obtaining a table with N*M cells, since there are cross-cell objects in the table to be reconstructed, the third table structure needs to be further modified. In a possible implementation, the position coordinates of at least one cross-cell object indicated by the sixth recognition result are matched with the position coordinates of cells in the second table structure, and the cells with matching position coordinates are updated to cross-cell objects to obtain an updated third table structure.

[0081] Exemplarily, if the second table structure is a table of 3 (rows) * 5 (columns), based on the sixth recognition result, it is determined that the cell in the 2nd row and 3rd column and the cell in the 2nd row and 4th column are cross-cell objects, then the cell in the 2nd row and 3rd column and the cell in the 2nd row and 4th column are merged into one cross-cell object.

[0082] Step 304D: Set first identification information for the first cell in the third table structure based on the fourth recognition result.

[0083] Since the text content of the column header object in the table may have special formats, such as bold, different fonts, etc., it is necessary to identify the cell where the column header object is located and specially mark the cell so that a special format can be set for the text content in the cell later. In a possible implementation, the position coordinates of at least one column header object indicated by the fourth recognition result are matched with the position coordinates of the cells in the third table structure, and first identification information is set for at least one first cell whose position coordinates match in the third table structure, and the first identification information is used to indicate that the first cell is the cell where the column header object is located.

[0084] Step 304E: Set second identification information for a second cell in the third table structure based on the fifth recognition result.

[0085] Similar to the column header object, the cell where the projected row header object is located also needs a special identification. In a corresponding possible implementation, the position coordinates of at least one projected row header object indicated by the fifth recognition result are matched with the position coordinates of the cells in the third table structure, and second identification information is set for at least one second cell whose position coordinates match in the third table structure, and the second identification information is used to indicate that the second cell is the cell where the projected row header object is located.

[0086] Step 304F: Generate table structure information of the table to be reconstructed based on the third table structure, the first identification information and the second identification information.

[0087] Furthermore, after constructing the table structure, ie, the third table structure, and some special identification information, ie, the first identification information and the second identification information, the table structure information of the table to be reconstructed can be generated.

[0088] Optionally, when the table structure recognition model also outputs a seventh recognition result, the position coordinates of the header object indicated by the seventh recognition result can be matched with the position coordinates of the cell in the third table structure, and third identification information can be set for the third cell with matching position coordinates in the third table structure. The third identification information is used to indicate that the third cell is the cell where the header object is located.

[0089] Since a table is essentially composed of a number of cells, the table structure information is obtained mainly for obtaining a number of cells contained in the table to be reconstructed, as well as the coordinates, row and column numbers, and other cell information of the cells. Correspondingly, in an illustrative example, step 304F may also include step 304F1 and step 304F2.

[0090] Step 304F1, determining the cell coordinates of at least one target cell in the third table structure.

[0091] After the third table structure is constructed, the number of target cells contained in the third table structure and the cell coordinates of each target cell can be obtained. The cell coordinates can be the position coordinates of the target cell in the third table structure; or after obtaining the position coordinates of the target cell in the third table structure, based on the position coordinates of the table to be rebuilt in the target image, they can be converted into the position coordinates of the target cell in the target image, so as to facilitate subsequent text content filling.

[0092] Step 304F2: Determine the table structure information of the table to be reconstructed based on the cell coordinates of at least one target cell, the first identification information, and the second identification information.

[0093] Furthermore, the first identification information is associated with a first cell in the target cells, the second identification information is associated with a second cell in the target cells, and based on the cell coordinates of at least one target cell contained in the third table structure, the table structure information of the table to be reconstructed is determined.

[0094] Step 305: Perform text recognition on the target image to obtain at least one text content and text coordinates corresponding to the text content.

[0095] The implementation of step 305 may refer to step 104, and this embodiment will not be described in detail here.

[0096] Step 306: Generate a target reconstruction table contained in the target image based on the table structure information, text coordinates and text content.

[0097] In order to accurately fill the table content, text filling is performed based on coordinate matching. For each target cell, if there is a text coordinate that matches the cell coordinates corresponding to the target cell, the text content corresponding to the text coordinate is filled into the target cell to generate a target reconstruction table contained in the target image.

[0098] In this embodiment, the table structure recognition task is converted into an object recognition problem of 6 categories of objects, and based on the recognized objects of several categories, the table structure is rebuilt in a preset construction order to achieve the purpose of table recognition and reconstruction that is independent of cell lines, thereby improving the accuracy of table reconstruction. In addition, column header objects, projected row header objects, and cross-cell object recognition are introduced into the 6-category object recognition, so that accurate recognition and reconstruction can be achieved for tables with special formats, further improving the accuracy of table reconstruction.

[0099] In other possible application scenarios, there is a need to rebuild a cross-page table, that is, a single table is located in two pictures, and the table reconstruction process is performed on one picture, that is, the cross-page table will be reconstructed into two tables. In order to realize the reconstruction of the cross-page table, it is necessary to determine whether it is a cross-page table, and after determining that it is a cross-page table, merge the tables.

[0100] Please refer to Figure 5 , which shows a flow chart of another table reconstruction method according to an exemplary embodiment of the present disclosure. The method is applied to an electronic device as an example for explanation. Figure 5 As shown, the method includes:

[0101] Step 501 , performing table detection on at least one target image to determine a table area image contained in the target image, wherein the table area image contains a single table to be reconstructed.

[0102] Step 502: for each table area image, perform table structure recognition on the table area image to obtain at least one table structure object included in the table to be reconstructed.

[0103] Step 503 : constructing a table structure based on at least one table structure object, and determining table structure information of the table to be reconstructed, wherein the table structure information at least includes the cell coordinates of at least one target cell included in the table to be reconstructed.

[0104] Step 504: Perform text recognition on the target image to obtain at least one text content and text coordinates corresponding to the text content.

[0105] Step 505: Generate a target reconstruction table contained in the target image based on the table structure information, text coordinates and text content.

[0106] The implementation of steps 501 to 505 may refer to the above embodiment, and will not be described in detail in this embodiment.

[0107] Step 506 : If the picture page numbers of the first picture and the second picture are continuous, it is detected whether the first reconstructed table and the second reconstructed table belong to the same table.

[0108] In a possible application scenario, when a user inputs multiple consecutive target images into an electronic device, there may be a problem of reconstructing a cross-page table. Taking at least one target image including a first image and a second image with consecutive page numbers as an example, the first image generates a first reconstructed table according to the table reconstruction process, and the second image generates a second reconstructed table according to the table reconstruction process. After the first reconstructed table and the second reconstructed table are generated, it is necessary to further determine whether the first reconstructed table and the second reconstructed table are the same table, that is, to detect whether the first reconstructed table and the second reconstructed table are cross-page tables.

[0109] Considering that there is only one table header in the same table and the table columns of the same table are the same, the table header and the number of columns can be combined to jointly determine whether the first reconstructed table and the second reconstructed table are cross-page tables. Correspondingly detecting whether the first reconstructed table and the second reconstructed table belong to the same table can also include step 506A1 and step 506A2.

[0110] Step 506A1 : if the first reconstructed table has a table header and the second reconstructed table does not have a table header, and the number of columns of the first reconstructed table is the same as the number of columns of the second reconstructed table, then determine that the first reconstructed table and the second reconstructed table belong to the same table.

[0111] Taking the example that the picture page number of the first picture is before the picture page number of the second picture, if the first reconstructed table has a header, the second reconstructed table does not have a header, and the number of columns of the first reconstructed table is the same as that of the second reconstructed table, then when the first picture and the second picture are pictures with consecutive page numbers, it can be determined that the first reconstructed table and the second reconstructed table are the same table.

[0112] Optionally, taking the example that the picture page number of the first picture is after the picture page number of the second picture, if the first reconstructed table does not have a table header, the second reconstructed table has a table header, and the number of columns of the first reconstructed table is the same as that of the second reconstructed table, then when the first picture and the second picture are pictures with consecutive page numbers, it can be determined that the first reconstructed table and the second reconstructed table are the same table.

[0113] In order to further improve the accuracy of identifying whether it is a cross-page table, in other possible implementations, the position of the first reconstructed table and the second reconstructed table in the picture can also be combined for determination. Taking the picture page number of the first picture before the picture page number of the second picture as an example, after determining that the first reconstructed table has a header, the second reconstructed table does not have a header, and the number of columns of the first reconstructed table is the same as the number of columns of the second reconstructed table, the first position of the first reconstructed table in the first picture and the second position of the second reconstructed table in the second picture are obtained. If the first position is at the end of the page of the first picture (at the same time, there is no other text or picture content below the first reconstructed table in the first picture), and the second position is at the beginning of the page of the second picture (at the same time, there is no other text or picture content above the second reconstructed table in the second picture), it can be determined that the first reconstructed table and the second reconstructed table belong to the same table across pages.

[0114] Step 506A2: if the first reconstructed table has a table header and the second reconstructed table does not have a table header, and the number of columns of the first reconstructed table is different from the number of columns of the second reconstructed table, then determine that the first reconstructed table and the second reconstructed table do not belong to the same table.

[0115] Some tables may not have a header. For example, the picture page number of the first picture is before the picture page number of the second picture. If the first reconstructed table has a header and the second reconstructed table does not have a header, but the number of columns of the first reconstructed table is different from that of the second reconstructed table, it is determined that the first reconstructed table and the second reconstructed table do not belong to the same table.

[0116] Optionally, taking the example that the picture page number of the first picture is after the picture page number of the second picture, if the first reconstructed table does not have a table header and the second reconstructed table has a table header, but the number of columns of the first reconstructed table is different from that of the second reconstructed table, it can be determined that the first reconstructed table and the second reconstructed table do not belong to the same table.

[0117] Optionally, taking the example that the picture page number of the first picture is before the picture page number of the second picture, after the first reconstructed table has a header, the second reconstructed table does not have a header, and the number of columns of the first reconstructed table is the same as the number of columns of the second reconstructed table, if the first position of the first reconstructed table and the second position of the second reconstructed table indicate that there are other text or pictures and other contents between the first reconstructed table and the second reconstructed table, it is determined that the first reconstructed table and the second reconstructed table do not belong to the same table.

[0118] Step 507 : If the first reconstruction table and the second reconstruction table belong to the same table, merge the first reconstruction table and the second reconstruction table into a third reconstruction table.

[0119] In the case where it is determined that the first reconstruction table and the second reconstruction table belong to the same table, the first reconstruction table and the second reconstruction table may be combined into a third reconstruction table in the order of page numbers.

[0120] Optionally, when generating the first reconstruction table and the second reconstruction table, the reconstructed table can be saved in HTML format; correspondingly, when merging the first reconstruction table and the second reconstruction table, the HTML format generated files of the two tables can be merged in sequence into one HTML format generated file.

[0121] In this embodiment, a method of comprehensively judging cross-page tables by table headers and column numbers is provided, so that after reconstructing the table based on a single target image, automatic merging and generation of cross-page tables can still be achieved, further improving the authenticity and restoration of table reconstruction.

[0122] Please refer to Figure 6 , which is a schematic diagram of the structure of a table reconstruction device provided by an embodiment of the present disclosure. For example, Figure 6 As shown, the device 600 includes:

[0123] A table detection module 601 is used to perform table detection on at least one target image to determine a table area image contained in the target image, wherein the table area image contains a single table to be reconstructed;

[0124] A structure recognition module 602 is used to perform table structure recognition on each table area image to obtain at least one table structure object included in the table to be reconstructed;

[0125] A structure building module 603 is used to build a table structure based on the at least one table structure object, and determine the table structure information of the table to be rebuilt, wherein the table structure information at least includes the cell coordinates of at least one target cell included in the table to be rebuilt;

[0126] A text recognition module 604 is used to perform text recognition on the target image to obtain at least one text content and text coordinates corresponding to the text content;

[0127] The table reconstruction module 605 is used to generate a target reconstruction table contained in the target image based on the table structure information, the text coordinates and the text content.

[0128] Optionally, the structure recognition module 602 is further used to:

[0129] Inputting the table area image into a table structure recognition model to obtain an object recognition result of at least one type of object output by the table structure recognition model, wherein different output units of the table structure recognition model are used to output object recognition results of different types of objects;

[0130] Based on the object recognition result of the at least one type of object, the at least one table structure object included in the table to be reconstructed is determined.

[0131] Optionally, the table structure recognition model includes six output units;

[0132] The structure recognition module 602 is further used for:

[0133] The table area image is input into the table structure recognition model to obtain a first recognition result of the table overall object output by the first output unit, a second recognition result of the table row object output by the second output unit, a third recognition result of the table column object output by the third output unit, a fourth recognition result of the column header object output by the fourth output unit, a fifth recognition result of the projected row header object output by the fifth output unit, and a sixth recognition result of the cross-cell object output by the sixth output unit.

[0134] Optionally, the structure recognition module 602 is further used to:

[0135] If the first recognition result indicates the existence of the table as a whole object, the at least one table structure object included in the table to be reconstructed is determined based on the second recognition result, the third recognition result, the fourth recognition result, the fifth recognition result and the sixth recognition result.

[0136] Optionally, the structure building module 603 is further used for:

[0137] Based on the second recognition result, construct a first table structure of the table to be reconstructed;

[0138] Update the first table structure based on the third recognition result to obtain a second table structure of the table to be reconstructed;

[0139] Based on the sixth recognition result, the second table structure is updated to obtain a third table structure of the table to be reconstructed;

[0140] Setting first identification information for a first cell in the third table structure based on the fourth recognition result;

[0141] Setting second identification information for a second cell in the third table structure based on the fifth recognition result;

[0142] The table structure information of the table to be rebuilt is generated based on the third table structure, the first identification information and the second identification information.

[0143] Optionally, the structure building module 603 is further used for:

[0144] determining the cell coordinates of at least one target cell in the third table structure;

[0145] The table structure information of the table to be reconstructed is determined based on the cell coordinates of the at least one target cell, the first identification information, and the second identification information.

[0146] Optionally, the table reconstruction module 605 is further used to:

[0147] For each of the target cells, if there are text coordinates matching the cell coordinates corresponding to the target cell, the text content corresponding to the text coordinates is filled into the target cell to generate a target reconstruction table contained in the target image.

[0148] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0149] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0150] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

[0151] refer to Figure 7, a block diagram of an electronic device 700 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0152] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0153] A plurality of components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 may be any type of device capable of inputting information to the electronic device 700, and the input unit 706 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 707 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 may include but is not limited to a disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0154] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, Figure 1 , Figure 3 , Figure 5 The method shown may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 700 via ROM 702 and / or communication unit 709. In some embodiments, computing unit 701 may be configured to execute the computer program by any other suitable means (e.g., by means of firmware). Figure 1 , Figure 3 , Figure 5 The method shown.

[0155] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0156] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0157] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0158] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0159] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0160] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A table reconstruction method, characterized in that: The method comprises: Performing table detection on at least one target image to determine a table area image contained in the target image, wherein the table area image contains a single table to be reconstructed; For each of the table area images, perform table structure recognition on the table area image to obtain at least one table structure object included in the table to be reconstructed; Building a table structure based on the at least one table structure object, determining table structure information of the table to be rebuilt, wherein the table structure information at least includes cell coordinates of at least one target cell included in the table to be rebuilt; Performing text recognition on the target image to obtain at least one text content and text coordinates corresponding to the text content; Based on the table structure information, the text coordinates and the text content, a target reconstruction table included in the target image is generated.

2. The method according to claim 1, characterized in that The performing table structure recognition on the table area image to obtain at least one table structure object contained in the table to be reconstructed includes: Inputting the table area image into a table structure recognition model to obtain an object recognition result of at least one type of object output by the table structure recognition model, wherein different output units of the table structure recognition model are used to output object recognition results of different types of objects; Based on the object recognition result of the at least one type of object, the at least one table structure object included in the table to be reconstructed is determined.

3. The method according to claim 2, characterized in that The table structure recognition model includes six output units; The step of inputting the table area image into a table structure recognition model to obtain an object recognition result of at least one type of object output by the table structure recognition model includes: The table area image is input into the table structure recognition model to obtain a first recognition result of the table overall object output by the first output unit, a second recognition result of the table row object output by the second output unit, a third recognition result of the table column object output by the third output unit, a fourth recognition result of the column header object output by the fourth output unit, a fifth recognition result of the projected row header object output by the fifth output unit, and a sixth recognition result of the cross-cell object output by the sixth output unit.

4. The method according to claim 3, characterized in that The determining, based on the object recognition result of the at least one type of object, the at least one table structure object included in the table to be reconstructed comprises: If the first recognition result indicates the existence of the table as a whole object, the at least one table structure object included in the table to be reconstructed is determined based on the second recognition result, the third recognition result, the fourth recognition result, the fifth recognition result and the sixth recognition result.

5. The method according to claim 4, characterized in that The step of constructing a table structure based on the at least one table structure object and determining table structure information of the table to be rebuilt includes: Based on the second recognition result, construct a first table structure of the table to be reconstructed; Update the first table structure based on the third recognition result to obtain a second table structure of the table to be reconstructed; Based on the sixth recognition result, the second table structure is updated to obtain a third table structure of the table to be reconstructed; Setting first identification information for a first cell in the third table structure based on the fourth recognition result; Setting second identification information for a second cell in the third table structure based on the fifth recognition result; The table structure information of the table to be rebuilt is generated based on the third table structure, the first identification information and the second identification information.

6. The method according to claim 5, characterized in that The generating the table structure information of the table to be reconstructed based on the third table structure, the first identification information and the second identification information includes: determining the cell coordinates of at least one target cell in the third table structure; The table structure information of the table to be reconstructed is determined based on the cell coordinates of the at least one target cell, the first identification information, and the second identification information.

7. The method according to claim 6, characterized in that The generating, based on the table structure information, the text coordinates and the text content, a target reconstruction table contained in the target image includes: For each of the target cells, if there are text coordinates matching the cell coordinates corresponding to the target cell, the text content corresponding to the text coordinates is filled into the target cell to generate a target reconstruction table contained in the target image.

8. A table reconstruction device, characterized in that: The device comprises: A table detection module, used to perform table detection on at least one target image, and determine a table area image contained in the target image, wherein the table area image contains a single table to be reconstructed; A structure recognition module, used for performing table structure recognition on each of the table area images to obtain at least one table structure object contained in the table to be reconstructed; A structure building module, used to build a table structure based on the at least one table structure object, and determine the table structure information of the table to be rebuilt, wherein the table structure information at least includes the cell coordinates of at least one target cell included in the table to be rebuilt; A text recognition module, used to perform text recognition on the target image to obtain at least one text content and text coordinates corresponding to the text content; A table reconstruction module is used to generate a target reconstruction table contained in the target image based on the table structure information, the text coordinates and the text content.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images