Table restoration method and device, electronic equipment and computer program product

Abstract

The invention provides a table restoration method and device, electronic equipment and a computer program product, and the method comprises the steps: extracting the table position information of a to-be-restored table and the cell position information of each cell in the to-be-restored table from a target image containing the to-be-restored table, and carrying out the restoration of the table according to the table position information; the structure information of the table to be restored is extracted from the target image, the structure information comprises the cell content of each cell and the address of the cell content in the table to be restored, and then the size of each cell and the address of each cell in the table to be restored are determined according to the cell position information; according to the structure information, the size of each cell and the address of each cell in the table to be restored, the size of each cell, the content of each cell and the layout of the table can be restored, and the purpose of restoring the table in the digital processing process is achieved.

Description

Technical Field

[0001] The present application relates to the technical field of table recognition, and in particular to a table restoration method, device, electronic device and computer program product. Background Art

[0002] Table is an important form of information expression, which can organize data into a standard structure to facilitate information retrieval and comparison. In modern data processing, it is often necessary to digitize the table. How to restore the table during the digitization process has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0003] In view of this, the present application proposes a method, device, electronic device and computer program product for restoring a table, which method can restore the table during the digital processing of the table.

[0004] The technical solutions proposed in this application are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for restoring a table, comprising:

[0006] Extracting table position information of the table to be restored and cell position information of each cell in the table to be restored from a target image including the table to be restored;

[0007] According to the table position information, extracting structural information of the table to be restored from the target image; the structural information includes the cell content of each cell and the address of the cell content in the table to be restored;

[0008] The size of each cell and the address of each cell in the table to be restored are determined according to the cell position information, and the cell restoration is performed according to the structural information and the size of each cell and the address of each cell in the table to be restored.

[0009] Furthermore, in the above method, extracting the table position information of the table to be restored and the cell position information of each cell in the table to be restored from the target image containing the table to be restored includes:

[0010] Inputting the prompt information and the target image containing the table to be restored into the table detection network, so that the table detection network extracts the table position information of the table to be restored and the cell position information of each cell in the table to be restored from the target image;

[0011] The prompt information is used to instruct the table detection network to perform a task of extracting the table position information and the cell position information from the target image.

[0012] Furthermore, in the above method, the table detection network includes an image encoding unit, a prompt information encoding unit and a decoding unit;

[0013] The step of inputting the prompt information and the target image containing the table to be restored into the table detection network, so that the table detection network extracts the table position information of the table to be restored and the cell position information of each cell in the table to be restored from the target image, comprises:

[0014] The target image is encoded by the image encoding unit to obtain image features; the prompt information is encoded by the prompt information encoding unit to obtain prompt information encoding features; the image features and the prompt information encoding features are decoded by the decoding unit to obtain the table position information and the cell position information output by the decoding unit.

[0015] Furthermore, in the above method, extracting the structural information of the table to be restored from the target image according to the table position information includes:

[0016] The table position information and the target image are both input into the large language model, so that the large language model extracts the structural information of the table to be restored from the target image according to the table position information.

[0017] Furthermore, in the above method, the step of inputting both the table position information and the target image into a large language model so that the large language model extracts the structural information of the table to be restored from the target image according to the table position information includes:

[0018] Performing encoding processing on the target image by an image encoding unit to obtain image features;

[0019] The table position information is encoded by a prompt information encoding unit to obtain a table position information encoding feature;

[0020] The image features and the table position information encoding features are identified by the large language model to obtain the structural information output by the large language model.

[0021] Furthermore, in the above method, determining the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restoring the cell according to the structure information and the size of each cell and the address of each cell in the table to be restored, includes:

[0022] The cell position information, the structural information and the target image are all input into the large language model, so that the large language model determines the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restores the cells according to the structural information and the size of each cell and the address of each cell in the table to be restored.

[0023] Furthermore, in the above method, the cell position information, the structure information and the target image are all input into the large language model, so that the large language model determines the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restores the cell according to the structure information and the size of each cell and the address of each cell in the table to be restored, including:

[0024] Performing encoding processing on the target image by an image encoding unit to obtain image features;

[0025] The cell position information is encoded by a prompt information encoding unit to obtain a cell position information encoding feature, and the structure information is encoded by the prompt information encoding unit to obtain a structure information encoding feature;

[0026] The image features, the cell position information encoding features and the structure information encoding features are identified through the large language model to achieve cell restoration.

[0027] In a second aspect, an embodiment of the present application provides a table restoration device, comprising:

[0028] A first extraction module is used to extract table position information of the table to be restored and cell position information of each cell in the table to be restored from a target image containing the table to be restored;

[0029] A second extraction module is used to extract the structural information of the table to be restored from the target image according to the table position information; the structural information includes the cell content of each cell and the address of the cell content in the table to be restored;

[0030] The restoration module is used to determine the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restore the cell according to the structural information and the size of each cell and the address of each cell in the table to be restored.

[0031] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0032] A memory and a processor; wherein the memory is used to store programs; and the processor is used to implement any of the methods described above by running the programs in the memory.

[0033] In a fourth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the computer program implements any one of the above methods. Optionally, the computer program can be stored in a readable storage medium of a computer device or in the cloud; the processor of the computer device reads the computer program from the readable storage medium or the cloud.

[0034] The table restoration method proposed in the present application extracts table position information of the table to be restored and cell position information of each cell in the table to be restored from a target image containing the table to be restored, and extracts structural information of the table to be restored from the target image based on the table position information, wherein the structural information includes the cell content of each cell and the address of the cell content in the table to be restored, and then determines the size of each cell and the address of each cell in the table to be restored based on the cell position information, so as to perform cell restoration based on the structural information and the size of each cell and the address of each cell in the table to be restored. Such an arrangement can restore the size of each cell, the content of each cell and the layout of the table, thereby achieving the purpose of restoring the table during the digital processing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0036] Figure 1 It is a flowchart of a table restoration method provided in an embodiment of the present application.

[0037] Figure 2 It is a schematic diagram of a table to be restored provided in an embodiment of the present application.

[0038] Figure 3It is a structural diagram of a table detection network provided in an embodiment of the present application.

[0039] Figure 4 It is a schematic diagram of a process for obtaining structural information provided in an embodiment of the present application.

[0040] Figure 5 It is a flowchart of restoring a table based on structural information, target image and cell location information provided in an embodiment of the present application.

[0041] Figure 6 It is a structural schematic diagram of a table restoration device provided in an embodiment of the present application.

[0042] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] Tables are an important form of information expression. They can organize data into a standard structure to facilitate information retrieval and comparison. In modern data processing, tables often need to be digitized. Due to the characteristics of table data such as variable structure, diverse forms, numerous cell combinations, and complex row and column relationships, it is currently impossible to accurately parse the table structure during the digital processing process.

[0045] Specifically, tables can have infinite possible combinations based on certain layout rules, including complex operations such as splitting rows and columns, merging rows and columns, and cells are not completely aligned. In addition, in addition to the common wired tables, there are also table forms such as tables with few lines. Especially for tables with few lines, the alignment of cells is more complicated, and sometimes even ambiguous, making it difficult to correctly restore the structure and layout of the table. In addition, there are hand-drawn tables in real life, which brings more challenges and difficulties to the recognition of table structure.

[0046] The current mainstream approach is to first detect the cells and text content in the table, then use text recognition technology to parse the text content, and finally integrate and reconstruct the cells according to preset rules to restore the structure of the table. Although this method can parse and reconstruct the table layout to a certain extent, it is overly dependent on detection results and involves complex rule processing, resulting in poor robustness in practical applications and difficulty in forming a set of generally applicable post-processing rules.

[0047] Therefore, how to restore a table has become a technical problem that needs to be solved urgently by those skilled in the art. The present application proposes a table restoration method, device, electronic device and computer program product. The technical solution restores the table by restoring the cells according to the structural information of the cells, the size of each cell and the address of each cell in the table to be restored.

[0048] Specifically, the embodiment of the present application proposes a table restoration method, which can be executed by an electronic device, which can be any device with data and instruction processing functions, for example, a notebook computer, a tablet computer, a desktop computer, a mobile device (for example, a mobile phone, a personal digital assistant, a dedicated messaging device) and other types of user terminals or a combination of any two or more of these electronic devices, or a server. Figure 1 As shown, the method includes:

[0049] S101 . Extracting table position information of the table to be restored and cell position information of each cell in the table to be restored from a target image including the table to be restored.

[0050] The above-mentioned table to be restored refers to a table that needs to be restored. The type of the table to be restored is not limited in the embodiments of the present application. It can be a table with lines or a table with few lines, or an electronic table or a hand-drawn table, etc. The target image containing the table to be restored can only contain the table to be restored, or can contain other content in addition to the table to be restored, which is not limited in the present embodiment.

[0051] The table position information of the table to be restored can indicate the position of the table to be restored in the target image, that is, the table to be restored can be located in the target image based on the table position information. In some embodiments, the table position information refers to the coordinates of the table to be restored, for example, the table position information includes the coordinates of each vertex of the table to be restored. The table to be restored is generally a rectangle, and the position of the rectangle can be determined by the coordinates of two diagonally opposite vertices of the rectangle. Based on this, in some embodiments, the table position information at least includes the coordinates of two diagonally opposite vertices of the table to be restored, for example Figure 2 In the table to be restored shown, the table position information at least includes the coordinates of point A and point C, or the coordinates of point B and point D.

[0052] The cell position information of each cell in the table to be restored can indicate the position of each cell in the table to be restored in the target image, that is, each cell can be located in the target image based on the cell position information. In some embodiments, the cell position information refers to the coordinates of each cell in the table to be restored, for example, the cell position information includes the coordinates of each vertex of each cell. Each cell in the table to be restored is generally also a rectangle, so in some embodiments, the cell position information includes at least the coordinates of two diagonally opposite vertices of each cell, for example Figure 2 In the table to be restored shown, the cell position information of cell a at least includes the coordinates of point E and point G, or the coordinates of point F and point H.

[0053] In an embodiment of the present application, the above-mentioned table location information and cell location information are extracted from the target image. In some embodiments, the above-mentioned table location information and cell location information are extracted from the target image using a pre-trained first information extraction model. Specifically, a large number of sample images containing tables can be obtained as training samples, and the epitope location information and cell location information in the sample images can be marked as training labels. During training, the training samples are input into the first information extraction model to obtain the prediction results output by the first information extraction model. By comparing the prediction results output by the first information extraction model and the training labels, the loss of the first information extraction model is determined. With the goal of reducing the loss of the first information extraction model, the parameters of the first information extraction model are adjusted, and then the above-mentioned training process is repeated until the loss function value, accuracy and other indicators of the first information extraction model meet the requirements, and the first information extraction model training is completed. The above-mentioned first information extraction model can be obtained based on any neural network model training, or based on a pre-trained model training, such as a pre-trained large model similar to ChatGPT.

[0054] The target image is input into a pre-trained first information extraction model, so that the first information extraction model extracts table position information of the table to be restored and cell position information of each cell in the table to be restored from the target image containing the table to be restored.

[0055] S102: extracting structural information of the table to be restored from the target image according to the table position information.

[0056] The above structural information includes the cell content of each cell and the address of the cell content in the table to be restored. The cell content of each cell refers to the specific content filled in the cell of the table to be restored, and the address of the cell content in the table to be restored refers to the position of the cell content in the table to be restored, including the row number and column number of the cell content. For example, Figure 2 In the table, the cell content of cell a is "95", and the address of "95" in the table is row 4, column 4. In some embodiments, the html (Hyper Text Markup Language) representation of the table to be restored is used as the structural information of the table to be restored.

[0057] In an embodiment of the present application, the structural information of the table to be restored is extracted from the target image based on the table position information. In some embodiments, the structural information of the table to be restored is extracted from the target image using a pre-trained second information extraction model. Specifically, a large number of sample images containing tables can be obtained, the table position information in the sample images can be extracted, the sample images and the table position information can be used as training samples, and the structural information in the training samples can be marked as training labels. The specific training process is the same as the training process of the first information extraction model in the above-mentioned embodiment. The second information extraction model can be trained with reference to the training method of the first information extraction model, which will not be elaborated here. The above-mentioned second information extraction model can be obtained based on any neural network model training, or can be obtained based on a pre-trained model training, such as a pre-trained large model similar to ChatGPT.

[0058] The target image and the table position information are input into a pre-trained second information extraction model, so that the second information extraction model extracts the structural information of the table to be restored from the target image according to the table position information.

[0059] S103, determining the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restoring the cells according to the structure information and the size of each cell and the address of each cell in the table to be restored.

[0060] The size of the cell can be determined based on the cell position information. Specifically, the cell position information includes at least the coordinates of the two diagonally opposite vertices of each cell, and the size of each cell can be determined based on the coordinates of the two diagonally opposite vertices of each cell. Figure 2 In the illustrated embodiment, the cell position information of cell a includes the coordinates of point F and point H. The coordinates of point F are (x, y), and the coordinates of point H are (u, v). Thus, the length of cell a is (xu) and the width is (yv).

[0061] Based on the cell position information, the address of the cell in the table to be restored can be determined. The address of the cell in the table to be restored refers to the specific position of the cell in the table to be restored, including the row and column number of the cell. For example, Figure 2In the table, the address of cell a in the table is row 4 and column 4. Specifically, the cells can be restored according to the cell position information, that is, each cell is placed according to the coordinates in the cell position information, and the layout of each cell in the table to be restored is restored. After the table layout is restored, the row and column number of each cell can be clearly known, and the address of each cell in the table to be restored can be obtained.

[0062] The above structural information includes the cell content of each cell and the address of the cell content in the table to be restored. After obtaining the cell content of each cell and the address of the cell content in the table to be restored according to the steps of the above embodiment, the cell content is filled into the corresponding cell according to the address, and the cell can be restored. For example, the cell content with the address of the first row and the first column is filled into the cell with the address of the first row and the first column, the cell content with the address of the first row and the second column is filled into the cell with the address of the first row and the second column, the cell content with the address of the first row and the third column is filled into the cell with the address of the first row and the third column, ..., until the cell is restored.

[0063] The above embodiment extracts the table position information of the table to be restored and the cell position information of each cell in the table to be restored from the target image containing the table to be restored, and extracts the structural information of the table to be restored from the target image based on the table position information, wherein the structural information includes the cell content of each cell and the address of the cell content in the table to be restored, and then determines the size of each cell and the address of each cell in the table to be restored based on the cell position information, so as to perform cell restoration based on the structural information and the size of each cell and the address of each cell in the table to be restored. Such an arrangement can restore the size of each cell, the content of each cell and the layout of the table, thereby achieving the purpose of restoring the table during the digital processing process.

[0064] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment extract the table position information of the table to be restored and the cell position information of each cell in the table to be restored from the target image containing the table to be restored, which may specifically include the following steps:

[0065] The prompt information and the target image containing the table to be restored are input into the table detection network, so that the table detection network extracts the table position information of the table to be restored and the cell position information of each cell in the table to be restored from the target image.

[0066] Prompt refers to text or instructions input when interacting with the table detection network, which is used to guide the table detection network to generate specific output content. The above prompt information in this embodiment is used to instruct the table detection network to perform the task of extracting table location information and cell location information from the target image. In some embodiments, the prompt information is set to "Please output the table location information and cell location information in the figure."

[0067] It should be noted that the specific training process of the table detection network is the same as the training process of the first information extraction model in the above embodiment. The table detection network can be trained with reference to the training method of the first information extraction model, which will not be described here. The above-mentioned table detection network can be trained based on any neural network model, or based on a pre-trained model, such as a pre-trained large model similar to ChatGPT.

[0068] In an embodiment of the present application, both the prompt information and the target image are input into a table detection network, and the prompt information is used to instruct the table detection network to perform the task of extracting table location information and cell location information from the target image, so that the table detection network can understand and generate more accurate and relevant content.

[0069] As an optional implementation, another embodiment of the present application discloses that the table detection network of the above embodiment includes an image encoding unit, a prompt information encoding unit and a decoding unit. The steps of the above embodiment input the prompt information and the target image containing the table to be restored into the table detection network, so that the table detection network extracts the table position information of the table to be restored and the cell position information of each cell in the table to be restored from the target image, which may specifically include the following steps:

[0070] The target image is encoded by the image encoding unit to obtain image features; the prompt information is encoded by the prompt information encoding unit to obtain prompt information encoding features; the image features and prompt information encoding features are decoded by the decoding unit to obtain table position information and cell position information output by the decoding unit.

[0071] The Segment Anything Model (SAM) can be used as the basic architecture of the table detection network. Figure 3 As shown, the table detection network of this embodiment includes an image encoder, a prompt encoder, and a decoding unit. The decoding unit may be a mask decoder.

[0072] The image encoding unit is used to encode the target image to obtain the image feature image embed In some embodiments, the image encoding unit uses a ViT (Vision Transformer) model pre-trained by MAE (Masked Autoencoders). Each block can be expressed by the following mathematical formula:

[0073]

[0074] z' l =MSA(LN(z l-1 ))+z l-1 ,l=1…L

[0075] z l =MLP(LN(z' l ))+z' i l=1…L

[0076]

[0077] The prompt information encoding unit is used to encode the prompt information to obtain the prompt information encoding feature text embed In some embodiments, the prompt encoder may use a pre-trained CLIP (Contrastive Language-Image Pre-training) text encoder.

[0078] The decoding unit is used to decode the image features and the prompt information encoding features to obtain the table position information and the cell position information output by the decoding unit. Specifically, the image features and the prompt information encoding features are spliced ​​and input into the decoding unit, and the decoding unit decodes the spliced ​​features to obtain the table position information and the cell position information. In some embodiments, the table position information includes the coordinates Pos of two vertices of the table that are diagonally opposite to each other. table (x1, y1; x2, y2), the cell location information includes the coordinates of the two vertices of each cell diagonally opposite to each other. cell (x3,y3;x4,y4).

[0079] In the above embodiment, the table position information and the cell position information can be extracted, so as to restore the above table based on the table position information and the cell position information.

[0080] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment extract the structural information of the table to be restored from the target image according to the table position information, which may specifically include the following steps:

[0081] The table position information and the target image are both input into the large language model, so that the large language model extracts the structural information of the table to be restored from the target image according to the table position information.

[0082] The table location information can be used as prompt information and input into a large language model (LLM) together with the target image, so that the large language model can perform the task of extracting the structural information of the table to be restored from the target image.

[0083] It should be noted that the specific training process of the large language model is the same as the training process of the second information extraction model in the above embodiment. The table detection network can be trained with reference to the training method of the second information extraction model, which will not be described in detail here.

[0084] In an embodiment of the present application, both the table location information and the target image are input into the large language model, and the table location information is used as prompt information to instruct the large language model to perform the task of extracting the structural information of the table to be restored from the target image, so that the large language model can understand and generate more accurate and relevant content.

[0085] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment input both the table position information and the target image into the large language model, so that the large language model extracts the structural information of the table to be restored from the target image according to the table position information, which may specifically include the following steps:

[0086] The target image is encoded by the image encoding unit to obtain image features. The table position information is encoded by the prompt information encoding unit to obtain table position information encoding features. The image features and table position information encoding features are identified by the large language model to obtain structural information output by the large language model.

[0087] Specifically, Figure 4 As shown, in this embodiment, the target image is encoded to obtain the image feature image embed The above-mentioned image encoding unit can be used to encode the target image to obtain image features; a separate image encoding unit can also be set to encode the target image to obtain image features; in addition, the image features extracted in the above-mentioned embodiment can also be directly used, which is not limited in this embodiment. It should be noted that Figure 4 In the embodiment shown, the image feature image extracted in the above embodiment is directly used. embed .

[0088] The table position information is encoded using the prompt information encoding unit to obtain the table position information encoding feature tableembed The above-mentioned prompt information encoding unit may be used to encode the table position information to obtain the table position information encoding feature; a separate prompt information encoding unit may also be set to encode the table position information to obtain the table position information encoding feature, which is not limited in this embodiment.

[0089] The large language model is used to identify the image features and the table position information encoding features to obtain the structural information output by the large language model. Specifically, the image features and the table position information encoding features are spliced ​​and input into the large language model, and the spliced ​​features are identified by the large language model to obtain the structural information.

[0090] The above process can be described by the following formula:

[0091] table embed =Prompt_Encoder(Pos table (x1,y1,x2,y2))

[0092] html=LLM([table embed ,image embed ])

[0093] Wherein, html represents the structural information of the table to be restored. In this embodiment, the html representation of the table to be restored is used as the structural information of the table to be restored.

[0094] In the above embodiment, the structural information of the table to be restored can be extracted, so as to restore the above table based on the structural information of the table to be restored.

[0095] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment determine the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restore the cell according to the structure information and the size of each cell and the address of each cell in the table to be restored, which may specifically include the following steps:

[0096] The cell position information, structural information and target image are all input into the large language model, so that the large language model determines the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restores the cells according to the structural information and the size of each cell and the address of each cell in the table to be restored.

[0097] The cell position information and structure information are used as prompt information and are input into the large language model together with the target image, so that the large language model can perform the task of determining the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restoring the cells according to the structure information and the size of each cell and the address of each cell in the table to be restored.

[0098] In an embodiment of the present application, the cell position information, structural information and target image are all input into the large language model, and the cell position information and structural information are used as prompt information to instruct the large language model to perform the tasks of determining the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restoring the cells according to the structural information and the size of each cell and the address of each cell in the table to be restored, so that the large language model can understand and generate more accurate and relevant content.

[0099] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment input the cell position information, the structural information and the target image into the large language model, so that the large language model determines the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restores the cell according to the structural information and the size of each cell and the address of each cell in the table to be restored, which may specifically include the following steps:

[0100] The target image is encoded by the image encoding unit to obtain image features. The cell position information is encoded by the prompt information encoding unit to obtain cell position information encoding features. The structural information is encoded by the prompt information encoding unit to obtain structural information encoding features. The image features, cell position information encoding features and structural information encoding features are identified by the large language model to achieve cell restoration.

[0101] After the above steps, all cell location information has been obtained And the HTML representation of the table. However, since the cell position information output by the table detection network is in a disordered state, that is, the table detection network is not completely sorted according to the human reading order, and the output is from the first cell to the last cell, it is impossible to confirm the address of the cell in the table to be restored, and it is impossible to restore the layout information of the table to be restored. Therefore, the cell position can be reordered according to the steps of the following embodiment, that is, ordered cell information can be obtained.

[0102] Specifically, Figure 5 As shown, in this embodiment, the target image is encoded to obtain the image feature image embedThe above-mentioned image encoding unit can be used to encode the target image to obtain image features; a separate image encoding unit can also be set to encode the target image to obtain image features; in addition, the image features extracted in the above-mentioned embodiment can also be directly used, which is not limited in this embodiment. It should be noted that Figure 5 In the embodiment shown, the image feature image extracted in the above embodiment is directly used. embed .

[0103] The cell position information is encoded using the prompt information encoding unit to obtain the cell position information encoding feature The structure information is encoded by the prompt information encoding unit to obtain the structure information encoding feature html embed The above-mentioned prompt information encoding unit can be used to encode the cell position information and the structural information; a separate prompt information encoding unit can also be set to encode the cell position information and the structural information, which is not limited in this embodiment.

[0104] The large language model is used to identify image features, cell location information encoding features and structural information encoding features to achieve cell restoration. Specifically, the image features, cell location information encoding features and structural information encoding features are spliced ​​and input into the large language model, and the spliced ​​features are identified by the large language model to facilitate the ordering of cells so that the layout of the cells is consistent with the layout of the table to be identified, and then the cell contents are filled into the corresponding cells according to the addresses in the table to be restored.

[0105] The above process can be described by the following formula:

[0106] html embed =Prompt_Encoder(html)

[0107]

[0108] In the above-mentioned embodiment, the cell position information, structure information and target image can be used to achieve the purpose of restoring the table.

[0109] Corresponding to the above table restoration method, the present application embodiment also discloses a table restoration device, see Figure 6 As shown, the device comprises:

[0110] The first extraction module 100 is used to extract the table position information of the table to be restored and the cell position information of each cell in the table to be restored from the target image containing the table to be restored;

[0111] The second extraction module 110 is used to extract the structural information of the table to be restored from the target image according to the table position information; the structural information includes the cell content of each cell and the address of the cell content in the table to be restored;

[0112] The restoration module 120 is used to determine the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restore the cell according to the structure information and the size of each cell and the address of each cell in the table to be restored.

[0113] As an optional implementation, another embodiment of the present application discloses that the first extraction module 100 in the above embodiment is specifically used for:

[0114] The prompt information and the target image containing the table to be restored are both input into the table detection network, so that the table detection network can extract the table position information of the table to be restored and the cell position information of each cell in the table to be restored from the target image; the prompt information is used to instruct the table detection network to perform the task of extracting the table position information and the cell position information from the target image.

[0115] As an optional implementation, another embodiment of the present application discloses that the table detection network in the above embodiment includes an image encoding unit, a prompt information encoding unit and a decoding unit; the first extraction module 100 in the above embodiment is specifically used for:

[0116] The target image is encoded by the image encoding unit to obtain image features; the prompt information is encoded by the prompt information encoding unit to obtain prompt information encoding features; the image features and prompt information encoding features are decoded by the decoding unit to obtain table position information and cell position information output by the decoding unit.

[0117] As an optional implementation, another embodiment of the present application discloses that the second extraction module 110 in the above embodiment is specifically used for:

[0118] The table position information and the target image are both input into the large language model, so that the large language model extracts the structural information of the table to be restored from the target image according to the table position information.

[0119] As an optional implementation, another embodiment of the present application discloses that the second extraction module 110 in the above embodiment is specifically used for:

[0120] The target image is encoded by the image encoding unit to obtain image features. The table position information is encoded by the prompt information encoding unit to obtain table position information encoding features. The image features and table position information encoding features are identified by the large language model to obtain structural information output by the large language model.

[0121] As an optional implementation, another embodiment of the present application discloses that the restoration module 120 in the above embodiment is specifically used for:

[0122] The cell position information, structural information and target image are all input into the large language model, so that the large language model determines the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restores the cells according to the structural information and the size of each cell and the address of each cell in the table to be restored.

[0123] As an optional implementation, another embodiment of the present application discloses that the restoration module 120 in the above embodiment is specifically used for:

[0124] The target image is encoded by the image encoding unit to obtain image features. The cell position information is encoded by the prompt information encoding unit to obtain cell position information encoding features. The structural information is encoded by the prompt information encoding unit to obtain structural information encoding features. The image features, cell position information encoding features and structural information encoding features are identified by the large language model to achieve cell restoration.

[0125] Specifically, the device provided in this embodiment belongs to the same application concept as the method provided in the above embodiment of this application, can execute the method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. For the technical details not fully described in this embodiment, please refer to the specific processing content of the method provided in the above embodiment of this application, which will not be repeated here.

[0126] The functions implemented by the above modules can be implemented by the same or different processors respectively, and the embodiments of the present application are not limited thereto.

[0127] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0128] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0129] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0130] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0131] An embodiment of the present application further provides a control device, which includes a processor and an interface circuit. The processor in the control device is connected to an input-output component through the interface circuit of the control device.

[0132] The input-output component specifically refers to a hardware component that enables a user to input information and output information to the user, such as a microphone, keyboard, handwriting tablet, touch screen, display, speaker, printer, etc.

[0133] The above-mentioned interface circuit can be any interface circuit that can realize the data communication function, for example, it can be a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIE circuit, etc.

[0134] The processor in the control device is a circuit with signal processing capability, which realizes the purpose of restoring the table in the digital processing process by executing any one of the table restoration methods introduced in the above embodiments. The specific implementation of the processor can refer to the above processor implementation, and the embodiments of the present application are not strictly limited.

[0135] When the control device is applied to a device with a human-computer interaction function, the input and output components of the control device may be input components and output components on the device, such as a microphone, a keyboard, a handwriting tablet, a touch screen, a display, an audio player, etc. At the same time, the processor of the control device may be a CPU or GPU, etc. provided by the device, and the interface circuit of the control device may be an interface circuit between the information input component of the device and a processor such as a CPU or GPU.

[0136] Corresponding to the above table restoration method, the present application embodiment also discloses an electronic device, see Figure 7 As shown, the electronic device includes:

[0137] Memory 200 and processor 210;

[0138] The memory 200 is connected to the processor 210 and is used to store programs;

[0139] The processor 210 is configured to implement the table restoration method disclosed in any of the above embodiments by running the program stored in the memory 200 .

[0140] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0141] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0142] A bus may include a pathway that transfers information between components of a computer system.

[0143] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0144] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0145] The memory 200 stores a program for executing the technical solution of the present application, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes computer operation instructions. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0146] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0147] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0148] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0149] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of the table restoration method provided in the above embodiment of the present application.

[0150] In addition to the above methods and devices, the embodiments of the present application may also be a computer program product, which includes a computer program. When the computer program is executed by a processor, the table restoration method provided in any of the above embodiments of the present application may be executed. Optionally, the computer program may be stored in a readable storage medium or in the cloud of a computer device; the processor of the computer device reads the computer program from the readable storage medium or the cloud.

[0151] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0152] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK).

[0153] In addition, the embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes each step of the table restoration method provided in the above embodiment.

[0154] The computer readable storage medium may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0155] Specifically, the specific working contents of each part of the above-mentioned electronic device, computer program product and storage medium, as well as the specific processing contents when the computer program product or the computer program on the above-mentioned storage medium is executed by the processor, can be found in the contents of each embodiment of the restoration method in the above-mentioned table, and will not be repeated here.

[0156] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0157] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0158] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0159] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0160] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, such as multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0161] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0163] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0164] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0165] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0166] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for restoring a table, characterized in that: include: Extracting table position information of the table to be restored and cell position information of each cell in the table to be restored from a target image including the table to be restored; Extracting structural information of the table to be restored from the target image according to the table position information; The structural information includes the cell content of each cell and the address of the cell content in the table to be restored; The size of each cell and the address of each cell in the table to be restored are determined according to the cell position information, and the cell restoration is performed according to the structural information and the size of each cell and the address of each cell in the table to be restored.

2. The method according to claim 1, characterized in that The step of extracting table position information of the table to be restored and cell position information of each cell in the table to be restored from the target image containing the table to be restored includes: Inputting the prompt information and the target image containing the table to be restored into the table detection network, so that the table detection network extracts the table position information of the table to be restored and the cell position information of each cell in the table to be restored from the target image; The prompt information is used to instruct the table detection network to perform a task of extracting the table position information and the cell position information from the target image.

3. The method according to claim 2, characterized in that The table detection network includes an image encoding unit, a prompt information encoding unit and a decoding unit; The step of inputting the prompt information and the target image containing the table to be restored into the table detection network, so that the table detection network extracts the table position information of the table to be restored and the cell position information of each cell in the table to be restored from the target image, comprises: The target image is encoded by the image encoding unit to obtain image features; the prompt information is encoded by the prompt information encoding unit to obtain prompt information encoding features; the image features and the prompt information encoding features are decoded by the decoding unit to obtain the table position information and the cell position information output by the decoding unit.

4. The method according to claim 1, characterized in that The step of extracting the structure information of the table to be restored from the target image according to the table position information includes: The table position information and the target image are both input into the large language model, so that the large language model extracts the structural information of the table to be restored from the target image according to the table position information.

5. The method according to claim 4, characterized in that The step of inputting both the table position information and the target image into a large language model so that the large language model extracts the structural information of the table to be restored from the target image according to the table position information includes: Performing encoding processing on the target image by an image encoding unit to obtain image features; The table position information is encoded by a prompt information encoding unit to obtain a table position information encoding feature; The image features and the table position information encoding features are identified by the large language model to obtain the structural information output by the large language model.

6. The method according to claim 1, characterized in that Determining the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restoring the cell according to the structure information and the size of each cell and the address of each cell in the table to be restored, includes: The cell position information, the structural information and the target image are all input into the large language model, so that the large language model determines the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restores the cells according to the structural information and the size of each cell and the address of each cell in the table to be restored.

7. The method according to claim 6, characterized in that The step of inputting the cell position information, the structure information and the target image into a large language model so that the large language model determines the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restoring the cell according to the structure information and the size of each cell and the address of each cell in the table to be restored includes: Performing encoding processing on the target image by an image encoding unit to obtain image features; The cell position information is encoded by a prompt information encoding unit to obtain a cell position information encoding feature, and the structure information is encoded by the prompt information encoding unit to obtain a structure information encoding feature; The image features, the cell position information encoding features and the structure information encoding features are identified through the large language model to achieve cell restoration.

8. A table restoration device, characterized in that: include: A first extraction module is used to extract table position information of the table to be restored and cell position information of each cell in the table to be restored from a target image containing the table to be restored; A second extraction module is used to extract the structural information of the table to be restored from the target image according to the table position information; the structural information includes the cell content of each cell and the address of the cell content in the table to be restored; The restoration module is used to determine the size of each cell and the address of each cell in the table to be restored according to the cell position information, and restore the cell according to the structural information and the size of each cell and the address of each cell in the table to be restored.

9. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor is used to implement the method according to any one of claims 1 to 7 by running the program in the memory.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images