Table restoration method and apparatus

Abstract

The present application provides a table restoration method and apparatus based on optical character recognition, for use in respectively performing structure recognition on rows and columns in a table, and using text detection to recognize texts in a same row to more accurately recognize a row textbox, thereby achieving more accurate table structure restoration. The method comprises: first, performing textbox detection on input data to obtain a scanning result, wherein the input data may specifically comprise an image or other non-readable documents, etc., the input data comprises an input table, and the scanning result comprises information of at least one textbox in the input table, such as information of the coordinates, length, width or area of the textbox; then, performing text detection on the at least one textbox to determine a row textbox, and on the basis of the at least one textbox, determining a column textbox; and then, merging the row textbox and the column textbox to obtain an output table, wherein the output table is generally a readable table.

Description

A method and apparatus for restoring a table

[0001] This application claims priority to Chinese Patent Application No. 202410535017.0, filed on April 29, 2024, entitled "A Method and Apparatus for Reconstructing a Table", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of optical character recognition, and in particular to a method and apparatus for restoring a table. Background Technology

[0003] In the context of digitalization across industries, there is a need to digitize electronic documents accumulated over many years. Images and scanned documents are the main document formats. Manually entering information is very labor-intensive. To improve operational efficiency, it is necessary to perform optical character recognition (OCR) to recognize and restore various scanned documents.

[0004] Tables are a commonly used data format, offering clear, concise, and efficient expression. They are widely used in electronic documents across various industries, and restoring the table structure is a crucial step in the digitization of scanned documents. Complex and dense table structures, especially in wireless formats, present a significant challenge for OCR recognition and restoration.

[0005] Therefore, improving the accuracy of table structure restoration has become an urgent problem to be solved. Summary of the Invention

[0006] This application provides a table reconstruction method and apparatus for performing structural recognition on table rows and columns separately, and using text detection to identify text in the same row, thereby more accurately identifying text boxes and achieving more accurate table structure reconstruction.

[0007] In view of this, in a first aspect, this application provides a table restoration method, comprising: firstly, performing text box detection on input data to obtain a scanning result, wherein the input data may specifically include an image or other unreadable document, the input data includes an input table, and the scanning result includes information of at least one text box in the input table, such as the coordinates, length, width, or area of ​​the text box representing the text box area; subsequently, performing text detection on the at least one text box to determine row text boxes, and determining column text boxes based on the at least one text box; subsequently, merging the row text boxes and column text boxes to obtain an output table, wherein the output table is typically a readable table, and is higher resolution and has readability and writability compared to the input table.

[0008] In this embodiment, during the detection of rows in the table area, detection is performed from both row and column dimensions. Text detection is incorporated into the identification of row text boxes, allowing for accurate identification of text boxes within the same row by leveraging text semantics. For example, the text semantics between different rows within the same cell are usually coherent, thus accurately identifying cases where the same cell contains text from different rows, leading to more precise row text box identification. This allows for subsequent row and column merging based on more accurate row text boxes, resulting in a more accurate output table.

[0009] In one possible implementation, the aforementioned method may further include: performing text recognition on at least one text box to obtain the text within the at least one text box. Accordingly, the aforementioned text detection of the table area to determine the line text boxes may include: performing natural language processing (NLP) on the table area in the input data to identify the line text boxes, that is, using NLP to identify the semantic meaning of the text in each text box in the input data, thereby accurately identifying the text in the line text boxes, and thus identifying accurate line text boxes. For example, in the case where there are multiple lines of text in a text box, NLP can be used to identify the text content belonging to the same text box, achieving more accurate line text box identification.

[0010] In one possible implementation, the aforementioned natural language processing of the text in at least one text box to identify line text boxes includes: inputting input data into a line detection model and outputting a line detection box, which can be used to represent the region of each row in the input table; determining an initial line text box based on the line detection box and each text box in the scanning result, for example, filling each text box in the scanning result into the corresponding line detection box based on its coordinates; and performing natural language processing on the text in the initial line text box to identify the line text box.

[0011] In this embodiment, a pre-trained row detection model can be used to output the row detection box corresponding to the input table. That is, the region corresponding to the row of the input table is divided from the input data, and the initial row text box is obtained based on the region. NLP is then performed to refine the initial row text box, thereby obtaining a more accurate row text box.

[0012] In one possible implementation, the aforementioned determination of column text boxes based on at least one text box includes: clustering the at least one text box based on its coordinates to obtain column-clustered text boxes, for example, clustering the coordinates of the text boxes and classifying text boxes with similar column coordinates as text boxes in the same column; and determining column text boxes based on the column-clustered text boxes, for example, classifying text boxes in a cluster as text boxes in a column, i.e., column text boxes. In this embodiment of the application, clustering can be used to identify text boxes in the same column, thereby accurately identifying text boxes in the same column based on the column coordinates of each text box.

[0013] In one possible implementation, the aforementioned determination of column text boxes based on at least one text box includes: taking input data as input to a pre-trained column detection model, outputting column detection boxes, which can be used to represent the area corresponding to each column in the input table; then determining column text boxes based on the column detection boxes and the aforementioned at least one text box, for example, filling each text box into the corresponding column detection box based on the coordinates of each text box, and outputting the column text boxes.

[0014] In this embodiment of the application, a pre-trained column detection model can also be used to perform the detection task and output the column regions in the table area. In this way, the detection capability of the model can be used to accurately identify the column regions of the table and output more accurate column text boxes.

[0015] In one possible implementation, the aforementioned determination of column text boxes based on at least one text box includes: first, identifying text boxes in a table area of ​​the input data; clustering the coordinates of the text boxes in the table area to obtain column cluster text boxes; furthermore, using the input data as input to a pre-trained column detection model to output column detection boxes; subsequently fusing the column cluster text boxes and the class detection boxes to obtain column text boxes, for example, filling each text box in the column cluster text boxes into column detection boxes whose coordinates are close to those of each cluster center column, and outputting the column text boxes.

[0016] In this embodiment, clustering and column detection models can be combined to identify column text boxes, thereby combining multiple methods to determine column text boxes, providing multiple guarantees for the identification of column text boxes, and further improving the accuracy of the output column text boxes.

[0017] In one possible implementation, the aforementioned method further includes: performing layout analysis on the input data and outputting a table area, the table area including the region of the input table within the input data; wherein, if a row detection model exists, the table area is used as input for the row detection model, and if a column detection model exists, the table area is used as input for the column detection model. In this embodiment, a layout analysis step can be added to distinguish multiple types of regions that may exist in the input data, such as distinguishing headers or footers, thereby further filtering out regions unrelated to the table and achieving table region detection.

[0018] In one possible implementation, the aforementioned step of inputting input data into the layout analysis model and outputting a table area includes: correcting the tilt angle of the input data to obtain corrected input data; and inputting the corrected input data into the layout analysis model to output a table area. Therefore, in this embodiment, a tilt correction step can be added to accommodate situations where the table area in the input data has a certain tilt angle.

[0019] Secondly, this application provides a table restoration method, characterized by comprising: first, performing text box detection on input data to obtain a scanning result, wherein the input data includes an input table, and the scanning result includes information of at least one text box in the input table, such as the coordinates, length, width, or area of ​​the text box representing the text box area; subsequently, determining row text boxes based on the position of at least one text box; determining column text boxes based on at least one text box, wherein the column text boxes are obtained by clustering at least one text box, and / or, the column text boxes are obtained by column detection boxes output by a column detection model, wherein the column detection boxes represent the area of ​​each column in the input table; merging the row text boxes and column text boxes to obtain an output table, wherein the row text boxes include text boxes in the same row in the output table, and the column text boxes include text boxes in the same row in the output table.

[0020] In this embodiment, clustering and / or column detection models are used to accurately identify the column text boxes in the input table, thereby improving the accuracy of column reconstruction and resulting in a more accurate output table. Furthermore, compared to distinguishing column text boxes solely based on their coordinates, the clustering method used in this application constrains the coordinates of text boxes within the same column, making their coordinates more similar. The column detection model accurately identifies the region where each column in the input table is located, allowing text boxes to be filled into the corresponding column regions, resulting in more accurate column text boxes.

[0021] In one possible implementation, when the column text box is determined by combining the clustering of at least one text box with the column detection box output by the column detection model, the aforementioned determination of the column text box based on at least one text box includes: clustering the text boxes in the table area to obtain the column clustering text box; inputting the input table area in the input data to the column detection model and outputting the column detection box; and fusing the column clustering text box and the column detection box to obtain the column text box.

[0022] In this embodiment, the column text boxes output by clustering can constrain the coordinates of text boxes in the same column, making the coordinates of text boxes in the same column more similar; and by using the column detection model, the region where each column in the input table is located can be accurately identified, and then the text boxes can be filled into the corresponding column regions to obtain more accurate column text boxes.

[0023] In one possible implementation, when the column text boxes are clustered together with at least one text box, the aforementioned determination of column text boxes based on at least one text box includes: clustering the at least one text box based on its coordinates to obtain column-clustered text boxes, for example, clustering the coordinates of the text boxes and classifying text boxes with similar column coordinates as text boxes in the same column; and determining column text boxes based on the column-clustered text boxes, for example, classifying text boxes in one cluster as text boxes in a column, i.e., column text boxes. In this embodiment of the application, clustering can be used to identify text boxes in the same column, thereby accurately identifying text boxes in the same column based on the column coordinates of each text box.

[0024] In one possible implementation, when the column text boxes are determined based on column detection boxes output by a column detection model, the aforementioned determination of column text boxes based on at least one text box includes: taking input data as input to a pre-trained column detection model, outputting column detection boxes that can represent the regions corresponding to each column in the input table; subsequently determining column text boxes based on the column detection boxes and the aforementioned at least one text box, for example, filling each text box into its corresponding column detection box based on the coordinates of each text box, and outputting the column text boxes. Therefore, by using a pre-trained column detection model to perform the detection task and outputting the regions of columns in the table area, the detection capabilities of the model can be utilized to accurately identify the regions of table columns, thereby outputting more accurate column text boxes.

[0025] In one possible implementation, the aforementioned method may further include: performing text recognition on at least one text box to obtain the text within the at least one text box. Accordingly, the aforementioned determination of line text boxes based on at least one text box may include: performing natural language processing (NLP) on the table area in the input data to identify line text boxes, that is, using NLP to identify the semantic meaning of the text in each text box in the input data, thereby accurately identifying the text in the line text boxes, and thus identifying accurate line text boxes. For example, in the case where a text box contains multiple lines of text, NLP can be used to identify the text content belonging to the same text box, achieving more accurate line text box identification.

[0026] In one possible implementation, the aforementioned natural language processing of the text in at least one text box to identify line text boxes includes: inputting input data into a line detection model and outputting a line detection box, which can be used to represent the region of each row in the input table; determining an initial line text box based on the line detection box and each text box in the scanning result, for example, filling each text box in the scanning result into the corresponding line detection box based on its coordinates; and performing natural language processing on the text in the initial line text box to identify the line text box.

[0027] In this embodiment, a pre-trained row detection model can be used to output the row detection box corresponding to the input table. That is, the region corresponding to the row of the input table is divided from the input data, and the initial row text box is obtained based on the region. NLP is then performed to refine the initial row text box, thereby obtaining a more accurate row text box.

[0028] In one possible implementation, the aforementioned method further includes: performing layout analysis on the input data and outputting a table area, the table area including the region of the input table within the input data; wherein, if a row detection model exists, the table area is used as input for the row detection model, and if a column detection model exists, the table area is used as input for the column detection model. In this embodiment, a layout analysis step can be added to distinguish multiple types of regions that may exist in the input data, such as distinguishing headers or footers, thereby further filtering out regions unrelated to the table and achieving table region detection.

[0029] In one possible implementation, the aforementioned step of inputting input data into the layout analysis model and outputting a table area includes: correcting the tilt angle of the input data to obtain corrected input data; and inputting the corrected input data into the layout analysis model to output a table area. Therefore, in this embodiment, a tilt correction step can be added to accommodate situations where the table area in the input data has a certain tilt angle.

[0030] Thirdly, this application provides a form restoration apparatus, comprising:

[0031] The scanning module is used to perform text box detection on the input data to obtain the scanning result. The input data includes an input table, and the scanning result includes information from at least one text box in the input table.

[0032] The line processing module is used to perform text detection on at least one text box to determine the line text box;

[0033] The column processing module is used to determine the column text boxes based on at least one text box.

[0034] The merge module is used to merge row text boxes and column text boxes to obtain the output table.

[0035] The effects achieved by the third aspect and any optional implementation of the third aspect can be referred to the description of the first aspect or any optional implementation of the first aspect mentioned above, and will not be repeated here.

[0036] In one possible implementation, the line processing module is specifically used for: performing text recognition on at least one text box to obtain the text in at least one text box; and performing natural language processing on the text in at least one text box to recognize the line text box.

[0037] In one possible implementation, the row processing module is specifically used for: inputting input data into the row detection model and outputting row detection boxes, the row detection boxes being used to represent the region of each row in the input table; determining initial row text boxes based on the row detection boxes and the scanning results; and performing natural language processing on the text in the initial row text boxes to identify the row text boxes.

[0038] In one possible implementation, the column processing module is specifically used for: clustering at least one text box to obtain column clustered text boxes; and determining column text boxes based on the column clustered text boxes.

[0039] In one possible implementation, the column processing module is specifically used to: input input data into the column detection model, output column detection boxes, the column detection boxes being used to represent the area of ​​each column in the input table; and determine column text boxes based on the column detection boxes and the scanning results.

[0040] In one possible implementation, the column processing module is specifically used for: clustering text boxes in a table area to obtain column clustering text boxes; inputting input data into a column detection model and outputting column detection boxes, which are used to represent the area of ​​each column in the input table; and fusing the column clustering text boxes and the column detection boxes to obtain column text boxes.

[0041] In one possible implementation, the apparatus further includes: a layout analysis module for inputting input data into a layout analysis model and outputting a table area, the table area including the area of ​​the input table in the input data; wherein, in the presence of a row detection model, the table area is used as input to the row detection model, and in the presence of a column detection model, the table area is used as input to the column detection model.

[0042] In one possible implementation, the device further includes: a correction module for correcting the tilt angle of the input data to obtain corrected input data;

[0043] The layout analysis module is specifically used to input the corrected input data into the layout analysis model and output a table area.

[0044] Fourthly, this application provides a form restoration apparatus, comprising:

[0045] The scanning module is used to detect text boxes in the input data and obtain the scanning results. The input data includes an input table, and the scanning results include information from at least one text box in the input table.

[0046] The line processing module is used to determine the line text box based on at least one text box;

[0047] The column processing module is used to determine column text boxes based on at least one text box. The column text boxes are obtained by clustering at least one text box, and / or the column text boxes are obtained based on column detection boxes output by the column detection model. The column detection boxes represent the area of ​​each column in the input table.

[0048] The merge module is used to merge row text boxes and column text boxes to obtain an output table. The row text boxes include text boxes in the same row as the text boxes in the output table, and the column text boxes include text boxes in the same row as the text boxes in the output table.

[0049] The effects achieved by the fourth aspect and any optional implementation of the fourth aspect can be referred to the description of the second aspect or any optional implementation of the second aspect mentioned above, and will not be repeated here.

[0050] In one possible implementation, when the column text box is determined by combining the clustering of at least one text box with the column detection box output by the column detection model, the column processing module is specifically used to: cluster the text boxes in the table area to obtain the column clustering text box; input the input table area in the input data to the column detection model and output the column detection box; and fuse the column clustering text box and the column detection box to obtain the column text box.

[0051] Fifthly, embodiments of this application provide a computing device, including a processor and a memory; the processor of at least one computing device is configured to execute instructions stored in the memory of at least one computing device to cause the computing device to perform method steps as in any implementation of the first or second aspect.

[0052] In a sixth aspect, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs method steps as in any implementation of the first aspect or the second aspect.

[0053] In a seventh aspect, embodiments of this application provide a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform a method as described in either the first or second aspect.

[0054] Eighthly, embodiments of this application provide a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, enable the cluster of computing devices to perform a method as described in either the first or second aspect.

[0055] Ninthly, embodiments of this application provide a chip including at least one processor and an interface; the at least one processor obtains program instructions or data through the interface; the at least one processor is used to execute program line instructions to implement the method in any implementation of the first aspect or the second aspect. Attached Figure Description

[0056] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0057] Figure 2 is a flowchart illustrating a table restoration method provided in this application;

[0058] Figure 3 is a schematic diagram of an application scenario provided in this application;

[0059] Figure 4 is a schematic diagram of another application scenario provided in this application;

[0060] Figure 5 is a schematic diagram of another application scenario provided in this application;

[0061] Figure 6 is a schematic diagram of another application scenario provided in this application;

[0062] Figure 7 is a schematic diagram of another application scenario provided in this application;

[0063] Figure 8 is a flowchart illustrating another table restoration method provided in this application;

[0064] Figure 9 is a schematic diagram of another application scenario provided in this application;

[0065] Figure 10 is a flowchart illustrating another table restoration method provided in this application;

[0066] Figure 11 is a flowchart illustrating another table restoration method provided in this application;

[0067] Figure 12 is a schematic diagram of another application scenario provided by this application;

[0068] Figure 13 is a schematic diagram of another application scenario provided by this application;

[0069] [Corrected according to Rule 91 06.03.2025] Figure 14 is a flowchart illustrating another table restoration method provided in this application;

[0070] [Corrected according to Rule 91 06.03.2025] Figure 15 is a flowchart illustrating another table restoration method provided in this application;

[0071] [Corrected according to Rule 91 06.03.2025] Figure 16 is a structural schematic diagram of a table restoration device provided in this application;

[0072] [Corrected according to Rule 91 06.03.2025] Figure 17 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0073] [Corrected according to Rule 91 06.03.2025] Figure 18 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0074] [Correction based on Rule 91 06.03.2025] Figure 19 is a schematic diagram of another computing device cluster structure provided in an embodiment of this application.

[0075] [Corrected according to detailed rules 91 06.03.2025][Deleted]

[0076] [Corrected according to detailed rules 91 06.03.2025][Deleted]

[0077] [Corrected according to detailed rules 91 06.03.2025][Deleted]

[0078] [Corrected according to detailed rules 91 06.03.2025][Deleted] Detailed Implementation

[0079] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0080] This application relates to the application of neural networks and natural language processing (NLP). In order to better understand the solutions of this application, the relevant terms and concepts of neural networks that may be involved in this application will be introduced below.

[0081] (1) Neural Network

[0082] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes input data and an intercept of 1 as input can output the following:

[0083] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weight parameters are denoted by b, which represents the bias of the neural unit. f is the activation function of the neural unit, used to introduce non-linear characteristics into the neural network to convert the input signal into the output signal. The output signal of this activation function can be used as the input to the next convolutional layer; the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple individual neural units, meaning the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0084] (2) Deep Neural Networks

[0085] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple intermediate layers. Based on the position of these layers, the internal neural network of a DNN can be divided into three categories: input layer, intermediate layers, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are intermediate layers, or hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0086] Although DNNs appear complex, each layer can be represented as a linear relational expression: in, It is the input vector. It is the output vector. is the offset vector, also known as the bias parameter; w is the weight matrix (also called coefficients); and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is also quite large. The definitions of these parameters in DNNs are as follows: Taking the coefficient w as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0087] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0088] It's important to note that the input layer does not have a W parameter. In deep neural networks, more intermediate layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0089] (3) Language model (LM)

[0090] Language models play a crucial role in Natural Language Processing (NLP), where their task is to predict the probability of a sentence occurring in a language. For example, a language model is typically constructed as a probability distribution p(s) of a string s, where p(s) attempts to reflect the frequency of string s as a sentence. It can be applied to scenarios such as text recognition or machine translation. In the embodiments of this application, the NLP models mentioned below include language models.

[0091] (4) Optical character recognition (OCR)

[0092] This typically refers to using electronic devices (such as scanners, cameras, or mobile phones) to scan physical documents (e.g., printed documents on paper or other materials), converting light signals into electrical signals to generate corresponding images, and then performing character recognition on the images to identify the content within them. For example, an electronic device can be used to scan a paper document, which may contain printed tables. This involves using optical methods to convert the printed characters into a black-and-white dot matrix image file, and then using recognition software to convert the text in the image into text format for further editing by word processing software.

[0093] (5) Convolutional Recurrent Neural Network (CRNN)

[0094] It is a text recognition model used for end-to-end recognition of text sequences of variable length. Instead of segmenting individual characters first, it transforms text recognition into a time-dependent sequence learning problem, which is image-based sequence recognition and can recognize longer text sequences.

[0095] CRNN comprises a convolutional neural network (CNN) feature extraction layer and an attention-based and bidirectional long short-term memory (BLSTM) sequence feature extraction layer, enabling end-to-end joint training. It utilizes BLSTM and CTC (connectionist temporal classification) components to learn contextual relationships in character images, effectively improving text recognition accuracy and making the model more robust. During prediction, the front end uses a standard CNN network to extract features from the text image, then uses BLSTM to fuse the feature vectors to extract contextual features of the character sequence, obtaining the probability distribution of each feature column. Finally, the text sequence is predicted using a transcription layer (CTC rule).

[0096] The system architecture and methodological steps provided in this application are described below.

[0097] First, the method provided in this application can be applied to table scanning scenarios, by scanning an image containing a table to restore the table in the image, restoring it to an editable table or a higher resolution table.

[0098] The method provided in this application can be deployed in various electronic devices, such as server clusters, cloud platforms, personal computers, smartphones, or smart cars.

[0099] In one possible implementation, the method provided in this application can be deployed on computing devices such as personal computers, computer workstations, smartphones, tablets, laptops, and smart cars. Users can directly use computing devices to execute the method provided in the embodiments of this application to restore the table.

[0100] For example, the method provided in this application can be applied to user devices, such as smartphones, tablets, or devices capable of deploying OCR recognition. Users can use the user device to scan entities requiring table reconstruction, for example, using the user device's camera or infrared scanning probe to obtain scan data of the entity. The table in the entity can then be reconstructed using the method provided in this application.

[0101] In one possible implementation, the method provided in this application can be deployed on a server cluster, cloud platform, or other devices with computing capabilities.

[0102] For example, the method provided in this application can be deployed in a cloud platform to provide services to users through cloud services.

[0103] For example, Figure 1 shows a schematic diagram of the structure of a cloud service system provided in this application. As shown in Figure 1, the cloud service system 10 may include a computing device 11 and a client 12.

[0104] The computing device 11 may specifically include a server cluster or a cloud platform, or it may be other devices with computing capabilities. Optionally, the computing device 11 may work in conjunction with other computing devices, such as data storage, routers, load balancers, etc. The computing device 11 may use data from the data storage system or call program code in the data storage system to implement the method steps provided in the embodiments of this application.

[0105] The computing device 11 can provide services to users in the form of a client. Users can operate on the client 12 to achieve data interaction with the computing device 11 or request services from the computing device 11. This client can be deployed on personal computers, computer workstations, smartphones, tablets, laptops, and smart cars, etc.

[0106] In one implementation, the computing device 11 is used to implement the method provided in the embodiments of this application to restore the input image table for the client and send the output table to the client.

[0107] Among them, client 12 is an optional device. In actual application scenarios, the computing device can read input data from its own storage space or from the data center, restore the input data into a table, and then output the restored output table.

[0108] In summary, the methods provided in this application embodiment can be applied to electronic devices, that is, the aforementioned computing device 11 can be various electronic devices, such as server clusters, cloud platforms, personal computers, smartphones or smart cars, etc.

[0109] Based on the architecture shown in Figure 1 above, the method provided in this application can be executed by a computing device 11, which may specifically include a server cluster or a cloud platform, and can provide services to users through a client. Alternatively, the computing device 11 may also be other devices with computing capabilities.

[0110] For example, the method provided in this application can be deployed on a cloud platform to provide table restoration services to users through a client. Users can input images on the client, specifically by capturing images with an image sensor or scanning with a scanning probe, and then transmit the input images to the cloud platform. The cloud platform then uses the method provided in this application to restore the table and obtain higher-resolution table data.

[0111] In practical applications, under the backdrop of digitalization across various industries, there is a need to digitize electronic documents accumulated over many years. Images and scanned documents are the primary file formats. Manual data entry is extremely labor-intensive. To improve operational efficiency, OCR recognition and reconstruction of various scanned documents are necessary. Among various document types, tables, as the most commonly used data organization format, can express information as clearly, concisely, and efficiently as possible, and are widely present in electronic documents across various industries. Reconstructing the table structure is a crucial step in the digitization of scanned documents. However, the reconstruction of complex and dense wireless table structures has always been a pain point and a challenge in table structure reconstruction, and also a common and difficult problem in business scenarios.

[0112] In some existing scenarios for table reconstruction, whether it's vision-based table cell detection and table line segmentation, or direct end-to-end reconstruction of table structure encoding, it's difficult to solve the problem of complex wireless table reconstruction.

[0113] For example, in an existing table reconstruction scheme, the text box detection and text recognition results of the wireless table are first obtained. Then, for text boxes that are mistakenly merged or split, the text boxes are split and merged. Finally, based on the obtained accurate text box positions, the row and column information is obtained in a more systematic way. However, for complex tables, the workload of splitting and merging the table is very large, and the accuracy requirements are also very high. Therefore, the accuracy of the table reconstruction achieved by this scheme is low, and it is only suitable for simple table scenarios.

[0114] For example, in an existing table restoration scheme, the coordinates of text boxes in the wireless table are first detected. The difference in coordinates between adjacent rows of text boxes determines whether they belong to the same unit. Whether the text boxes intersect with the extension line of the text box's border indicates whether they are in the same row. The horizontal coordinate of each text box allows adjustment of the text boxes in each column, generating column boundary lines. The distance from the boundary lines determines whether to merge or split the text boxes. Finally, the cross-column text boxes are split based on the coordinate information of each character within each text box.

[0115] For example, in an existing table reconstruction scheme, the coordinates of text boxes in the wireless table are first detected. The distance between text boxes is used to determine whether multiple lines of text boxes belong to a single cell. Then, the relative coordinates of the merged cells are input into a deep learning model to decode the final row and column information of each text box. Clearly, this scheme involves complex cell splitting and merging steps, and many steps rely on distance to determine whether to separate or merge, requiring precise distance recognition. However, this distance recognition may not be accurate in different table scenarios, making the scheme unreliable. Furthermore, its recognition capability is poor for dense wireless tables.

[0116] For example, in an existing table reconstruction scheme, a deep learning model is used to combine spatial position regression of table cells with logical position regression between cells, simultaneously outputting table cell position and row / column information. However, the tables learned by deep learning models are usually very regular tables, and their ability to reconstruct tables in complex scenarios is poor.

[0117] Therefore, this application provides a table restoration method with stronger generalization ability, which can be applied to both simple and complex table restoration scenarios, and can accurately restore tables in various scenarios.

[0118] The method and process provided in this application are described below.

[0119] Referring to Figure 2, a flowchart of a table restoration method provided in this application is shown below.

[0120] 201. Perform text box detection on the input data to obtain the scanning results.

[0121] The input data includes an input table, and the scan result may include information from one or more text boxes. Specifically, the input table can be represented as an image within the input data or a non-readable / writable table.

[0122] The input data may specifically include RGB images, grayscale images, or infrared images, etc. The input image may include received images or acquired data. When the input data includes acquired images, for example, the method provided in this application can be deployed in a table restoration device, and the input data may be an image acquired using the infrared probe of the table restoration device or an image acquired using the camera of the table restoration device.

[0123] Pre-trained models can be used to detect text boxes in input data, that is, to identify text boxes in the input data and output information about at least one text box. The information about each text box can include information such as the position, size, or shape of the text box.

[0124] Specifically, OCR can be performed on the input data to detect text box regions within the input data, resulting in a scan result. This scan result can include information about one or more text boxes in the input data, such as the coordinates, region, or size of each text box.

[0125] Furthermore, OCR recognition can be performed on the input data to identify the text content included in the input data. Accordingly, the scan result can further include the text content in the text box areas of the table area of ​​the input data, so that a table with editable text content in the text boxes can be generated subsequently.

[0126] Optionally, to improve the accuracy of the table area in the scanned results, further processing such as correction or layout analysis can be performed on the table area, which will be introduced below.

[0127] In one possible implementation, the input data can be tilted, and the corrected input data can be output after tilt correction. After obtaining the scanning results, the tilt angle of each text box in the scanning results can be calculated using the coordinates of each text box. This tilt angle can be used as the tilt angle of the input data, and the input data can be corrected based on this tilt angle, such as by rotating the input data, to obtain the corrected input data. This corrected input data can then be used to identify table areas, row detection boxes, or column detection boxes, etc.

[0128] Correspondingly, when the input data is skewed, the input table within the input data will also be skewed. Therefore, the detected text boxes can be skewed to obtain more regular text boxes, thereby reducing the impact of the skew angle on subsequent table reconstruction. For example, after calculating the skew angle, one or more initial text boxes can be rotated based on this skew angle to correct them to alignment with the standard coordinate system, resulting in more standard text boxes. This allows for subsequent processing based on more standard text boxes, eliminating the need to calculate the skew angle in later processing stages and improving the detection efficiency of subsequent row and column text boxes.

[0129] In one possible implementation, the scan results can also be analyzed for layout to further identify the precise table areas.

[0130] Specifically, input data can be fed into a layout analysis model (or replaced with other layout analysis algorithms), or after skew correction, the corrected input data can be fed into the layout analysis model to output a table area. This table area represents the region where the input table is located in the input data, and can specifically include information such as the corner coordinates, center coordinates, length, width, or area of ​​the table area. This table area can be used as input for subsequent row detection models or column detection models. Typically, in addition to the output table, the input data may also include areas such as headers, footers, titles, or images. Through the layout analysis model, the various regions in the input data can be distinguished, thereby accurately identifying the region corresponding to the input table in the input data, i.e., outputting one or more text boxes in the input table. Therefore, in this embodiment, the layout analysis model can be used to crop the input data and output the text boxes of the table area in the input data to avoid the influence of other layout content in the input data on the subsequent table restoration points.

[0131] A table can be divided into rows and columns. In this embodiment, steps 202 and 203 can be performed from the dimensions of rows and columns, respectively. The method provided in this embodiment can be applied to wired tables and / or wireless tables. A wired table is a table that includes table lines, and a wireless table is a table that does not have table lines.

[0132] Before introducing steps 202 and 203, for ease of understanding, we will first introduce the input table, row text box, and column text box.

[0133] For example, the structure of the input form can be as shown in Figure 3, which includes multiple text boxes, each of which can be filled with information to save the new Xenia. The text boxes in the input form are divided into rows and columns horizontally and vertically, respectively.

[0134] Correspondingly, a row text box is a text box located in the same row, as shown in Figure 4. Text boxes located in the same table row are called row text boxes. There may be one or more rows of text boxes in the same input table, and there may be one or more text boxes in a row. A column text box is a text box located in the same column, as shown in Figure 5. Text boxes located in the same table column are called column text boxes. There may be one or more columns of text boxes in the same input table, and there may be one or more text boxes in a column.

[0135] 202. Perform text detection on at least one text box to determine the line text box.

[0136] After performing the aforementioned step 201, information from one or more text boxes in the input table can be output, and text detection can be performed on the one or more text boxes, that is, the text content in the one or more text boxes can be detected, thereby identifying text boxes on the same line based on the text content.

[0137] Specifically, based on the text boxes output in step 201, text boxes on the same line can be initially filtered out. Then, the text content within the text boxes is examined. For example, the semantics of the text in each text box are identified, and based on the contextual semantics between adjacent text boxes, it is determined whether there are any misjudgments regarding text boxes on different lines. For instance, whether different lines within the same text box are divided into multiple text boxes, and correcting for cases where the same text box is divided into multiple text boxes, thereby further refining the screening of text boxes on the same line, identifying the text content within the same text box, and accurately determining the text boxes on the same line.

[0138] Optionally, in step 201 above, if the text in each text box has been identified, for example, if OCR recognition has been performed on the content of the text boxes, then in step 201, the text content in each text box is also output. In this case, the text content in each text box can be directly processed by NLP. If the text content in each text box is not output in step 201 above, then in this step 202, it is also necessary to identify the text content corresponding to each text box. For example, OCR recognition or other text recognition steps can be performed in this step, and then NLP processing can be performed on the identified text content.

[0139] Specifically, the aforementioned detection of text content in text boxes can include performing NLP processing on the text in each text box, such as using an NLP-enabled model, to identify the semantics of the text in each text box within the table area. Based on the semantics of the text in each text box, the text contained within each text box is determined, thereby dividing the table area into row text boxes. In this embodiment, NLP can be performed on the text in the table area to identify the specific content of the text in each text box based on semantic recognition. This allows for accurate differentiation of row text boxes based on semantics, resulting in more accurate output of row text boxes, i.e., text boxes within the same table row.

[0140] In some scenarios, there may be misjudgments that classify different text boxes in the same text box as multiple text boxes. Therefore, in the embodiments of this application, NLP can be further used to identify different text boxes that should be classified into the same text box based on the contextual semantics between adjacent text boxes, thereby further correcting the identified initial line text boxes and outputting more accurate line text boxes.

[0141] Optionally, during the initial screening of text boxes in the same row, the coordinates of each text box can be used for initial screening, or a pre-trained row detection model can be used to output the coordinates of the rows in the input table, thereby initially dividing the text boxes output in the aforementioned step 201 and outputting the initial row text boxes.

[0142] For example, input data (or skew-corrected input data) or a table area within the input data (such as the table area output by the aforementioned layout analysis model) can be input into a row detection model to output row detection boxes. These row detection boxes can represent rows in the input table; for instance, the row detection model can output the center or corner coordinates of the row detection box to mark the area where each row in the input table is located. Then, by combining one or more text boxes output by the text box detection model with the row detection boxes, initial row text boxes are identified. For example, these one or more text boxes can be divided into corresponding row detection boxes to form initial row text boxes in a single row. Finally, NLP is performed on the text in the initial row text boxes to further identify row text boxes based on the NLP results.

[0143] Furthermore, in conjunction with the aforementioned layout analysis steps, this section describes the layout analysis steps using an input image as the input data. This input image includes an input table and may also include other areas unrelated to the input table, such as headers, footers, or titles. Therefore, this application also provides an optional implementation: before inputting the input data into the row detection model, the table area in the input data can be identified and used as the input to the row detection model. For example, as shown in Figure 6, when the input data is an image, the image may include table areas or other areas, such as headers, footers, or titles. The image can be cropped to retain the table area, or the input table can be extracted from the image, thereby using the table area corresponding to the input table as the input to the row detection model. This reduces the interference of areas unrelated to the table in the image on the row detection model, improving the accuracy of the output results of the row detection model.

[0144] 203. Determine the column text box.

[0145] For columns in the table, the text boxes can be divided into columns in the input table based on the coordinates of each text box output in step 201 above, and the text boxes in the same column in the input table, i.e., column text boxes, can be output.

[0146] Specifically, based on the coordinates of each text box output in step 201 above, text boxes with similar column coordinates can be designated as text boxes in the same column, thereby outputting text boxes for each column.

[0147] Furthermore, the method for determining the column text box in the embodiments of this application may also employ clustering and / or column detection models to determine the column text box. For details, please refer to the description of step 803, which will not be repeated here.

[0148] It should be noted that this application does not limit the execution order of steps 202 and 203. Step 202 can be executed first, or step 203 can be executed first, or steps 202 and 203 can be executed simultaneously. The specific order can be adjusted according to the actual application scenario.

[0149] 204. Merge the row text boxes and column text boxes to obtain the output table.

[0150] After obtaining the row text boxes and column text boxes, you can merge the row text boxes and column text boxes to get the output table.

[0151] Specifically, after obtaining the row text boxes and column text boxes, the rows and columns in the table can be restored respectively. For example, as shown in Figure 7, the column text boxes are filled into the columns of the table, and the row text boxes are filled into the rows of the table, thus obtaining the complete output table.

[0152] In addition, in the aforementioned step 201 or step 202, the text content included in each text box is also identified, and the identified text content in each text box can be filled into each text box to obtain an output table containing text content.

[0153] The aforementioned input tables are typically images or unreadable / unwritable tables that cannot be directly read or modified. In contrast, the output table in this application's embodiment has higher resolution compared to the input table. Furthermore, when the output table includes text content, it also possesses readability and writability, facilitating further data extraction or rewriting by the user.

[0154] In this embodiment, row text boxes and column text boxes can be detected separately. When detecting row text boxes, text content detection can be used to more accurately distinguish the text content within the same row text box, thus achieving more accurate row text box detection. Subsequently, by fusing the row and column text boxes, a complete table is reconstructed, achieving table reconstruction. Furthermore, for dense and complex wireless tables, the method provided in this application, using text content detection, can also more accurately identify the text content within the same row text box, thereby achieving more accurate table reconstruction. Therefore, the method provided in this embodiment has stronger generalization ability and can be applied to both simple and complex table reconstruction scenarios, accurately reconstructing tables in various scenarios.

[0155] The foregoing has described a table restoration method provided in this application, which focuses on the recognition of table rows. It identifies the text content in the same text box through text detection to improve the accuracy of table row recognition.

[0156] This application also provides a table restoration method that focuses on the identification of table columns. By accurately restoring the columns in the table, a more accurate output table can be restored.

[0157] Referring to Figure 8, a flowchart of another table restoration method provided in this application is shown below.

[0158] 801. Perform text box detection on the input data to obtain the scanning results.

[0159] Step 801 can be referred to the description of step 201 above, and will not be repeated here.

[0160] 802. Determine a line text box based on the at least one text box.

[0161] In this embodiment of the application, after performing the aforementioned step 201, information of one or more text boxes in the input table can be output. The text boxes can be divided based on their positions. The relative positions of the text boxes can be identified based on their positions, thereby dividing text boxes with similar row coordinates into the same row and outputting the row text boxes.

[0162] The difference between this step and step 202 is that text detection is not required to identify line text boxes here, which reduces the computational load of table reconstruction and improves the efficiency of table reconstruction.

[0163] Of course, the text detection method described in step 202 above can also be used to output the line text. For details, please refer to the description of step 202 above, which will not be repeated here.

[0164] 803. Use clustering and / or column detection models to determine column text boxes.

[0165] In the embodiments of this application, the methods for determining column text boxes may include one or more, such as using a column detection model to detect column text boxes, clustering text boxes in a table area to determine column text boxes, or merging the column text boxes detected by the column detection model with the column text boxes obtained by clustering, etc. The methods for determining column text boxes are described below.

[0166] 1. Column-based detection model

[0167] The process involves inputting input data into a column detection model, or inputting a table area from the input data into the column detection model, and outputting a column detection box. This column detection box can be used to represent the area of ​​each column in the input table. Subsequently, based on the column detection box and one or more text boxes in the scan results, the text boxes in each column are determined, resulting in column text boxes.

[0168] In this embodiment, a column detection model can be used to determine the region of each column in the input table, and then, by combining the position of each text box in the scanning result, the text boxes in the same column can be identified, so as to distinguish the columns in the input table.

[0169] Optionally, similar to the data input to the row detection model described above, the input image includes an input table and may also include other regions unrelated to the input table, such as headers, footers, or titles. Therefore, this application also provides an optional implementation where, before inputting the input data to the column detection model, table regions in the input data can be identified and used as input to the column detection model. This reduces the interference of unrelated regions in the input data on the column detection model, thereby improving the accuracy of the column detection model's output.

[0170] 2. Based on clustering

[0171] One or more text boxes included in the aforementioned scan results can be clustered to divide the one or more text boxes into multiple classes. Each class is used as a column text box in the input table, and the output column text box is generated.

[0172] Specifically, the column coordinates of one or more text boxes can be clustered, as shown in Figure 9. The coordinates on the x-axis of one or more text boxes can be clustered, and the text boxes can be divided into one or more columns according to the clustering results. That is, a column of clustered text boxes is used as a column text box.

[0173] Therefore, in this embodiment of the application, clustering can be used to divide text boxes in the same column into the same column, thereby more accurately identifying the column text boxes.

[0174] 3. Based on column detection model and clustering

[0175] In one possible implementation, the aforementioned column detection model and clustering can be combined to further identify more accurate column text boxes.

[0176] Specifically, the input data is fed into the column detection model, and the output column detection box is generated. This column detection box can be used to represent the region of each column in the input table. The one or more text boxes included in the aforementioned scanning results are clustered to classify the one or more text boxes into multiple categories, and the column clustering text box is output. Then, the column clustering text box and the column detection box are merged, for example, by filling the text boxes in the column clustering text box into the corresponding column detection box, so as to output the final column text box.

[0177] Therefore, in this embodiment, the column detection model and clustering method can be combined to more accurately identify text boxes in the same column through dual protection, thereby achieving more accurate table reconstruction from the column dimension.

[0178] 804. Merge the row text boxes and column text boxes to obtain the output table.

[0179] After obtaining the row text boxes and column text boxes, you can merge the row text boxes and column text boxes to obtain the restored output table.

[0180] For details on step 804, please refer to step 204 above, which will not be repeated here.

[0181] Therefore, in this embodiment, clustering and / or column detection models are used to accurately identify the column text boxes in the input table, thereby improving the accuracy of column reconstruction and resulting in a more accurate output table. Compared to distinguishing column text boxes solely based on their coordinates, the clustering method used in this application constrains the coordinates of text boxes within the same column, making their coordinates more similar. Furthermore, the column detection model accurately identifies the region where each column in the input table is located, allowing text boxes to be filled into the corresponding column regions for more accurate column text boxes. Additionally, row and column text boxes can be merged, achieving a more accurate reconstruction of the input table and producing a highly accurate output table.

[0182] The foregoing has provided a general overview of the method provided in this application. The following section will further describe the method flow in conjunction with specific application scenarios.

[0183] Taking a document scanning scenario as an example, refer to Figure 10, which shows a flowchart of another table restoration method provided in this application.

[0184] First, a scanning device can be used to scan the printed paper document containing the input table to obtain a scanned image. OCR detection is then performed on the scanned image, i.e., step 1001, outputting one or more text boxes in the scanned image. Optionally, image correction or layout analysis can be further performed on the scanned image, i.e., steps 1002 and 1003, outputting a tilt-corrected image and the table area within the image. Subsequently, table row restoration and table column restoration can be performed, i.e., steps 1004 and 1005, where table row restoration restores the row text boxes in the input table, and table column restoration restores the column text boxes in the input table. Based on the output results of table row restoration and table column restoration, table restoration is performed, i.e., step 1006, outputting the final output table.

[0185] The aforementioned steps are described in detail below. Referring to the implementation steps corresponding to Figures 2 and 8, different steps can be selected and deployed depending on the scenario. For example, when restoring table rows, the following methods can be used: Method 1: Determine the initial row text boxes based on coordinates, and further determine the row text boxes using NLP; Method 2: Use a row detection model to identify row detection regions, and further determine the row text boxes using NLP; Method 3: Use a row detection model to output row detection boxes, and output row text boxes based on the row detection boxes, etc. When restoring table columns, the following methods can be used: Method a: Determine the column text boxes using text box clustering; Method b: Determine the column text boxes using a list detection model; Method c: Combine text box clustering and column detection models, etc. Furthermore, for some optional steps, such as orientation correction or layout analysis, it is also possible to choose whether to execute them based on the actual application scenario.

[0186] In specific implementation, one of the aforementioned methods 1, 2, and 3 can be combined with one of methods a, b, and c. In this application embodiment, some possible combinations are described below without exhaustive listing. In actual application scenarios, the required methods can be adaptively selected to restore rows and columns respectively, and this application does not impose any limitations on this.

[0187] For example, when restoring the table structure, clustering can be used, column detection models can be used for column detection, or a combination of clustering and column detection models can be used. When restoring line text boxes, line detection models can be used, NLP models can be used for line text box detection, or a combination of line detection models and NLP models can be used. The different implementation methods are described below.

[0188] Implementation Method 1

[0189] Referring to Figure 11, a flowchart of another table restoration method provided in this application is shown below.

[0190] 1101. OCR detection and recognition.

[0191] The input for OCR detection and recognition is an image, such as the aforementioned input data or scanned image, collectively referred to here as the input image. Specifically, the input image can be an image acquired by an image sensor or a received image. The input image includes an input table, as shown in step 1101 of Figure 11. The input image may include an input table, as well as other text or images.

[0192] The OCR detection and recognition step in this process can be divided into OCR detection and OCR recognition steps, with OCR recognition being an optional step. The input for OCR detection is the aforementioned input data or scanned image, and the output is one or more detected text boxes. The input for OCR recognition is one or more text boxes output by OCR detection, along with the aforementioned input data or scanned image, and the output is the content contained within each text box. The OCR detection and OCR recognition steps are described below.

[0193] 1. OCR detection:

[0194] Specifically, OCR detection can detect text boxes in an input image, such as the coordinates, shape, or size of the text boxes in the input image.

[0195] Alternatively, models suitable for text box detection, such as Differentiable Binarization Net (DBNet), DBNet++, and Progressive Scale Expansion Network (PSENet), can be used.

[0196] In one scenario, the method provided in this application embodiment can be deployed on a cloud platform, providing table restoration services to users via a client. The client can be deployed on a user's computing device, such as a mobile phone, tablet, personal computer, or other terminal. The user can use the image sensor on their computing device to scan the document to be restored, thereby obtaining an image containing the document, and then transmitting the image to the cloud platform. For example, when a user needs to restore a table contained in a printed document, they can use the camera on their terminal to scan the document, thereby capturing an image containing the document, and then transmitting the image to the cloud platform, where the cloud platform performs OCR detection on the image.

[0197] In one scenario, the method provided in this application embodiment can be deployed on a user's computing device, such as a mobile phone, tablet, personal computer, or other terminal. The user can use the image sensor set on the computing device to scan the document that needs to be restored, thereby obtaining an image containing the document. For example, when a user needs to restore a table contained in a printed document, they can use the camera of the terminal deployed with this application embodiment to scan the document, thereby capturing an image containing the document, and then the computing device performs OCR detection on the image.

[0198] For example, in one possible scenario, the method provided in this application embodiment can be deployed as an end-to-end document OCR recognition system. Users utilize sensors to scan printed documents on paper, thereby obtaining data containing information about the physical document. Specifically, sensors can be used to scan the document entity or image to be restored. These sensors can include RGB image sensors, laser sensors, or other sensors capable of acquiring information about the entity. After scanning the document, the sensors can acquire the scanned document information and generate a corresponding image. For example, an RGB image sensor can be used to acquire a document image and output the corresponding RGB image; or, a laser sensor can be used to scan the document and output the corresponding document image. An image sensor can be used to acquire the image corresponding to the document entity. Specifically, the image sensor acquires the light signal from the document entity, converts the light signal into an electrical signal and outputs it. Decoding the electrical signal output by the image sensor yields the scanned image.

[0199] 2. OCR recognition:

[0200] Furthermore, OCR recognition can be performed on each text box to identify the text content within each box, so that the text content in the table can be restored later. Models such as CRNN, Master (a network for text recognition), and SVTR (scene text recognition with a single visual model) can be used for text recognition to output the text content included in the table.

[0201] 1102. Direction correction.

[0202] Specifically, the tilt angle of the image can be estimated based on the text box data output by OCR detection, thereby correcting the image. For example, the image can be rotated to make it tilted within a preset range, which is called a corrected image for easy distinction. Alternatively, the text boxes detected by OCR can be rotated to obtain text boxes with tilt angles within a preset range.

[0203] For example, as shown in Figure 12, when a certain tilt angle is detected in the image, the tilt angle of the table line of the text box relative to the text box in the standard coordinate system can be calculated based on the coordinates of the text box detected by OCR, and the image can be rotated to make the image with an angle within a certain range relative to the standard coordinate system; or the text box detected by OCR can be rotated to make the text box with an angle within a certain range relative to the standard coordinate system.

[0204] Step 1102 is an optional step. For example, if the input image is not tilted relative to the standard coordinate system, step 1102 does not need to be executed. Whether to execute step 1102 can be determined according to the actual application scenario, which will not be elaborated here.

[0205] 1103. Page layout analysis.

[0206] The input for the layout analysis step may include the aforementioned input image, the corrected image output in step 1102, etc. Here, the corrected image output in step 1102 is used as the input for the layout analysis step as an example. The layout analysis of the corrected image can be performed, for example, by using a layout analysis model. The layout analysis model is used to identify various types of regions in the corrected image, and then to identify the region corresponding to the input table.

[0207] Typically, layout analysis models can detect areas such as headers, footers, tables, titles, and images in a document. In this embodiment, table areas need to be restored; therefore, after layout analysis, the table areas can be output. Specifically, pre-trained models such as YOLOv8 (a network for object recognition), Faster R-CNN (an R-CNN for detection), and DINO (a visual model) can be used for layout analysis to output the positions of table areas in the corrected image, such as the coordinates of each corner point within the table area.

[0208] After performing layout analysis, the OCR detection and recognition results output from steps 1101 to 1103, along with the layout analysis recognition results, can be input into the following branches. For example, for the row and column branches of a table, the recognition of row text boxes and column text boxes can be processed separately. Steps 1104-1105 are the detection and recognition steps for row text boxes, and steps 1106-1108 are the detection and recognition steps for column text boxes, which will be described below.

[0209] 1104. Determine the initial line text box based on coordinates.

[0210] Based on the coordinates of the text boxes in the previously identified table area, text boxes in the same row can be identified. For example, the direction corresponding to the column of the table can be used as the direction of the vertical axis, and the direction corresponding to the row of the table can be used as the direction of the horizontal axis. The distance between the vertical coordinates of each text box can be calculated, and the text boxes can be divided according to their vertical coordinates. For example, different text boxes whose vertical coordinates differ by less than a preset difference can be divided into the same row, thereby dividing the table area into one or more rows of text boxes and outputting the initial row of text boxes.

[0211] For example, the initial row text box can be as shown in Figure 13, which includes one or more text boxes to facilitate the restoration of rows in the table area.

[0212] 1105. Using an NLP model, output a line text box.

[0213] The input to the NLP model can include the initial line text box output in step 1104 and the OCR recognition result output in step 1101. The OCR recognition result can include the text in each text box. Specifically, the NLP model (i.e., the model that can be applied to perform NLP) can be used to recognize the semantics of the text in each text box, thereby semantically grouping the text content belonging to the same text box into the same text box to accommodate situations where there are multiple lines of text in a text box.

[0214] The input data type of an NLP model can be determined based on the input data type during pre-training, or the training data can be collected based on the type of input data to train the NLP model. For example, in one scenario, the text content of each of the aforementioned text boxes can be used as input to the NLP model, and the output can be the semantic meaning of the text in each text box or the text content belonging to the same text box. Thus, the semantic understanding of the NLP model can be used to identify the text content belonging to the same text box, thereby more accurately identifying one or more lines of text belonging to the same text box.

[0215] For example, during the training of an NLP model, natural language corpora are collected, and by randomly segmenting sentences, the first half of any sentence is combined with the second half of another sentence to obtain negative samples. A natural language processing model is then trained to perform binary classification to determine whether a sentence is coherent. Specifically, a pre-trained BERT model can be used, extracting input sentence vectors and connecting a classification layer to the output for binary classification, using cross-entropy loss as the loss function to train the model. To ensure robustness to application scenarios, relevant corpora from specific scenarios can be collected for model training. After training, adjacent lines of text can be input into the model to determine coherence; if coherent, they are merged; otherwise, they are not. Furthermore, the NLP model provided in this application can also be implemented using a large model, leveraging its capabilities to determine the coherence of adjacent lines of text.

[0216] For example, one possible table structure is shown in Table 1:

[0217] Table 1

[0218] In Table 1, the text box content "XXX, Nanhu Street, Luohu District, Shenzhen, Guangdong Province" is divided into two rows: one row reads "Luohu, Shenzhen, Guangdong Province," and the other row reads "XXX, Nanhu Street, Luohu District." In this embodiment, an NLP model can be used to identify the semantics of the text content in the table area. By combining the semantics between the texts, if the probability that "Luohu, Shenzhen, Guangdong Province" and "XXX, Nanhu Street, Luohu District" can be concatenated is higher than a preset value, then "Luohu, Shenzhen, Guangdong Province" and "XXX, Nanhu Street, Luohu District" can be considered as the content of the same text box. Therefore, based on the contextual semantics of each text, content belonging to the same text box can be identified, and the row text boxes in the table area can be accurately identified.

[0219] 1106. Text box clustering.

[0220] For columns in a table, clustering can be used to identify text boxes within those columns. Specifically, the coordinates of text boxes in the table area can be clustered, grouping text boxes belonging to the same column together, thus enabling the detection of columns within the table area.

[0221] Specifically, text box clustering can use clustering algorithms such as K-means and K-means++. For example, the K-means algorithm is used here. First, the text boxes in the table area are retrieved, and the x-coordinate of the center point of each text box is calculated. The number of cluster centers is set to 2. The K-means algorithm is used to cluster all the center points, and the mean error dist1 from each center point's x-coordinate to its respective cluster center is calculated. The number of cluster centers is increased to 3, and the above clustering steps are repeated to obtain the mean error dist2. When the value of dist1 - dist2 is less than the threshold threshold1, clustering is stopped, and text boxes in the same column are obtained. Otherwise, the number of cluster centers is increased until clustering stops.

[0222] 1107. Output column detection boxes using the column detection model.

[0223] For columns in the input table, a column detection model can be used to identify the area of ​​each column in the table region. For example, the table region output in step 1103 above can be used as the input of the column detection model to output the information of the column detection box, specifically including the coordinates of the center point, corner points, length, width or area of ​​the column detection box.

[0224] [Corrected according to Rule 91 06.03.2025] This column detection box marks the area of ​​each column in the table area so that each text box can be filled into the corresponding column area later.

[0225] 1108. Determine the column text box.

[0226] Based on the column detection results output in steps 1106 and 1107, the column text boxes output by clustering and the column text boxes output by the column detection model can be merged. For example, the text boxes of each cluster output can be filled into the column detection boxes output by the column detection model, thereby ensuring the accuracy of the final output column text boxes in multiple ways.

[0227] It should be noted that when restoring the columns in the input table, either step 1107 or step 1108 can be performed, or both steps can be performed.

[0228] For example, only step 1107 can be executed. In this case, the column text boxes can be determined based on the column clustering text boxes output after the text box clustering in step 1107. For example, text boxes of the same cluster category can be directly used as column text boxes in the table column.

[0229] If only step 1108 is executed, then based on the column detection box output in step 1108, each text box can be filled into the corresponding column detection box according to the coordinates of each text box output in step 1101, and the text box in the filled column detection box, i.e., the column text box, can be output.

[0230] It should be noted that this application does not limit the execution order of steps 1104 and 1106. Step 1104 can be executed first, or step 1106 can be executed first, or steps 1104 and 1106 can be executed simultaneously. The specific order can be determined according to the actual application scenario. The execution order of steps 1104 and 1106 here is only an example and is not intended to be a limitation.

[0231] 1109. Merge rows and columns to restore the table.

[0232] After identifying the row text boxes and column text boxes in the table area, the table is restored based on the detection results of the row text boxes and column text boxes, resulting in an editable or high-definition table, which is the output table.

[0233] [Corrected according to Rule 91 06.03.2025] For example, after outputting the row text boxes (carrying the text in each text box) and column text boxes, the row text boxes and column text boxes are merged. For example, the row text boxes (and the text in each text box) and column text boxes are filled into the rows and columns of the output table respectively. For example, the first text box in a row is represented as C1, and the second text box r1 in a column is represented as C1_r1. The text content corresponding to C1 is then filled into the current cell to complete the filling of the current cell.

[0234] In this embodiment, rows and columns of the table can be detected and identified separately. When detecting row text boxes, text recognition can be used to accurately identify the text content within the same text box based on whether the text content in each text box matches the text in adjacent text boxes, thus accurately distinguishing rows in the table. When detecting column text boxes, clustering or column detection models can be used to detect them, accurately identifying the column structure in the table. Therefore, in this embodiment, both row and column text boxes in the table can be detected very accurately, thus accurately reconstructing the table structure. Especially for dense and complex wireless tables, the method provided in this application can accurately reconstruct the rows and columns of the table, resulting in a highly accurate table reconstruction.

[0235] Implementation Method 2

[0236] [Corrected according to Rule 91 06.03.2025] Referring to Figure 14, a flowchart of another table restoration method provided in this application is shown below.

[0237] 1601. OCR detection and recognition.

[0238] Step 1601 can be referred to the description of step 401 above, and will not be repeated here.

[0239] 1602. Use a table detection model to detect table regions.

[0240] In the aforementioned Figure 11, step 1103, the layout analysis can be replaced by a pre-trained table detection model. The input of this table detection model can include an input image (not shown in the figure), which can be used to detect table regions in the input image and output table detection boxes corresponding to the table regions. Thus, the input image can be cropped based on the table detection boxes to output the corresponding table regions.

[0241] [Corrected according to Rule 91 06.03.2025] For example, the input image may typically include areas such as headers, footers, tables, or titles. This input image is used as input to a table detection model to identify the table areas and output the corresponding table detection boxes, such as the center point coordinates, corner point coordinates, length, width, or area of ​​the table area.

[0242] The table area output in step 1602 can be input into steps 1603 and 1605 for subsequent detection of the table's rows and columns.

[0243] 1603. Determine the line text box based on the line detection model.

[0244] Specifically, the table area output in step 1602 can be used as the input to the row detection model, and the row detection boxes in the table area can be output, such as the center point coordinates, corner point coordinates, length, width or area of ​​the row detection boxes.

[0245] [Corrected according to Rule 91 06.03.2025] For example, the column text box output by the row detection model marks one or more rows in the table area. Combined with the coordinates of each text box output in step 1601 above, each text box can be filled into the corresponding row detection box, so as to output the row text box from the dimension of the row, that is, the text box in the same table row.

[0246] 1604. Text box clustering.

[0247] 1605. Output column detection boxes using the column detection model.

[0248] 1606. Determine the column text box.

[0249] 1607. Merge rows and columns to restore the table.

[0250] Steps 1605-16016 can be referred to the description of steps 406-409 above, and will not be repeated here.

[0251] In this embodiment, detection is performed separately for rows and columns in the table. When performing column detection, clustering or column detection models can be combined to detect column text boxes, thereby accurately identifying the column structure in the table. Therefore, the method provided in this embodiment can achieve more accurate table reconstruction. Furthermore, compared to Figure 11 above, this embodiment eliminates the need for NLP processing, i.e., step 1101 is unnecessary. For example, if the row detection model has high output accuracy, row text boxes can be accurately identified, reducing the number of row text box recognition operations and improving the efficiency of outputting row text boxes. Moreover, the aforementioned layout analysis step is replaced by using a table detection model to output table regions, thereby utilizing the detection capabilities of the table detection model to output table regions in the image for subsequent processing.

[0252] Implementation Method 3

[0253] [Corrected according to Rule 91 06.03.2025] Referring to Figure 15, a flowchart of another table restoration method provided in this application is shown below.

[0254] 1901. OCR detection and recognition.

[0255] 1902, Direction Correction.

[0256] 1903. Page Layout Analysis.

[0257] Steps 1901 to 1903 can be found in the description of steps 1101 to 1103 above, and will not be repeated here.

[0258] 1904. Determine the initial line text box based on the line detection model.

[0259] Step 1904 can be referred to the description of step 1609 above. The difference is that the output line text box is used as the initial text box so that NLP can continue to be executed in the subsequent step 1905.

[0260] 1905. Use NLP models to identify the same text box.

[0261] Step 1905 can be referred to the description of step 1105 above, and will not be repeated here.

[0262] The difference lies in the fact that, in the implementation of this application, a line detection model and an NLP model are combined to more accurately identify the line text box through dual protection.

[0263] Specifically, after executing step 1904, the line detection boxes output by the line detection model are used to fill the text boxes output in step 1901 into the line detection boxes, and the initial line text boxes are output. Then, the data included in the initial line text boxes is input into the NLP model, and the NLP model is used to identify the semantics of the text included in each text box. Based on the contextual semantics between adjacent text boxes, it is determined whether the content included in adjacent text boxes is contextual text, and then it is determined whether adjacent text boxes are identified as the same text box. If different text boxes are identified as the same text box, the different text boxes are merged, which is equivalent to refining the initial text boxes and outputting more accurate line text boxes.

[0264] 1906. Text box clustering.

[0265] 1907. Output column detection boxes using the column detection model.

[0266] 1908. Determine the column text box.

[0267] 1909. Merge rows and columns to restore the table.

[0268] Steps 1905 to 1909 can be found in the description of steps 1105 to 1109 above, and will not be repeated here.

[0269] [Corrected according to Rule 91, 06.03.2025] Therefore, in this embodiment, detection is performed separately for rows and columns in the table. When detecting rows, a row detection model is added to detect row text boxes, and the text semantics recognized by the NLP model are combined to identify the text content in the same text box, thus achieving accurate detection of row text boxes. Furthermore, when detecting columns, clustering or column detection models can be used to detect column text boxes, thereby accurately identifying the column structure in the table. Therefore, in this embodiment, both row and column text boxes in the table can be detected very accurately, thus accurately restoring the table structure. Especially for dense and complex wireless tables, the method provided in this application can very accurately restore the rows and columns in the table, thus achieving very accurate table restoration.

[0270] The method flow provided in this application has been described above. The structure of the apparatus for performing the method provided in this application is described below.

[0271] [Corrected according to Rule 91, 06.03.2025] Referring to Figure 16, this application provides a table restoration device, comprising:

[0272] The scanning module 2001 is used to perform text box detection on the input data to obtain the scanning result. The input data includes an input table, and the scanning result includes information from at least one text box in the input table.

[0273] The line processing module 2002 is used to perform text detection on at least one text box to determine the line text box;

[0274] Column processing module 2003 is used to determine column text boxes based on at least one text box;

[0275] The merge module 2004 is used to merge row text boxes and column text boxes to obtain an output table.

[0276] In one possible implementation, the line processing module 2002 is specifically used for: performing text recognition on at least one text box to obtain the text in at least one text box; performing natural language processing on the text in at least one text box to recognize the line text box.

[0277] In one possible implementation, the row processing module 2002 is specifically used for: inputting input data into the row detection model, outputting row detection boxes, the row detection boxes being used to represent the region of each row in the input table; determining initial row text boxes based on the row detection boxes and the scanning results; performing natural language processing on the text in the initial row text boxes to identify the row text boxes.

[0278] In one possible implementation, the column processing module 2003 is specifically used for: clustering at least one text box to obtain a column clustering text box; and determining a column text box based on the column clustering text box.

[0279] In one possible implementation, the column processing module 2003 is specifically used to: input input data into the column detection model, output column detection boxes, the column detection boxes being used to represent the area of ​​each column in the input table; and determine column text boxes based on the column detection boxes and the scanning results.

[0280] In one possible implementation, the column processing module 2003 is specifically used for: clustering the text boxes in the table area to obtain column clustering text boxes; inputting input data into the column detection model and outputting column detection boxes, the column detection boxes being used to represent the area of ​​each column in the input table; and fusing the column clustering text boxes and the column detection boxes to obtain column text boxes.

[0281] In one possible implementation, the apparatus further includes: a layout analysis module 2005, configured to input input data into a layout analysis model and output a table area, the table area including the area of ​​the input table in the input data; wherein, in the presence of a row detection model, the table area is used as input to the row detection model, and in the presence of a column detection model, the table area is used as input to the column detection model.

[0282] In one possible implementation, the device further includes: a correction module 2006, used to correct the tilt angle of the input data to obtain corrected input data;

[0283] The layout analysis module is specifically used to input the corrected input data into the layout analysis model and output a table area.

[0284] [Correction 06.03.2025 according to Rule 91] In conjunction with the structure of the table restoration device shown in FIG16 above, in the embodiments of this application, the table restoration device may include different functions.

[0285] The scanning module 2001 is used to perform text box detection on the input data and obtain the scanning result. The input data includes an input table, and the scanning result includes information of at least one text box in the input table.

[0286] Line processing module 2002 is used to determine a line text box based on at least one text box;

[0287] The column processing module 2003 is used to determine column text boxes based on at least one text box, wherein the column text box is obtained by clustering at least one text box, and / or the column text box is obtained based on the column detection box output by the column detection model, wherein the column detection box represents the area of ​​each column in the input table;

[0288] The merge module 2004 is used to merge row text boxes and column text boxes to obtain an output table. The row text boxes include text boxes in the same row as the text boxes in the output table, and the column text boxes include text boxes in the same row as the text boxes in the output table.

[0289] In one possible implementation, when the column text box is determined by combining the clustering of at least one text box with the column detection box output by the column detection model, the column processing module 2002 is specifically used to: cluster the text boxes in the table area to obtain the column clustering text box; input the input table area in the input data to the column detection model and output the column detection box; and fuse the column clustering text box and the column detection box to obtain the column text box.

[0290] In one possible implementation, when the column text boxes are clustered together with at least one text box, the column processing module 2002 is specifically used to: cluster the at least one text box based on its coordinates to obtain column clustered text boxes, for example, clustering the coordinates of the text boxes and taking text boxes with similar column coordinates as text boxes in the same column; and determine column text boxes based on the column clustered text boxes, for example, taking text boxes in a cluster as text boxes in a column, i.e., column text boxes. In this embodiment of the application, clustering can be used to identify text boxes in the same column, thereby accurately identifying text boxes in the same column based on the column coordinates of each text box.

[0291] [Corrected according to Rule 91, 06.03.2025] In one possible implementation, when the column text boxes are determined by column detection boxes output by the column detection model, the column processing module 2002 is specifically used to: take the input data as input to a pre-trained column detection model, output column detection boxes, which can be used to represent the area corresponding to each column in the input table; then determine the column text boxes based on the column detection boxes and at least one of the aforementioned text boxes, for example, filling each text box into the corresponding column detection box based on the coordinates of each text box, and outputting the column text boxes. Therefore, by using a pre-trained column detection model to perform the detection task and outputting the column areas in the table area, the detection capability of the model can be used to accurately identify the areas of the table columns, thereby outputting more accurate column text boxes.

[0292] Each module in the aforementioned table restoration device can be implemented in software or hardware. For example, the implementation of the row processing module will be described below. Similarly, the implementation of other modules, such as the scanning module, column processing module, merging module, correction module, or layout analysis module, can refer to the implementation of the row processing module.

[0293] As an example of a software functional unit, a line processing module may include code running on a compute instance. This compute instance can be at least one of a physical host (compute device), a virtual machine, a container, or other compute devices. Furthermore, the aforementioned compute devices can be one or more. For example, a line processing module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed within the same region or in different regions. Similarly, the multiple hosts / virtual machines / containers used to run the code can be distributed within the same Availability Zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0294] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a single region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0295] As an example of a hardware functional unit, a line processing module may include at least one computing device, such as a server. Alternatively, a line processing module may also be a device implemented using a central processing unit (CPU), application-specific integrated circuit (ASIC), programmable logic device (PLD), complex programmable logic device (CPLD), field-programmable gate array (FPGA), generic array logic (GAL), data processing unit (DPU), neural network processing unit (NPU), system-on-chip (SoC), offloading card, accelerator card, etc. The aforementioned PLD can be implemented using CPLD, FPGA, GAL, or any combination thereof.

[0296] The row processing module includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the row processing module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the row processing module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0297] As an example of a software functional unit, a table restoration device may include code running on a computing instance. This computing instance can be at least one of a physical host (computing device), a virtual machine, a container, or other computing devices. Furthermore, the aforementioned computing devices may be one or more. For example, the table restoration device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed within the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code can be distributed within the same Availability Zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0298] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a single region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0299] As an example of a hardware functional unit, a table restoration device may include at least one computing device, such as a server. Alternatively, the table restoration device may also be a device implemented using an ASIC or a PLD. The aforementioned PLD may be implemented using a CPLD, FPGA, GAL, or any combination thereof.

[0300] The table restoration device includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the multiple computing devices can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0301] This application also provides a chip system including a processor and a power supply circuit. The power supply circuit supplies power to the processor, which executes the operation steps corresponding to the method provided in this application. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices such as a DPU, NPU, XPU, SoC, offload card, or accelerator card.

[0302] [Corrected according to Article 91, 06.03.2025] This application also provides a computing device 100. As shown in FIG17, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 100 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 100.

[0303] [Corrected according to Rule 91, 06.03.2025] Bus 102 may be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc. The Unified Bus is also known as the Lingqu Bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, only one line is used in Figure 17, but this does not mean that there is only one bus or one type of bus. Bus 104 may include a path for transmitting information between various components of computing device 100 (e.g., memory 106, processor 104, communication interface 108). The Unified Bus is also known as the Lingqu Bus.

[0304] The processor 104 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0305] Memory 106 may include volatile memory, such as random access memory (RAM). Processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 106 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0306] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 106, or two or more types of storage media can be configured to realize the function of memory 106. This application does not limit this.

[0307] The memory 106 stores executable program code, and the processor 104 executes the executable program code to implement the functions of the various modules mentioned in FIG. 7, thereby implementing the method provided in this application. That is, the memory 106 stores instructions for executing the method provided in this application.

[0308] Alternatively, the memory 106 stores executable code, which the processor 104 executes to implement the functions of the aforementioned scanning module, row processing module, column processing module, merging module, correction module, or layout analysis module, thereby implementing the method provided in this application. That is, the memory 106 stores instructions for executing the method provided in this application.

[0309] The communication interface 103 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0310] As one possible implementation, the computing device 100 may also include a chip system, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, and the processor executes the operation steps corresponding to the method provided in this application. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices or AI chips such as a DPU, NPU, XPU, SoC, offload card, or accelerator card.

[0311] As one possible implementation, the computing device 100 may include various types of processors 104, meaning the computing device 100 is a heterogeneous device. For example, the computing device 100 may include a CPU and a GPU, and at least one of the processors 104 may execute the operation steps corresponding to the method provided in this application. For the sake of brevity, further details are omitted here.

[0312] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0313] [Correction based on Rule 91, 06.03.2025] As shown in FIG18, the computing device cluster includes at least one computing device 100. The memory 106 of one or more computing devices 100 in the computing device cluster may store the same instructions for performing the methods provided in this application.

[0314] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the methods provided in this application. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for executing the methods provided in this application.

[0315] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the joint testing device. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of one or more of the aforementioned scanning module, row processing module, column processing module, merging module, correction module, or layout analysis module.

[0316] [Corrected according to Rule 91, 06.03.2025] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. The network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 19 illustrates one possible implementation. As shown in Figure 19, two computing devices 100A and 100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 106 in computing device 100A stores instructions for performing the functions of a scanning module, row processing module, column processing module, merging module, correction module, or layout analysis module. Simultaneously, the memory 106 in computing device 100B stores instructions for performing the functions of the scanning module, row processing module, column processing module, merging module, correction module, or layout analysis module.

[0317] [Correction based on Rule 91, 06.03.2025] It should be understood that the function of computing device 100A shown in Figure 19 can also be performed by multiple computing devices 100. Similarly, the function of computing device 100B can also be performed by multiple computing devices 100.

[0318] [Correction based on Rule 91 06.03.2025] The connection method between the computing device clusters shown in Figure 19 can be considered in light of the fact that the method provided in this application requires a large amount of computing power, needs to achieve load balancing, or requires a large amount of data storage, etc. Therefore, it is considered to deploy different modules in different computing devices. For example, the functions implemented by the row processing module are executed by computing device 100A, and the functions implemented by the column processing module are executed by computing device 100B.

[0319] [Corrected according to Rule 91, 06.03.2025] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster described in Figures 18 and 19. The difference is that the memory 106 in one or more computing devices 100 in this computing device cluster can store the same instructions for executing the method provided in this application.

[0320] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the methods provided in this application. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for executing the methods provided in this application.

[0321] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions for executing some functions of the table restoration apparatus provided in this application. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of one or more modules such as the scanning module, row processing module, column processing module, merging module, correction module, or layout analysis module.

[0322] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the method provided in this application.

[0323] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform the method provided in this application.

[0324] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A form restoration method characterized by, The method comprises: performing text box detection on input data to obtain a scan result, the input data comprising an input table, the scan result comprising information of at least one text box in the input table; performing text detection on the at least one text box to determine a row text box; determining a column text box according to the at least one text box; merging the row text box and the column text box to obtain an output table.

2. The method of claim 1, wherein, The method further comprises: performing text recognition on the at least one text box to obtain text in the at least one text box; the performing text detection on the at least one text box to determine a row text box comprises: performing natural language processing on the text in the at least one text box to recognize the row text box.

3. The method of claim 2, wherein, The performing natural language processing on the text in the at least one text box to recognize the row text box comprises: inputting the input data into a row detection model to output a row detection box, the row detection box being used to represent a region of each row in the input table; determining an initial row text box according to the row detection box and the scan result; performing natural language processing on text in the initial row text box to recognize the row text box.

4. The method according to any one of claims 1-3, characterized in that, The determining a column text box according to the at least one text box comprises: performing clustering on the at least one text box to obtain a column clustered text box; determining the column text box according to the column clustered text box.

5. The method according to any one of claims 1-3, characterized in that, The determining a column text box according to the at least one text box comprises: inputting the input data into a column detection model to output a column detection box, the column detection box being used to represent a region of each column in the input table; determining the column text box according to the column detection box and the scan result.

6. The method according to any one of claims 1-3, characterized in that, The determining a column text box according to the at least one text box comprises: performing clustering on text boxes in the table region to obtain a column clustered text box; inputting the input data into a column detection model to output a column detection box, the column detection box being used to represent a region of each column in the input table; fusing the column clustered text box and the column detection box to obtain the column text box.

7. The method of any one of claims 3, 5, or 6, wherein, The method further comprises: performing layout analysis on the input data to output a table region, the table region comprising a region of the input table in the input data; wherein, in the presence of a row detection model, the table region is used as input of the row detection model, and in the presence of a column detection model, the table region is used as input of the column detection model.

8. The method of claim 7, wherein, The inputting the input data into a layout analysis model to output a table region comprises: correcting an inclination angle of the input data to obtain corrected input data; inputting the corrected input data into the layout analysis model to output the table region.

9. A form restoration method characterized by, The method comprises: performing text box detection on input data to obtain a scan result, the input data comprising an input table, the scan result comprising information of at least one text box in the input table; determining a row text box according to the at least one text box; determine column text boxes according to the at least one text box, the column text boxes being obtained by clustering the at least one text box, and / or the column text boxes being obtained according to column detection boxes output by a column detection model, the column detection boxes representing regions of each column in the input table; merge the row text boxes and the column text boxes to obtain an output table, the row text boxes including text boxes in the same row in the output table, and the column text boxes including text boxes in the same column in the output table.

10. The method of claim 9, wherein, In a case where the column text boxes are determined in combination of clustering the at least one text box and the column detection boxes output by the column detection model, the determining the column text boxes according to the at least one text box includes: clustering the text boxes in the table region to obtain column clustered text boxes; inputting the input table region in the input data into a column detection model to output the column detection boxes; fusing the column clustered text boxes and the column detection boxes to obtain the column text boxes.

11. A table reconstruction apparatus characterized by comprising: includes: a scanning module, configured to perform text box detection on input data to obtain a scanning result, the input data including an input table, and the scanning result including information of at least one text box in the input table; a row processing module, configured to determine row text boxes by performing text detection on the at least one text box; a column processing module, configured to determine column text boxes according to the at least one text box; a merging module, configured to merge the row text boxes and the column text boxes to obtain an output table.

12. The apparatus of claim 11, wherein, The row processing module is specifically configured to: perform text recognition on the at least one text box to obtain text in the at least one text box; perform natural language processing on the text in the at least one text box to identify the row text boxes.

13. The apparatus of claim 12, wherein, The row processing module is specifically configured to: input the input data into a row detection model to output row detection boxes, the row detection boxes being used to represent regions of each row in the input table; determine initial row text boxes according to the row detection boxes and the scanning result; perform natural language processing on text in the initial row text boxes to identify the row text boxes.

14. The apparatus of any one of claims 11-13, wherein, The column processing module is specifically configured to: cluster the at least one text box to obtain column clustered text boxes; determine the column text boxes according to the column clustered text boxes.

15. The apparatus of any one of claims 11-13, wherein, The column processing module is specifically configured to: input the input data into a column detection model to output column detection boxes, the column detection boxes being used to represent regions of each column in the input table; determine the column text boxes according to the column detection boxes and the scanning result.

16. The apparatus of any one of claims 11-13, wherein, The column processing module is specifically configured to: cluster the text boxes in the table region to obtain column clustered text boxes; input the input data into a column detection model to output column detection boxes, the column detection boxes being used to represent regions of each column in the input table; fuse the column clustered text boxes and the column detection boxes to obtain the column text boxes.

17. The apparatus of any one of claims 13, 15, or 16, wherein, The apparatus further includes: a layout analysis module, configured to input the input data into a layout analysis model to output a table region, the table region including a region of the input table in the input data; In the presence of a row detection model, the table region is used as input to the row detection model, and in the presence of a column detection model, the table region is used as input to the column detection model.

18. The apparatus of claim 17, wherein, The device further comprises: a correction module configured to correct a tilt angle of the input data to obtain corrected input data; the layout analysis module is specifically configured to input the corrected input data into the layout analysis model and output the table region.

19. A table reconstruction apparatus characterized by comprising: Comprise: a scanning module configured to perform text box detection on input data to obtain a scanning result, the input data comprising an input table, and the scanning result comprising information of at least one text box in the input table; a row processing module configured to determine a row text box according to the at least one text box; a column processing module configured to determine a column text box according to the at least one text box, the column text box being obtained by clustering the at least one text box, and / or the column text box being obtained according to a column detection model outputting a column detection box representing a region of each column in the input table; a merging module configured to merge the row text box and the column text box to obtain an output table, the row text box comprising text boxes in the same row in the output table, and the column text box comprising text boxes in the same column in the output table.

20. The apparatus of claim 19, wherein, In the case of combining clustering of the at least one text box and determination of the column detection box according to the column detection model, the column text box determined according to the at least one text box comprises: clustering text boxes in the table region to obtain column clustered text boxes; inputting the input table region in the input data into a column detection model to output the column detection box; fusing the column clustered text boxes and the column detection box to obtain the column text box.

21. A computing device, comprising: The computing device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the operation steps of the method of any one of claims 1 to 10.

22. A cluster of computing devices, characterized in that, Comprise at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the operation steps of the method of any one of claims 1 to 10.

23. A computer program product comprising instructions, characterized in that, When the instructions are executed by the computing device cluster, the computing device cluster performs the operation steps of the method of any one of claims 1 to 10.

24. A computer-readable storage medium, characterized in that, Comprise computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the operation steps of the method of any one of claims 1 to 10.

Images