Table restoration method and apparatus

Abstract

The present application provides a table restoration method and apparatus based on optical character recognition, for use in respectively performing structure recognition on rows and columns in a table, and using text detection to recognize texts in a same row to more accurately recognize a row textbox, thereby achieving more accurate table structure restoration. The method comprises: first, performing textbox detection on input data to obtain a scanning result, wherein the input data may specifically comprise an image or other non-readable documents, etc., the input data comprises an input table, and the scanning result comprises information of at least one textbox in the input table, such as information of the coordinates, length, width or area of the textbox; then, performing text detection on the at least one textbox to determine a row textbox, and on the basis of the at least one textbox, determining a column textbox; and then, merging the row textbox and the column textbox to obtain an output table, wherein the output table is generally a readable table.

Description

Table restoration method and device

[0001] The present application claims priority to the Chinese patent application No. 202410535017.0, filed on April 29, 2024, and entitled "Table restoration method and device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of optical character recognition, and in particular to a table restoration method and device. BACKGROUND

[0003] Under the background of digitalization of various industries, it is necessary to digitize the electronic files accumulated for many years in various industries, and pictures and scanned documents are the main file forms. Manual information input consumes a lot of manpower. In order to improve the efficiency of business operation, it is necessary to perform optical character recognition (OCR) recognition and restoration on various scanned documents.

[0004] Tables are commonly used data formats that can express as clearly, concisely and efficiently as possible, and are widely used in electronic documents in various industries. Restoring the table structure is an important step in the digitization of scanned files. In particular, for complex and dense wireless table structures, it is a difficult point for OCR recognition and restoration of table structures.

[0005] Therefore, how to improve the accuracy of table structure restoration has become a problem to be solved. SUMMARY

[0006] The present application provides a table restoration method and device for performing structure recognition on table rows and columns respectively, and using text detection to identify text in the same row, more accurately identifying the row text box, and realizing more accurate table structure restoration.

[0007] Therefore, in a first aspect, the present application provides a table restoration method, which includes: first, performing text box detection on input data to obtain a scanning result, the input data can specifically include images or other unreadable documents, etc., and the input data includes an input table. The scanning result includes information of at least one text box in the input table, such as the coordinates, length, width or area of the text box, etc. representing the information of the text box area; then performing text detection on the at least one text box to determine a row text box, and determining a column text box according to the at least one text box; then merging the row text box and the column text box to obtain an output table, which is usually a readable table, and is more clear and readable than the input table.

[0008] In the embodiments of the present application, in the process of detecting the rows of the table region, detection is performed from the dimension of the row and the dimension of the column respectively, and in the process of identifying the row text box, text detection is combined for identification, so that the text semantics can be used to accurately judge the text boxes in the same row. For example, the text semantics between different rows in a same cell is usually coherent, so that the case of including different row texts in a same cell can be accurately distinguished, and the identification of the row text box is more accurate. Therefore, the subsequent row-column merging based on the more accurate row text box can obtain a more accurate output table.

[0009] In a possible implementation, the foregoing method can further include: performing text recognition on the at least one text box to obtain text in the at least one text box. Correspondingly, the foregoing text detection on the table region to determine the row text box can include: performing natural language processing (NLP) on the table region in the input data to identify the row text box, that is, using NLP to recognize the text semantics in each text box in the input data, so as to accurately identify the text of the row text box, and further accurately identify the row text box. For example, for the case that multiple rows of text exist in a text box, NLP can be used to recognize the text content belonging to the same text box, to achieve more accurate identification of the row text box.

[0010] In a possible implementation, the foregoing NLP on the text in the at least one text box to identify the row text box includes: inputting the input data into a row detection model to output a row detection box, which can be used to represent the region of each row in the input table; determining an initial row text box according to the row detection box and each text box in the scanning result, for example, filling each text box in the scanning result into the corresponding row detection box based on the coordinates of the text box; and performing NLP on the text in the initial row text box to identify the row text box.

[0011] In the embodiments of the present application, a pre-trained row detection model can be used to output the row detection box corresponding to the input table, that is, to divide the region corresponding to the row of the input table from the input data, to obtain an initial row text box based on the region, and to further perform NLP to refine the initial row text box, so as to obtain a more accurate row text box.

[0012] In a possible implementation, the aforementioned determining column text boxes according to the at least one text box includes: clustering the at least one text box based on coordinates of the at least one text box to obtain column clustered text boxes, such as clustering the coordinates of the text boxes, and taking text boxes with similar column coordinates in the text boxes as text boxes in the same column; and determining column text boxes according to the column clustered text boxes, such as taking text boxes in a cluster as text boxes in a column, that is, column text boxes. In the implementation of the present application, the clustering manner can be used to identify text boxes in the same column, so as to accurately identify text boxes in the same column based on column coordinates of the text boxes.

[0013] In a possible implementation, the aforementioned determining column text boxes according to the at least one text box includes: taking the input data as input of a pre-trained column detection model, and outputting column detection boxes, which can be used to represent regions corresponding to each column in the input table; and then determining column text boxes according to the column detection boxes and the aforementioned at least one text box, such as filling each text box into a corresponding column detection box based on coordinates of the text boxes, and outputting column text boxes.

[0014] In the implementation of the present application, the pre-trained column detection model can also be used to perform a detection task to output regions of columns in the table region, so that the detection capability of the model can be used to accurately identify regions of the columns in the table, and then more accurate column text boxes can be output.

[0015] In a possible implementation, the aforementioned determining column text boxes according to the at least one text box includes: first, identifying text boxes in a table region in the input data; clustering coordinates of the text boxes in the table region to obtain column clustered text boxes; in addition, taking the input data as input of a pre-trained column detection model, and outputting column detection boxes; and then fusing the column clustered text boxes and the column detection boxes to obtain column text boxes, such as filling each text box in the column clustered text boxes into a column detection box with a similar column coordinate, and outputting column text boxes.

[0016] In the implementation of the present application, the column text boxes can be identified in combination with the clustering and the column detection model, so that the column text boxes can be determined in combination with multiple manners, which provides multiple guarantees for identification of the column text boxes, and further improves accuracy of the output column text boxes.

[0017] In a possible implementation, the method further includes: performing layout analysis on the input data to output a table region, the table region including a region of the input table in the input data; and wherein the table region is used as an input of the row detection model when the row detection model exists, and the table region is used as an input of the column detection model when the column detection model exists. In the implementation of the present application, a layout analysis step can be added to distinguish multiple types of regions that can exist in the input data, such as distinguishing a header or footer region, so as to further filter out regions irrelevant to the table and achieve detection of the table region.

[0018] In a possible implementation, the method of inputting the input data into the layout analysis model and outputting the table region includes: correcting an inclination angle of the input data to obtain corrected input data; and inputting the corrected input data into the layout analysis model to output the table region. Therefore, in the implementation of the present application, an inclination correction step can be added to adapt to a case where the table region in the input data has a certain inclination angle.

[0019] In a second aspect, the present application provides a table restoration method, which is characterized by comprising: first, performing text box detection on input data to obtain a scanning result, the input data including an input table, and the scanning result including information of at least one text box in the input table, such as coordinates, length, width or area of the text box, etc. representing information of a text box region; then determining row text boxes according to positions of the at least one text box; determining column text boxes according to the at least one text box, the column text boxes being obtained by clustering the at least one text box, and / or the column text boxes being obtained from column detection boxes output by a column detection model, the column detection boxes representing a region of each column in the input table; and merging the row text boxes and the column text boxes to obtain an output table, the row text boxes including text boxes in the same row in the output table, and the column text boxes including text boxes in the same column in the output table.

[0020] In the implementation of the present application, for the column part in the input table, the column text boxes in the table are accurately recognized by using the clustering and / or column detection model, so as to improve the restoration accuracy of the columns in the input table and obtain a more accurate output table. Compared with distinguishing the column text boxes in the table based on only the coordinates of the text boxes, the column text boxes output by the clustering can constrain the coordinates of the text boxes in the same column to be closer to each other, and the column detection model can accurately identify the region of each column in the input table, so as to fill the text boxes into the corresponding column region and obtain more accurate column text boxes.

[0021] In a possible implementation, in a case where the column text boxes are determined in combination with clustering of the at least one text box and column detection boxes output by the column detection model, the foregoing determining the column text boxes from the at least one text box includes: clustering the text boxes in the table region to obtain column clustered text boxes; inputting the input table region in the input data into the column detection model to output column detection boxes; and fusing the column clustered text boxes and the column detection boxes to obtain the column text boxes.

[0022] In the implementation, the column text boxes output by the clustering can constrain the coordinates of the text boxes in the same column to be closer to each other, and the column detection model can accurately identify the region of each column in the input table, and then the text boxes can be filled into the corresponding column region to obtain more accurate column text boxes.

[0023] In a possible implementation, in a case where the column text boxes are determined in combination with clustering of the at least one text box, the foregoing determining the column text boxes from the at least one text box includes: clustering the at least one text box based on the coordinates of the at least one text box to obtain column clustered text boxes, for example, clustering the coordinates of the text boxes, and taking the text boxes with similar column coordinates as the text boxes in the same column; and determining the column text boxes from the column clustered text boxes, for example, taking the text boxes in one cluster as the text boxes in one column, that is, the column text boxes. In the implementation, the clustering can be used to identify the text boxes in the same column, so that the text boxes in the same column can be accurately identified based on the column coordinates of the text boxes.

[0024] In a possible implementation, in a case where the column text boxes are determined from column detection boxes output by the column detection model, the foregoing determining the column text boxes from the at least one text box includes: inputting the input data into the pre-trained column detection model as input to output column detection boxes, which can be used to represent the region of each column in the input table; and then determining the column text boxes from the column detection boxes and the foregoing at least one text box, for example, filling the text boxes into the corresponding column detection boxes based on the coordinates of the text boxes to output the column text boxes. Therefore, the pre-trained column detection model is used to perform the detection task to output the region of the column in the table region, so that the detection capability of the model can be used to accurately identify the region of the table column, and then more accurate column text boxes can be output.

[0025] In a possible implementation, the method can further include: performing text recognition on the at least one text box to obtain text in the at least one text box, and accordingly, the determining the row text box according to the at least one text box can include: performing natural language processing (NLP) on the table region in the input data to identify the row text box, that is, using NLP to identify the text semantics in each text box in the input data, so as to accurately identify the text of the row text box, and further identify the accurate row text box. For example, in the case that there are multiple rows of text in a text box, NLP can be used to identify the text content belonging to the same text box, to achieve more accurate row text box identification.

[0026] In a possible implementation, the performing NLP on the text in the at least one text box to identify the row text box includes: inputting the input data into a row detection model to output a row detection box, the row detection box can be used to represent the region of each row in the input table; determining an initial row text box according to the row detection box and each text box in the scanning result, for example, filling each text box in the scanning result into the corresponding row detection box based on the coordinates of the text box; and performing NLP on the text in the initial row text box to identify the row text box.

[0027] In the embodiments of the present application, a pre-trained row detection model can be used to output the row detection box corresponding to the input table, that is, to divide the region corresponding to the rows of the input table from the input data, to obtain an initial row text box based on the region, and further perform NLP to refine the initial row text box, so as to obtain a more accurate row text box.

[0028] In a possible implementation, the method further includes: performing layout analysis on the input data to output a table region, the table region including a region of the input table in the input data; and in the case that the row detection model exists, the table region is used as input of the row detection model, and in the case that the column detection model exists, the table region is used as input of the column detection model. In the embodiments of the present application, a layout analysis step can be added to distinguish multiple types of regions that can exist in the input data, such as distinguishing the header or footer regions, so as to further filter out regions unrelated to the table, and to achieve detection of the table region.

[0029] In a possible implementation, the inputting the input data into the layout analysis model to output the table region includes: correcting an inclination angle of the input data to obtain corrected input data; and inputting the corrected input data into the layout analysis model to output the table region. Therefore, in the embodiments of the present application, an inclination correction step can be added to adapt to the case that the table region in the input data has a certain inclination angle.

[0030] In a third aspect, the present application provides a table restoration device, comprising:

[0031] a scanning module, configured to perform text box detection on input data to obtain a scanning result, the input data comprising an input table, and the scanning result comprising information of at least one text box in the input table;

[0032] a row processing module, configured to determine a row text box by performing text detection on the at least one text box;

[0033] a column processing module, configured to determine a column text box according to the at least one text box;

[0034] a merging module, configured to merge the row text box and the column text box to obtain an output table.

[0035] Effects achieved by the third aspect and any possible implementation manner of the third aspect can refer to the description of the first aspect or any possible implementation manner of the first aspect, and will not be described here.

[0036] In a possible implementation manner, the row processing module is specifically configured to: perform text recognition on the at least one text box to obtain text in the at least one text box; and perform natural language processing on the text in the at least one text box to identify the row text box.

[0037] In a possible implementation manner, the row processing module is specifically configured to: input the input data into a row detection model to output a row detection box, the row detection box being used to represent a region of each row in the input table; determine an initial row text box according to the row detection box and the scanning result; and perform natural language processing on text in the initial row text box to identify the row text box.

[0038] In a possible implementation manner, the column processing module is specifically configured to: perform clustering on the at least one text box to obtain column clustered text boxes; and determine the column text box according to the column clustered text boxes.

[0039] In a possible implementation manner, the column processing module is specifically configured to: input the input data into a column detection model to output a column detection box, the column detection box being used to represent a region of each column in the input table; and determine the column text box according to the column detection box and the scanning result.

[0040] In a possible implementation manner, the column processing module is specifically configured to: perform clustering on text boxes in a table region to obtain column clustered text boxes; input the input data into a column detection model to output a column detection box, the column detection box being used to represent a region of each column in the input table; and fuse the column clustered text boxes and the column detection box to obtain the column text box.

[0041] In a possible implementation, the apparatus further includes a layout analysis module configured to input the input data into a layout analysis model, and output a table region, the table region including a region of the input table in the input data; and wherein the table region is used as an input of the row detection model when the row detection model exists, and the table region is used as an input of the column detection model when the column detection model exists.

[0042] In a possible implementation, the apparatus further includes a correction module configured to correct a tilt angle of the input data to obtain corrected input data.

[0043] The layout analysis module is specifically configured to input the corrected input data into the layout analysis model, and output the table region.

[0044] In a fourth aspect, the present application provides a table restoration apparatus, including:

[0045] The scanning module is configured to perform text box detection on the input data to obtain a scanning result, the input data including an input table, and the scanning result including information of at least one text box in the input table.

[0046] The row processing module is configured to determine a row text box according to the at least one text box.

[0047] The column processing module is configured to determine a column text box according to the at least one text box, the column text box being obtained by clustering the at least one text box, and / or the column text box being obtained according to a column detection frame output by a column detection model, the column detection frame representing a region of each column in the input table.

[0048] The merging module is configured to merge the row text box and the column text box to obtain an output table, the row text box including text boxes in the same row in the output table, and the column text box including text boxes in the same column in the output table.

[0049] Effects of the fourth aspect and any possible implementation of the fourth aspect can refer to the description of the second aspect or any possible implementation of the second aspect, which will not be repeated here.

[0050] In a possible implementation, when the column text box is determined by combining clustering of the at least one text box and the column detection frame output by the column detection model, the column processing module is specifically configured to: cluster the text boxes in the table region to obtain column clustered text boxes; input the input table region in the input data into the column detection model to output the column detection frame; and fuse the column clustered text boxes and the column detection frame to obtain the column text box.

[0051] In a fifth aspect, an embodiment of the present application provides a computing device, comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device performs the method steps in any implementation manner of the first aspect or the second aspect.

[0052] In a sixth aspect, an embodiment of the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method steps in any implementation manner of the first aspect or the second aspect.

[0053] In a seventh aspect, an embodiment of the present application provides a computer program product comprising instructions which, when executed by a computing device cluster, cause the computing device cluster to perform the method in any implementation manner of the first aspect or the second aspect.

[0054] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer program instructions, which, when executed by a computing device cluster, cause the computing device cluster to perform the method in any implementation manner of the first aspect or the second aspect.

[0055] In a ninth aspect, an embodiment of the present application provides a chip, comprising at least one processor and an interface; the at least one processor acquires program instructions or data through the interface; the at least one processor is configured to execute the program instructions to implement the method in any implementation manner of the first aspect or the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0056] FIG. 1 is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0057] FIG. 2 is a flowchart of a table restoration method provided by the present application;

[0058] FIG. 3 is a schematic diagram of an application scenario provided by the present application;

[0059] FIG. 4 is a schematic diagram of another application scenario provided by the present application;

[0060] FIG. 5 is a schematic diagram of another application scenario provided by the present application;

[0061] FIG. 6 is a schematic diagram of another application scenario provided by the present application;

[0062] FIG. 7 is a schematic diagram of another application scenario provided by the present application;

[0063] FIG. 8 is a flowchart of another table restoration method provided by the present application;

[0064] FIG. 9 is a schematic diagram of another application scenario provided by the present application;

[0065] FIG. 10 is a schematic diagram of another table restoration method provided by the present application;

[0066] FIG. 11 is a schematic diagram of another table restoration method provided by the present application;

[0067] FIG. 12 is a schematic diagram of another application scenario provided by the present application;

[0068] FIG. 13 is a schematic diagram of another application scenario provided by the present application;

[0069] [Corrected according to Rule 91 on 06.03.2025] FIG. 14 is a schematic diagram of another table restoration method provided by the present application;

[0070] [Corrected according to Rule 91 on 06.03.2025] FIG. 15 is a schematic diagram of another table restoration method provided by the present application;

[0071] [Corrected according to Rule 91 on 06.03.2025] FIG. 16 is a schematic diagram of a table restoration apparatus provided by the present application;

[0072] [Corrected according to Rule 91 on 06.03.2025] FIG. 17 is a schematic diagram of a computing device provided by an embodiment of the present application;

[0073] [Corrected according to Rule 91 on 06.03.2025] FIG. 18 is a schematic diagram of a computing device cluster provided by an embodiment of the present application;

[0074] [Corrected according to Rule 91 on 06.03.2025] FIG. 19 is a schematic diagram of another computing device cluster provided by an embodiment of the present application.

[0075] [Corrected according to Rule 91 on 06.03.2025][Deleted]

[0076] [Corrected according to Rule 91 on 06.03.2025][Deleted]

[0077] [Corrected according to Rule 91 on 06.03.2025][Deleted]

[0078] [Corrected according to Rule 91 on 06.03.2025][Deleted] DETAILED DESCRIPTION

[0079] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0080] The embodiments of the present application relate to neural networks and related applications of natural language processing (NLP). In order to better understand the solutions of the embodiments of the present application, the related terms and concepts of neural networks that may be involved in the embodiments of the present application will be introduced as follows.

[0081] (1) Neural network

[0082] A neural network can be composed of neural units, and a neural unit can refer to an operation unit with x s (input data) and an intercept 1 as inputs. The output of the operation unit can be:

[0083] where s = 1, 2, … n, n is a natural number greater than 1, W s is a weight parameter of x s , b is a bias of the neural unit. f is an activation function of the neural unit, which is used to introduce a nonlinear characteristic into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. The neural network is a network formed by connecting multiple single neural units, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neural units.

[0084] (2) Deep neural network

[0085] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple intermediate layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, intermediate layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the number of intermediate layers is the intermediate layer, or the hidden layer. The layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer.

[0086] Although DNN looks complicated, each layer can be represented as a linear relationship expression: where, is an input vector, is an output vector, is a bias vector or called bias parameter, w is a weight matrix (also called coefficient), and a() is an activation function. Each layer is just an operation on the input vector to get the output vector Due to the large number of layers in DNN, the number of coefficients W and bias vectors is also relatively large. These parameters in DNN are defined as follows: taking the coefficient w as an example: suppose in a three-layer DNN, the linear coefficient from the 4th neuron of the second layer to the 2nd neuron of the third layer is defined as The superscript 3 represents the layer number of the coefficient W, and the subscript corresponds to the output third layer index 2 and the input second layer index 4.

[0087] In summary, the coefficient from the kth neuron of the L-1th layer to the jth neuron of the Lth layer is defined as

[0088] It should be noted that the input layer has no W parameter. In a deep neural network, more intermediate layers enable the network to better depict complex situations in the real world. In theory, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is a process of learning weight matrices, and the ultimate goal is to obtain the weight matrices of all layers of the trained deep neural network (weight matrices formed by many layers of vectors W).

[0089] (3) Language Model (LM)

[0090] Used for natural language processing (NLP), which plays an important role in NLP, and its task is to predict the probability of a sentence appearing in the language. For example, a language model is usually constructed as a probability distribution p(s) of a string s, where p(s) attempts to reflect the frequency of the string s as a sentence. It can be applied to text recognition or machine translation scenarios. In the embodiments of the present application, the NLP model mentioned below can include a language model.

[0091] (4) Optical Character Recognition (OCR)

[0092] Generally refers to using electronic devices (such as scanners, cameras or mobile phones and other devices) to scan physical documents (such as paper printed documents or other materials printed documents, etc.), convert optical signals into electrical signals to generate corresponding images, and perform character recognition on the images to recognize the content in the images. For example, an electronic device can be used to scan a paper document that may contain printed table content, i.e., for printed characters, an optical method is used to convert the text in the paper document into a black and white dot matrix image file, and the text in the image is converted into a text format by recognition software for further editing and processing by word processing software.

[0093] (5) Convolutional recurrent neural network (CRNN)

[0094] It is a model for recognizing text sequences of indefinite length end-to-end, without cutting individual characters first, but converting text recognition into a time-dependent sequence learning problem, i.e., sequence recognition based on images, which can recognize longer text sequences.

[0095] CRNN includes a convolutional neural network (CNN) feature extraction layer and a sequence feature extraction layer based on an attention mechanism and a bi-directional long short-term memory (BLSTM), which can be trained end-to-end. The BLSTM and the connectionist temporal classification (CTC) component learn the context relationship in the character image, thereby effectively improving the text recognition accuracy and making the model more robust. In the prediction process, the front end uses a standard CNN network to extract the features of the text image, the BLSTM is used to fuse the feature vectors to extract the context features of the character sequence, then the probability distribution of each column feature is obtained, and finally the text sequence is obtained by the transcription layer (CTC rule) for prediction.

[0096] The system architecture and method steps provided by the present application are described below.

[0097] First, the method provided by the present application can be applied to the scene of table scanning, and the table in the image is restored to an editable table or a higher definition table by scanning the image containing the table.

[0098] The method provided by the present application can be deployed in various electronic devices, such as server clusters, cloud platforms, personal computers, smart phones or smart cars, etc.

[0099] In a possible implementation, the method provided in the present application can be deployed on a computing device such as a personal computer, a computer workstation, a smart phone, a tablet computer, a notebook computer, a smart car, and the like, and a user can directly use the computing device to perform the method provided in the embodiments of the present application to realize table restoration.

[0100] For example, the method provided in the present application can be applied to a user device such as a smart phone, a tablet computer, or a device capable of deploying OCR recognition, and the like. A user can use the user device to scan an entity to be restored, for example, using a camera or an infrared scanning probe of the user device to obtain scanning data of the entity. Then, the method provided in the present application is used to restore the table in the entity.

[0101] In a possible implementation, the method provided in the present application can be deployed on a server cluster, a cloud platform, or other computing device.

[0102] For example, the method provided in the present application can be deployed on a cloud platform to provide services for users through cloud services.

[0103] For example, FIG. 1 shows a structural schematic diagram of a cloud service system provided in the present application. As shown in FIG. 1, the cloud service system 10 can include a computing device 11 and a client 12.

[0104] The computing device 11 can specifically include a server cluster or a cloud platform, or the computing device can also be other computing devices. Optionally, the computing device 11 can cooperate with other computing devices such as a data storage, a router, a load balancer, and the like. The computing device 11 can use data in a data storage system or call program code in the data storage system to implement the method steps provided in the embodiments of the present application.

[0105] The computing device 11 can provide services for users in the form of a client. A user can operate in the client 12 to interact with the computing device 11 or request the computing device 11 to provide services. The client can be specifically deployed on a personal computer, a computer workstation, a smart phone, a tablet computer, a notebook computer, a smart car, and the like.

[0106] In an implementation, the computing device 11 is configured to implement the method provided in the embodiments of the present application to realize table restoration of an input image of the client and send an output table to the client.

[0107] The client 12 is an optional device. In an actual application scenario, the computing device can read input data from a storage space of the computing device or from a data center, restore the input data to an output table, and output the restored output table.

[0108] In general, the method provided by the embodiments of the present application can be applied to an electronic device, i.e., the aforementioned computing device 11 can be various electronic devices, such as a server cluster, a cloud platform, a personal computer, a smart phone, or a smart car, etc.

[0109] In combination with the architecture shown in the aforementioned FIG. 1, the method provided by the present application can be executed by the computing device 11, which specifically can include a server cluster or a cloud platform, can provide services for users in the manner of a client, or the computing device 11 can also be other devices with computing capabilities.

[0110] For example, the method provided by the present application can be deployed in a cloud platform to provide a table restoration service for users in the manner of a client. The user can input an image in the client, the input manner specifically can be to take an image through an image sensor, or to scan through a scanning probe, etc., and transmit the input image to the cloud platform, which restores the table through the method provided by the present application to obtain a more high-definition table data after restoration.

[0111] In actual application scenarios, under the background of digitalization of various industries, it is necessary to digitize the electronic files accumulated for many years in various industries, among which pictures and scanned documents are the main file forms. Manual information input consumes a lot of manpower. In order to improve the efficiency of business operation, it is necessary to perform OCR recognition and restoration on various scanned documents. Among various documents, tables are the most commonly used data organization form, can express as clearly, concisely and efficiently as possible, and widely exist in electronic documents of various industries. Restoring the table structure is an important step in the digitization of scanned files. The restoration of complex and dense wireless table structure has always been a pain point and difficulty problem of table structure restoration, and is also a common problem and difficulty problem in business scenarios.

[0112] In some existing scenarios for table restoration, whether it is based on visual table cell detection and table line segmentation, or directly restores the table structure code in an end-to-end manner, it is difficult to solve the problem of complex wireless table restoration.

[0113] For example, in an existing table restoration scheme, first, the text box detection and text recognition results of the wireless table are obtained, then the splitting and merging of the text box are performed for the text box that is mis-merged or mis-split, and then the row and column information is obtained according to the accurate text box position and more rules. However, for a complex table, the splitting and merging of the table is a very large workload, and the accuracy requirement is also very high, so the table restoration accuracy of the implementation is low, and this scheme can only be applied to a simple table scenario.

[0114] For example, in an existing table restoration scheme, the coordinates of the text box of the wireless table are first detected, whether it is the same cell is determined by the difference of the adjacent line text box coordinates, whether the position of the text box is in the same line can be determined according to whether it intersects with the extension line of the text box middle line; the text boxes in each column can be adjusted by the horizontal coordinates of each text box, the column boundary line is generated, and whether to merge or split is determined according to the distance between the boundary line. Finally, the cross-column text box is split according to the coordinate information of each character in each text box.

[0115] For example, in an existing table restoration scheme, the coordinates of the text box of the wireless table are first detected, whether it is the same cell is determined by the difference of the adjacent line text box coordinates, whether the position of the text box is in the same line can be determined according to whether it intersects with the extension line of the text box middle line; the text boxes in each column can be adjusted by the horizontal coordinates of each text box, the column boundary line is generated, and whether to merge or split is determined according to the distance between the boundary line. Finally, the cross-column text box is split according to the coordinate information of each character in each text box.

[0116] For example, in an existing table restoration scheme, the coordinates of the text box of the wireless table are first detected, whether it is the same cell is determined by the difference of the adjacent line text box coordinates, whether the position of the text box is in the same line can be determined according to whether it intersects with the extension line of the text box middle line; the text boxes in each column can be adjusted by the horizontal coordinates of each text box, the column boundary line is generated, and whether to merge or split is determined according to the distance between the boundary line. Finally, the cross-column text box is split according to the coordinate information of each character in each text box.

[0117] Therefore, the present application provides a table restoration method with stronger generalization ability, which can be applied to simple table restoration scenes and complex table restoration scenes, and can accurately restore tables in various scenes.

[0118] The method flow provided by the present application is introduced below.

[0119] Referring to FIG. 2, the flowchart of the table restoration method provided by the present application is as follows.

[0120] 201. Text box detection is performed on the input data to obtain a scanning result.

[0121] The input data includes an input table, and the scanning result can include information of one or more text boxes. The input table can be an image or a non-readable and non-writable table in the input data.

[0122] The input data can specifically include an RGB image, a grayscale image, an infrared image, or the like. The input image can include a received image or collected data. In the case where the input data includes collected images, the method provided in the present application can be deployed in a table restoration device, and the input data can be an image collected by an infrared probe of the table restoration device or an image collected by a camera of the table restoration device.

[0123] The pre-trained model can be used to detect text boxes in the input data, i.e., to identify text boxes in the input data, and output information of at least one text box. The information of each text box can include information such as a position, a size, or a shape of the text box.

[0124] Specifically, the input data can be subjected to OCR detection, so as to detect a text box region in the input data and obtain a scanning result. That is, the scanning result can include information of one or more text boxes in the input data, such as coordinates, regions, or sizes of the text boxes in the input data.

[0125] Further, the input data can be subjected to OCR recognition, i.e., to recognize text content included in the input data. Accordingly, the scanning result can further include the recognized text content in the text box region in the table region of the input data, so that a table including editable text content in the text box can be generated subsequently.

[0126] Optionally, to improve the accuracy of the table region in the scanning result, the table region can be further subjected to correction or layout analysis, which will be introduced below.

[0127] In one possible implementation, the input data can be subjected to inclination correction, and corrected input data is output after the inclination correction. After the scanning result is obtained, the coordinates of the text boxes in the scanning result can be used to calculate an inclination angle of the text boxes, the inclination angle is taken as an inclination angle of the input data, and the input data is corrected based on the inclination angle, e.g., the input data is rotated, so as to obtain corrected input data, so that the table region, the row detection box, or the column detection box in the corrected input data can be recognized subsequently.

[0128] Correspondingly, when the input data has a tilt, the input table in the input data also has a tilt accordingly, and thus the detected text box can be corrected for tilt, so that a more regular text box after correction is obtained, thereby reducing the influence of the tilt angle on subsequent table restoration. For example, after the tilt angle is calculated, one or more first initial text boxes output can be rotated based on the tilt angle, so that the one or more first initial text boxes are corrected to align with the standard coordinate system, to obtain a more standard text box, so that subsequent processing can be performed according to the more standard text box, without the need to increase the tilt angle for calculation in the subsequent processing stage, and the detection efficiency of the row text box and the column text box can be improved.

[0129] In a possible implementation, the scanning result can also be analyzed for layout, so as to further identify an accurate table region.

[0130] Specifically, the input data can be input to a layout analysis model (which can also be replaced by other layout analysis algorithms), or after the tilt correction, the corrected input data is input to the layout analysis model, and a table region is output, which represents the region of the input table in the input data, and can specifically include the corner coordinates, center point coordinates, length, width or area of the table region, and the like. The table region can be used as input of a subsequent row detection model or column detection model. Generally, in addition to the output table, the input data can also include regions such as headers, footers, titles or images, and through the layout analysis model, the various regions in the input data can be distinguished, so as to accurately identify the region corresponding to the input table in the input data, that is, to output one or more text boxes in the input table. Therefore, in the embodiments of the present application, the layout analysis model can be used to crop the input data to output the text boxes in the table region of the input data, so as to avoid the influence of other layout contents in the input data on the subsequent table restoration point.

[0131] The table can be divided into rows and columns. In the embodiments of the present application, the following steps 202 and 203 can be detected from the dimensions of rows and columns respectively. The method provided in the embodiments of the present application can be applied to wired tables and / or wireless tables. The wired table is a table including table lines, and the wireless table is a table without table lines.

[0132] Before introducing the following steps 202 and 203, the input table, row text box and column text box are introduced first for the convenience of understanding.

[0133] For example, the structure of the input table can be as shown in FIG. 3, which includes a plurality of text boxes, and each text box can be filled with information to save new information. The text boxes in the input table are divided into rows and columns according to the horizontal direction and the vertical direction respectively.

[0134] Correspondingly, the row text boxes, i.e. the text boxes in the same row, are shown in FIG. 4. The text boxes in the same table row are referred to as row text boxes. There can be one or more rows of text boxes in the same input table, and there can be one or more text boxes in a row.

[0135] 202. Perform text detection on the at least one text box to determine the row text boxes.

[0136] After performing the foregoing step 201, the information of the one or more text boxes in the input table can be outputted. Text detection is performed on the one or more text boxes, i.e. the text content in the one or more text boxes is detected, so as to identify the text boxes in the same row based on the text content.

[0137] Specifically, the text boxes in the same row can be preliminarily screened based on the text boxes outputted in the foregoing step 201. Then, the text content in the text boxes is detected. For example, the semantics of the text in each text box is identified, so as to determine whether the text boxes in different rows are misjudged based on the context semantics between adjacent text boxes. For example, whether different rows in the same text box are divided into multiple text boxes is determined, and the case that the same text box is divided into multiple text boxes is corrected, so as to further screen the text boxes in the same row, identify the text content in the same text box, and accurately determine the row text boxes, i.e. the text boxes in the same row.

[0138] Optionally, in the foregoing step 201, if the text in each text box has been identified, for example, the content of the text box has been recognized by OCR, then the text content in each text box is also outputted in step 201. In this case, the text content in each text box can be directly subjected to subsequent NLP processing. If the text content in each text box is not outputted in the foregoing step 201, then the text content corresponding to each text box needs to be identified in the present step 202, for example, OCR recognition or other text recognition steps are performed in the present step, and the identified text content is subjected to NLP processing.

[0139] Specifically, the aforementioned detecting the text content in the text boxes can specifically include performing NLP processing on the text in each text box, such as performing NLP processing by using a model capable of NLP, so as to identify the semantics of the text in each text box in the table region, and then determine the text included in each text box according to the semantics of the text in each text box, so as to divide the table region into row text boxes. In the embodiments of the present application, NLP can be performed on the text in the table region, so as to identify the specific content of the text in each text box based on the identified semantics, and then the accurate division of the row text boxes can be realized based on the semantics, and more accurate row text boxes, i.e., the text boxes in the same table row, can be output.

[0140] In some scenarios, it is possible that different text boxes in the same text box are divided into multiple text boxes by mistake, and therefore, in the embodiments of the present application, the NLP can be further used to identify different text boxes that should be divided into the same text box based on the context semantics between adjacent text boxes, so as to further correct the initial row text boxes identified, and output more accurate row text boxes.

[0141] Optionally, in the process of performing preliminary screening of the text boxes in the same row, the coordinates of each text box can be used for preliminary screening, or a pre-trained row detection model can be used to output the coordinates of the rows in the input table, so as to preliminarily divide the text boxes output in the aforementioned step 201, and output the initial row text boxes.

[0142] For example, the input data (or the input data after tilt correction) or the table region in the input data (such as the table region output by the aforementioned layout analysis model) can be input into the row detection model, and a row detection box is output. The row detection box can be used to represent the rows in the input table, such as the center point coordinates or the corner point coordinates of the row detection box output by the row detection model, so as to mark the regions where each row in the input table is located; then, one or more text boxes output by the text box detection are combined with the row detection box, and the initial row text boxes are identified, such as dividing the one or more text boxes into the corresponding row detection box to form the initial row text boxes in the same row; and then, NLP is performed on the text in the initial row text boxes, and the row text boxes are further identified based on the NLP result.

[0143] In addition, in combination with the aforementioned layout analysis step, the layout analysis step is introduced here taking the input data as an input image as an example. The input image includes an input table and can also include other regions unrelated to the input table, such as a header, a footer, or a title region. Therefore, in the embodiments of the present application, an optional embodiment is also provided, in which the table region in the input data can be recognized before the input data is input into the row detection model, and the table region is taken as the input of the row detection model. For example, as shown in FIG. 6, in the case where the input data is an image, the image can include a table region or other regions such as a header, a footer, or a title region. The image can be cropped to retain the table region therein, or the input table can be cut from the image, so that the table region corresponding to the input table is taken as the input of the row detection model, thereby reducing the interference of regions unrelated to the table in the image on the row detection model, to improve the accuracy of the output result of the row detection model.

[0144] 203, determine column text boxes.

[0145] For the columns in the table, the text boxes can be divided into columns in the input table based on the coordinates of the text boxes output in step 201, and text boxes in the same column in the input table, i.e., column text boxes, are output.

[0146] Specifically, the text boxes with similar column coordinates can be taken as text boxes in the same column according to the coordinates of the text boxes output in step 201, so that the column text boxes are output.

[0147] In addition, the manner of determining the column text boxes in the embodiments of the present application can also use clustering and / or a column detection model to determine the column text boxes, which can be referred to the description of step 803 below, and will not be described here.

[0148] It should be noted that the present application does not limit the execution order of steps 202 and 203, and steps 202 can be executed first, or steps 203 can be executed first, or steps 202 and 203 can be executed simultaneously, which can be adjusted according to actual application scenarios.

[0149] 204, merge the row text boxes and the column text boxes to obtain an output table.

[0150] After obtaining the row text boxes and the column text boxes, the row text boxes and the column text boxes can be merged to obtain an output table.

[0151] Specifically, after obtaining the row text boxes and the column text boxes, the rows and the columns in the table can be restored respectively, for example, as shown in FIG. 7, the column text boxes are filled into the columns of the table, and the row text boxes are filled into the rows of the table, so that a complete output table is obtained.

[0152] In addition, in the step 201 or the step 202, the text content included in each text box is also identified, and the identified text content in each text box can be filled into each text box to obtain an output table containing the text content.

[0153] The input table is usually an image or a non-readable and non-writable table, which cannot be directly read or modified. The output table output by the embodiment of the present application has higher definition than the input table, and further has the characteristics of being readable and writable when the text content is included in the output table, so as to facilitate the user to further extract data or rewrite data in the table.

[0154] In the embodiment of the present application, the row text box and the column text box can be detected respectively. In the detection of the row text box, the text content detection can be used to accurately distinguish the text content included in the same row text box, so as to realize more accurate detection of the row text box. Then, the complete table is restored by fusing the row text box and the column text box, so as to realize the table restoration. For the dense and complex wireless table, the text content detection can also be used to more accurately identify the text content included in the same row text box by the method provided in the present application, so as to realize more accurate table restoration. Therefore, the method provided in the embodiment of the present application has stronger generalization ability, and can be applied to simple table restoration scenes and complex table restoration scenes, and can accurately restore the table in various scenes.

[0155] The foregoing describes a table restoration method provided by the present application, which focuses on the identification of table rows, and identifies the text content in the same text box by text detection, so as to improve the identification accuracy of the table rows.

[0156] The present application also provides a table restoration method, which focuses on the identification of table columns, and restores a more accurate output table by accurately restoring the columns in the table.

[0157] Referring to FIG. 8, a flowchart of another table restoration method provided by the present application is shown as follows.

[0158] 801, text box detection is performed on input data to obtain a scanning result.

[0159] The step 801 can refer to the description of the foregoing step 201, and will not be described here.

[0160] 802, a row text box is determined according to the at least one text box.

[0161] In the embodiment of the present application, after the foregoing step 201 is performed, the information of one or more text boxes in the input table can be output, and each text box can be divided based on the position of the one or more text boxes, and the relative positions in each text box are identified according to the position of each text box, so that the text boxes representing similar row coordinates are divided into the same row, and the row text boxes are output.

[0162] The difference between this step and the foregoing step 202 is that the row text boxes can be identified without using the text detection method, so as to reduce the calculation amount of the table restoration and improve the efficiency of the table restoration.

[0163] Of course, the row text can also be output based on the text detection method in the foregoing step 202, and the specific steps can be referred to the introduction of the foregoing step 202, which will not be described here.

[0164] 803. The column text boxes are determined using a clustering and / or column detection model.

[0165] In the embodiment of the present application, the method for determining the column text boxes can include one or more, such as using a column detection model to detect the column text boxes, or clustering the text boxes in the table region to determine the column text boxes, or fusing the column text boxes obtained by clustering the column text boxes detected by the column detection model and the column text boxes obtained by clustering, and the following will introduce the determination methods of the column text boxes.

[0166] 1. Based on a column detection model

[0167] In this embodiment of the present application, the input data is input into the column detection model, or the table region in the input data is input into the column detection model, and the column detection box is output, which can be used to represent the region of each column in the input table. Then, based on the column detection box and one or more text boxes in the scanning result, the text boxes in each column are determined to obtain the column text boxes.

[0168] In the embodiment of the present application, the column detection model can be used to determine the region of each column in the input table, so as to identify the text boxes in the same column in combination with the positions of the text boxes in the scanning result, so as to distinguish the columns in the input table.

[0169] Alternatively, similar to the foregoing data input into the row detection model, the input image includes the input table and can also include other regions unrelated to the input table, such as header, footer or title regions. Therefore, in the embodiment of the present application, an optional embodiment is also provided, and before the input data is input into the column detection model, the table region in the input data can also be identified, and the table region is used as the input of the column detection model. The regions unrelated to the table in the input data can reduce the interference of the column detection model, so as to improve the accuracy of the output result of the column detection model.

[0170] 2、based on clustering

[0171] The one or more text boxes included in the aforementioned scanning result can be clustered, so as to divide the one or more text boxes into multiple categories, and each category is taken as a column of text boxes in the input table, and the column text boxes are output.

[0172] Specifically, the column coordinates of the one or more text boxes can be clustered, as shown in FIG. 9, the coordinates on the x-axis of the one or more text boxes can be clustered, and the one or more text boxes are divided into one or more columns of text boxes according to the clustering result, that is, one column of clustered text boxes is taken as a column of text boxes.

[0173] Therefore, in the embodiments of the present application, the text boxes in the same column can be divided into the same column by using the clustering manner, so as to more accurately identify the column text boxes.

[0174] 3、based on a column detection model and clustering

[0175] In a possible implementation, the column detection model and the clustering can be combined to further identify more accurate column text boxes.

[0176] Specifically, the input data is input into the column detection model, and a column detection box is output, which can be used to represent the area of each column in the input table; the one or more text boxes included in the aforementioned scanning result are clustered, so as to divide the one or more text boxes into multiple categories, and the column clustered text boxes are output; and then the column clustered text boxes and the column detection box are fused, for example, the text boxes in the column clustered text boxes are filled into the corresponding column detection box, so as to output the final column text boxes.

[0177] Therefore, in the embodiments of the present application, the column detection model and the clustering manner can be combined, and the text boxes in the same column can be more accurately identified through double protection, so as to more accurately restore the table from the dimension of the column.

[0178] 804、merge the row text boxes and the column text boxes to obtain an output table.

[0179] After obtaining the row text boxes and the column text boxes, the row text boxes and the column text boxes can be fused to obtain the restored output table.

[0180] The step 804 can refer to the aforementioned step 204, and will not be described here.

[0181] Therefore, in the embodiments of the present application, for the column part in the input table, the column text box in the table is accurately recognized by using the clustering and / or column detection model, so as to improve the restoration accuracy of the column in the input table and obtain a more accurate output table. Compared with distinguishing the column text box in the table based on the coordinates of the text box, the column text box output by the clustering can constrain the coordinates of the text box in the same column, so that the coordinates of the text box in the same column are closer. By using the column detection model, the area of each column in the input table can be accurately recognized, and then the text box can be filled into the corresponding column area, so that a more accurate column text box is obtained. The row text box and the column text box can be fused, so that the input table is more accurately restored, and a very accurate output table is output.

[0182] The foregoing has briefly introduced the method provided by the present application. In the following, the method process provided by the present application is further introduced in combination with a specific application scenario.

[0183] Taking the document scanning scenario as an example, referring to FIG. 10, the flowchart of another table restoration method provided by the present application is as follows.

[0184] Firstly, the paper printed document entity containing the input table can be scanned by using the scanning device to obtain a scanned image, and the scanned image is subjected to OCR detection, that is, step 1001 is executed, and one or more text boxes in the scanned image are output. Optionally, the scanned image can be further subjected to image correction or layout analysis, that is, steps 1002 and 1003 are executed, and the image after inclination correction and the table area in the image are output. Then, the table row restoration and the table column restoration can be executed respectively, that is, steps 1004 and 1005 are executed, wherein the table row restoration is to restore the row text box in the input table, and the table column restoration is to restore the column text box in the input table. The table restoration is performed based on the output results of the table row restoration and the table column restoration, that is, step 1006 is executed, and the final output table is output.

[0185] The foregoing steps are described below. In combination with the foregoing embodiment steps of FIG. 2 and FIG. 8, different steps can be selected and deployed according to different scenarios. For example, for table row restoration, the following methods can be used: method 1, using coordinate-based determination of initial row text boxes, combined with NLP to further determine row text boxes; method 2, using a row detection model to identify row detection regions, combined with NLP to further determine row text boxes; method 3, using a row detection model to output row detection boxes, and based on the row detection boxes to output row text boxes. For table column restoration, the following methods can be used: method a, using text box clustering to determine column text boxes; method b, using a table detection model to determine column text boxes; method c, combining text box clustering and column detection model, etc. In addition, for some optional steps such as direction correction or layout analysis, etc., whether to perform them can also be selected according to the actual application scenario.

[0186] In specific implementation, one of the foregoing methods 1, 2, and 3 can be combined with one of methods a, b, and c. In the embodiments of the present application, some possible combination methods are introduced below, and are not exhaustive. In actual application scenarios, the required methods can be selected to restore rows and columns respectively, and the present application does not limit this.

[0187] For example, when restoring the table structure, clustering can be used, column detection model can be used for column detection, or clustering and column detection model can be used for detection, etc. When restoring row text boxes, row detection model can be used for row text box detection, NLP model can be used for row text box detection, or row detection model and NLP model can be used, etc. Different implementation methods are introduced below.

[0188] Implementation Method One

[0189] Referring to FIG. 11, the flowchart of another table restoration method provided by the present application is as follows.

[0190] 1101, OCR detection and identification.

[0191] The input of OCR detection and identification is an image, such as the foregoing input data or scanned image, which is collectively referred to as input image here. The input image can be an image collected by an image sensor, or an image received. The input image includes an input table, as shown in step 1101 of FIG. 11. The input image can include an input table, and in addition, can include other text or image content, etc.

[0192] In the step, the OCR detection and recognition can be divided into an OCR detection step and an OCR recognition step, and the OCR recognition is an optional step. The input of the OCR detection is the input data or the scanned image, and the output is one or more text boxes detected. The input of the OCR recognition is the one or more text boxes detected by the OCR detection and the input data or the scanned image, and the output is the content included in each text box. The OCR detection and the OCR recognition will be introduced respectively.

[0193] 1. OCR detection

[0194] Specifically, the OCR detection can detect the text box in the input image, such as the coordinates, shape, or size of the text box in the input image.

[0195] Optionally, a differentiable binarization network (DBNet), DBNet++, a progressive scale expansion network (PSENet), or other models that can be used for text box detection can be used.

[0196] In one scenario, the method provided by the embodiment of the present application can be deployed in a cloud platform, and a table restoration service can be provided for users in the form of a client. The client can be deployed in a computing device of a user, such as a mobile phone, a tablet computer, a personal computer, or other terminals, and the user can use an image sensor provided by the computing device to scan a document to be restored, so as to obtain an image including the document, and transmit the image to the cloud platform. For example, when a user needs to restore a table included in a printed document, the user can use a camera of the terminal to scan the document, so as to capture an image including the document, and transmit the image to the cloud platform, and the cloud platform performs OCR detection on the image.

[0197] In one scenario, the method provided by the embodiment of the present application can be deployed in a computing device of a user, such as a mobile phone, a tablet computer, a personal computer, or other terminals, and the user can use an image sensor provided by the computing device to scan a document to be restored, so as to obtain an image including the document. For example, when a user needs to restore a table included in a printed document, the user can use a camera of the terminal deployed with the embodiment of the present application to scan the document, so as to capture an image including the document, and then the computing device performs OCR detection on the image.

[0198] For example, in a possible scenario, the method provided by the embodiment of the present application can be deployed in an end-to-end document OCR recognition system, and a user scans a paper document printed on paper by using a sensor to obtain data containing information of the entity document. Specifically, the document entity or image to be restored can be scanned by using a sensor, which can specifically include an RGB image sensor, a laser sensor, or other sensors that can be used to collect information of the entity. After the document is scanned, the information of the scanned document can be collected by using the sensor, and a corresponding image can be generated. For example, the RGB image sensor can be used to collect the document image, and output the corresponding RGB image of the document; or the laser sensor can be used to scan the document, and output the corresponding document image. The image sensor can be used to collect the image corresponding to the document entity. Specifically, the image sensor collects the optical signal from the document entity, converts the optical signal into an electrical signal and outputs it, and the electrical signal output by the image sensor is decoded to obtain the scanned image.

[0199] 2. OCR recognition:

[0200] Further, the OCR recognition can also be performed on each text box to recognize the text content in each text box, so that the text content in the table can be restored when the table is restored subsequently. For example, a model such as CRNN, Master (a network for text recognition), SVTR (scene text recognition with a single visual model) can be used for text recognition to output the text content included in the table.

[0201] 1102. Direction correction.

[0202] Specifically, the inclination angle of the image can be estimated according to the text box data output by the OCR detection, so that the image is corrected, such as being rotated, so that the image is rotated to an image with an inclination angle within a predetermined range. For the sake of distinction, it is called a corrected image, or the text box detected by the OCR is rotated to obtain a text box with an inclination angle within a predetermined range.

[0203] For example, as shown in FIG. 12, when it is detected that the image has a certain inclination angle, such as the inclination angle of the table line of the text box relative to the standard coordinate system can be calculated according to the text box coordinates detected by the OCR, and the image is rotated, so that the image is rotated to an image with an angle within a certain range relative to the standard coordinate system; or the text box detected by the OCR is rotated to rotate the text box to an angle within a certain range relative to the standard coordinate system.

[0204] The step 1102 is an optional step. For example, if the input image does not have an inclination relative to the standard coordinate system, the step 1102 can be omitted. Whether the step 1102 is performed can be determined according to an actual application scenario, and details are not described herein.

[0205] 1103, layout analysis.

[0206] The layout analysis step can input the aforementioned input image, the corrected image output by the step 1102, and the like. In this embodiment, the corrected image output by the step 1102 is taken as an example, and the layout analysis is performed on the corrected image. For example, a layout analysis model can be used to identify various types of regions in the corrected image, and then identify the region corresponding to the input table.

[0207] Generally, the layout analysis model can detect the header, footer, table, title, image, and the like in the document. In this embodiment, the table region needs to be restored, so the table region part can be output after the layout analysis. Specifically, a pre-trained YOLOv8 (a network for target recognition), FasterRCNN (a RCNN for detection), DINO (a visual model), and the like can be used for layout analysis to output the position of the table region in the corrected image, such as the coordinates of each corner point in the table region.

[0208] After the layout analysis, the OCR detection and recognition results output by the steps 1101 to 1103 and the layout analysis recognition results can be input into the following multiple branches. For example, the branches for the rows and columns of the table can process the recognition of the row text box and the column text box, respectively. The steps 1104 to 1105 are detection and recognition steps for the row text box, and the steps 1106 to 1108 are detection and recognition steps for the column text box, which are described below.

[0209] 1104, determining an initial row text box based on coordinates.

[0210] The text boxes in the same row can be identified based on the coordinates of the text boxes in the identified table region. For example, the direction corresponding to the columns of the table can be taken as the vertical coordinate direction, and the direction corresponding to the rows of the table can be taken as the horizontal coordinate direction. The distance between the vertical coordinates of the text boxes can be calculated, and the text boxes can be divided according to the vertical coordinates. For example, different text boxes with a difference in vertical coordinates less than a preset difference value are divided into the same row, so that the table region is divided into one or more rows of text boxes, and the initial row text box is output.

[0211] For example, the initial row text box can be as shown in FIG. 13, which includes one or more text boxes to facilitate restoring the rows in the table area.

[0212] 1105. Output the row text box by using the NLP model.

[0213] The input of the NLP model can include the initial row text box output in step 1104 and the OCR recognition result output in step 1101, and the OCR recognition result can include the text in each text box. Specifically, the NLP model (i.e., a model applicable to performing NLP) can be used to identify the text semantics in each text box, so as to semantically divide the text content belonging to the same text box into the same text box to adapt to the case that multiple rows of text exist in one text box.

[0214] The input data type of the NLP model can be determined based on the input data type during pre-training, or the NLP model can be trained based on the type of input data. For example, in one scenario, the text content in each of the aforementioned text boxes can be used as the input of the NLP model, and the text semantics in each text box or the text content belonging to the same text box can be output, so as to use the semantic understanding of the NLP model to identify the text content belonging to the same text box, thereby more accurately identifying the case that one or more rows of text belong to the same text box.

[0215] For example, in the process of training the NLP model, by collecting natural corpus, by randomly punctuating, the first half of any sentence is combined with the second half of another sentence to obtain negative samples, by training a natural language processing model, whether it is a coherent sentence is judged by binary classification, specifically, a pre-trained Bert model can be used, the input sentence vector is extracted by the model, a classification layer is output at the end, binary classification is performed, and cross entropy loss is used as the loss function to train the model. In order to be robust to application scenarios, some corpus corresponding to the application scenarios can also be collected for training the model. After the model is trained, the text of adjacent rows can be input into the model to judge whether it is coherent. If it is coherent, it is merged, otherwise it is not merged. In addition, the NLP model provided in the present application can also be implemented by using a large model to determine whether the adjacent row text is coherent.

[0216] For example, a possible table structure can be as shown in Table 1:

[0217] Table 1

[0218] In Table 1, the text box content "Guangdong Shenzhen Luohu District Nanhu Street XXX" is divided into two lines, one line is "Guangdong Shenzhen Luohu", and the other line is "District Nanhu Street XXX". In the embodiment of the application, the semantic of the text content in the table region can be recognized by using the NLP model, and the probability of splicing between "Guangdong Shenzhen Luohu" and "District Nanhu Street XXX" can be higher than a preset value in combination with the semantic between texts, that is, "Guangdong Shenzhen Luohu" and "District Nanhu Street XXX" can be regarded as the content of the same text box. Thus, the content belonging to the same text box can be recognized based on the context semantic of each text, and the line text box in the table region can be accurately recognized.

[0219] 1106, text box clustering.

[0220] For the columns in the table, the clustering method can be used to determine the column clustering text box in the table. Specifically, the coordinates of the text box in the table region can be clustered, so that the text boxes belonging to the same column are clustered into a class, thereby detecting the columns in the table region.

[0221] Specifically, the text box clustering can use Kmeans, Kmeans++, and the like clustering algorithm. Exemplarily, the Kmeans algorithm is used as an example here, first, the text box of the table region is taken, the horizontal coordinate of the center point of each text box is calculated, the number of clustering centers is set to 2, all the horizontal coordinates of the center points of the boxes are clustered using the Kmeans algorithm, and the average error dist1 of each center point horizontal coordinate to the belonging clustering center is calculated; the number of clustering centers is increased to 3, and the above clustering step is repeated to obtain the average error dist2, when the value of dist1-dist2 is less than a threshold threshold1, the clustering is stopped, and the column text box is obtained, otherwise the clustering center is continuously increased until the clustering is stopped.

[0222] 1107, output column detection box using column detection model.

[0223] For the columns in the input table, the column detection model can be used to recognize the region of each column in the table region. For example, the table region output by the foregoing step 1103 is taken as the input of the column detection model, and the information of the column detection box is output, including the center point coordinates, the corner point coordinates, the length, the width or the area of the column detection box, and the like.

[0224] [According to the correction of Rule 91 on 06.03.2025] The column detection box marks the region of each column in the table region, so that the subsequent text box can be filled into the corresponding column region.

[0225] 1108, determine column text box.

[0226] The column detection result output by the step 1106 and the step 1107 respectively can be merged with the column text box output by the clustering and the column text box output by the column detection model. For example, the text box of each class output by the column clustering is filled into the column detection box output by the column detection model, so as to ensure the accuracy of the column text box finally output by multiple ways.

[0227] It should be noted that when the column in the input table is restored, the step 1107 and the step 1108 can be executed, or both the step 1107 and the step 1108 can be executed.

[0228] For example, only the step 1107 can be executed. In this case, the column text box can be determined based on the column clustering text box output by the step 1107, for example, the text box of the same clustering category is directly used as the column text box in the table column.

[0229] In the case of only executing the step 1108, the text box in the column detection box filled after the step 1108 can be output based on the coordinates of each text box output by the step 1101, that is, the column text box.

[0230] It should be noted that the execution order of the step 1104 and the step 1106 is not limited in the present application. The step 1104 can be executed first, or the step 1106 can be executed first. The step 1104 and the step 1106 can also be executed simultaneously. The execution order of the step 1104 and the step 1106 is only exemplary and is not limited.

[0231] 1109, merging the row and column to restore the table.

[0232] After the row text box and the column text box in the table area are determined respectively, the table can be restored based on the detection result of the row text box and the column text box, so as to obtain the restored editable or high-definition table, that is, the output table.

[0233] [Corrected according to Rule 91 on June 3, 2025] For example, after the row text box (carrying the text in each text box) and the column text box are output, the row text box and the column text box are merged, for example, the row text box (and the text in each text box) and the column text box are filled into the row and the column of the output table, for example, the first text box in a certain row is represented as C1, and the coordinate corresponding to the second text box in a certain column is r1. The cell can be represented as C1_r1, and the text content corresponding to C1 is filled into the current cell, so as to complete the filling of the current cell.

[0234] In the embodiments of the present application, the rows and columns of the table can be detected and recognized respectively. When detecting the row text boxes, the text recognition method can be used to accurately recognize the text content in the same text box based on whether the text content in each text box and the text in the adjacent text box constitute a whole, so as to accurately distinguish the rows in the table. When detecting the column text boxes, the clustering or column detection model can be used to detect the column text boxes, so as to accurately recognize the column structure in the table. Therefore, in the embodiments of the present application, the row text boxes and the column text boxes in the table can be accurately detected, so as to accurately restore the table structure. Especially for dense and complex wireless tables, the method provided by the present application can accurately restore the rows and columns in the table, so as to accurately restore the table.

[0235] Embodiment two

[0236] [According to the rules 91 correction 06.03.2025] Referring to FIG. 14, the flowchart of another table restoration method provided by the present application is as follows.

[0237] 1601, OCR detection and recognition.

[0238] Wherein, step 1601 can refer to the description of the foregoing step 401, which will not be repeated here.

[0239] 1602, detecting the table region by using the table detection model.

[0240] Wherein, the layout analysis in step 1103 in the foregoing FIG. 11 can be replaced by a pre-trained table detection model to perform. The input of the table detection model can include an input image (not shown in the figure), which can be used to detect the table region in the input image, and output the table detection box corresponding to the table region, so that the input image can be cropped according to the table detection box, and the corresponding table region is output.

[0241] [According to the rules 91 correction 06.03.2025] For example, the input image may include regions such as headers, footers, tables or titles, etc. The input image is input into the table detection model to identify the table region therein, and output the information of the table detection box corresponding to the table region, such as the center point coordinates, corner point coordinates, length, width or area of the table region.

[0242] The table region output by step 1602 can be input to step 1603 and step 1605 to perform subsequent detection of the rows and columns of the table.

[0243] 1603, determining the row text box based on the row detection model.

[0244] Specifically, the table region output by the foregoing step 1602 can be taken as an input of the row detection model, and a row detection box in the table region is output, such as the center point coordinates, corner point coordinates, length, width, or area of the row detection box, and the like.

[0245] [According to Rule 91 correction 06.03.2025] For example, the column text box output by the row detection model marks the region of one or more rows in the table region, and in combination with the coordinates of each text box output by the foregoing step 1601, each text box can be filled into the corresponding row detection box, so as to output the row text box from the dimension of the row, that is, the text box in the same table row.

[0246] 1604, text box clustering.

[0247] 1605, output column detection box by using column detection model.

[0248] 1606, determine column text box.

[0249] 1607, merge rows and columns to restore the table.

[0250] Among them, the steps 1605-16016 can refer to the description of the foregoing steps 406-409, which will not be described here.

[0251] In the embodiment of the application, the rows and columns in the table are detected respectively, and when detecting the columns, the clustering or the column detection model can be combined to realize the detection of the column text box, so that the column structure in the table can be accurately recognized. Therefore, by using the method provided in the embodiment of the application, more accurate table restoration can be realized. In addition, compared with the foregoing FIG. 11, in the embodiment of the application, the step of NLP processing does not need to be performed, that is, the foregoing step 1101 does not need to be performed, such as in the case that the output accuracy of the row detection model is high, that is, the row text box can be accurately recognized, reducing the recognition operation for the row text box and improving the efficiency of outputting the row text box. And the foregoing layout analysis step is replaced by using the table detection model to output the table region, so that the detection capability of the table detection model can be used to output the table region in the image, so as to facilitate the subsequent processing of the table region.

[0252] Embodiment three

[0253] [According to Rule 91 correction 06.03.2025] Referring to FIG. 15, a flowchart of another table restoration method provided by the application is shown as follows.

[0254] 1901, OCR detection and recognition.

[0255] 1902, direction correction.

[0256] 1903, layout analysis.

[0257] Steps 1901 to 1903 can refer to the description of the aforementioned steps 1101 to 1103, which will not be repeated here.

[0258] 1904, determining initial row text boxes based on the row detection model.

[0259] Step 1904 can refer to the description of the aforementioned step 1609, with the difference that the output row text boxes are used as initial text boxes to continue NLP in the subsequent step 1905.

[0260] 1905, identifying the same text box using the NLP model.

[0261] Step 1905 can refer to the description of the aforementioned step 1105, which will not be repeated here.

[0262] The difference is that in the embodiments of the present application, the row detection model and the NLP model are combined to more accurately identify the row text boxes through double protection.

[0263] Specifically, after step 1904 is performed, the row detection model outputs the row detection box, and the text boxes output by step 1901 are filled into the row detection box to output the initial row text boxes; then the data included in the initial row text boxes is input into the NLP model, and the NLP model is used to identify the semantics of the text included in each text box, so as to identify whether the content included in the adjacent text boxes is context text based on the context semantics between the adjacent text boxes, and further determine whether to determine the adjacent text boxes as the same text box. In the case of determining different text boxes as the same text box, the different text boxes are merged, which is equivalent to refining the initial text boxes, and more accurate row text boxes are output.

[0264] 1906, text box clustering.

[0265] 1907, outputting column detection boxes using the column detection model.

[0266] 1908, determining column text boxes.

[0267] 1909, merging and restoring the table by rows and columns.

[0268] Steps 1905 to 1909 can refer to the description of the aforementioned steps 1105 to 1109, which will not be repeated here.

[0269] [Rule 91 correction 06.03.2025] Therefore, in the embodiments of the present application, detection is performed on rows and columns in the table respectively. When detection is performed on rows, a row detection model is added to detect the row text box, and the text content in the same text box is recognized in combination with the text semantic recognized by the NLP model, so as to accurately detect the row text box. When detection is performed on columns, clustering or a column detection model can be used to detect the column text box, so that the column structure in the table can be accurately recognized. Therefore, in the embodiments of the present application, the row text box and the column text box in the table can be accurately detected, so that the table structure can be accurately restored. Especially for dense and complex wireless tables, the method provided by the present application can accurately restore the rows and columns in the table, so that the table can be accurately restored.

[0270] The foregoing introduces the method provided by the present application, and the structure of the device for executing the method provided by the present application is introduced below.

[0271] [Rule 91 correction 06.03.2025] Referring to FIG. 16, a table restoration device provided by the present application includes:

[0272] The scanning module 2001 is configured to perform text box detection on the input data to obtain a scanning result, the input data including an input table, and the scanning result including information of at least one text box in the input table.

[0273] The row processing module 2002 is configured to perform text detection on the at least one text box to determine a row text box.

[0274] The column processing module 2003 is configured to determine a column text box according to the at least one text box.

[0275] The merging module 2004 is configured to merge the row text box and the column text box to obtain an output table.

[0276] In a possible implementation, the row processing module 2002 is specifically configured to: perform text recognition on the at least one text box to obtain text in the at least one text box; and perform natural language processing on the text in the at least one text box to recognize the row text box.

[0277] In a possible implementation, the row processing module 2002 is specifically configured to: input the input data into a row detection model to output a row detection box, the row detection box being used to represent the area of each row in the input table; determine an initial row text box according to the row detection box and the scanning result; and perform natural language processing on the text in the initial row text box to recognize the row text box.

[0278] In a possible implementation, the column processing module 2003 is specifically configured to: cluster the at least one text box to obtain column clustered text boxes; and determine the column text boxes according to the column clustered text boxes.

[0279] In a possible implementation, the column processing module 2003 is specifically configured to: input the input data into a column detection model to output column detection boxes, the column detection boxes being used to represent regions of each column in the input table; and determine the column text boxes according to the column detection boxes and the scanning result.

[0280] In a possible implementation, the column processing module 2003 is specifically configured to: cluster the text boxes in the table region to obtain column clustered text boxes; input the input data into a column detection model to output column detection boxes, the column detection boxes being used to represent regions of each column in the input table; and fuse the column clustered text boxes and the column detection boxes to obtain the column text boxes.

[0281] In a possible implementation, the apparatus further includes a layout analysis module 2005, configured to input the input data into a layout analysis model to output a table region, the table region including a region of the input table in the input data; and in the case that the row detection model exists, the table region is used as input of the row detection model, and in the case that the column detection model exists, the table region is used as input of the column detection model.

[0282] In a possible implementation, the apparatus further includes a correction module 2006, configured to correct an inclination angle of the input data to obtain corrected input data.

[0283] The layout analysis module is specifically configured to input the corrected input data into a layout analysis model to output the table region.

[0284] [Corrected according to Rule 91 on 06.03.2025] In combination with the structure of the table restoration apparatus shown in the foregoing FIG. 16, in an embodiment of the present application, the table restoration apparatus can include implementations of different functions.

[0285] The scanning module 2001 is configured to perform text box detection on input data to obtain a scanning result, the input data including an input table, and the scanning result including information of at least one text box in the input table.

[0286] The row processing module 2002 is configured to determine row text boxes according to the at least one text box.

[0287] The column processing module 2003 is configured to determine column text boxes according to the at least one text box, the column text boxes being obtained by clustering the at least one text box, and / or the column text boxes being obtained according to column detection boxes output by a column detection model, the column detection boxes representing regions of each column in the input table.

[0288] The merging module 2004 is configured to merge the row text boxes and the column text boxes to obtain the output table. The row text boxes include text boxes in the same row in the output table, and the column text boxes include text boxes in the same column in the output table.

[0289] In a possible implementation, in a case where the column text boxes are determined according to the column clustering of the at least one text box and the column detection frame output by the column detection model, the column processing module 2002 is specifically configured to: cluster the text boxes in the table region to obtain column clustered text boxes; input the input table region in the input data into the column detection model to output a column detection frame; and fuse the column clustered text boxes and the column detection frame to obtain the column text boxes.

[0290] In a possible implementation, in a case where the column text boxes are determined according to the column clustering of the at least one text box, the column processing module 2002 is specifically configured to: cluster the at least one text box based on the coordinates of the at least one text box to obtain column clustered text boxes, for example, clustering the coordinates of the text boxes, and regarding text boxes with similar column coordinates in the text boxes as text boxes in the same column; and determining the column text boxes according to the column clustered text boxes, for example, regarding the text boxes in a cluster as the text boxes in a column, that is, the column text boxes. In the embodiments of the present application, the clustering manner can be used to identify the text boxes in the same column, so as to accurately identify the text boxes in the same column based on the column coordinates of the text boxes.

[0291] [Corrected according to Rule 91 on 06.03.2025] In a possible implementation, in a case where the column text boxes are determined according to the column detection frame output by the column detection model, the column processing module 2002 is specifically configured to: input the input data into the pre-trained column detection model as an input, and output a column detection frame, which can be used to represent the region corresponding to each column in the input table; and then determine the column text boxes according to the column detection frame and the at least one text box, for example, fill the text boxes into the corresponding column detection frame based on the coordinates of the text boxes, and output the column text boxes. Therefore, the pre-trained column detection model is used to perform the detection task to output the region of the column in the table region, so that the detection capability of the model can be used to accurately identify the region of the table column, and then more accurate column text boxes are output.

[0292] Each module in the foregoing table restoration apparatus can be implemented by software or by hardware. For example, the implementation of the row processing module is described below. Similarly, the implementation of other modules, such as the scanning module, the column processing module, the merging module, the correction module, or the layout analysis module, can refer to the implementation of the row processing module.

[0293] As an example of a software functional unit, a row processing module can include code running on a compute instance. The compute instance can be at least one of a physical host (computing device), a virtual machine, a container, and the like. Further, the computing device can be one or more. For example, a row processing module can include code running on multiple hosts / virtual machines / containers. Note that the multiple hosts / virtual machines / containers used to run the application can be distributed in the same region, or in different regions. The multiple hosts / virtual machines / containers used to run the code can be distributed in the same AZ, or in different AZs, each of which includes one data center or multiple data centers in close geographical proximity. Typically, a region can include multiple AZs.

[0294] Similarly, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same VPC, or in multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs in the same region, and cross-region communication between VPCs in different regions, requires a communication gateway to be set up in each VPC to achieve interconnection between VPCs.

[0295] As an example of a hardware functional unit, a row processing module can include at least one computing device, such as a server, and the like. Alternatively, a row processing module can also be a device implemented using a central processing unit (CPU), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system on chip (SoC), an offload card, an acceleration card, and the like. The PLD can be a CPLD, an FPGA, a GAL, or any combination thereof.

[0296] The multiple computing devices included in the row processing module can be distributed in the same region or in different regions. The multiple computing devices included in the row processing module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the row processing module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, acceleration cards, and the like.

[0297] As an example of a software functional unit, the module can be code running on a compute instance. The compute instance can be at least one of a physical host (computing device), a virtual machine, a container, and the like. Further, the computing device can be one or more. For example, the table restoration apparatus can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers running the code can be distributed in the same AZ or in different AZs, and each AZ includes one data center or multiple data centers in close geographical proximity. Generally, one region can include multiple AZs.

[0298] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same VPC or in multiple VPCs. Generally, one VPC is set up in one region. Communication between two VPCs in the same region and cross-region communication between VPCs in different regions requires a communication gateway to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0299] As an example of a hardware functional unit, the module can be at least one computing device, such as a server, and the like. Alternatively, the table restoration apparatus can also be a device implemented using an ASIC or a PLD, and the like. The PLD can be a CPLD, an FPGA, a GAL, or any combination thereof.

[0300] The multiple computing devices included in the table restoration apparatus can be distributed in the same region or in different regions. The multiple computing devices included in the table restoration apparatus can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the table restoration apparatus can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0301] The application further provides a chip system, which comprises a processor and a power supply circuit, the power supply circuit being configured to supply power to the processor, and the processor being configured to perform the operation steps corresponding to the method provided by the application. For brevity, details are not repeated here. The processor can be implemented by a GPU, or can be implemented by a DPU, NPU, XPU, SoC, offload card, acceleration card, or other computing device.

[0302] [According to Rule 91 Correction 06.03.2025] The application further provides a computing device 100. As shown in FIG. 17, the computing device 100 comprises a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate through the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the number of processors and memories in the computing device 100 is not limited by the application.

[0303] [According to Rule 91 Correction 06.03.2025] The bus 102 can be a peripheral component interconnect Express (PCIe) bus or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The unified bus is also referred to as a bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 17, but it does not mean that there is only one bus or only one type of bus. The bus 104 can include a path for transmitting information between various components (e.g., the memory 106, the processor 104, and the communication interface 108) of the computing device 100. The unified bus can also be referred to as a bus.

[0304] The processor 104 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, an acceleration card, or other computing device.

[0305] The memory 106 can include volatile memory, such as random access memory (RAM). The processor 104 can also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). In addition, the memory 106 can also be implemented by storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0306] It is worth noting that the same type of storage medium can be configured to implement the function of the memory 106 in the same computing device, or two or more types of storage media can be configured to implement the function of the memory 106, which is not limited in the present application.

[0307] The memory 106 stores executable program code, and the processor 104 executes the executable program code to respectively implement the functions of the various modules mentioned in the foregoing FIG. 7, thereby implementing the method provided by the present application. That is, the memory 106 stores instructions for executing the method provided by the present application.

[0308] Alternatively, the memory 106 stores executable program code, and the processor 104 executes the executable program code to respectively implement the functions of the foregoing scanning module, row processing module, column processing module, merging module, correction module, or layout analysis module, thereby implementing the method provided by the present application. That is, the memory 106 stores instructions for executing the method provided by the present application.

[0309] The communication interface 103 uses a transceiver module such as, but not limited to, a network interface card, a transceiver, to implement communication between the computing device 100 and other devices or communication networks.

[0310] As a possible implementation, the computing device 100 can also include a chip system including a processor and a power supply circuit for performing power supply for the processor, and the processor is used to perform the operation steps corresponding to the method provided in the present application. For the sake of brevity, it will not be repeated here. Among them, the processor can be implemented by GPU, and can also be implemented by DPU, NPU, XPU, SoC, offload card, acceleration card and other computing devices or AI chips.

[0311] As a possible implementation, the computing device 100 can include multiple types of processors 104, that is, the computing device 100 is a heterogeneous device, for example, the computing device 100 includes CPU and GPU, and at least one of the processors 104 in the computing device 100 can perform the operation steps corresponding to the method provided in the present application. For the sake of brevity, it will not be repeated here.

[0312] The embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a desktop computer, a notebook computer, or a terminal device such as a smart phone.

[0313] [According to Rule 91 correction 06.03.2025] As shown in FIG. 18, the computing device cluster includes at least one computing device 100. The memory 106 in one or more computing devices 100 in the computing device cluster can store the same instructions for executing the method provided in the present application.

[0314] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster can also respectively store partial instructions for executing the method provided in the present application. In other words, the combination of one or more computing devices 100 can collectively execute the instructions for executing the method provided in the present application.

[0315] It should be noted that the memories 106 in different computing devices 100 in the computing device cluster can store different instructions, respectively used to perform part of the functions of the joint test device. That is, the instructions stored in the memories 106 in different computing devices 100 can implement the functions of one or more of the aforementioned scanning module, row processing module, column processing module, merging module, correction module or layout analysis module.

[0316] [Rule 91 Correction 06.03.2025] In some possible implementations, one or more of the computing devices in the cluster of computing devices can be connected through a network. Among others, the network can be a wide area network or a local area network, etc. FIG. 19 shows one possible implementation. As shown in FIG. 19, two computing devices 100A and 100B are connected through a network. Specifically, the computing devices are connected to the network through a communication interface in each of the computing devices. In this class of possible implementations, the memory 106 in the computing device 100A stores instructions for performing the functions of the scanning module, the row processing module, the column processing module, the merging module, the correction module, or the layout analysis module. Meanwhile, the memory 106 in the computing device 100B stores instructions for performing the functions of the scanning module, the row processing module, the column processing module, the merging module, the correction module, or the layout analysis module.

[0317] [Rule 91 Correction 06.03.2025] It should be understood that the functions of the computing device 100A shown in FIG. 19 can also be accomplished by multiple computing devices 100. Similarly, the functions of the computing device 100B can also be accomplished by multiple computing devices 100.

[0318] [Rule 91 Correction 06.03.2025] The manner in which the computing devices in the cluster shown in FIG. 19 are connected can be such that a large amount of computing power is needed to implement the methods provided in the present application, load balancing needs to be implemented, or a large amount of data storage is needed, etc. Therefore, different modules are deployed in different computing devices, such as the functions implemented by the row processing module are executed by the computing device 100A, and the functions implemented by the column processing module are executed by the computing device 100B.

[0319] [Rule 91 Correction 06.03.2025] The present application also provides another cluster of computing devices. The connection relationship between the computing devices in the cluster can be similar to the connection manners of the clusters of computing devices described with reference to FIG. 18 and FIG. 19. The difference is that the memory 106 in one or more of the computing devices 100 in the cluster can store the same instructions for implementing the methods provided in the present application.

[0320] In some possible implementations, the memory 106 of one or more of the computing devices 100 in the cluster can also respectively store partial instructions for implementing the methods provided in the present application. In other words, the combination of one or more of the computing devices 100 can collectively execute the instructions for implementing the methods provided in the present application.

[0321] It should be noted that the memories 106 in different computing devices 100 in the computing device cluster can store different instructions for performing part of the functions of the table restoration apparatus provided in the present application. That is, the instructions stored in the memories 106 in different computing devices 100 can implement the functions of one or more of the scanning module, the row processing module, the column processing module, the merging module, the correction module, or the layout analysis module.

[0322] The embodiments of the present application further provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform the method provided in the present application.

[0323] The embodiments of the present application further provide a computer readable storage medium. The computer readable storage medium can be any available medium that the computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium contains instructions, which instruct the computing device to perform the method provided in the present application.

[0324] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A form restoration method characterized by, The method comprises: performing text box detection on input data to obtain a scan result, the input data comprising an input table, the scan result comprising information of at least one text box in the input table; performing text detection on the at least one text box to determine a row text box; determining a column text box according to the at least one text box; merging the row text box and the column text box to obtain an output table.

2. The method of claim 1, wherein, The method further comprises: performing text recognition on the at least one text box to obtain text in the at least one text box; the performing text detection on the at least one text box to determine a row text box comprises: performing natural language processing on the text in the at least one text box to recognize the row text box.

3. The method of claim 2, wherein, The performing natural language processing on the text in the at least one text box to recognize the row text box comprises: inputting the input data into a row detection model to output a row detection box, the row detection box being used to represent a region of each row in the input table; determining an initial row text box according to the row detection box and the scan result; performing natural language processing on text in the initial row text box to recognize the row text box.

4. The method according to any one of claims 1 to 3, characterized in that, The determining a column text box according to the at least one text box comprises: performing clustering on the at least one text box to obtain a column clustered text box; determining the column text box according to the column clustered text box.

5. The method according to any one of claims 1-3, characterized in that, The determining a column text box according to the at least one text box comprises: inputting the input data into a column detection model to output a column detection box, the column detection box being used to represent a region of each column in the input table; determining the column text box according to the column detection box and the scan result.

6. The method according to any one of claims 1-3, characterized in that, The determining a column text box according to the at least one text box comprises: performing clustering on text boxes in the table region to obtain a column clustered text box; inputting the input data into a column detection model to output a column detection box, the column detection box being used to represent a region of each column in the input table; fusing the column clustered text box and the column detection box to obtain the column text box.

7. The method of any one of claims 3, 5, or 6, wherein, The method further comprises: performing layout analysis on the input data to output a table region, the table region comprising a region of the input table in the input data; wherein, in the presence of a row detection model, the table region is used as input of the row detection model, and in the presence of a column detection model, the table region is used as input of the column detection model.

8. The method of claim 7, wherein, The inputting the input data into a layout analysis model to output a table region comprises: correcting an inclination angle of the input data to obtain corrected input data; inputting the corrected input data into the layout analysis model to output the table region.

9. A form restoration method characterized by, The method comprises: performing text box detection on input data to obtain a scan result, the input data comprising an input table, the scan result comprising information of at least one text box in the input table; determining a row text box according to the at least one text box; determine column text boxes according to the at least one text box, the column text boxes being obtained by clustering the at least one text box, and / or the column text boxes being obtained according to column detection boxes output by a column detection model, the column detection boxes representing regions of each column in the input table; merge the row text boxes and the column text boxes to obtain an output table, the row text boxes including text boxes in the same row in the output table, and the column text boxes including text boxes in the same column in the output table.

10. The method of claim 9, wherein, In a case where the column text boxes are determined in combination of clustering the at least one text box and the column detection boxes output by the column detection model, the determining the column text boxes according to the at least one text box includes: clustering the text boxes in the table region to obtain column clustered text boxes; inputting the input table region in the input data into a column detection model to output the column detection boxes; fusing the column clustered text boxes and the column detection boxes to obtain the column text boxes.

11. A table reconstruction apparatus characterized by comprising: includes: a scanning module, configured to perform text box detection on input data to obtain a scanning result, the input data including an input table, and the scanning result including information of at least one text box in the input table; a row processing module, configured to determine row text boxes by performing text detection on the at least one text box; a column processing module, configured to determine column text boxes according to the at least one text box; a merging module, configured to merge the row text boxes and the column text boxes to obtain an output table.

12. The apparatus of claim 11, wherein, The row processing module is specifically configured to: perform text recognition on the at least one text box to obtain text in the at least one text box; perform natural language processing on the text in the at least one text box to identify the row text boxes.

13. The apparatus of claim 12, wherein, The row processing module is specifically configured to: input the input data into a row detection model to output row detection boxes, the row detection boxes being used to represent regions of each row in the input table; determine initial row text boxes according to the row detection boxes and the scanning result; perform natural language processing on text in the initial row text boxes to identify the row text boxes.

14. The apparatus of any one of claims 11-13, wherein, The column processing module is specifically configured to: cluster the at least one text box to obtain column clustered text boxes; determine the column text boxes according to the column clustered text boxes.

15. The apparatus of any one of claims 11-13, wherein, The column processing module is specifically configured to: input the input data into a column detection model to output column detection boxes, the column detection boxes being used to represent regions of each column in the input table; determine the column text boxes according to the column detection boxes and the scanning result.

16. The apparatus of any one of claims 11-13, wherein, The column processing module is specifically configured to: cluster the text boxes in the table region to obtain column clustered text boxes; input the input data into a column detection model to output column detection boxes, the column detection boxes being used to represent regions of each column in the input table; fuse the column clustered text boxes and the column detection boxes to obtain the column text boxes.

17. The apparatus of any one of claims 13, 15, or 16, wherein, The apparatus further includes: a layout analysis module, configured to input the input data into a layout analysis model to output a table region, the table region including a region of the input table in the input data; In the presence of a row detection model, the table region is used as input to the row detection model, and in the presence of a column detection model, the table region is used as input to the column detection model.

18. The apparatus of claim 17, wherein, The device further comprises: a correction module configured to correct a tilt angle of the input data to obtain corrected input data; the layout analysis module is specifically configured to input the corrected input data into the layout analysis model and output the table region.

19. A table reconstruction apparatus characterized by comprising: Comprise: a scanning module configured to perform text box detection on input data to obtain a scanning result, the input data comprising an input table, and the scanning result comprising information of at least one text box in the input table; a row processing module configured to determine a row text box according to the at least one text box; a column processing module configured to determine a column text box according to the at least one text box, the column text box being obtained by clustering the at least one text box, and / or the column text box being obtained according to a column detection model outputting a column detection box representing a region of each column in the input table; a merging module configured to merge the row text box and the column text box to obtain an output table, the row text box comprising text boxes in the same row in the output table, and the column text box comprising text boxes in the same column in the output table.

20. The apparatus of claim 19, wherein, In the case of combining clustering of the at least one text box and determination of the column detection box according to the column detection model, the column text box determined according to the at least one text box comprises: clustering text boxes in the table region to obtain column clustered text boxes; inputting the input table region in the input data into a column detection model to output the column detection box; fusing the column clustered text boxes and the column detection box to obtain the column text box.

21. A computing device, comprising: The computing device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the operation steps of the method of any one of claims 1 to 10.

22. A cluster of computing devices, characterized in that, Comprise at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the operation steps of the method of any one of claims 1 to 10.

23. A computer program product comprising instructions, characterized in that, When the instructions are executed by the computing device cluster, the computing device cluster performs the operation steps of the method of any one of claims 1 to 10.

24. A computer-readable storage medium, characterized in that, Comprise computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the operation steps of the method of any one of claims 1 to 10.

Images