Table parsing method, device, computer equipment and storage medium

Abstract

The present application is in the field of natural language processing technology of artificial intelligence technology, and provides a table parsing method, device, computer equipment and storage medium, wherein the method includes: obtaining a picture containing a table, determining the position of the table in the picture, and obtaining the table position of the table; determining the position of text information in the table, and obtaining the text information position of the text information; according to the table position and the text information position, calling a pre-trained sequence annotation model to predict whether the text information of all two adjacent cells in the table does not belong to the same sentence or the same paragraph; when it is determined that the text information of all two adjacent cells in the table does not belong to the same sentence or the same paragraph, taking the table as the target table, parsing the text information of each cell in the target table, and obtaining structured data, thereby using the sequence annotation model to comprehensively parse the table and generate structured data, thereby realizing the parsing of the table in the picture format.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing of artificial intelligence technology. Specifically, the present application relates to a table parsing method, apparatus, computer equipment and storage medium. Background Art

[0002] In daily work such as the Internet and finance, there are various tables, which are often unstructured data in the form of images. Such unstructured data cannot be extracted and stored well, so it is necessary to use table parsing methods to convert unstructured data in tables into structured data. Existing table parsing methods cannot parse tables in images, have limitations, and have poor parsing effects. Summary of the invention

[0003] The main purpose of this application is to provide a table parsing method, device, computer equipment and storage medium to parse tables in pictures and improve the parsing effect.

[0004] In order to achieve the above-mentioned object of the invention, the present application provides a table parsing method, which comprises:

[0005] Get the image containing the table;

[0006] Determine the position of the table in the picture to obtain the table position of the table;

[0007] Determine the position of the text information in the table to obtain the text information position of the text information;

[0008] According to the table position and the text information position, calling a pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph;

[0009] When it is determined that the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph, the table is used as a target table;

[0010] Parse the text information of each cell in the target table to obtain structured data.

[0011] Furthermore, after calling the pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph, the method further includes:

[0012] When it is determined that the text information of two adjacent cells in the table belongs to the same sentence or the same paragraph, the two adjacent cells whose text information belongs to the same sentence or the same paragraph are merged to obtain the target table.

[0013] Preferably, the parsing of the text information of each cell in the target table to obtain structured data includes:

[0014] Generate a row relationship matrix and a column relationship matrix of the target table;

[0015] Constructing a row undirected graph according to the row relationship matrix, and constructing a column undirected graph according to the column relationship matrix;

[0016] Extracting cells of each row and each column in the target table according to the row undirected graph and the column undirected graph;

[0017] The text information in the cells of each row and each column is parsed in turn to obtain the structured data.

[0018] Preferably, the step of obtaining a picture containing a table includes:

[0019] Obtaining an input multimedia file; wherein the multimedia file contains a table;

[0020] When it is determined that the multimedia file is in PDF format, the multimedia file is converted into a unified picture format to obtain a picture containing a table.

[0021] Furthermore, the calling of the pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph also includes:

[0022] Acquire a training sample set; wherein the training sample set includes a sample image, a reference table position of a reference table in the sample image, a reference text information position of text information in the reference table, and annotation information indicating whether text information of every two adjacent cells in the reference table does not belong to the same sentence or the same paragraph;

[0023] Inputting the training sample set into a preset neural network model for training to obtain training results;

[0024] Determining whether the training result meets the requirements;

[0025] When it is determined that the training result meets the requirements, the trained neural network model is used as a sequence labeling model.

[0026] Preferably, the determining whether the training result meets the requirement includes:

[0027] Calculating the loss value of the trained neural network model according to the training results and a preset loss function;

[0028] Determining whether the loss value is lower than a preset loss value;

[0029] When it is determined that the loss value is lower than the preset loss value, it is determined that the training result meets the requirements.

[0030] Preferably, calling a pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph includes:

[0031] Calling a pre-trained sequence labeling model to obtain text information of every two adjacent cells in the table;

[0032] Calculate the correlation between the text information of every two adjacent cells;

[0033] When the correlation degree between the text information of every two adjacent cells is less than the preset correlation degree, it is determined that the text information of all the two adjacent cells in the table do not belong to the same sentence or the same paragraph.

[0034] The present application also provides a table parsing device, which includes:

[0035] The acquisition module is used to obtain the image containing the table;

[0036] A first determining module, used to determine the position of the table in the picture, and obtain the table position of the table;

[0037] A second determination module is used to determine the position of the text information in the table to obtain the text information position of the text information;

[0038] A prediction module, used to call a pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph according to the table position and the text information position;

[0039] A determination module, configured to use the table as a target table when it is determined that the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph;

[0040] The parsing module is used to parse the text information of each cell in the target table to obtain structured data.

[0041] The present application also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0042] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0043] The present application provides a table parsing method, device, computer equipment and storage medium, which obtain a picture containing a table, determine the position of the table in the picture, and obtain the table position of the table; determine the position of text information in the table, obtain the text information position of the text information, and call a pre-trained sequence annotation model to predict whether the text information of all two adjacent cells in the table does not belong to the same sentence or the same paragraph according to the table position and the text information position; when it is determined that the text information of all two adjacent cells in the table does not belong to the same sentence or the same paragraph, the table is used as a target table, and the text information of each cell in the target table is parsed to obtain structured data, thereby using the sequence annotation model to comprehensively parse the table and generate structured data, thereby realizing the parsing of the table in the picture format; at the same time, before the structured data is generated, by judging whether the text information of all two adjacent cells in the table belongs to the same paragraph or the same sentence, parsing errors and semantic ambiguity are avoided, thereby realizing accurate parsing of the table. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a table parsing method according to an embodiment of the present application;

[0045] Figure 2 This is a schematic block diagram of the structure of a table parsing device according to an embodiment of the present application;

[0046] Figure 3 A schematic block diagram of the structure of a computer device according to an embodiment of the present application.

[0047] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] This application proposes a table parsing method, and the embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0050] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0051] A table parsing method proposed in the present application takes a server as the execution entity. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0052] This table parsing method is used to solve the technical problem that the current table parsing method cannot parse the table in the picture, has limitations, and has poor parsing effect. Figure 1 In one embodiment, the table parsing method includes:

[0053] S11, obtaining a picture containing a table;

[0054] S12, determining the position of the table in the picture, and obtaining the table position of the table;

[0055] S13, determining the position of the text information in the table, and obtaining the text information position of the text information;

[0056] S14, according to the table position and the text information position, calling a pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph;

[0057] S15, when it is determined that the text information of all two adjacent cells in the table does not belong to the same sentence or the same paragraph, taking the table as the target table;

[0058] S16. Parse the text information of each cell in the target table to obtain structured data.

[0059] As described in the above step S11, when the terminal where the user is located initiates a parsing request to the server to parse an image containing a table, the parsing request contains the image to be parsed or the image information of the image, and the image information is used to query and obtain the image from the database. The server responds to the parsing request, extracts the image containing the table from the parsing request, or queries and obtains the image from the database according to the image information in the parsing request.

[0060] As described in the above step S12, this embodiment determines the position of the table in the picture and obtains the table position of the table. Specifically, the position of the table in the picture can be determined by using a pre-trained table detection model. The table detection model is a neural network model trained with a large amount of training data, and its function is to detect the position of the table in the picture.

[0061] In one embodiment, the lower left corner position of the image may be set as the starting coordinate, the table position of the table may be determined based on the positional relationship between the table and the starting coordinate, and the table position may be expressed in the form of coordinates.

[0062] In one embodiment, the training method of the table detection model can be: obtaining training data, the training data including a standard picture and an annotation box that annotates the table in the standard picture; inputting the training data into the table detection model to be trained, predicting the table in the standard picture through the segmentation network of the table detection model to be trained, outputting a table prediction segmentation map, predicting a table prediction box whose area ratio to the standard picture is greater than a threshold through the detection network of the table detection model to be trained, and outputting position information of the table prediction box; updating the network parameters of the table detection model to be trained according to the difference between the table prediction segmentation map and the annotation box, and the difference between the position information of the table prediction box and the target annotation box, until the loss function value of the table detection model to be trained is not greater than a preset value, so as to obtain a table detection model, which can improve the detection accuracy of the table.

[0063] As described in the above step S13, the present embodiment can also pre-train the text detection model. In terms of training techniques, corrosion and expansion image processing technology and data enhancement processing technology are used to increase the diversity of images. The images after data enhancement processing are subjected to corrosion and expansion processing to obtain images with more obvious text features. The processed images are detected with a text detection model to obtain the position of the text information in the table, and the text information position of the text information is obtained.

[0064] The text detection model of this embodiment uses the image processing technology of corrosion and expansion in the prediction and training process. Compared with the original image for prediction, the more blurred text information will make the features more obvious, improving the accuracy and recall rate of the text detection model. In addition, compared with the model that directly predicts the merged cells, the use of the text detection model to predict text information does not need to consider too complex spatial layout, which greatly reduces the difficulty of the task and makes the model easier to converge, so the robustness and performance of the model will be better.

[0065] As described in step S14 above, this embodiment pre-trains a sequence labeling model to predict whether the text information of all two adjacent cells in the table does not belong to the same sentence or the same paragraph, that is, to determine whether two sentences belong to the same paragraph or the same sentence. The model structure of the sequence labeling model can be albert+lstm+crf, and other pre-trained models can be used to replace albert, such as MASS, UniLM, ERNIE, SpanBERT, RoBERTa. Preferably, the sequence labeling model is albert.

[0066] When the sequence labeling model is used, two sentences are input into the model at the same time. If the prediction result is BE, they are from the same sentence, that is, the two cells should be merged into the same cell.

[0067] As described in the above steps S15-S16, when it is determined that the text information in all two adjacent cells in the image does not belong to the same sentence or the same paragraph, a merged target table is obtained, and the cells in the target table are calculated pairwise to determine whether the row and column coordinates of the center of a cell are within the row and column coordinate range of another cell. A text recognition model is used to identify the text information in each cell to obtain complete structured data, and the structured data is stored in the database.

[0068] It should be noted that in this embodiment, structured data is obtained after parsing the table of unstructured data. The unstructured data refers to data whose field length is variable, and each field record can be composed of repeatable or non-repeatable sub-fields, such as full text, images, sounds, movies, hypermedia and other information. Unstructured data includes office documents, texts, pictures, XML, HTML, various reports, images and audio / video information in all formats. Structured data is row data, which is stored in the database and can be logically expressed in a two-dimensional table structure.

[0069] The present application provides a table parsing method, which obtains a picture containing a table, determines the position of the table in the picture, and obtains the table position of the table; determines the position of text information in the table, obtains the text information position of the text information, and calls a pre-trained sequence annotation model to predict whether the text information of all two adjacent cells in the table does not belong to the same sentence or the same paragraph according to the table position and the text information position; when it is determined that the text information of all two adjacent cells in the table does not belong to the same sentence or the same paragraph, the table is used as a target table, and the text information of each cell in the target table is parsed to obtain structured data, thereby using the sequence annotation model to comprehensively parse the table and generate structured data, thereby realizing the parsing of the table in the picture format; at the same time, before the structured data is generated, by judging whether the text information of all two adjacent cells in the table belongs to the same paragraph or the same sentence, parsing errors and semantic ambiguity are avoided, thereby realizing accurate parsing of the table.

[0070] In one embodiment, after calling a pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph, the method may further include:

[0071] When it is determined that the text information of two adjacent cells in the table belongs to the same sentence or the same paragraph, the two adjacent cells whose text information belongs to the same sentence or the same paragraph are merged to obtain the target table.

[0072] This embodiment pre-trains a sequence labeling model to determine whether two text information in any two adjacent cells are from the same paragraph or sentence, that is, to determine whether two sentences belong to the same paragraph or the same sentence. When the sequence labeling model is used, two sentences are input into the model at the same time. If the prediction result is BE, it is from the same sentence, that is, the two cells should be merged into the same cell, and finally the merged target table is obtained to achieve accurate parsing of the table.

[0073] When it is determined that the text information of two adjacent cells in the table belongs to the same sentence or the same paragraph, the two adjacent cells whose text information belongs to the same sentence or the same paragraph are merged, and the merged table is used as the target table. The cells in the target table are calculated pairwise to determine whether the row and column coordinates of the center of a cell are within the row and column coordinate range of another cell. The text recognition model is used to identify the text information in each cell to obtain complete structured data, and the structured data is stored in the database to avoid separately parsing the text information belonging to the same sentence or the same paragraph, which leads to semantic ambiguity in the parsed text information.

[0074] In one embodiment, parsing the text information of each cell in the target table to obtain structured data may specifically include:

[0075] Generate a row relationship matrix and a column relationship matrix of the target table;

[0076] Constructing a row undirected graph according to the row relationship matrix, and constructing a column undirected graph according to the column relationship matrix;

[0077] Extracting cells of each row and each column in the target table according to the row undirected graph and the column undirected graph;

[0078] The text information in the cells of each row and each column is parsed in turn to obtain the structured data.

[0079] This embodiment generates a row relationship matrix and a column relationship matrix of the target table, constructs a row undirected graph according to the row relationship matrix, constructs a column undirected graph according to the column relationship matrix, extracts cells of each row and column in the target table according to the row undirected graph and the column undirected graph, and finally identifies the text information in the cells of each row and column to obtain the structured data. Therefore, by using the connectivity graph post-processing method, compared with using a model to predict the structure of the table, the present application does not need to additionally mark the spatial structure of the table as training data, and is more efficient and convenient to use.

[0080] In one embodiment, the row relationship matrix represents the association result of each text information in the same row, and the column relationship matrix represents the association result of each text information in the same column. This embodiment can construct an undirected graph based on the association result of the text information. Specifically, each word in the association result of the text information can be used as a node, and the association relationship in the association result of the text information can be used as an edge to construct an undirected graph, so as to obtain an undirected graph. For example, the association result of the text information has a total of 6 words "A", "B", "C", "D", "E", and "F". The word "A" has an association relationship with "C", the word "B" has an association relationship with "D", and the word "E" has an association relationship with "F". Then, the 6 words are used as nodes and the three association relationships are used as edges to construct an undirected graph.

[0081] In one embodiment, obtaining the picture containing the table may specifically include:

[0082] Obtaining an input multimedia file; wherein the multimedia file contains a table;

[0083] When it is determined that the multimedia file is in PDF format, the multimedia file is converted into a unified picture format to obtain a picture containing a table.

[0084] This embodiment can uniformly convert the input multimedia files into image formats based on the user's format conversion instructions. The multimedia file contains a table. For example, when the input multimedia file is in PDF format, pymupdf can be used to convert the multimedia file in PDF format into a picture. There are many ways for the user to give format conversion instructions. For example, the user can give a format conversion instruction by clicking a preset button in the table page; the user can also give a format conversion instruction by clicking a preset shortcut key on the keyboard; the user can also give a format conversion instruction by clicking a preset button in the right-click menu of the mouse. Here, there is no limitation on the way the user gives the format conversion instruction. Accordingly, there is no limitation on the way to receive the user's format conversion instruction.

[0085] In addition, the present embodiment can also call the API (Application Programming Interface) in the computer device for converting canvas data into pictures, so as to obtain pictures corresponding to the table. Since there can be multiple tables, there may also be multiple pictures generated. When the number of pictures is large, if the user transmits the pictures one by one, the operation is cumbersome and prone to problems such as missing transmission and multiple transmission. To avoid the above problems, after obtaining the pictures, a file of a specified format containing all the pictures can be generated based on the obtained pictures, and the file can be directly transmitted during transmission.

[0086] In one embodiment, calling a pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph may also include:

[0087] Acquire a training sample set; wherein the training sample set includes a sample image, a reference table position of a reference table in the sample image, a reference text information position of text information in the reference table, and annotation information indicating whether text information of every two adjacent cells in the reference table does not belong to the same sentence or the same paragraph;

[0088] Inputting the training sample set into a preset neural network model for training to obtain training results;

[0089] Determining whether the training result meets the requirements;

[0090] When it is determined that the training result meets the requirements, the trained neural network model is used as a sequence labeling model.

[0091] In this embodiment, the obtained sample images and the reference table position of the reference table in the sample images, the reference text information position of the text information in the reference table, and the annotation information indicating whether the text information of each two adjacent cells in the reference table does not belong to the same sentence or the same paragraph are used as the training sample set. Among them, the form of the sample images is not specifically limited, but each sample image needs to contain a reference table, and the reference table position of the reference table and the reference text information position of the text information in the reference table are determined, and these positions are expressed in the form of coordinates.

[0092] In one embodiment, the annotation information can be expressed in the form of letters, Chinese characters or numbers. For example, when the text information of two adjacent cells in the reference table belongs to the same sentence or the same paragraph, "1" can be marked in the middle of the two adjacent cells to indicate that the text information of the two adjacent cells belongs to the same sentence or the same paragraph; when the text information of two adjacent cells in the reference table does not belong to the same sentence or the same paragraph, "0" can be marked in the middle of the two adjacent cells to indicate that the text information of the two adjacent cells does not belong to the same sentence or the same paragraph.

[0093] Among them, the data volume of the training sample set needs to be greater than the preset threshold to ensure the training effect of the sequence labeling model. When the data volume of the training sample set is lower than the preset threshold, the data enhancement processing method can be used to perform data enhancement processing on the sample images in the training sample set. Specifically, it can be a data enhancement processing method such as flipping, rotating or color adjustment of the sample image. In addition, it can also be a combination of multiple data enhancement processing methods. For example, the sample image can be flipped along the horizontal line first, and then the flipped sample image can be rotated 15 degrees clockwise to obtain the sample image after data enhancement processing, and add it to the training sample set so that the data volume of the training sample set is greater than the preset threshold. Among them, the preset threshold can be determined based on the historical training effect, such as being set to 10,000 images.

[0094] After obtaining the training sample set, the training sample set is input into the preset neural network model for iterative training. When the iteration conditions are met, the training of the neural network model is completed. For example, the number of training times of the neural network model is set. When the neural network model is trained to the number of training times, it indicates that the neural network model has completed the training. The training results of the trained neural network model are obtained to determine whether the training results meet the requirements. When it is determined that the training results meet the requirements, the trained neural network model is used as a sequence labeling model. For example, when the prediction results of 500 sample images out of 1,000 sample images predicted by the trained neural network model match the labeling information, the prediction accuracy is 50%. This accuracy is lower than the preset accuracy, and it is determined that the training results do not meet the requirements. It is necessary to adjust the parameters of the neural network model and then train it again until the training results meet the requirements.

[0095] In one embodiment, the determining whether the training result meets the requirement may specifically include:

[0096] Calculating the loss value of the trained neural network model according to the training results and a preset loss function;

[0097] Determining whether the loss value is lower than a preset loss value;

[0098] When it is determined that the loss value is lower than the preset loss value, it is determined that the training result meets the requirements.

[0099] In this embodiment, after training the neural network model, a preset loss function can be used to calculate the loss value of the neural network model after training. When the loss value meets the preset threshold or is less than the preset loss value, that is, the requirement is met, it indicates that the neural network model meets the training requirements and the training of the neural network model is completed. Among them, the loss function is used to evaluate the degree of difference between the predicted value and the true value of the neural network model. The better the loss function, the better the performance of the neural network model.

[0100] When the loss value of the neural network model is not less than the preset loss value, forward propagation can be performed in the neural network structure of the neural network model according to the loss value, relevant parameters of the neural network model can be adjusted, and the adjusted neural network model can be retrained based on the reset relevant parameters until the loss value of the neural network model is less than the preset loss value, thereby obtaining a trained sequence labeling model.

[0101] In one embodiment, calling a pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph may specifically include:

[0102] Calling a pre-trained sequence labeling model to obtain text information of every two adjacent cells in the table;

[0103] Calculate the correlation between the text information of every two adjacent cells;

[0104] When the correlation degree between the text information of every two adjacent cells is less than the preset correlation degree, it is determined that the text information of all the two adjacent cells in the table do not belong to the same sentence or the same paragraph.

[0105] This embodiment can call a pre-trained sequence labeling model to obtain the text information of every two adjacent cells in the table, and then calculate the correlation between the text information of every two adjacent cells. When calculating the correlation, the text information of every two adjacent cells can be converted into vector form respectively, and the similarity of the vectors of the two text information is calculated. When the similarity is greater than the preset similarity, it indicates that the correlation between the two text information is greater than the preset correlation. The cells corresponding to the two text information with a correlation greater than the preset correlation are merged to obtain the target table.

[0106] When the correlation between the text information of every two adjacent cells is less than the preset correlation, it is determined that the text information of all two adjacent cells in the table does not belong to the same sentence or the same paragraph. The two text information can be accurately distinguished through correlation calculation, and the text information that does not belong to the same sentence or the same paragraph can be parsed separately, avoiding semantic ambiguity in the parsed text information and improving the accuracy of table parsing.

[0107] Reference Figure 2 In an embodiment of the present application, a table parsing device is also provided, comprising:

[0108] An acquisition module 11 is used to acquire a picture containing a table;

[0109] A first determining module 12, used to determine the position of the table in the picture, and obtain the table position of the table;

[0110] A second determination module 13, used to determine the position of the text information in the table, and obtain the text information position of the text information;

[0111] A prediction module 14 is used to call a pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph according to the table position and the text information position;

[0112] A determination module 15, configured to use the table as a target table when it is determined that the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph;

[0113] The parsing module 16 is used to parse the text information of each cell in the target table to obtain structured data.

[0114] When the user's terminal initiates a parsing request to the server to parse an image containing a table, the parsing request contains the image to be parsed or image information of the image, and the image information is used to query and obtain the image from a database. The server responds to the parsing request, extracts the image containing the table from the parsing request, or queries and obtains the image from the database based on the image information in the parsing request.

[0115] This embodiment determines the position of the table in the picture and obtains the table position of the table. Specifically, the position of the table in the picture can be determined by using a pre-trained table detection model. The table detection model is a neural network model trained with a large amount of training data, and its function is to detect the position of the table in the picture.

[0116] In one embodiment, the lower left corner position of the image may be set as the starting coordinate, the table position of the table may be determined based on the positional relationship between the table and the starting coordinate, and the table position may be expressed in the form of coordinates.

[0117] In one embodiment, the training method of the table detection model can be: obtaining training data, the training data including a standard picture and an annotation box that annotates the table in the standard picture; inputting the training data into the table detection model to be trained, predicting the table in the standard picture through the segmentation network of the table detection model to be trained, outputting a table prediction segmentation map, predicting a table prediction box whose area ratio to the standard picture is greater than a threshold through the detection network of the table detection model to be trained, and outputting position information of the table prediction box; updating the network parameters of the table detection model to be trained according to the difference between the table prediction segmentation map and the annotation box, and the difference between the position information of the table prediction box and the target annotation box, until the loss function value of the table detection model to be trained is not greater than a preset value, so as to obtain a table detection model, which can improve the detection accuracy of the table.

[0118] This embodiment can also pre-train the text detection model. In terms of training techniques, corrosion and expansion image processing technology and data enhancement processing technology are used to increase the diversity of images. The images after data enhancement processing are subjected to corrosion and expansion processing to obtain images with more obvious text features. The processed images are detected using a text detection model to obtain the position of the text information in the table and the text information position of the text information.

[0119] The text detection model of this embodiment uses the image processing technology of corrosion and expansion in the prediction and training process. Compared with the original image for prediction, the more blurred text information will make the features more obvious, improving the accuracy and recall rate of the text detection model. In addition, compared with the model that directly predicts the merged cells, the use of the text detection model to predict text information does not need to consider too complex spatial layout, which greatly reduces the difficulty of the task and makes the model easier to converge, so the robustness and performance of the model will be better.

[0120] This embodiment pre-trains a sequence labeling model to predict whether the text information of all two adjacent cells in a table does not belong to the same sentence or the same paragraph, that is, to determine whether two sentences belong to the same paragraph or the same sentence. The model structure of the sequence labeling model can be albert+lstm+crf, and other pre-trained models can be used to replace albert, such as MASS, UniLM, ERNIE, SpanBERT, RoBERTa. Preferably, the sequence labeling model is albert.

[0121] When the sequence labeling model is used, two sentences are input into the model at the same time. If the prediction result is BE, they are from the same sentence, that is, the two cells should be merged into the same cell.

[0122] When it is determined that the text information in all two adjacent cells in the image does not belong to the same sentence or the same paragraph, the merged target table is obtained, and the cells in the target table are calculated pairwise to determine whether the row and column coordinates of the center of a cell are within the row and column coordinate range of another cell. The text recognition model is used to identify the text information in each cell, obtain complete structured data, and store the structured data in the database.

[0123] It should be noted that in this embodiment, structured data is obtained after parsing the table of unstructured data. The unstructured data refers to data whose field length is variable, and each field record can be composed of repeatable or non-repeatable sub-fields, such as full text, images, sounds, movies, hypermedia and other information. Unstructured data includes office documents, texts, pictures, XML, HTML, various reports, images and audio / video information in all formats. Structured data is row data, which is stored in the database and can be logically expressed in a two-dimensional table structure.

[0124] As described above, it can be understood that the various components of the table parsing device proposed in the present application can realize the functions of any one of the table parsing methods described above, and the specific structure will not be repeated.

[0125] Reference Figure 3In the embodiment of the present application, a computer device is also provided, whose internal structure can be as follows Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a storage medium and an internal memory. The storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the storage medium. The database of the computer device is used to store relevant data of the table parsing method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a table parsing method is implemented.

[0126] The processor executes the table parsing method, including:

[0127] Get the image containing the table;

[0128] Determine the position of the table in the picture to obtain the table position of the table;

[0129] Determine the position of the text information in the table to obtain the text information position of the text information;

[0130] According to the table position and the text information position, calling a pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph;

[0131] When it is determined that the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph, the table is used as a target table;

[0132] Parse the text information of each cell in the target table to obtain structured data.

[0133] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a table parsing method is implemented, including the steps of:

[0134] Get the image containing the table;

[0135] Determine the position of the table in the picture to obtain the table position of the table;

[0136] Determine the position of the text information in the table to obtain the text information position of the text information;

[0137] According to the table position and the text information position, calling a pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph;

[0138] When it is determined that the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph, the table is used as a target table;

[0139] Parse the text information of each cell in the target table to obtain structured data.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in the present application and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0141] In summary, the greatest beneficial effects of this application are:

[0142] The present application provides a table parsing method, device, computer equipment and storage medium, which obtain a picture containing a table, determine the position of the table in the picture, and obtain the table position of the table; determine the position of text information in the table, obtain the text information position of the text information, and call a pre-trained sequence annotation model to predict whether the text information of all two adjacent cells in the table does not belong to the same sentence or the same paragraph according to the table position and the text information position; when it is determined that the text information of all two adjacent cells in the table does not belong to the same sentence or the same paragraph, the table is used as a target table, and the text information of each cell in the target table is parsed to obtain structured data, thereby using the sequence annotation model to comprehensively parse the table and generate structured data, thereby realizing the parsing of the table in the picture format; at the same time, before the structured data is generated, by judging whether the text information of all two adjacent cells in the table belongs to the same paragraph or the same sentence, parsing errors and semantic ambiguity are avoided, thereby realizing accurate parsing of the table.

[0143] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, device, article or method including the element.

[0144] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A table parsing method, characterized in that: include: Get the image containing the table; Determine the position of the table in the picture to obtain the table position of the table; Determine the position of the text information in the table to obtain the text information position of the text information; According to the table position and the text information position, calling a pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph; When it is determined that the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph, the table is used as a target table; Parsing the text information of each cell in the target table to obtain structured data; The determining the position of the table in the picture to obtain the table position of the table includes: The position of the table in the image is determined by a pre-trained table detection model, and the training method of the table detection model is: Acquire training data, wherein the training data includes a standard picture and a labeling box that labels a table in the standard picture; Input the training data into the table detection model to be trained, predict the table in the standard picture through the segmentation network of the table detection model to be trained, output the table prediction segmentation map, predict the table prediction box whose area ratio to the standard picture is greater than a threshold through the detection network of the table detection model to be trained, and output the position information of the table prediction box; According to the difference between the table prediction segmentation map and the annotation box, and the difference between the position information of the table prediction box and the target annotation box, the network parameters of the table detection model to be trained are updated until the loss function value of the table detection model to be trained is not greater than a preset value, so as to obtain the table detection model; The calling of the pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph also includes: Acquire a training sample set; wherein the training sample set includes a sample image, a reference table position of a reference table in the sample image, a reference text information position of text information in the reference table, and annotation information indicating whether text information of every two adjacent cells in the reference table does not belong to the same sentence or the same paragraph. When the data volume of the training sample set is lower than a preset threshold, a data enhancement processing method is used to perform data enhancement processing on the sample image in the training sample set, specifically, data enhancement processing methods such as flipping, rotating or color adjustment of the sample image are performed to make the data volume of the training sample set greater than the preset threshold. Alternatively, by combining a plurality of data enhancement processing methods, the sample image is first flipped along a horizontal line, and then the flipped sample image is rotated 15 degrees clockwise to obtain a sample image after data enhancement processing, and the sample image is added to the training sample set to make the data volume of the training sample set greater than the preset threshold. Inputting the training sample set into a preset neural network model for training to obtain training results; Determining whether the training result meets the requirements; When it is determined that the training result meets the requirements, the trained neural network model is used as a sequence labeling model.

2. The method according to claim 1, characterized in that After calling the pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph, the method further includes: When it is determined that the text information of two adjacent cells in the table belongs to the same sentence or the same paragraph, the two adjacent cells whose text information belongs to the same sentence or the same paragraph are merged to obtain the target table.

3. The method according to claim 1, characterized in that The parsing of the text information of each cell in the target table to obtain structured data includes: Generate a row relationship matrix and a column relationship matrix of the target table; Constructing a row undirected graph according to the row relationship matrix, and constructing a column undirected graph according to the column relationship matrix; Extracting cells of each row and each column in the target table according to the row undirected graph and the column undirected graph; The text information in the cells of each row and each column is parsed in turn to obtain the structured data.

4. The method according to claim 1, characterized in that: The step of obtaining a picture containing a table includes: Obtaining an input multimedia file; wherein the multimedia file contains a table; When it is determined that the multimedia file is in PDF format, the multimedia file is converted into a unified picture format to obtain a picture containing a table.

5. The method according to claim 1, characterized in that The determining whether the training result meets the requirement includes: Calculating the loss value of the trained neural network model according to the training results and a preset loss function; Determining whether the loss value is lower than a preset loss value; When it is determined that the loss value is lower than the preset loss value, it is determined that the training result meets the requirements.

6. The method according to claim 1, characterized in that The calling of the pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph includes: Calling a pre-trained sequence labeling model to obtain text information of every two adjacent cells in the table; Calculate the correlation between the text information of every two adjacent cells; When the correlation between the text information of every two adjacent cells is less than the preset correlation, it is determined that the text information of all the two adjacent cells in the table do not belong to the same sentence or the same paragraph.

7. A table parsing device, used to implement the method according to any one of claims 1 to 6, characterized in that: include: The acquisition module is used to obtain the image containing the table; A first determining module, used to determine the position of the table in the picture, and obtain the table position of the table; A second determination module is used to determine the position of the text information in the table to obtain the text information position of the text information; A prediction module, used to call a pre-trained sequence labeling model to predict whether the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph according to the table position and the text information position; A determination module, configured to use the table as a target table when it is determined that the text information of all two adjacent cells in the table do not belong to the same sentence or the same paragraph; The parsing module is used to parse the text information of each cell in the target table to obtain structured data.

8. A computer device, characterized in that: include: processor; Memory; The memory stores a computer program, and the processor implements the table parsing method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the table parsing method according to any one of claims 1 to 6.

Images