Table semantic information extraction method, system and device based on cell coordinate optimization and medium

Abstract

The application discloses a table semantic information extraction method based on cell coordinate optimization, which comprises the following steps: performing OCR identification on a data set containing multiple types of forms to obtain a table information file containing characters, cells and labels; meanwhile, the same data set is input into a multi-modal pre-training model to predict cells, calculate the coordinates of the predicted cells, and correct the table information obtained by the OCR identification by using the coordinates of the predicted cells to obtain corrected table information; inputting the corrected table information into the multi-modal pre-training model to train a semantic entity recognition model, performing semantic recognition by the semantic entity recognition model, extracting table semantic information, obtaining a corrected semantic recognition prediction result, and classifying the prediction result. The problems of low semantic recognition accuracy, great difficulty in table extraction, and low table extraction accuracy are solved.

Description

Technical Field

[0001] This invention relates to the field of computer information processing technology, specifically to a method for extracting semantic information from tables based on cell coordinate optimization. Background Technology

[0002] With the development of information technology, the importance of spreadsheets is undeniable. While some deep learning-based solutions exist for table recognition, their accuracy is not ideal. Existing table recognition technologies, for tables with incomplete lines (e.g., no lines, partial lines, or unclear lines), primarily rely on text region recognition. This method demands high-quality table images and structural integrity, and suffers from low accuracy. For tables with complete lines, they rely on recognizing cell lines to identify cells, a method that is complex, unstable, and inaccurate. Therefore, existing table recognition technologies cannot truly meet the ever-increasing demands for table recognition.

[0003] For example, grid-based methods focus on the fusion of multimodal information at the image level. Text is mostly character-level, and the embedding of text and structural information is relatively simple, such as the Chargrid algorithm. Token-based methods encode positional, visual, and other feature information into a multimodal model and pre-train on large-scale datasets, achieving good results in downstream tasks. Examples include BERT, RoBERTa, LayoutLM, LayoutLMv2, LayoutXLM, and StrucText. GCN-based methods attempt to learn the structural information between images and text, thus solving the problem of open-set information extraction (templates not seen in the training set), such as GCN and SDMGR. End-to-end methods combine existing OCR text recognition and table information extraction tasks into a unified network for joint learning, with mutual reinforcement during the learning process, such as the Trie algorithm.

[0004] Publication number CN113269153A, entitled "A Table Recognition Method and Apparatus", discloses a table recognition method, which involves acquiring an image to be recognized; inputting the image to be recognized into a trained table detection model to obtain the table coordinate information and table line position information of the table image in the image to be recognized; determining the position information and text information of each cell in the table image; and generating the table information corresponding to the table image.

[0005] Publication number CN115546809A, titled "A Table Structure Recognition Method Based on Cell Constraints and Its Application," describes a table structure recognition method based on cell constraints. It inputs different table images into corresponding target detection models according to table type to output the table's position information, obtaining information about the entire row and column, including the coordinates of any two non-adjacent vertices in each row and column; identifies all text and its borders in the table; obtains the table's structure information through machine translation; corrects the cells in the table based on the position information, row and column information, and table structure information; and clusters the coordinates of the corrected cells to generate the table's cell structure. This application can be combined with any optical symbol recognition method to analyze table structure from electronic documents end-to-end through complete table recognition technology.

[0006] The aforementioned end-to-end method places the existing OCR text recognition and table information extraction tasks into a unified network for joint learning and mutual reinforcement during the learning process. However, this method is not accurate enough for table extraction using text recognition algorithms. For simple table extraction, although it solves the problem of relatively simple embedding of text and structural information to a certain extent, table extraction using text recognition algorithms is more difficult and the table extraction accuracy is not high. Summary of the Invention

[0007] This invention addresses the problems of high difficulty in table extraction using text recognition algorithms, difficulty in extracting complex tables, and insufficient accuracy in table extraction in existing technologies. It proposes a table semantic information extraction method based on cell coordinate optimization, aiming to improve the accuracy of table extraction using text recognition.

[0008] In view of this, according to one aspect of this application, a method for extracting semantic information from tables based on cell coordinate optimization is proposed. This method performs OCR recognition on a dataset containing various types of forms to obtain a table information file containing text, cells, and labels. Simultaneously, the same dataset is input into a multimodal pre-trained model to predict cells, calculate the predicted cell coordinates, and use these coordinates to correct the table information obtained from the OCR recognition, resulting in corrected table information. The corrected table information is then input into the multimodal pre-trained model, and after training, a semantic entity recognition model is obtained. This model performs semantic recognition to extract the semantic information from the tables, resulting in corrected semantic recognition prediction results, which are then classified.

[0009] Further optimization involves inputting the table information file of the OCR recognition results into a pre-trained model for training, performing semantic entity SER recognition, obtaining semantic prediction indicators, determining the recognition accuracy threshold, analyzing errors in semantic entity SER recognition based on the semantic prediction indicators, judging the accuracy of cell semantic recognition based on the error analysis, and if the cell recognition does not reach the recognition accuracy threshold, then inputting the table information file of the OCR recognition results into the pre-trained model again for semantic entity SER recognition until the recognition accuracy threshold requirement is met, and extracting the table semantic information.

[0010] Further preferably, the step of inputting the same dataset into the pre-trained model to obtain the predicted cell includes: inputting the XFUND dataset into the structured text understanding model StrucTexT model, using multimodal information across text, images and layout to pre-train the StrucTexT model to obtain the pre-trained model, the pre-trained model predicts the sentence length in the output table cell and pairs the cell orientation, wherein the encoder completes the segment-character alignment of the text in the table cell.

[0011] Further optimization involves constructing an additional fragment ID embedding during the model input stage to associate visual and textual features at different granularities. Simultaneously, a two-dimensional position embedding is added to incorporate layout cues. Hadamard product units are invoked to fuse encoded features at different levels and patterns to achieve advanced feature fusion.

[0012] Further preferably, the step of inputting the corrected table information into the multimodal pre-trained model further includes: the table text is obtained by calling the text mapping unit through the OCR engine to obtain the text; a visual feature map is obtained through visual encoding; the information output from the visual or text mapping is a one-dimensional coordinate map and a 2D rectangular coordinate position map, which are then input into the encoding layer and encoded using a multimodal transformer encoder to obtain an output vector; the output vector is used to perform a semantic entity recognition task to classify each detected text; and a dual affine attention mechanism classifier is used to extract the relationship of the text content in the image based on the relationship pairs to obtain a multimodal embedding of the vector mapping.

[0013] Further optimization involves using information from text, image, and layout aspects to integrate the information into a mapping vector for an input image with preset conditions for OCR results. Then, multimodal embeddings are fed into the transformer network of the pre-trained model to obtain rich semantic features. The transformer network completes cross-modal fusion by establishing the interaction between different modal inputs based on classification relationships. The structured text understanding module receives the text and image features encoded by the mapping vector, performs entity recognition, and extracts relationships for entity labeling and linking.

[0014] According to another aspect of this application, a table semantic information recognition system based on cell coordinate optimization is proposed, including: an OCR recognition module, a structured text understanding StrucTexT model, a document correction unit, a multimodal pre-trained model LayoutXLM, and a semantic entity recognition module. The OCR recognition module performs OCR recognition on a dataset containing various types of forms to obtain a table information file containing text, cells, and labels. Simultaneously, the StrucTexT model is pre-trained on the same dataset to obtain predicted cells and calculate the predicted cell coordinates. The document correction unit uses the predicted cell coordinates to correct the obtained table information to obtain corrected table information. The corrected table information is input into the multimodal pre-trained model, and after training, a semantic entity recognition model is obtained. The semantic entity recognition model performs semantic recognition, extracts the semantic information of the table, obtains the corrected semantic recognition prediction result, and classifies the prediction result.

[0015] Further optimization involves inputting the table information file of the OCR recognition results into a pre-trained model for training, performing semantic entity SER recognition, obtaining semantic prediction indicators, determining the recognition accuracy threshold, analyzing errors in semantic entity SER recognition based on the semantic prediction indicators, judging the accuracy of cell semantic recognition based on the error analysis, and if the cell recognition does not reach the recognition accuracy threshold, then inputting the table information file of the OCR recognition results into the pre-trained model again for semantic entity SER recognition until the recognition accuracy threshold requirement is met, and extracting the table semantic information.

[0016] Further optimization involves constructing an additional segment ID embedding during the model input stage to associate visual and textual features at different granularities. Simultaneously, a two-dimensional position embedding is added to incorporate table layout cues. Hadamard product units are invoked to fuse encoded features at different levels and patterns to achieve advanced feature fusion, thereby completing the understanding and labeling of structured text segments in the XFUND dataset.

[0017] Further preferably, the step of inputting the corrected table information into the multimodal pre-trained model further includes: obtaining text from the table text by calling the text mapping unit through the OCR engine; obtaining visual feature mapping through visual encoding; outputting one-dimensional coordinate mapping and two-dimensional rectangular coordinate position mapping through visual or text mapping; inputting them together into the encoding layer; encoding them using a multimodal transformer encoder to obtain an output vector; classifying each detected text using the output vector for semantic entity recognition; and extracting the relationship between the text content in the image based on the relationship pair using a dual affine attention mechanism classifier to obtain the multimodal embedding of the vector mapping.

[0018] According to another aspect of this application, an electronic device is also provided, comprising: a processor; and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the table semantic information extraction method based on cell coordinate optimization described above.

[0019] According to another aspect of this application, a non-transitory computer-readable storage medium storing computer instructions is also provided, the computer instructions being used to cause the computer to perform the table semantic information extraction method based on cell coordinate optimization described above.

[0020] This application first performs optical character recognition on a dataset containing various types of forms to obtain a table information file containing text, cells, and labels. Simultaneously, a structured text understanding model is pre-trained on the same dataset to obtain predicted cells. The predicted cell coordinates are then used to correct the table information file. The corrected table information file is then input into a multimodal pre-trained model for training. Finally, a semantic entity recognition module performs semantic recognition to obtain the corrected semantic recognition prediction result, which shows a significant improvement in accuracy compared to the previous version. This application features a simple structure, high accuracy, and solves problems such as low semantic recognition accuracy, low precision in extracting semantic information from tables, and high difficulty in table extraction. It also possesses good transferability, versatility, and scalability. Attached Figure Description

[0021] Figure 1 The diagram shown is a schematic flowchart of a table recognition method based on cell coordinate optimization in an exemplary embodiment of this application.

[0022] Figure 2 The diagram shown is a schematic representation of the pre-trained model LayoutXLM structure in an exemplary embodiment of this application.

[0023] Figure 3 The diagram shown is a structural block diagram of an exemplary electronic device that can be used to implement embodiments of this application. Detailed Implementation

[0024] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0025] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0026] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0027] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0028] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0029] This application proposes a method for extracting semantic information from tables based on cell coordinate optimization.

[0030] First, optical character recognition is performed on a dataset containing various types of forms to obtain a table information file containing text, cells, and labels. Simultaneously, the StrucTexT model, a structured text understanding model, is pre-trained on the same dataset to obtain predicted cells. The table information file is then corrected using the coordinates of the predicted cells. The corrected table information file is then input into the LayoutXLM model, a multimodal pre-trained model, for training. Finally, the semantic entity recognition module performs semantic recognition to obtain the corrected semantic recognition prediction results. The accuracy is significantly improved compared to the original, thus solving the problem of low accuracy in extracting semantic information from tables.

[0031] The table recognition function is mainly implemented by the following models: single-line text detection model - DB, single-line text recognition model - CRNN, table structure cell model, and coordinate prediction model - RARE. The single-line text detection model detects the coordinates of single-line text in the image. These coordinates are then input into the single-line text recognition model to obtain the recognition result, which includes the table structure of the image and the coordinates of the table's cells. The prediction model, based on the table's structural information and cell coordinates, combines the coordinates of the single-line text, the recognition result, and the cell coordinates to output the cell recognition result. The cell recognition result and the table structure together construct the table's HTML string. For the same dataset, using the cell coordinates obtained from the Structured Text Understanding (StrucTexT) pre-trained model to correct the cell information of the table recognition file obtained by the Optical Character Recognition (OCR) unit, and then calling the Semantic Entity Recognition (SER) unit for recognition, can effectively improve the accuracy of table semantic information prediction.

[0032] This application provides a further detailed description using the XFUND dataset as an example in its exemplary instance.

[0033] The XFUND dataset is a multilingual dataset for the Key Information Extraction (KIE) task. It contains form data in seven different languages, all of which have been manually labeled in key-value pair format. Each language dataset contains 199 form data points, divided into a training set of 149 forms and a test set of 50 forms.

[0034] Key information extraction is an important task in Document VQA, which mainly involves extracting the required key information from images, such as extracting name and citizen identification number information from an ID card. The types of this information are often fixed under a specific task, but they are different between different tasks.

[0035] Using Semantic Entity Recognition (SER), text recognition and classification in images can be performed.

[0036] like Figure 1 The diagram shows a flowchart of a table recognition method based on cell coordinate optimization in an exemplary embodiment of this application. It includes:

[0037] The XFUND dataset, a multilingual dataset containing various types of form data, is acquired. First, an OCR unit performs corrected optical character recognition (OCR) on the XFUND dataset to obtain a table information file containing text, cells, and labels. Simultaneously, the same XFUND dataset is input into a pre-trained model (e.g., the StrucTexT model) for training to obtain a cell prediction model. This model is then used to predict cells in the form to be recognized. Predicted cell coordinates are calculated based on the predicted cell information, and these coordinates are used to correct the obtained table information, resulting in corrected table information. This corrected table information includes text, cells, and labels from a JSON file. The corrected table information is then input into a multimodal pre-trained model (e.g., the LayoutXLM model) for training to obtain a semantic recognition model. This model performs semantic recognition on the table information file to complete the SER semantic recognition task, extracting the semantic information from the table.

[0038] Furthermore, a semantic prediction index is constructed, and a recognition accuracy threshold is determined. Based on the semantic prediction index, a bad case analysis is performed on the semantic entity SER recognition to improve prediction accuracy. The accuracy of cell semantic recognition is judged based on the error case analysis. When cell recognition is not accurate enough and does not reach the recognition accuracy threshold, the table information file of the OCR recognition result is input into the pre-trained model for training, performing semantic entity SER recognition, calculating the recognition accuracy, and performing a bad case analysis on the semantic entity SER recognition results based on the semantic prediction index and recognition accuracy threshold to further accurately extract table semantic information until the recognition accuracy is met, obtaining accurately recognized cells.

[0039] The pre-trained model is trained on the XFUND dataset. This application's embodiments use the StrucTexT model training on the XFUND dataset as an example for detailed explanation.

[0040] The StrucTexT model is used to integrate and learn image cell information. The XFUND dataset is input into the StrucTexT model for training, and the prediction results of cell information are obtained.

[0041] During pre-training, the StrucTexT model invokes masked visual language task, sentence length prediction task, and paired cell orientation task units to incorporate multimodal information across text, images, and layout. Furthermore, in the exemplary embodiments of this application, the StrucTexT model introduces a segment-character aligned encoder layer on top of the transformer to handle entity labeling and entity linking tasks at different granularities.

[0042] The XFUND dataset is input into the StrucTexT model for pre-training. Multimodal information across text, images, and layout is used to pre-train the StrucTexT model to obtain a pre-trained model. The XFUND dataset to be identified is input into the pre-trained model to predict the sentence length in the cells of the output table and match the orientation of the cells. The encoder completes the segment-character alignment of the text in the table cells and identifies the semantic information of the table in the output cells.

[0043] like Figure 2 The diagram shown is a schematic diagram of the pre-trained LayoutXLM model structure in an exemplary embodiment of this application.

[0044] LayoutXLM is a multilingual extension of the LayoutLMv2 model. LayoutLMv2 learns cross-modal interactions between visual and textual information using the Transformer architecture, integrating image information during the pre-training phase. Furthermore, in addition to the masked visual language model, LayoutLMv2 uses two new training objectives: image-text matching and image masking prediction. Here, visual embedding represents visual mapping, text embedding represents text mapping, 2D position embedding represents 2D rectangular coordinate position mapping, and position embedding represents one-dimensional coordinate mapping. visualEncoder represents visual encoding, Feature Map represents visual feature mapping, OCR System represents OCR engine, and Relation Extraction represents relation extraction.

[0045] First, the table text is obtained by calling the Text Embedding unit of the OCR engine OCR System. Then, a visual feature map is obtained through visual encoding. The visual or text embedding outputs the following: a one-dimensional coordinate mapping (position embedding, such as numerical information) and a 2D rectangular coordinate positioning mapping (2D position embedding, such as position coordinate information (x...)). ij :y ijThe text is input into the Transformer Encoder Layers, which employs a Multi-Modal Transformer Encoder Layer with Spatial-Aware Self-Attention to encode the output vectors. For the XFUND dataset, there are four categories of output vectors: QUESTION, ANSWER, HEADER, and OTHER, yielding the corresponding categories and OCR recognition results. The output vectors are then used for Semantic Entity Recognition to classify each detected text, such as dividing it into questions and answers, and then finding the corresponding answers for each question (e.g., E1&E2, E1&E3, E2&E3). A Biaffine Attention Classifier is then used to perform Relation Extraction, extracting the corresponding relations based on the relation pairs. Based on the Relation Extraction task, relations can be extracted from the text content in the image, resulting in a multi-modal embedding of the vector mapping.

[0046] For an input image containing predefined OCR results, such as bounding boxes and text fragments, the feature embedding stage in vector mapping leverages various information from text, image, and layout aspects to integrate the information into a mapping vector for subsequent processing.

[0047] The multimodal embeddings obtained above are then fed into the transformer network in the multimodal transformer encoder layers of the pre-trained model to obtain rich semantic features.

[0048] Then, the transformer network completes cross-modal (text, layout, and image) fusion by establishing the interaction between different modal inputs based on classification relationships.

[0049] Finally, the Transformer Encoder Layers of the structured text understanding module receive text and image features encoded by mapping vectors, perform entity recognition for entity labeling, and extract relationships for entity linking.

[0050] StrucTexT's unified framework combines features from different levels and patterns, effectively improving the understanding of various document structures. A transformer encoder (transformer network) is introduced into StrucTexT.

[0051] StrucTexT learns cross-modal knowledge from images of fragments and word tags. Furthermore, an additional fragment ID embedding is constructed during the model input stage to correlate visual and textual features at different granularities. Simultaneously, a 2D positional embedding is appended during the model input stage to address layout cues. Subsequently, a Hadamard product unit is invoked to fuse encoded features from different levels and modalities for high-level feature fusion. The transform encoder is located at the center of StrucTexT's unified framework, thus enabling StrucTexT to support both segment-level and tag-level tasks in structured text understanding within a single framework.

[0052] Predictions are made using the LayoutXLM model. Multimodal pre-training of text, layout, and images achieves state-of-the-art (SOTA) performance on visually rich document understanding tasks, demonstrating significant potential for joint learning across different modalities. The trained LayoutXLM model is a multimodal pre-trained model for multilingual document understanding, designed to bridge the language barriers in visually rich document understanding.

[0053] The corrected tabular information file is fed into the LayoutXLM model along with the original dataset (such as the XFUND dataset) for prediction. The LayoutXLM model receives information from three different modes, including text, layout, and image, which are encoded as text embeddings, layout embeddings, and visual embeddings, respectively. The text and image embeddings are concatenated and then the layout embedding is added to obtain the input embedding. The input embedding is encoded by a multimodal transformer with a spatially aware self-attention mechanism. Finally, the output contextual representation can be used in subsequent task-specific layers.

[0054] Complete the SER semantic recognition task. Relation extraction involves classifying each detected text, such as dividing it into question and answer pairs. Then, find the corresponding answer for each question. Based on the RE task, it is possible to extract relations from the text content in an image, such as determining question pairs.

[0055] The Semantic Entity Recognition (SER) module classifies the results predicted by the LayoutXLM model, classifying each detected text. Semantic information refers to the semantic information represented by the text in each cell of the table, mainly including header, question, answer, and other. Based on the position of the text in the table, each text is divided into four categories: header, question, answer, and other.

[0056] Correct the table information file output by the OCR recognition engine.

[0057] Because the predicted cell information in the table information file obtained by OCR is not accurate enough, there may be cases such as missed cell recognition, cell recognition area being too large, and cell recognition area being too small.

[0058] Therefore, the table information file obtained by training the StrucTexT model on the XFUND dataset is analyzed using badcase to obtain the cell information file that needs to be corrected, and the cell information in the table information file obtained by OCR is modified.

[0059] This application first addresses the problem of inaccurate cell information, and then, based on this, addresses the problem of inaccurate extraction of semantic information from tables.

[0060] This embodiment uses sample text for prediction and obtains a semantic prediction index based on the evaluation results. The semantic prediction results obtained from the classification are evaluated, and the obtained semantic prediction index is used to perform bad case analysis. The analysis results are shown in the table below, where box represents the rectangular region obtained by OCR recognition.

[0061] Table 1: Results of Bad Case Analysis Based on Semantic Prediction Indicators

[0062] box missing identification The box is too large. The box is too small. Region identification error total 169 238 539 946 percentage 0.179 0.251 0.570 1

[0063] Based on the analysis results, Python software was used to correct the tables according to the rules, accurately extracting semantic information. The corrected tables, with their more accurate semantic information compared to the uncorrected tables, showed improvements in overall evaluation metrics.

[0064] Table 2 shows a comparison between predicted and uncorrected metrics. It compares the uncorrected and corrected table information for cell recognition and semantic information extraction. The results show that prediction accuracy, recall, and the overall performance index (F1) all improved significantly.

[0065] Table 2: Comparison of Forecast Indicators and Uncorrected Indicators

[0066]

[0067] Using the specific dataset XFUND, we first performed OCR recognition on the XFUND dataset to obtain a table of text recognition results (JSON file), which was then fed into the StrucTexT model for training to obtain the predicted cell information. Next, the JSON file trained by the StrucTexT model was used to modify the cell information in the OCR-generated JSON file, resulting in a new JSON file. This new JSON file, along with the XFUND dataset, was then fed into the LayoutXLM model for prediction and classification. The semantic prediction results obtained from the classification were evaluated to obtain prediction metrics. Compared with the uncorrected metrics, the table above shows that the prediction accuracy, recall, and F-score all improved significantly.

[0068] refer to Figure 3 The present invention describes a structural block diagram of an electronic device 300 that can serve as a server or client of this application, which is an example of a hardware device that can be applied to various aspects of this application. The electronic device is intended to represent various forms of digital electronic computer devices, such as desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0069] like Figure 3 As shown, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. The RAM 303 may also store various programs and data required for the operation of the device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0070] Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, output unit 307, storage unit 308, and communication unit 309. Input unit 306 can be any type of device capable of inputting information to electronic device 300. Input unit 306 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 307 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 308 may include, but is not limited to, disk and optical disk. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0071] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above. For example, in some embodiments, the reconstruction and decomposition of the muscle movement trajectory of the signature stroke based on its original trajectory, and the decomposition of its logarithmic velocity curve, can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 300 via ROM 302 and / or communication unit 309. In some embodiments, the computing unit 301 can be configured by any other suitable means (e.g., by means of firmware) to perform a signature handwriting dynamic acquisition implementation method.

[0072] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0073] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0074] As used in this application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0075] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0076] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0077] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

Claims

1. A table semantic information extraction method based on cell coordinate optimization, characterized in that, The data set containing multiple types of forms is subjected to optical character OCR recognition to obtain table information containing characters, cells and labels; meanwhile, the same data set is input into a multi-modal pre-training model StrucTexT to train a cell prediction model, and the cell prediction model is called to perform cell prediction on the form to be recognized; the predicted cell coordinates are calculated according to the predicted cell information, the table information obtained by the OCR recognition is corrected using the predicted cell coordinates to obtain corrected table information, and the obtained table information includes characters, cells and labels of a table information file in a json file; the corrected table information is input into a multi-modal pre-training model LayoutXLM, a semantic entity recognition model is obtained after training, semantic recognition is performed by the semantic entity recognition model, table semantic information is extracted, and a corrected semantic recognition prediction result is obtained, and the prediction result is classified.

2. The method of claim 1, wherein, Further comprising, inputting the table information of the OCR recognition result into the pre-training model for training and performing semantic entity SER recognition, obtaining a semantic prediction index, determining an identification accuracy threshold, analyzing the error condition of the semantic entity SER recognition according to the semantic prediction index, judging the accuracy of the cell semantic recognition according to the error condition analysis, and when the cell recognition does not reach the identification accuracy threshold, inputting the table information file of the corrected OCR recognition result into the pre-training model for semantic entity SER recognition again until the identification accuracy threshold requirement is met, and extracting table semantic information.

3. The method of claim 1, wherein, The same data set is input into a pre-training model to obtain a predicted cell, including: inputting an XFUND data set into a structured text understanding model StrucTexT model, pre-training the StrucTexT model using multi-modal information across texts, images and layouts to obtain a cell prediction model, the cell prediction model performing semantic recognition according to the corrected table information to output the sentence length in the table cell and pair the direction of the cell, and the encoder completes the segment-character alignment of the characters in the table cell.

4. The method of claim 1, wherein, An additional segment ID embedding is constructed in the model input stage to associate visual and text features at different granularities, and a two-dimensional position embedding is additionally added to involve layout clues, a Hadamard product unit is called to fuse encoding features of different levels and modes to realize high-level feature fusion.

5. The method according to one of claims 1 to 4, characterized in that The inputting of the corrected table information into the multi-modal pre-training model further includes that table text is called by an OCR engine to obtain text Text, visual features are obtained by visual coding to obtain a visual feature map Feature Map, and information obtained by visual or text mapping outputs one-dimensional coordinate mapping positions and two-dimensional rectangular coordinate mapping positions, which are input into an encoding layer together, encoded by a multi-modal transformer encoder to obtain an output vector, each detected text is classified by a semantic entity recognition task through the output vector, and the relationship extraction of the text content in the image is completed by a double-affine attention mechanism classifier according to relationship pairs to obtain a vector-mapped multi-modal embedding.

6. The method of claim 4, wherein, For an input image with a preset condition of an OCR result, information processing is integrated into a mapping vector using information from the aspects of text, image and layout; the multi-mode embedding is input into a transformer network of a pre-trained model to obtain rich semantic features, and the transformer network completes cross-modal fusion according to the classification relationship to establish the interaction between different mode inputs; a structured text understanding module receives the text and image features encoded through the mapping vector, performs entity recognition and extracts relationships, and performs entity tagging and linking.

7. A system for identifying table semantic information based on cell coordinate optimization, characterized in that, Comprise: An OCR recognition module, a structured text understanding StrucTexT model, a file correction unit, a multi-modal pre-training model LayoutXLM, and a semantic entity recognition module, the OCR recognition module performs OCR recognition on a data set containing multiple types of forms to obtain a table information file containing text, cells and labels; at the same time, the StrucTexT model is pre-trained on the same data set to obtain a predicted cell, calculate the coordinates of the predicted cell, and the file correction unit uses the predicted cell coordinates to correct the obtained table information to obtain corrected table information; the corrected table information is input into the multi-modal pre-training model, and a semantic entity recognition model is obtained after training, which performs semantic recognition and extracts table semantic information to obtain a corrected semantic recognition prediction result, and the prediction result is classified.

8. The system of claim 7, wherein, Further comprise, input the table information file of the OCR recognition result into the pre-training model for training, perform semantic entity SER recognition, obtain semantic prediction indicators, determine an identification accuracy threshold, analyze the error situation of the semantic entity SER recognition according to the semantic prediction indicators, judge the accuracy of the cell semantic recognition according to the error situation analysis, when the cell recognition does not reach the identification accuracy threshold, input the corrected table information file of the OCR recognition result into the pre-training model for semantic entity SER recognition again, until the identification accuracy threshold requirement is met, and the table semantic information is extracted.

9. The system according to claim 7 or 8, characterized in that, The XFUND data set is input into the structured text understanding model StrucTexT model, and the StrucTexT model is pre-trained using multi-modal information across text, image and layout to obtain a cell prediction model, which performs semantic recognition according to the corrected table information and outputs the sentence length in the table cell and the direction of the paired cell, and completes the segment-character alignment of the text in the table cell through an encoder; an additional segment ID embedding is constructed in the model input stage to associate visual and text features at different granularities, and a two-dimensional position embedding is additionally added to involve table layout clues, a Hadamard product unit is called to fuse encoding features of different levels and modes to realize high-level feature fusion, and structured text segmentation understanding and tagging in the XFUND data set are completed.

10. The system of claim 7 or 8, wherein, The inputting the revised table information into the multi-modal pre-training model further includes that the table text obtains text through a text mapping unit called by an OCR engine, and obtains visual feature mapping through visual coding; information output one-dimensional coordinate mapping and two-dimensional rectangular coordinate position mapping are obtained through visual or text mapping, and are input into an encoding layer together; a multi-modal transformer encoder is used for encoding to obtain an output vector; each detected text is classified through a semantic entity recognition task based on the output vector; a double-affine attention mechanism classifier is used for extracting the relationship of the text content in the image according to the relationship to obtain vector mapping multi-modal embedding.

11. An electronic device, comprising: Comprise: a processor; and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the table semantic information extraction method based on cell coordinate optimization according to any one of claims 1-6.

12. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to cause the computer to perform the table semantic information extraction method based on cell coordinate optimization according to any one of claims 1-6.

Images