Test report element extraction method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of intelligent file processing, and discloses an element extraction method and device of a test report, computer equipment and a storage medium, the method comprises the following steps: obtaining scanning document documents of a plurality of paper test reports, and screening out a multi-modal format document from the scanning document documents; identifying and extracting text features, picture features and table features in the multi-modal format document based on an optical character identification technology model; performing paragraph combination on the extracted text features; and extracting structured data from the text features, the picture features and the table features after paragraph combination by adopting a multi-modal element extraction model to obtain a complete electronic document corresponding to the multi-modal format document. Based on an optical character recognition technology, character features, image features and table features in various paper test reports are accurately recognized and extracted by using a number, image and character recognition and extraction technology, and the recognition and extraction accuracy and extraction efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent file processing, and in particular to a method, device, computer equipment and storage medium for extracting elements from a test report. Background Art

[0002] At present, the traditional way of sending and receiving documents is to scan the documents into electronic files through scanners, and then upload them to the relevant information system for document processing. During the document processing process, it is often necessary to extract the official document elements in the documents. At present, the identification of official document elements in office business is mainly based on manual methods, which manually identify the official document elements in the documents and manually enter the official document content elements into the business information processing system.

[0003] Taking new energy sites as an example, there are a large number of paper and PDF versions of test reports and materials in the new energy production management work. These data are mainly stored in the form of paper files, which makes data extraction, integration and reuse more difficult. Therefore, it is necessary to identify and extract the content elements on the paper test reports. However, it is usually necessary to manually identify the official document elements and manually enter them into the business information processing system. This method is labor-intensive, error-prone, and repetitive. If the volume of document processing increases, the manpower and time costs consumed are huge. In addition, the identification and extraction of information on official document elements need to have a high accuracy rate to avoid information entry errors. Therefore, a method for extracting elements from new energy site test reports is needed. Summary of the Invention

[0004] In view of this, the present invention provides a method, apparatus, computer device and storage medium for extracting elements of a test report to solve the problem of low extraction efficiency and inaccuracy caused by manual extraction of elements of a test report.

[0005] In a first aspect, the present invention provides a method for extracting elements from a test report, the method comprising:

[0006] Obtain scanned documents of multiple paper test reports and filter out multimodal format documents from the scanned documents;

[0007] Identify and extract text features, image features, and table features in multimodal documents based on optical character recognition technology models;

[0008] Merge the extracted text features into paragraphs;

[0009] A multimodal feature extraction model is used to extract structured data from the text features, image features and table features after the paragraphs are merged to obtain a complete electronic document corresponding to the multimodal format document.

[0010] The present invention provides a method for extracting elements from test reports. The method is based on optical character recognition technology (Office of Civilian Requirements, abbreviated as OCR technology) and utilizes digital, image, and text recognition and extraction technologies to accurately identify and extract text features, image features, and table features in various paper test reports. The extracted text features are then merged into paragraphs, and multiple lines of text can be merged into paragraphs, thereby improving the accuracy and efficiency of recognition and extraction.

[0011] In an optional embodiment, the optical character recognition technology model includes a text recognition model, a layout parsing model, a text classification model, and a table recognition model;

[0012] Identify and extract text features, image features, and table features from multimodal documents based on the optical character recognition technology model, including:

[0013] Using a text recognition model to identify text features in a multimodal format document, and using the text classification model to extract the identified text features;

[0014] A layout parsing model is used to divide image documents in multimodal format documents into regions and locate and extract image features;

[0015] A table recognition model is used to identify and extract table features in multimodal format documents to obtain an electronic version of the table.

[0016] The present invention provides a method for extracting elements from a test report. The method uses a text recognition model to extract text features from multimodal paper documents. It also obtains text position information in a table through a table recognition model, and obtains pixel information of the entire image based on a layout parsing model. Ultimately, it achieves element extraction in text format, and achieves significant improvements in accuracy and versatility.

[0017] In an optional embodiment, the text classification model is a deep learning model;

[0018] A text recognition model is used to identify text features in multimodal documents, and a text classification model is used to extract the identified text features, including:

[0019] Use text recognition models to generate unlabeled data and pseudo-labels corresponding to text regions in multimodal documents;

[0020] Filter out unlabeled data and pseudo labels that meet a preset confidence threshold from unlabeled data and pseudo labels;

[0021] The preset small model is trained using unlabeled data and pseudo labels that meet the preset confidence threshold to identify text areas;

[0022] A deep learning model is used to extract text features in text areas.

[0023] The present invention provides a method for extracting elements from a test report. The method adopts a text recognition model to generate unlabeled data and pseudo-labels corresponding to text regions in a multimodal format document; the unlabeled data and pseudo-labels that meet a preset confidence threshold are screened out from the unlabeled data and pseudo-labels; the unlabeled data and pseudo-labels that meet the preset confidence threshold are used to train a preset small model to identify the text region, and finally a deep learning model is used to extract text features from the text region. The text recognition model is used to improve the efficiency of generating unlabeled data and pseudo-labels, and the deep learning model is used to improve the accuracy of text feature extraction.

[0024] In an optional embodiment, a layout parsing model is used to divide image documents in a multimodal document into regions and locate and extract image features, including:

[0025] A rule-based approach is used to identify key regions containing textual content from image documents in multimodal documents, and image features are extracted from the key regions.

[0026] or,

[0027] A template-based approach is used to match image documents in multimodal documents, identify and extract image features.

[0028] The present invention provides a method for extracting elements from a test report, which adopts a rule-based approach to identify key areas containing text content from image documents in a multimodal document and extract image features from the key areas, or adopts a template-based approach to match image documents in a multimodal document, identify and extract image features, and facilitate the subsequent extraction of text elements from image features.

[0029] In an optional embodiment, the table recognition model includes a cell positioning module, a table structure analysis module, a text recognition module and a table construction module;

[0030] The table recognition model is used to identify and extract table features in multimodal documents to obtain an electronic version of the table, including:

[0031] The cell positioning module is used to locate the cell boundaries of the table in the multimodal format document and identify the rows and columns of the cells;

[0032] A table structure analysis module is used based on the rows and columns of cells to determine the layout and structure of the table, thereby obtaining a plurality of structured cells;

[0033] Use the text recognition module to recognize the text in each cell and extract the text content in the cell;

[0034] The table construction module is used to fill the text content in the cell into multiple structured cells to obtain an electronic version of the table.

[0035] The present invention provides a method for extracting elements from a test report, which obtains text position information and cell information through a table recognition model, ultimately filling the text content into multiple structured cells and automatically obtaining an electronic version of the table, thereby improving the accuracy and efficiency of extracting table features in multimodal paper documents.

[0036] In an optional implementation, merging the extracted text features into paragraphs includes:

[0037] Use word embedding technology to convert text features into feature vectors for each line of text, and determine whether the feature vectors of text in two adjacent lines need to be merged;

[0038] When merging is required, a binary classification model is used to perform encoding, fusion, and decoding on the text feature vectors in sequence to obtain the merged paragraph.

[0039] The present invention provides a method for extracting elements from a test report, which can restore the recognized text into paragraphs and determine whether two adjacent lines of text need to be merged in the paragraph merging and splitting tasks, so that multiple lines of text can be merged into paragraphs, which is convenient for subsequent element extraction.

[0040] In an optional embodiment, the multimodal feature extraction model includes a named entity recognition module, an entity linking module, and a relationship extraction module;

[0041] A multimodal feature extraction model is used to extract structured data from the text features, image features, and table features after merging paragraphs to obtain a complete electronic document corresponding to the multimodal format document, including:

[0042] The named entity recognition module is used to identify multiple entities from the text features, image features, and table features after the paragraphs are merged, and the corresponding annotation information and entity type are added to each entity;

[0043] Calculate the relevance of multiple entities to preset candidate entity objects, and link the entities to corresponding preset candidate entity objects using an entity linking module based on the relevance;

[0044] The relationship extraction module is used to extract the grammatical or semantic connection between any two entities from the preset candidate entity objects that have completed the link, and multiple entities are fused according to the grammatical or semantic connection to obtain a complete electronic document corresponding to the multimodal format document.

[0045] The present invention provides a method for extracting elements from a test report, which uses a named entity recognition module, an entity linking module, and a relationship extraction module to realize the extraction of structured data, and ultimately realizes the conversion of multimodal format documents into corresponding complete electronic documents, thereby improving the accuracy of extraction, being able to accurately convert multimodal paper documents into electronic documents, and facilitating the storage of electronic documents corresponding to the test report.

[0046] In an optional embodiment, the method for extracting elements from a test report further includes:

[0047] The data synchronization model is used to store the complete electronic document in the designated directory to form the electronic document of the test report, and the electronic document of the test report is transmitted to the corresponding business database.

[0048] The present invention provides a method for extracting elements from a test report, which uses a data synchronization model to store a complete electronic document in a designated directory to form an electronic document of the test report, and transmits the electronic document of the test report to a corresponding business database, thereby realizing the storage of the electronic document of the test report and being able to trace the test report.

[0049] In a second aspect, the present invention provides a device for extracting elements from a test report, the device comprising:

[0050] The document acquisition and screening module is used to obtain scanned documents of multiple paper test reports and screen out multimodal format documents from the scanned documents;

[0051] Feature recognition and extraction module, used to recognize and extract text features, image features, and table features in multimodal documents based on the optical character recognition technology model;

[0052] The paragraph merging module is used to merge the extracted text features into paragraphs;

[0053] The structured data extraction module is used to extract structured data from the text features, image features and table features after the paragraphs are merged using a multimodal feature extraction model to obtain a complete electronic document corresponding to the multimodal format document.

[0054] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method for extracting elements from a test report according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0055] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for extracting elements from a test report according to the first aspect or any corresponding embodiment thereof.

[0056] In a fifth aspect, the present invention provides a computer program product comprising computer instructions for causing a computer to execute the method for extracting elements from a test report according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 is a flow chart of a method for extracting elements from a test report according to an embodiment of the present invention;

[0059] Figure 2 is a flow chart of another method for extracting elements from a test report according to an embodiment of the present invention;

[0060] Figure 3 is a flow chart of another method for extracting elements from a test report according to an embodiment of the present invention;

[0061] Figure 4 is a flow chart of a method for extracting elements from another test report according to an embodiment of the present invention;

[0062] Figure 5 is a schematic diagram of a text recognition model according to an embodiment of the present invention;

[0063] Figure 6 is a schematic diagram of a text classification model according to an embodiment of the present invention;

[0064] FIG7( a ) is a schematic diagram of a paragraph merging model according to an embodiment of the present invention;

[0065] FIG7( b ) is a schematic diagram of a binary classification model according to an embodiment of the present invention;

[0066] Figure 8 is a schematic diagram of a table recognition model according to an embodiment of the present invention;

[0067] Figure 9 is a schematic diagram of named entity recognition according to an embodiment of the present invention;

[0068] Figure 10 is a schematic diagram of entity linking according to an embodiment of the present invention;

[0069] Figure 11 is a schematic diagram of relationship extraction according to an embodiment of the present invention;

[0070] Figure 12 is a structural block diagram of a device for extracting elements from a test report according to an embodiment of the present invention;

[0071] Figure 13 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0072] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0073] According to an embodiment of the present invention, an embodiment of a method for extracting elements from a test report is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0074] In this embodiment, a method for extracting elements from a test report is provided, which can be used in computer terminals, such as central processing units, servers, etc. Figure 1 : is a flow chart of a method for extracting elements from a test report according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0075] Step S101: obtain scanned documents of multiple paper test reports and filter out documents in multimodal formats.

[0076] Specifically, if Figure 4 As shown, multiple paper test reports are scanned by a scanning device to obtain scanned electronic documents of multiple paper test reports. Several scanned electronic documents are divided into single modality (one modality for information expression or transmission, pure text or pure image) and multimodality (multiple modalities, text and image combination) formats for feature extraction of text, pictures and table elements in the documents. The scanned electronic documents are stored in a shared directory, and the scanned electronic documents are obtained from the shared directory through data synchronization technology for subsequent processing operations.

[0077] It should be noted that for single-modality scanned electronic documents, on the basis of the existing information extraction model, the models of different information extraction tasks should be unified. In particular, for the extraction of information such as relationships and attributes with very few training samples, the current mainstream information extraction model can be used to ensure accuracy.

[0078] Step S102 : recognizing and extracting text features, image features, and table features in the multimodal document based on an optical character recognition technology model.

[0079] Specifically, optical character recognition (OCR) technology (Office of Civilian Requirements) is a technology that uses optical and computer technologies to convert text information on paper documents, images, photos, and other carriers into text data that can be edited and processed by computers. Through optical input methods such as scanning, the text on various bills, newspapers, books, manuscripts, and other printed materials is converted into image information, and then text recognition technology is used to convert this image information into usable text. Optical character recognition technology models include, but are not limited to, text recognition models, layout parsing models, text classification models, and table recognition models. These models can be used to extract text features, image features, and table features from multimodal documents.

[0080] Step S103: Merge the extracted text features into paragraphs.

[0081] Specifically, the scanned electronic document does not have paragraph information after being parsed by OCR technology, but only lines of text, which is very unfavorable for the subsequent extraction of elements. This embodiment can restore the recognized text to paragraphs, and determine whether two adjacent lines of text need to be merged in the paragraph merging and splitting tasks, and design a classification model based on this goal, so that multiple lines of text can be merged into paragraphs, which is convenient for subsequent element extraction.

[0082] In step S104 , a multimodal feature extraction model is used to extract structured data from the text features, image features, and table features after the paragraphs are merged, to obtain a complete electronic document corresponding to the multimodal format document.

[0083] Specifically, a multimodal feature extraction model is one that can simultaneously process and extract data from multiple modalities. In the real world, information often exists in multimodal forms, such as images, text, and sound. Traditional information extraction techniques primarily process data from a single modality, while multimodal extraction models can integrate information from multiple modalities, enabling a more comprehensive understanding and extraction of information.

[0084] Structured data refers to data with a fixed format and rules, typically presented in the form of tables, tree structures, or relational models. Structured data has clear data types and fields, allowing for operations such as data modeling and data analysis.

[0085] A multimodal feature extraction model is used to extract structured data from the text features, image features and table features after the paragraphs are merged, and finally a complete electronic document corresponding to the multimodal format document is obtained, realizing multimodal feature extraction.

[0086] The test report element extraction method provided in this embodiment is based on optical character recognition technology (Office of Civilian Requirements, referred to as OCR technology), and utilizes digital, image, and text recognition and extraction technology to accurately identify and extract text features, image features, and table features in various types of paper test reports. The extracted text features are then merged into paragraphs, and multiple lines of text can be merged into paragraphs, thereby improving the accuracy and efficiency of recognition and extraction.

[0087] In this embodiment, a method for extracting elements from a test report is provided, which can be used in computer terminals, such as central processing units, servers, etc. Figure 2 : is a flow chart of a method for extracting elements from a test report according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0088] Step S201: obtain scanned copies of multiple paper test reports and select multimodal format documents from the scanned documents. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0089] Step S202 : recognizing and extracting text features, image features, and table features in the multimodal document based on an optical character recognition technology model.

[0090] Specifically, the optical character recognition technology model includes a text recognition model, a layout parsing model, a text classification model, and a table recognition model. The above step S202 includes:

[0091] Step S2021: Use a text recognition model to identify text features in a multimodal document, and use a text classification model to extract the identified text features.

[0092] In some optional implementations, the text classification model is a deep learning model, and step S2021 includes:

[0093] Step a1: Use a text recognition model to generate unlabeled data and pseudo labels corresponding to text regions in a multimodal document.

[0094] Specifically, unlabeled data refers to data that has not been manually labeled or classified. This data does not have clear labels or classification information. Pseudo-labeling is the process of adding unlabeled data with high confidence to the training data.

[0095] like Figure 5 As shown, the text recognition model includes but is not limited to the high-precision Chinese scene text recognition model SVTR (Scene Text Recognition with a Single Visual Model), which is a single visual model for scene text recognition.

[0096] The high-precision Chinese scene text recognition model SVTR is used to accurately generate unlabeled data corresponding to text regions in multimodal documents and pseudo labels corresponding to each unlabeled data, including:

[0097] SVTR first decomposes the image text into patches called character components. After that, it repeatedly merges and / or combines component-level hybrid modules. The global and local hybrid modules are designed to perceive the relationship between character components and within characters, thereby achieving multi-level character component perception and forming a multi-level character component perception. Finally, the characters are predicted through a simple classifier to obtain unlabeled data and pseudo labels corresponding to each unlabeled data. Figure 5 In the data, Unlabeled Crop Text Image represents an image cropped from the original image containing text, and this image is not given any labels or classification information; SVTR-tiny represents a lightweight Vision Transformer model designed for text recognition tasks, where SVTR stands for Simple Vision Transformer model, and tiny refers to a model version with fewer parameters and lower computational resource requirements; CTC Decode (Connectionist Temporal Classification Decode) is a decoding algorithm used in sequence recognition tasks, and Pseudo Label is a common technique in the field of machine learning, especially in semi-supervised learning and weakly supervised learning. Pseudo labeling allows researchers and engineers to train models using unlabeled or partially labeled datasets.

[0098] Step a2: Filter out the unlabeled data and pseudo labels that meet a preset confidence threshold from the unlabeled data and pseudo labels.

[0099] Specifically, confidence refers to the degree of certainty of the model about a certain prediction result. Specifically for text recognition tasks, the confidence threshold refers to the internal score of the model on the accuracy of the recognition result after the text area is identified. The internal score is the confidence determination. When the confidence of the recognition result is higher than the set confidence threshold, the result is considered reliable and can be used as a pseudo-label; when the confidence is lower than the confidence threshold, the result may not be accurate enough and is therefore not adopted. The specific confidence threshold can be set according to the actual situation, usually selected from 0-1, which can be 0.7, 0.8, etc. Filter out unlabeled data and pseudo-labels that are higher than the preset confidence threshold from the unlabeled data and pseudo-labels to provide training data for subsequent small model training.

[0100] In step a3, the preset small model is trained using unlabeled data and pseudo labels that meet the preset confidence threshold to identify the text area.

[0101] Specifically, the preset small model usually refers to a model with fewer parameters, simpler structure, and lower computational complexity compared to an existing large model or benchmark model. It is a model that minimizes resource consumption and computational complexity while maintaining certain performance, such as a simplified version of a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0102] The purpose of training it is as follows:

[0103] Improved accuracy: Through the information provided by pseudo-labels, small models can learn useful features without a large amount of labeled data, thereby improving their recognition accuracy.

[0104] Enhanced generalization: In real-world scenarios, the model may encounter diversity not covered in the training data. Using pseudo-label training can help the model adapt to these changes and enhance its generalization ability in real-world scenarios.

[0105] Small models usually require fewer computing resources and can run on devices with weaker computing power. This embodiment takes this into consideration and uses unlabeled data that meets a preset confidence threshold and a small pseudo-label model for training to identify text areas, but does not rely solely on the small model for recognition.

[0106] Step a4: Use a deep learning model to extract text features in the text area.

[0107] Specifically, the schematic diagram of the text classification model is as follows Figure 6As shown in the figure, the deep learning model uses ALBERT+Seq2Seq+Attention as the multi-label text classification framework.

[0108] The combination of ALBERT, Seq2Seq, and Attention is a technology combination in natural language processing, primarily used for sequence-to-sequence conversion tasks such as machine translation and text summarization. ALBERT is a lightweight BERT model (Bidirectional Encoder Representations from Transformers, a pre-trained language model based on the Transformer architecture). In the field of natural language processing (NLP), BERT is a pre-trained language representation model proposed by Google in 2018. It leverages the Transformer architecture and can process text data bidirectionally, simultaneously considering contextual information in the text. Through techniques such as parameter sharing and embedding factorization, the model's parameter count is reduced, improving training and inference efficiency. ALBERT has made improvements in pre-training tasks, such as predicting sentence coherence rather than simple sentence prediction, further enhancing the model's performance. The Seq2Seq (sequence-to-sequence) model is an encoder-decoder architecture used in fields such as machine translation, text summarization, and dialogue generation. The encoder encodes the input sequence into a fixed-size vector, and the decoder generates the output sequence based on this vector. The attention mechanism enhances the model's ability to distinguish and identify information by assigning different weights to each element in the sequence. It adjusts the output sequence based on the importance of each word in the input sequence, thereby better capturing key information and improving model performance and accuracy.

[0109] ALBERT is used as a tool for extracting text features, which is used to circle and extract text features in text areas. Seq2Seq+Attention is used as a downstream task framework to capture the key information of the extracted text features, extract key text content, and improve the accuracy of text feature extraction.

[0110] Figure 6In [1], x1 represents the model input, and Ex1 represents the "encoded representation" of the input text x1, which is the conversion of text x1 into a numerical representation that the model can understand. This typically involves breaking down the vocabulary into smaller units (tokens) and then mapping these units to corresponding embedding vectors. Tx1 represents the length of the input sequence x1. When processing text, the BERT model typically requires all input sequences to have the same length. Therefore, shorter sequences require padding to a predetermined length, while longer sequences require truncation.

[0111] cls refers to a special tag for classification tasks. When processing text, the BERT model adds a special [CLS] tag (short for "Classification") at the beginning of the sequence. This tag is used to indicate the beginning of the sequence, and its output vector is usually used for downstream classification tasks. Ecls is a vector that contains the semantic information of the input sequence and is the embedded representation of the [CLS] tag. It is obtained by adding the word embedding, position embedding, and paragraph embedding of the [CLS] tag; C represents the output of the Transformer encoder. The Transformer encoder in the BERT model receives the input embedding representation (including word embedding, position embedding, and paragraph embedding) and processes the input data through multiple self-attention layers and feedforward neural network layers. C can be regarded as the context representation matrix after encoder processing, which contains the context information of each tag in the input sequence.

[0112] Dense refers to a fully connected layer (or dense layer), which is a layer in a neural network where each neuron is connected to all neurons in the previous layer. In the BERT model, the output Dense usually refers to the fully connected layer after the Transformer encoder for specific tasks (such as classification, regression, or sequence labeling). In classification tasks, the Dense layer is usually located at the end of the model to convert the encoder output vector into category probabilities.

[0113] Label: represents the true label or target value, and is the data used to guide model learning during model training. In supervised learning tasks, each input sample has a corresponding label, and the goal of the model is to learn how to map the input to these labels. For classification tasks, label is the index or category name of the category to which each sample belongs; for tagging tasks, label is the category label of each token in the sequence; the SEP marker is used to indicate the end of the first sentence and the beginning of the second sentence. When the input contains a single sentence or paragraph, the SEP marker is placed at the end of the sentence; when the input contains two sentences or paragraphs, the first sentence is followed by a SEP marker, followed by the second sentence, and the beginning of the second sentence is also followed by a SEP marker to distinguish between the two different text fragments. Esep represents the embedded representation of the [SEP] marker after passing through the Embedding layer of the BERT model. Tsep represents the result of transforming and processing the embedded representation of the [SEP] marker for a time step (Time Step), which is the output of the embedded vector of the marker after passing through these layers. Such output can be used to understand the relationship between the two sentences.

[0114] The element extraction method of the test report provided in this embodiment uses a text recognition model to generate unlabeled data and pseudo-labels corresponding to text areas in a multimodal format document; screens out unlabeled data and pseudo-labels that meet a preset confidence threshold from the unlabeled data and pseudo-labels; uses the unlabeled data and pseudo-labels that meet the preset confidence threshold to train a preset small model to identify the text area, and finally uses a deep learning model to realize the extraction of text features in the text area. The text recognition model is used to improve the efficiency of generating unlabeled data and pseudo-labels, and the deep learning model is used to improve the accuracy of text feature extraction.

[0115] In step S2022, a layout parsing model is used to segment the image document within the multimodal document and locate and extract image features. Specifically, a layout parsing model refers to an algorithm or system used to analyze and understand the arrangement of elements in a document or image, such as rule-based and template-based methods.

[0116] The layout parsing model can adopt models such as DocLayout (diverse document layout detection), LayoutLM (multimodal basic model), and TextMind (text analysis model). The key areas located by the layout parsing model of this embodiment are text, titles, tables, pictures, etc. The layout analysis algorithm is developed based on PaddleDetection's lightweight model PP-PicoDet, which includes three types of models: English, Chinese, and table layout analysis. The model supports the detection of 10 types of areas: Text, Title, Figure, Figurecaption, Table, Tablecaption, Header, Footer, Reference, and Equation. Table layout analysis supports the detection of Table areas.

[0117] In some optional implementations, the above step S2022 includes:

[0118] Step b1: using a rule-based approach to identify key areas containing textual content from image documents in a multimodal document, and extracting image features from the key areas.

[0119] Specifically, the rule-based approach relies on predefined rules to parse the layout. The predefined rules can be based on the document's formatting standards or typical layout patterns.

[0120] For example, for a specific document type (such as an invoice, a resume, etc.), a series of rules can be set to identify and extract key information, where the area where the information is located is the key area, and image features are extracted from the key area.

[0121] Step b2: Match the image documents in the multimodal document using a template-based approach to identify and extract image features.

[0122] Specifically, the template-based approach involves creating a series of standard layout templates, each corresponding to a specific layout structure. During parsing, the input document is matched against the standard layout templates to identify and extract image feature information. The template-based approach is suitable for documents with a fixed layout that rarely changes.

[0123] The element extraction method of the test report provided in this embodiment adopts a rule-based approach to identify key areas containing text content from image documents in multimodal documents, and extract image features from the key areas, or adopts a template-based approach to match image documents in multimodal documents, identify and extract image features, so as to facilitate the subsequent extraction of text elements in image features.

[0124] Step S2023: Using a table recognition model, identify and extract table features in the multimodal document to obtain an electronic version of the table.

[0125] In some optional implementations, the table recognition model includes a cell positioning module, a table structure analysis module, a text recognition module, and a table construction module. Step S2023 includes:

[0126] In step c1, a cell positioning module is used to locate the cell boundaries of a table in a multimodal format document, and identify the rows and columns of the cells.

[0127] Specifically, the schematic diagram of the table recognition model is as follows Figure 8 As shown, the coordinates of the single-line text in the table image are detected by the single-line text detection model, and then sent to the table recognition model to obtain the recognition result. The cell positioning module of this embodiment adopts the OCR technology model, uses OCR technology to identify the cell boundaries in the table, and locates the rows and columns of the table through image processing technology (such as edge detection, contour recognition, etc.) or deep learning models (such as convolutional neural networks).

[0128] Step c2: using a table structure analysis module to determine the layout and structure of the table based on the rows and columns of the cells, and obtaining a plurality of structured cells.

[0129] Specifically, the table structure analysis module is used to analyze the position and size of cells and determine the layout and structure of the table, including identifying merged cells, table headers, table footers, and blank areas in the table.

[0130] Step c3: Use a text recognition module to recognize the text in each cell and extract the text content in the cell.

[0131] Specifically, a text recognition module is used to perform OCR recognition on the text in each cell to extract the text content in the cell.

[0132] Figure 8In

[15] , text detection algorithms are used to identify text areas within an image, particularly the location of text within a table. This may include detecting table boundaries, rows, and columns, as well as text boxes within the table. Text detection algorithms can use various computer vision techniques, such as edge detection, contour detection, or deep learning methods. Cropping: After detecting text areas, the system crops these areas for separate processing. Cropping reduces the amount of data that needs to be considered in subsequent processing and helps improve text recognition accuracy. Text recognition algorithms use optical character recognition (OCR) technology to convert the text in the cropped image into an editable text format. The text recognition algorithm analyzes each character in the image, identifies the letters or numbers they represent, and converts them into their corresponding text encoding. Cell text aggregation: In a table, each cell may contain one or more text elements. The purpose of this step is to aggregate text elements belonging to the same cell to form a complete cell text. This often requires considering the cell layout and structure to ensure the correct combination of text elements.

[0133] Step c4: Use a table construction module to fill the text content in the cell into multiple structured cells to obtain an electronic version of the table.

[0134] Specifically, based on the results of cell positioning and table structure analysis, the table is structured, which specifically involves the following steps:

[0135] Determine the number of rows and columns in a table; identify and mark merged cells; and split the table into individual cells, each containing one or more text elements. Fill the corresponding cells in the spreadsheet with the OCR results for each cell. For example, if using Excel spreadsheet software, this can be done through a programming interface (such as Openpyxl or xlrd / xlwt).

[0136] For example, Figure 8 In Excel, the identified and aggregated table data is converted into Excel or other spreadsheet formats. The purpose of this step is to facilitate data analysis and further processing by users. The exported Excel file usually retains the structure of the original table, including the layout of rows, columns, and cells. In the context of table data processing, "cell coordinate aggregation" refers to the process of organizing and merging cells in a table according to their coordinate information. Table structure prediction is an important part of understanding table data, mainly referring to the automatic identification and prediction of the structural information of a table given a table image or table data.

[0137] The element extraction method of the test report provided in this embodiment obtains the position information and cell information of the text through the table recognition model, and finally fills the text content into multiple structured cells, automatically obtaining an electronic version of the table, thereby improving the extraction accuracy and efficiency of table features in multimodal paper documents.

[0138] Step S203: Merge the extracted text features into paragraphs. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0139] In step S204, a multimodal feature extraction model is used to extract structured data from the text features, image features, and table features after the paragraphs are merged, to obtain a complete electronic document corresponding to the multimodal format document. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0140] The method for extracting elements from a test report provided in this embodiment uses a text recognition model to extract text features from multimodal paper documents. It also uses a table recognition model to obtain text position information in a table, and uses a layout parsing model to obtain pixel information from the entire image. Ultimately, the accuracy and versatility of extracting elements from text formats are greatly improved. This embodiment provides a method for extracting elements from a test report that can be used in computer terminals, such as central processing units, servers, and the like. Figure 3 : is a flow chart of a method for extracting elements from a test report according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0141] Step S301: obtain scanned copies of multiple paper test reports and select multimodal format documents from the scanned documents. Figure 2 Step S201 of the illustrated embodiment will not be described in detail here.

[0142] Step S302: Identify and extract text features, image features, and table features in the multimodal document based on the optical character recognition technology model. Figure 2 Step S202 of the illustrated embodiment will not be described in detail here.

[0143] Step S303: Merge the extracted text features into paragraphs.

[0144] Specifically, the above step S303 includes:

[0145] In step S3031, word embedding technology is used to convert text features into feature vectors for each line of text, and it is determined whether the feature vectors of text in two adjacent lines need to be merged.

[0146] Specifically, the input paragraph needs to be converted into a numerical form suitable for neural network processing. This can be achieved through word embedding technology, such as using pre-trained models such as Word2Vec, GloVe, or BERT to convert text into feature vectors for each line of text.

[0147] Figure 7(a) shows a schematic diagram of the paragraph merging model. In text recognition, the paragraph merging model determines whether two adjacent lines of text need to be merged. This determination involves a series of rules and algorithms, which can be referred to as follows:

[0148] ① Spatial proximity: If two lines of text are close vertically in the image, they may belong to the same paragraph or text block. ② Text alignment: If two lines of text are aligned horizontally and have similar font size and style, they may be continuous text. ③ Punctuation and grammar: The punctuation and grammatical structure in the text can also serve as a basis for judgment. For example, if the punctuation at the end of one sentence is immediately followed by the beginning of the next line, the two lines may need to be merged. The purpose of designing a classification model with this goal is to determine whether the recognized text needs to be merged into paragraphs.

[0149] More specifically, in terms of input feature selection, this embodiment uses features such as the text size of the upper and lower rows of characters, the left and right margins, the line spacing, the starting character type, and the ending character type to construct the input vector. In Figure 7(a), T1-T3 represent input features, and F1-F4 represent convolutional filters of different sizes or types, each designed to extract different text features from the input data.

[0150] Step S3032: When merging is required, a binary classification model is used to sequentially perform encoding, fusion, and decoding on the text feature vectors to obtain a merged paragraph.

[0151] Specifically, the schematic diagram of the binary classification model is shown in Figure 7(b). Using two fully connected layers to build a binary classification model, you can refer to the following content. Taking the Keras library in Python as an example, the binary classification model construction process is as follows:

[0152] 1. Import necessary libraries: You need to import relevant libraries in Keras;

[0153] 2. Build the model: Use the Sequential model, which is a linearly stacked layer model that can be added layer by layer.

[0154] 3. Add the first fully connected layer: The first fully connected layer maps the input features to the hidden layer. Assuming there are n_features input features, you can add a hidden layer with n_units neurons and use ReLU as the activation function.

[0155] 4. Add a second fully connected layer: The second fully connected layer maps the output of the hidden layer to the final output. For binary classification problems, the output layer usually has 1 neuron and uses the 'Sigmoid' activation function because it can output a value between 0 and 1, representing the probability of belonging to the positive class.

[0156] 5. Compile the model: When compiling the model, you need to select a loss function, an optimizer, and specify the evaluation metric. For binary classification problems, binary cross entropy loss (binary_crossentropy) and binary accuracy (accuracy) are commonly used.

[0157] 6. Training model: Use the training dataset to train the model.

[0158] 7. Evaluate the model: Use the test dataset to evaluate the performance of the model and finally obtain a binary classification model.

[0159] In an optional implementation, step S3032 includes:

[0160] In step d1, use the encoding layer to encode each paragraph. If using an RNN or LSTM, the word sequence of each paragraph can be encoded into a fixed-length vector. If using a Transformer model, the global information of the sequence can be retained, where:

[0161] 1) For RNN or LSTM: You can use a separate RNN or LSTM layer for each paragraph to obtain the contextual representation of each paragraph.

[0162] 2) For the Transformer model: The text sequences of the two paragraphs can be concatenated and then passed through the encoder of the Transformer model to obtain a context-aware representation.

[0163] Attention Mechanism: The attention mechanism is introduced to capture the relationship between different paragraphs. The attention mechanism can help the model identify which words or sentences determine the important lines of the merged paragraph.

[0164] Step d2: Input the encoded paragraph representation into the fusion layer for fusion. This can be achieved in many ways, for example:

[0165] 1) Interaction layer: An interaction layer can be designed to explicitly model the interaction information between two paragraphs.

[0166] 2) Concatenation and fully connected layers: The encoded representations of the two paragraphs are concatenated and then processed through one or more fully connected layers.

[0167] In step d3, a decoding layer is used to generate the merged paragraph. The decoding layer is a recurrent layer that generates the words of the merged paragraph one by one, or a Transformer decoder that takes into account the input context and previously generated words.

[0168] Step d3, Loss Function: During training, a loss function is required to evaluate the difference between the model-generated paragraphs and the actual merged paragraphs. Common loss functions include cross-entropy loss. Optimization and Training: Gradient descent or its variants (such as Adam) are used to optimize the model's parameters. During training, the model learns how to better encode, fuse, and decode paragraphs.

[0169] Step d4, post-processing: The merged text may require post-processing to improve readability, for example, correcting punctuation, removing unnecessary repetitions, etc., to finally obtain the merged paragraph.

[0170] In step S304, a multimodal feature extraction model is used to extract structured data from the text features, image features, and table features after the paragraphs are merged, to obtain a complete electronic document corresponding to the multimodal format document.

[0171] Specifically, the multimodal feature extraction model includes a named entity recognition module, an entity linking module, and a relationship extraction module. The above step S304 includes:

[0172] In step S3041, a named entity recognition module is used to identify multiple entities from the text features, image features, and table features after the paragraphs are merged, and corresponding annotation information and entity type are added to each entity.

[0173] Specifically, if Figure 9 As shown in the figure, Named Entity Recognition (NER) is an information extraction technology that aims to identify entities with specific meanings from text features. In this embodiment, entities refer to words or phrases in text that refer to specific objects or concepts, such as names of people, places, organizations, positions, and products.

[0174] Step S3042 : calculating the relevance between the plurality of entities and the preset candidate entity objects, and linking the entities to corresponding preset candidate entity objects using an entity linking module based on the relevance.

[0175] Specifically, an entity object refers to a specific entry or record in a knowledge base, representing a specific object or concept in the real world, and usually includes the following:

[0176] The name of an entity: such as the name of a person, a place, or an organization;

[0177] Attributes: Various properties or characteristics associated with an entity, such as a person's date of birth, the latitude and longitude coordinates of a location, the year an organization was founded, etc.

[0178] Description: A brief description or overview of the entity object, which may include information such as its history, function, and importance.

[0179] Calculating the relevance of multiple entities to preset candidate entity objects includes:

[0180] Suppose there is an entity e and a preset candidate entity set C, where each candidate entity c∈C has a corresponding label l c , the relevance S(e, c) between entity e and candidate entity c can be expressed by the following formula:

[0181] S(e, c) = ∑ f∈F w f ·f(e, c) (1);

[0182] Among them, S(e, c) is the relevance score between entity e and candidate entity c; F is a set of feature functions that extract the relevant information between the entity and the candidate entity, w f is the weight of the feature function f, indicating the importance of the feature in the scoring. f(e, c) is the result of applying the feature function f to entity e and candidate entity c. The feature function f may include but is not limited to the following:

[0183] String similarity: such as edit distance (Levenshtein distance), Jaccard similarity or cosine similarity.

[0184] Contextual match: How well the words surrounding the entity match the contextual words of the candidate entity.

[0185] Semantic similarity: the semantic distance between an entity and a candidate entity in the knowledge graph.

[0186] Entity type consistency: whether the entity and the candidate entity are of the same type.

[0187] Weight w f Learning can be done through empirical setting, grid search, or using machine learning algorithms such as logistic regression, SVM, or deep learning models.

[0188] like Figure 9The structure shown in the figure represents a combination of deep learning models: BERT processing: each word is input into the BERT model to obtain the context representation vector of each word; BiLSTM processing: the vector output by BERT is input into the BiLSTM model to further extract the relationship features between words; CRF processing: the output of BiLSTM is passed to the CRF model, and the CRF model is used to predict the label of each word; BERT provides powerful context information, BiLSTM further extracts sequence dependency features, and CRF can consider the dependency between labels, thereby improving the accuracy of annotation. This model combination performs well in Chinese NLP tasks, especially in tasks that require understanding complex context and sequence dependency.

[0189] B-Title: This tag is used to indicate the start of an entity, and this entity is a title. For example, when processing the references of a book, "B-Title" can be used to mark the beginning of the book title. In sequence tagging, if an entity is a title and it is the first word or character of the entity, then it will be marked as B-Title. I-Title: This tag indicates a part of an entity, and this entity is a title. If an entity is a title and it is not the first word or character of the entity, then it will be marked as I-Title. Here, "I" stands for "Inside", meaning that this word or character is located inside the title entity. B-Addressee: Similar to B-Title, this tag is used to indicate the start of an entity, but this entity refers to the recipient's address. When processing letters, emails, or other documents, the recipient's name and address can be regarded as an entity, and its beginning part will be marked as B-Addressee. I-Addressee: This tag indicates a part of an entity, and this entity is the recipient's address. If an entity is a recipient address and is not the first word or character of the entity, then it will be marked as I-Addressee.

[0190] like Figure 10 The process shown describes a deep learning model that combines text and image information to perform entity recognition and entity linking tasks;

[0191] Text: This step represents the raw text data input. In entity recognition and linking tasks, text is the basic data source, which contains the entity information that needs to be recognized and linked.

[0192] Glove Embeddings: Glove (Global Vectors for Word Representation) is a word embedding model that converts words in a text into fixed-size vector representations. These vectors capture the semantic information of the words and are used in subsequent model processing. In this step of the pipeline, the original text is converted into Glove word embeddings.

[0193] Candidate Generation: Candidate generation is the first step in entity recognition, aiming to identify possible entities from text. In this step, the model generates a list of candidate entities based on the text and image information. This may involve using text analysis techniques to identify named entities and image recognition techniques to identify possible entities in images.

[0194] Entity Ranking: After candidate generation, the generated candidate entities need to be ranked to determine which entities are most likely to match the mentions in the text. The entity ranking step usually involves a scoring mechanism that evaluates the candidate entities based on various features such as contextual information, frequency of entity occurrence, etc.

[0195] Textual Linking Results: The textual linking results step is to link the identified entities with entities in an external knowledge base or database. The goal of this step is to find the corresponding database entity for the entity mention in the text, thereby obtaining more context and attributes of the entity.

[0196] Cost Matrix: In optimization problems, a cost matrix is ​​a matrix that represents the cost of matching elements from one set to another. In entity recognition, data clustering, or other scenarios where matching elements is required, each row represents an element in the source set, each column represents an element in the target set, and each element (i, j) in the matrix represents the cost of matching the i-th element in the source set to the j-th element in the target set. This cost can be based on various distance metrics, such as Euclidean distance, Manhattan distance, etc.

[0197] Gromov-Wasserstein Distance, representing the Gromov-Wasserstein distance: The Gromov-Wasserstein distance is a method for measuring the distance between two probability distributions, which extends the classic Wasserstein distance (also known as the Earth Mover distance). This distance is very useful for comparing the distribution shapes of different data sets, especially when the structure or distribution of the data set cannot be simply measured using the standard Euclidean distance. The Gromov-Wasserstein distance takes into account the structural similarity between data sets and calculates the distance by optimizing a transportation plan that describes how to "move" one distribution to another with the least cost.

[0198] Transport Plan: A transport plan is a mathematical concept used in optimization problems, particularly when computing the Wasserstein distance. It describes how to redistribute "matter" or "resources" from one probability distribution to another with minimal total cost. When computing the Gromov-Wasserstein distance, a transport plan defines the optimal matching from one dataset to another, taking into account the similarities between data points and their relative importance in their respective datasets.

[0199] Images: This is the input stage of the process and refers to the raw image dataset. These images can come from various sources such as a camera, a database, or the internet.

[0200] Transformer: The Transformer is a deep learning architecture based on the self-attention mechanism, originally developed for natural language processing tasks. However, due to its powerful feature extraction and modeling capabilities, it has also been widely used in computer vision. In this step, the Transformer model is used to process the input image and extract image features.

[0201] Image Embedding: Similar to text, image embedding is the process of converting image content into a vector representation. This is typically accomplished using a convolutional neural network (CNN) or other image recognition models. Image embedding captures visual features in an image and can be combined with textual information in subsequent steps.

[0202] Pairwise Distance: Pairwise distance refers to a distance metric calculated between each pair of elements in a dataset. In image analysis, this usually means that for each pair of images in an image dataset, a numerical measure of similarity or difference is calculated between them. This distance can be based on a comparison of image content (such as color, texture, shape), or it can be based on the Euclidean distance, cosine similarity, or other similarity measures between image feature vectors. The result of the pairwise distance calculation can be represented in matrix form, where each element of the matrix represents a measure of the distance between a pair of images.

[0203] Visual Linking Results: Visual linking results refer to the results obtained after performing an image analysis task, which could be identifying objects, scenes, or relationships between images. In the context of visual linking, this typically involves identifying entities in an image and establishing relationships between them.

[0204] For example, if the task is to identify and link different views of the same object in a set of images, the visual linking results will include: a list of identified objects; associations between objects, such as which object appears in which images; and information about the object's location in the image, which may include bounding boxes or keypoints.

[0205] If the entity linking module is processing the word "Paris" in a text, it might select an entity object from the knowledge base that represents the French city of Paris. This entity object might contain the following information:

[0206] Unique identifier: For example, in DBpedia this might be http: / / dbpedia.org / page / Paris:

[0207] Entity Name: Paris;

[0208] Attributes: such as population, area, year of establishment, etc.

[0209] Relationship: such as Paris is the capital of France, the relationship between Paris and France, etc.;

[0210] Description: A short description of Paris, such as "Paris is the capital of France, famous for its fashion, cuisine, and tourist attractions like the Eiffel Tower.

[0211] In step S3043, a relationship extraction module is used to extract the grammatical or semantic relationship between any two entities from the preset candidate entity objects after the link is completed, and multiple entities are merged according to the grammatical or semantic relationship to obtain a complete electronic document corresponding to the multimodal format document.

[0212] For example, Figure 11As shown in the figure, H refers to the hidden states output by the BERT model. BERT processes the input text through a multi-layer Transformer encoder. Each layer outputs a hidden state vector that contains contextual information about the input text. For each input token (word or subword unit), BERT outputs a multi-dimensional hidden state vector.

[0213] Fully-connected: A fully connected layer (also called a dense layer or linear layer) is a neural network layer in which each neuron is connected to all neurons in the previous layer. In the context of BERT, fully connected layers are typically used to process the hidden state vector to extract or transform features.

[0214] Average, or average calculation, refers to the pooling operation on the hidden state vector output by BERT, usually taking the average of all hidden state vectors. This operation can merge the representation of each token into a single fixed-size vector, which helps to encode the overall semantic information of the text into a compact representation.

[0215] Fully-connected+activation: This indicates that an activation function is applied after the fully connected layer. Activation functions (such as ReLU, tanh, or sigmoid) are used to introduce nonlinear factors, allowing neural networks to learn and simulate more complex functions.

[0216] Fully-connected: Another fully connected layer to further transform or process the feature vectors generated in the previous steps.

[0217] Softmax: The softmax function is a commonly used activation function used in multi-classification tasks. It converts a real-valued vector into a probability distribution, where each element represents the probability of belonging to the corresponding category. In the context of BERT, softmax is often used in the final layer of classification tasks to determine the probability of each category.

[0218] The process can be expressed as follows:

[0219] H: BERT processes input text and outputs a hidden state vector.

[0220] Average: Calculate the average value of the hidden state vector to obtain a fixed-size vector representation.

[0221] Fully-connected: This vector representation is transformed through one or more fully connected layers.

[0222] Fully-connected+activation: Apply activation function after the fully connected layer to increase nonlinearity.

[0223] Fully-connected: Perform further feature transformation.

[0224] Softmax: At the end of the classification task, the softmax function is used to output the predicted probability of each category.

[0225] If two entities are given, it is necessary to obtain the grammatical or semantic connection between the entities, that is, the relationship, through the content of the two corresponding entity objects. Specifically, the connection is obtained through the content of the entity objects to realize relationship extraction. Finally, all the relationships between the entities are extracted, and multiple entities are merged according to the grammatical or semantic connection to obtain a complete electronic document corresponding to the multimodal format document.

[0226] In step S305, the data synchronization model is used to store the complete electronic document in a designated directory to form an electronic test report document, which is then transferred to the corresponding business database. Specifically, the data synchronization module stores the scanned data in a designated directory. The OCR recognition system monitors this directory and performs OCR on newly added documents, then saves and transfers the data to the business database.

[0227] The test report element extraction method provided in this embodiment can restore recognized text to paragraphs and, in paragraph merging and splitting tasks, determine whether two adjacent lines of text need to be merged, thereby merging multiple lines of text into paragraphs. The extraction of structured data is achieved using a named entity recognition module, an entity linking module, and a relationship extraction module, ultimately converting multimodal documents into corresponding complete electronic documents. This improves extraction accuracy and enables accurate conversion of multimodal paper documents into electronic documents, facilitating the storage of electronic documents corresponding to test reports.

[0228] As one or more specific application embodiments of the present invention, a further detailed description of the method for extracting elements from a test report provided by the present invention specifically includes:

[0229] Step S1: Scanning a paper document using a scanning device;

[0230] Step S2: Divide the documents into single-modality and multi-modality formats, and model them separately to extract features from the elements in the documents;

[0231] Step S3: For multimodal documents, the text in the document is recognized using OCR text recognition technology. Step S3 includes the following steps:

[0232] Step S31: Use the high-precision Chinese scene text recognition model SVTR to generate pseudo labels for unlabeled data and set a confidence threshold;

[0233] Step S32: mining and screening highly reliable unlabeled data and their pseudo labels;

[0234] Step S33: Use the filtered unlabeled data and its pseudo labels for small model training to improve the accuracy of the small model and its generalization ability in real scenarios.

[0235] Step S4: Divide the document in the form of an image into regions using a layout analysis model to locate key regions therein;

[0236] Step S5: extracting text features and labels of the document through a text classification model; Step S5 includes the following steps:

[0237] Step S51: Use ALBERT+Seq2Seq+Attention as the multi-label text classification framework;

[0238] Step S52: ALBERT is used as a tool for extracting text features to extract text features;

[0239] Step S53, Seq2Seq+Attention is a downstream task framework used to extract key information of text features.

[0240] Step S6: Calculate whether two adjacent lines of text need to be merged using the paragraph merging model; Step S6 includes the following steps:

[0241] Step S61: adopting a paragraph merging model implemented by a multi-layer neural network;

[0242] Step S62: Based on the text feature vector, a two-classification model is constructed using two fully connected layers;

[0243] Step S63: Merge and split paragraphs, calculate whether two adjacent lines of text need to be merged, and design a classification model based on this goal.

[0244] Step S7: Recognize the table document using the table recognition model; Step S7 includes the following steps:

[0245] Step S71: The coordinates of the single-line text in the table image are detected by the single-line text detection model, and then sent to the table recognition model to obtain the recognition result;

[0246] Step S72: The SLANet model can obtain the table structure information and cell coordinate information;

[0247] Step S73: combining the coordinates of the single-line text, the recognition result, and the coordinates of the cell to obtain the recognition result of the cell;

[0248] Step S74: construct an electronic form based on the cell recognition results and the form structure.

[0249] Step S8: extracting structured and semi-structured data through a multimodal feature extraction model and saving the data into a structured database;

[0250] Step S9: The scanned data is stored in a designated directory through the data synchronization module. The OCR recognition can monitor this directory and perform OCR text recognition on newly added documents. The data is then saved and transferred to the business database. Step S8 includes the following steps:

[0251] Step S81: Identify entities from the text through named entity recognition (NER), and add corresponding annotation information and entity types;

[0252] Step S82: Entity Linking (EL) is used to select a set of candidate entity objects from the knowledge base based on the given entity name, and then the entities are linked to the correct entity objects based on similarity.

[0253] Step S83: Given two entities, obtain the grammatical or semantic connection between the entities, that is, the relationship, so as to achieve relation extraction (RE).

[0254] The following technical difficulties may be encountered during the implementation process, which can be resolved in the following ways:

[0255] (1) Traditional OCR text recognition only performs image correction and text recognition on documents in pictures. However, OCR text recognition for energy industry documents has a complex recognition algorithm model with rich types, complex fields and diverse layouts, and there is a risk of performance not meeting expectations.

[0256] Solution: Try to design the model based on the mature algorithm framework in the deep learning community. When optimizing the algorithm model, adopt the principle of multiple rounds of iterative optimization and a small amount of optimization in each round to avoid detours.

[0257] (2) Deep learning requires a large amount of labeled data, and the labeling work requires a lot of manpower and time.

[0258] Solution: Leverage technology to save manpower. For example, software can automatically generate large numbers of structured tables from tabular annotation data, which can then be converted into images. For the annotated sample data required by knowledge extraction models, an iterative model of annotation, model training, model prediction, manual correction of prediction results, and model training can be adopted. Each round of correction of prediction results is equivalent to obtaining a larger annotated dataset. However, as the accuracy of model predictions gradually increases, the amount of manual correction work required gradually decreases.

[0259] (3) For some specific features to be extracted, there may be a small number of samples or zero samples, resulting in poor performance of the model in this category.

[0260] Solution: Design targeted unsupervised learning tasks based on the corpus type and the characteristics of the elements to be extracted. Through large-scale unsupervised learning on relevant domain corpora, the model can initially grasp and understand the grammatical, semantic, and contextual characteristics of the corpus type. Simultaneously, data augmentation operations are used to expand the corresponding sample size. Ultimately, the model can achieve few-shot or zero-shot learning on the elements. In the prior art, scanned documents usually have no paragraph information after OCR parsing, and only have lines of text, which is very unfavorable for the subsequent extraction of elements. The element extraction method for the test report provided in this embodiment can restore the recognized text to paragraphs, and judge whether two adjacent lines of text need to be merged in the paragraph merging and splitting tasks, and design a two-classification model based on this goal, so that multiple lines of text can be merged into paragraphs, which is convenient for subsequent element extraction work; the element extraction method for the test report provided in this embodiment can not only extract the text features of the document, but also obtain the position information of the text through the table recognition model, and obtain the pixel information of the entire image according to the layout parsing model, and finally realize the element extraction in the text format, so that in this way, the accuracy and versatility are greatly improved.

[0261] In this embodiment, a device for extracting elements from a test report is also provided. The device is used to implement the above-mentioned embodiments and preferred embodiments, and the details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0262] This embodiment provides a device for extracting elements from a test report, such as Figure 12 Shown, including:

[0263] The document acquisition and screening module 1201 is used to acquire scanned documents of multiple paper test reports and screen out multimodal format documents from the scanned documents.

[0264] The feature recognition and extraction module 1202 is used to recognize and extract text features, image features, and table features in a multimodal document based on an optical character recognition technology model.

[0265] The paragraph merging module 1203 is used to merge the extracted text features into paragraphs.

[0266] The structured data extraction module 1204 is used to extract structured data from the text features, image features and table features after the paragraphs are merged using a multimodal feature extraction model to obtain a complete electronic document corresponding to the multimodal format document.

[0267] In some optional implementations, the optical character recognition technology model includes a text recognition model, a layout parsing model, a text classification model, and a table recognition model, and the feature recognition and extraction module 1202 includes:

[0268] The text feature extraction unit is used to identify text features in a multimodal format document using a text recognition model and extract the identified text features using the text classification model.

[0269] The image feature extraction unit is used to divide the image documents in the multimodal format documents into regions using a layout analysis model, and locate and extract image features.

[0270] The table feature extraction unit is used to identify and extract table features in a multimodal format document using a table recognition model to obtain an electronic version of the table.

[0271] In some optional implementations, the text classification model is a deep learning model, and the text feature extraction unit includes:

[0272] The unlabeled data and pseudo-label recognition subunit is used to generate unlabeled data and pseudo-labels corresponding to text regions in multimodal format documents using a text recognition model.

[0273] The confidence judgment subunit is used to filter out unlabeled data and pseudo labels that meet a preset confidence threshold from the unlabeled data and pseudo labels.

[0274] The training subunit is used to train the preset small model using unlabeled data and pseudo labels that meet the preset confidence threshold to identify the text area.

[0275] The text feature extraction subunit is used to extract text features in the text area using a deep learning model.

[0276] In some optional implementations, the image feature extraction unit includes:

[0277] The first image feature subunit is configured to identify key areas containing text content from an image document in a multimodal document in a rule-based manner, and extract image features from the key areas.

[0278] The second image feature subunit is used to match image documents in the multimodal document in a template-based manner, and to identify and extract image features.

[0279] In some optional implementations, the table recognition model includes a cell positioning module, a table structure analysis module, a text recognition module, and a table construction module; and the table feature extraction unit includes:

[0280] The cell row and column identification subunit is used to locate the cell boundaries of the table in the multimodal format document using the cell positioning module and identify the rows and columns of the cells.

[0281] The structured subunit is used to determine the layout and structure of the table based on the rows and columns of the cells using a table structure analysis module to obtain a plurality of structured cells.

[0282] The cell text recognition subunit is used to recognize the text in each cell using a text recognition module and extract the text content in the cell.

[0283] The electronic form construction subunit is used to fill the text content in the cell into multiple structured cells using the table construction module to obtain an electronic form.

[0284] In some optional implementations, the paragraph merging module 1203 includes:

[0285] The paragraph merging judgment unit is used to convert text features into feature vectors of each line of text using word embedding technology, and to judge whether the feature vectors of text in two adjacent lines need to be merged.

[0286] The paragraph merging unit is used to use a binary classification model to perform encoding, fusion and decoding on the text feature vectors in sequence when merging is required to obtain the merged paragraph.

[0287] In some optional implementations, the multimodal feature extraction model includes a named entity recognition module, an entity linking module, and a relationship extraction module; the structured data extraction module 1204 includes:

[0288] The entity recognition unit is used to use the named entity recognition module to identify multiple entities from the text features, image features and table features after the paragraphs are merged, and add corresponding annotation information and entity type to each entity.

[0289] The entity linking unit is used to calculate the relevance between multiple entities and preset candidate entity objects, and based on the relevance, use the entity linking module to link the entities to corresponding preset candidate entity objects.

[0290] The relationship extraction unit is used to extract the grammatical or semantic connection between any two entities from the preset candidate entity objects that have completed the link using the relationship extraction module, and fuse multiple entities according to the grammatical or semantic connection to obtain a complete electronic document corresponding to the multimodal format document.

[0291] In some optional embodiments, the element extraction device of the test report further includes:

[0292] The data synchronization storage module is used to store the complete electronic document in a designated directory using a data synchronization model to form an electronic document of a test report, and transmit the electronic document of the test report to the corresponding business database.

[0293] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0294] The element extraction device of the test report in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0295] The embodiment of the present invention also provides a computer device having the above Figure 12 The element extraction device of the test report is shown.

[0296] See also Figure 13 , Figure 13 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 13As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 13 A processor 10 is taken as an example.

[0297] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0298] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0299] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0300] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0301] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 13 The bus connection is taken as an example.

[0302] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0303] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0304] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0305] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for extracting elements from a test report, characterized in that: The method comprises: Obtain scanned documents of multiple paper test reports, and filter out multimodal format documents from the scanned documents; Identify and extract text features, image features, and table features in multimodal documents based on the optical character recognition technology model; Merge the extracted text features into paragraphs; A multimodal feature extraction model is used to extract structured data from the text features, image features and table features after the paragraphs are merged to obtain a complete electronic document corresponding to the multimodal format document.

2. The method according to claim 1, characterized in that The optical character recognition technology model includes a text recognition model, a layout parsing model, a text classification model and a table recognition model; The optical character recognition technology model is used to identify and extract text features, image features and table features in multimodal format documents, including: Using the text recognition model to recognize text features in a multimodal format document, and using the text classification model to extract the recognized text features; The layout parsing model is used to divide the image document in the multimodal format document into regions, and locate and extract image features; The table recognition model is used to identify and extract table features in multimodal format documents to obtain an electronic version of the table.

3. The method according to claim 2, characterized in that The text classification model is a deep learning model; The text recognition model is used to recognize text features in a multimodal format document, and the text classification model is used to extract the recognized text features, including: The text recognition model is used to generate unlabeled data and pseudo labels corresponding to text regions in a multimodal format document; Filter out the unlabeled data and pseudo labels that meet a preset confidence threshold from the unlabeled data and pseudo labels; The preset small model is trained using unlabeled data and pseudo labels that meet the preset confidence threshold to identify the text area; A deep learning model is used to extract text features in text areas.

4. The method according to claim 2, characterized in that: The layout parsing model is used to divide the image document in the multimodal document into regions, and locate and extract image features, including: Using a rule-based approach to identify key areas containing text content from image documents in a multimodal document, and extracting image features from the key areas; or, A template-based approach is used to match image documents in multimodal documents to identify and extract image features.

5. The method according to claim 2, characterized in that: The table recognition model includes a cell positioning module, a table structure analysis module, a text recognition module and a table construction module; The table recognition model is used to recognize and extract table features in a multimodal format document to obtain an electronic version of the table, including: A cell positioning module is used to locate the cell boundaries of the table in the multimodal format document and identify the rows and columns of the cells; A table structure analysis module is used based on the rows and columns of cells to determine the layout and structure of the table, and a plurality of structured cells are obtained; Use the text recognition module to recognize the text in each cell and extract the text content in the cell; The table construction module is used to fill the text content in the cell into multiple structured cells to obtain an electronic version of the table.

6. The method according to claim 1, characterized in that The step of merging the extracted text features into paragraphs includes: Using word embedding technology to convert the text features into a text feature vector for each line, and determining whether the text feature vectors of two adjacent lines need to be merged; When merging is required, a binary classification model is used to sequentially perform encoding processing, fusion processing and decoding processing on the text feature vectors to obtain a merged paragraph.

7. The method according to claim 1, characterized in that The multimodal feature extraction model includes a named entity recognition module, an entity linking module and a relationship extraction module; The multimodal element extraction model is used to extract structured data from the text features, image features and table features after the paragraphs are merged to obtain a complete electronic document corresponding to the multimodal format document, including: The named entity recognition module is used to identify multiple entities from the text features, image features and table features after the paragraphs are merged, and the corresponding annotation information and entity type are added to each entity; Calculating the relevance of multiple entities to preset candidate entity objects, and linking the entities to corresponding preset candidate entity objects using an entity linking module based on the relevance; The relationship extraction module is used to extract the grammatical or semantic connection between any two entities from the preset candidate entity objects that have completed the link, and multiple entities are merged according to the grammatical or semantic connection to obtain a complete electronic document corresponding to the multimodal format document.

8. The method according to claim 1, characterized in that The method further comprises: The data synchronization model is used to store the complete electronic document in a designated directory to form an electronic document of a test report, and the electronic document of the test report is transmitted to a corresponding business database.

9. A device for extracting elements from a test report, characterized in that: The device comprises: A document acquisition and screening module, used to acquire scanned documents of multiple paper test reports, and screen out multimodal format documents from the scanned documents; A feature recognition and extraction module is used to recognize and extract text features, image features, and table features in multimodal format documents based on an optical character recognition technology model; A paragraph merging module is used to merge the extracted text features into paragraphs; The structured data extraction module is used to extract structured data from the text features, image features and table features after the paragraphs are merged using a multimodal feature extraction model to obtain a complete electronic document corresponding to the multimodal format document.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for extracting elements from a test report according to any one of claims 1 to 8 by executing the computer instructions.

Images