A method and system for extracting structured information

Abstract

The present invention discloses a method and system for extracting structured information, including: preprocessing the data of a tender document to obtain a preprocessed document; parsing the document structure of the preprocessed document; identifying the structure information in the preprocessed document after the document structure is parsed based on optical character recognition technology and natural language processing technology; classifying the extracted structure information into the corresponding document title level to generate a structure object tree table. The method can automatically and intelligently extract structured information from unstructured documents, improve the extraction efficiency and utilization efficiency of information, and also improve the accuracy of information extraction.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a method and system for extracting structured information. Background Art

[0002] At present, with the development of information technology, a large amount of document data exists in an unstructured form. Especially in the field of bidding, a large number of bidding documents exist in an unstructured form, which contains a wealth of key information, such as bidder qualification information, team member information, historical performance information, etc. At present, the extraction and utilization efficiency of this information is low, which seriously affects the extraction, processing and utilization efficiency of historical data. How to extract structured information from unstructured documents has become a problem that needs to be solved urgently.

[0003] Traditional manual sorting and screening is not only inefficient, but also easily affected by subjective factors, making it difficult to ensure the accuracy and consistency of information extraction. In addition, with the accelerated development of enterprise digital transformation, the number of unstructured documents has increased dramatically, and traditional manual information extraction methods can no longer meet the growing needs of information processing.

[0004] Existing technical means also have certain limitations in processing unstructured data. Traditional machine learning methods require a large amount of labeled data for training and are strictly dependent on document structure. They cannot process table, form, or image data, and have limitations when processing complex document content and long text input content. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems in the above-mentioned technology to a certain extent. To this end, the first purpose of the present invention is to propose a method for extracting structured information, which can automatically and intelligently extract structured information from unstructured documents, thereby improving the extraction efficiency and utilization efficiency of information, and also improving the accuracy of information extraction.

[0006] The second object of the present invention is to provide a system for extracting structured information.

[0007] To achieve the above object, a first embodiment of the present invention provides a method for extracting structured information, comprising:

[0008] Perform data preprocessing on the tender document to obtain a preprocessed file;

[0009] Parse the document structure of the preprocessed file;

[0010] Based on optical character recognition technology and natural language processing technology, identify the structural information in the pre-processed file after document structure analysis;

[0011] The extracted structure information is classified into the corresponding document title level to generate a tree table of structure objects.

[0012] According to some embodiments of the present invention, data preprocessing is performed on a bid document to obtain a preprocessed document, including:

[0013] The tender document is preprocessed, including removing invalid data, garbled data, element parsing, word segmentation, part-of-speech tagging, and named entity recognition to obtain a preprocessed file.

[0014] According to some embodiments of the present invention, based on optical character recognition technology and natural language processing technology, identifying the structure information in the preprocessed file after document structure parsing includes:

[0015] Based on optical character recognition technology and natural language processing technology, an algorithm model for extracting structural information and classifying document elements is constructed to determine the data category included in the preprocessed file; the data category includes at least one of text elements, picture elements and form elements;

[0016] Based on the structure extraction model, determine the structure objects and structure attributes included in each data category;

[0017] The structure information in the preprocessing file is determined according to the structure objects and structure attributes included in each data category.

[0018] According to some embodiments of the present invention, based on the structure extraction model, determining the structure objects and structure attributes included in each data category includes:

[0019] Convert the text elements into plain text and perform text preprocessing to obtain plain text segments; perform keyword retrieval and entity detection on the plain text segments, and determine a number of text pairs of the first keyword and the associated second keyword based on the context and the position information of the text elements in the document structure;

[0020] Based on the optical character recognition technology, the image elements are converted into text content, and the original image content is sliced ​​for object file storage to generate the object file address; based on the context keyword search, several first keywords in the text content are determined and associated with the text pairs of the second keyword;

[0021] For form elements, the recognized text content slices are formatted based on optical character recognition technology, and the original table slices are stored in object files to generate object file addresses; based on contextual keyword retrieval, several first keywords in the formatted text content are determined and associated with text pairs of the second keywords;

[0022] Filtering and semantically analyzing the first keyword and the second keyword to obtain a valid keyword text pair;

[0023] Convert the format of valid keyword text pairs to obtain JSON key-value pairs;

[0024] Assemble and summarize the JSON key-value pairs to obtain multi-layer JSON key-value pair pages and full text, and determine the structure objects and structure attributes included in each data category.

[0025] According to some embodiments of the present invention, the extracted structure information is classified into the corresponding document title level in the preset structure, and a structure object tree table is generated, including:

[0026] Based on the BERT deep learning model, the extracted structural information is classified into the corresponding document title level in the preset structure; the document title level is a multi-level structure, including business part and technical part;

[0027] Generate a tree table of structure objects based on the processed document title hierarchy.

[0028] According to some embodiments of the present invention, the method further includes: sending the structure object tree table to a user terminal;

[0029] or

[0030] An information sharing connection is established with the demand terminal based on the API interface, and the structure object tree table is sent to the demand terminal.

[0031] According to some embodiments of the present invention, the API interface is also used to:

[0032] Transmit the structure information to the demand terminal;

[0033] The cut unstructured data elements are synchronously transmitted to the demand terminal in the form of objects; the unstructured data elements include original picture elements and original chart elements.

[0034] According to some embodiments of the present invention, before determining the structure objects and structure attributes included in each data category based on the structure extraction model, the method further includes:

[0035] Get the test sample set and the corresponding sample structure information;

[0036] Input the test sample set into the structure extraction model to determine the preset structure information;

[0037] Compare the sample structure information with the preset structure information;

[0038] Calculate the The first structure parameter of the sample structure information of the test sample :

[0039]

[0040] in, For the The number of structure objects extracted from the sample structure information of the test sample; For the The number of structure attributes contained in a structure object; For the The normalized value of the i-th structure attribute included in the structure object; For the The normalized value of the jth structure attribute included in the structure object;

[0041] According to the first structural parameter of the sample structure information of each test sample, the matching degree with the second structural parameter of the preset structural information is calculated. :

[0042]

[0043] in, is the number of test samples included in the test sample set; The preset structure information output by the structure extraction model includes the second structure parameter of the a-th test sample;

[0044] The matching degree is compared with the preset matching degree. When it is determined that the matching degree is greater than the preset matching degree, it indicates that the structure extraction model is qualified; otherwise, it indicates that the structure extraction model is unqualified and the model parameters need to be adjusted.

[0045] To achieve the above object, a second embodiment of the present invention provides a system for extracting structured information, comprising:

[0046] A processing module is used to perform data preprocessing on the tender document to obtain a preprocessed file;

[0047] Parsing module, used to parse the document structure of the preprocessed file;

[0048] A recognition module, which is used to recognize the structural information in the preprocessed file after the document structure is parsed based on optical character recognition technology and natural language processing technology;

[0049] The generation module is used to classify the extracted structure information into the corresponding document title level and generate a tree table of structure objects.

[0050] The present invention proposes a method and system for extracting structured information, which have the following beneficial effects: 1. Improving the degree of automation of information extraction, reducing manual intervention and lowering costs; 2. Improving the accuracy and efficiency of information extraction to meet the needs of large-scale text processing; 3. Through structured information, it is convenient to store, retrieve and further analyze and process information; 4. The system has good scalability and can adapt to different business scenarios and needs; 5. It provides a user-friendly interface and rich API interfaces to facilitate user use and integration.

[0051] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0052] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0054] Figure 1 is a flow chart of a method for extracting structured information according to an embodiment of the present invention;

[0055] Figure 2 is a schematic diagram of preprocessing and identifying and extracting structural information in a preprocessing file according to an embodiment of the present invention;

[0056] Figure 3 is a schematic diagram of analyzing and processing structure objects and structure attributes included in each data category in a bid document according to an embodiment of the present invention;

[0057] Figure 4 is a block diagram of a system for extracting structured information according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0059] like Figure 1 As shown, the first embodiment of the present invention proposes a method for extracting structured information, including steps S1-S4:

[0060] S1. Preprocess the data of the tender document to obtain a preprocessed file;

[0061] S2, parsing the document structure of the preprocessed file;

[0062] S3, based on optical character recognition technology and natural language processing technology, identifying the structural information in the preprocessed file after document structure analysis;

[0063] S4. Classify the extracted structure information into corresponding document title levels and generate a structure object tree table.

[0064] The working principle of the above technical solution: the tender document is an unstructured document. The file format data in the tender document (PDF format and DOC format belong to unstructured formats) are segmented, and the document structure is analyzed and classified. The tender document data is preprocessed to obtain a preprocessed file, which is convenient for improving the accuracy of subsequent analysis and processing.

[0065] In this embodiment, the document structure is parsed for the preprocessed file, including document conversion, content management, text parsing engine optimization, etc. This process is implemented according to the specific technology stack (such as Python) and file format (such as Markdown, HTML, PDF, etc.). Determine the file format, different formats have different parsing methods and tools. For Markdown files: Use Python's Markdown library or PyPandoc (through Pandoc) to parse Markdown files and convert them into HTML or a format that is easier to handle. For HTML files: Use Python's BeautifulSoup or lxml library to parse HTML documents and extract the DOM tree. For PDF files: Use libraries such as PyPDF2 or PDFMiner.six to extract text content.

[0066] In this embodiment, based on optical character recognition technology and natural language processing technology, the structure information in the preprocessing file is identified, and the entity recognition and relationship extraction technology in the natural language processing technology are used to identify the structure information in the preprocessing file. First, the named entity recognition technology is used to identify the entities in the text, including the structure name, etc. Then, the relationship between the entities, such as the composition relationship or attribute relationship between the structures, can be identified by using the relationship extraction technology. The structure information in the preprocessing file can be effectively identified and extracted.

[0067] In this embodiment, the extracted structure information is classified into the corresponding document title level, a structure object tree table is generated, and the attributes and objects in the document data are extracted as the extraction part of the structure information. The structure information data is classified into the corresponding first-level, second-level, and third-level titles as the classification part of the structure information. Identify the document title and subtitle: Use natural language processing technology to identify the title and subtitle in the document, which can be extracted by the format, style or keywords of the title. Extract structure information: Traverse the document content and identify each structure and its attribute information, which can be achieved using entity recognition and relationship extraction technology. Classify into the corresponding document title level: Associate each structure and its attribute information with the corresponding document title level, and the classification relationship can be determined based on the location of the structure, the relationship between the title and the subtitle. Generate a structure object tree table: Generate a structure object tree table based on the classification relationship between the document title level and the structure information, so as to more intuitively display the organizational structure of the structure information in the document.

[0068] The beneficial effects of the above technical solution are as follows: an unstructured document element classification algorithm model is constructed through artificial intelligence and machine learning technologies, and structured information is extracted from key document information through optical character recognition and other technologies, so as to realize the extraction and classification of key information in the tender document, wherein the image recognition model can convert image information, information extraction can intelligently identify structural objects and attributes in the tender document, and element classification can intelligently classify structural objects, extract structured information entity objects and attributes in the document data through unstructured tender document information data, and output them according to the structural information classification and grading template. It can automatically and intelligently extract structured information from unstructured documents, improve the efficiency of information extraction and utilization, and also improve the accuracy of information extraction.

[0069] According to some embodiments of the present invention, data preprocessing is performed on a bid document to obtain a preprocessed document, including:

[0070] The tender document is preprocessed, including removing invalid data, garbled data, element parsing, word segmentation, part-of-speech tagging, and named entity recognition to obtain a preprocessed file.

[0071] The working principle and beneficial effects of the above technical solution: Eliminate invalid data: Eliminate invalid data in the bid document, such as headers, footers, blank lines, special characters, etc. Process garbled data: For garbled data that appears, use the corresponding character encoding method to decode or replace characters to repair the garbled data. Element parsing: Parse the bid document and extract important element information, such as titles, paragraphs, lists, tables, etc. Word segmentation: Use a word segmentation tool (such as the Embedding Chinese word segmentation library) to perform Chinese word segmentation on the bid document and divide the text into word units. Part-of-speech tagging: Tag the words after word segmentation, that is, determine the part of speech of each word in the sentence, such as nouns, verbs, adjectives, etc. Named entity recognition: Use a named entity recognition tool (such as TexSmart) to perform named entity recognition on the preprocessed text to identify entities such as names, place names, and organization names. Obtain a preprocessed file: After the above steps, save the preprocessing result as a new file, which already contains information such as word segmentation, part-of-speech tagging, and named entity recognition.

[0072] like Figure 2 As shown, according to some embodiments of the present invention, based on optical character recognition technology and natural language processing technology, identifying the structure information in the pre-processed file after document structure parsing includes:

[0073] Based on optical character recognition technology and natural language processing technology, an algorithm model for extracting structural information and classifying document elements is constructed to determine the data category included in the preprocessed file; the data category includes at least one of text elements, picture elements and form elements;

[0074] Based on the structure extraction model, determine the structure objects and structure attributes included in each data category;

[0075] The structure information in the preprocessing file is determined according to the structure objects and structure attributes included in each data category.

[0076] The working principle of the above technical solution: In this embodiment, an algorithm model for extracting structural information and classifying document elements is constructed based on optical character recognition technology and natural language processing technology to determine the data categories included in the preprocessed file, including: extracting features from the preprocessed file through optical character recognition technology and natural language processing technology; defining data categories; extracting structured information from the preprocessed file using natural language processing technology, such as entity recognition, keyword extraction, etc., according to the defined data categories; training the model based on machine learning algorithms such as naive Bayes, support vector machine (SVM), deep learning (such as LSTM, BERT, etc.), and obtaining an algorithm model for extracting structural information and classifying document elements. The information in the tender document is extracted and classified, wherein the information extraction can intelligently identify the structural objects and attributes in the tender document, and the element classification can intelligently classify the structural objects into the corresponding categories at each level.

[0077] According to the different objects and attributes of the text, pictures, forms and other data in the document, analyze the part to which the element data belongs, and output the part name containing the hierarchical relationship of the paragraph title

[0078] Example:

[0079] Input: Complete bid of Shenzhen XX Service Co., Ltd.

[0080] Output: Industry Qualification Certificate: Business Section - Qualification Certification Materials - Qualification Certification Documents

[0081] Situation of the project leader to be assigned: Technical part - project team

[0082] In this embodiment, the structure extraction model is trained based on a deep learning model and can identify structure objects and structure attributes, including the relationship between structure objects. Structural object extraction: The user inputs the bid document data, and the model needs to automatically extract the structure object based on the text data and return the index of the file location where the object is located.

[0083] Example:

[0084] Input: Complete bid of Shenzhen XX Service Co., Ltd.

[0085] Output: {"Information Security Management System Certification Certificate":{"Object File Address":"……":,"Object Location Index":[18,20],……}} (file location index information of the object);

[0086] Attribute extraction of structure objects: The user inputs the document data of the tender document, and the model needs to automatically extract the attributes of the structure object based on the recognized text data.

[0087] Example:

[0088] Input: Complete bid of Shenzhen XX Service Co., Ltd.

[0089] Output: {"Information Security Management System Certification Certificate": {"Initial Certification Date": "July 20, 2001","Validity Period": "June 16, 2024",...}} (object attribute information)

[0090] The beneficial effects of the above technical solution are as follows: the structure information in the preprocessing file is determined according to the structure objects and structure attributes included in each data category. The structure information data is classified into corresponding first-level, second-level, and third-level titles as the classification part of the structure information, and output according to the structure information classification and grading template.

[0091] like Figure 3 As shown, according to some embodiments of the present invention, based on the structure extraction model, determining the structure objects and structure attributes included in each data category includes:

[0092] The text elements are converted into plain text, and the text is preprocessed to obtain plain text fragments; the plain text fragments are subjected to keyword retrieval and entity detection, and a number of first keywords (attributes) and associated second keyword (value) text pairs are determined based on the context and the position information of the text elements in the document structure; the image elements are converted into text content based on optical character recognition technology, and the original image content is sliced ​​for object file storage to generate the object file address; based on context keyword retrieval, a number of first keywords (attributes) in the text content are determined and associated with the second keyword (value) text pairs;

[0093] For form elements, the recognized text content slices are formatted based on optical character recognition technology, and the original table slices are stored in object files to generate object file addresses; based on contextual keyword retrieval, several first keywords (attributes) in the formatted text content are determined and associated with the second keyword (value) text pairs;

[0094] Filter and semantically analyze the first keyword (attribute) and the second keyword (value) to obtain a valid keyword text pair;

[0095] Convert the format of valid keyword text pairs to obtain JSON key-value pairs;

[0096] Assemble and summarize the JSON key-value pairs to obtain multi-layer JSON key-value pair pages and full text, and determine the structure objects and structure attributes included in each data category.

[0097] The working principle of the above technical solution: In this embodiment, the text elements are converted into plain text, and the text is preprocessed to obtain plain text fragments; Text preprocessing: The text elements are converted into plain text, irrelevant information such as HTML tags, special characters, stop words, etc. are removed, and the text is cleaned to retain the text content. Plain text fragment extraction: The preprocessed text is segmented or sentence-processed to obtain plain text fragments, each of which can be a word, phrase, sentence, etc. Keyword extraction: Use keyword extraction algorithms, such as Fine-Grained NER, TF-IDF, TextRank, LDA and other methods to extract keywords from plain text fragments and determine the key concepts of each text fragment.

[0098] Perform keyword search and entity detection on plain text fragments. Based on the context and the location information of text elements in the document structure, determine several first keywords (attributes) and associate second keyword (value) text pairs, including: Keyword search: Based on the extracted keywords, establish a keyword index. You can use inverted index and other technologies to accelerate the search and query the index for text fragments containing specific keywords. Use pre-trained NER models or custom models to identify entities in the text, such as names of people, places, institutions, time, date, etc. NER tools can usually identify and classify key entities in the text. Context-based analysis: Using the results of dependency syntactic analysis, identify key verbs or noun phrases in the sentence, which can usually be used as the first keyword (attribute). Associated value: Find text fragments that have a direct relationship with these keywords (such as objects, modifiers, etc.) as the second keyword (value). Consider the document structure: If the text contains structural information such as titles and paragraphs, this information can be used to enhance the extraction of keywords and entity pairs. For example, noun phrases in titles may be more likely to be important attributes. Associating a second keyword (value) text pair: In a text segment containing a first keyword, a second keyword is determined based on a contextual relationship, and the first keyword and the second keyword form an associated text pair.

[0099] Example: Suppose there is a text: "Apple Inc. released its latest iPhone 15 in 2023."

[0100] Named Entity Recognition: Identify “Apple Inc.” as an organization name and “iPhone 15” as a product name.

[0101] Dependency syntactic analysis: Recognize that "released" is a verb phrase, its subject is "Apple", its object is "iPhone 15", and "in 2023" is a time adverbial.

[0102] Extract keywords and entity pairs: the first keyword can be "company name", and its value is "Apple Inc."; the second keyword can be "product release time", and its value is "2023"; the third keyword can be "product name", and its value is "iPhone 15".

[0103] Through the above steps, keywords and entity pairs can be effectively extracted from the text, providing strong support for subsequent text analysis, information extraction and other tasks.

[0104] In this embodiment, for picture elements, picture conversion and content slicing, optical character recognition technology is used to perform text recognition on picture elements, and the picture is converted into a text file. Then the text content is sliced, and it can be cut according to units such as paragraphs, sentences, and words. File storage and generation of object file address: The sliced ​​text content is stored as a file, and the object file address is generated in the file system for subsequent processing. Context keyword retrieval: Context keyword retrieval is performed in the sliced ​​text content to find and extract keywords. Natural language processing tools such as word segmentation, part-of-speech tagging, entity recognition and other technologies can be used to extract keywords from the text. Determine the first keyword: According to the results of the context keyword retrieval, determine the first keyword, which is usually a word with a higher weight in the text or has important meaning in a specific context. Associate the second keyword text pair: In the text segment containing the first keyword, based on the context relationship, determine the second keyword, and form an associated text pair with the first keyword and the second keyword.

[0105] In this embodiment, the form elements and form content are sliced ​​and stored in files: the text content in the form elements is identified using optical character recognition technology, and the content is sliced ​​and stored as a file. The content can be cut according to the structure, fields, etc. of the form and saved as a text file. Generate an object file address: save the sliced ​​text content as a file, and generate a second file address for subsequent processing and use. Context keyword search: perform context keyword search in the text content in the second file, find keywords and extract them. Natural language processing technology can be used to extract keywords and analyze the context of the text. Determine the second keyword: According to the results of the context keyword search, determine the second keyword, which is a word with a higher weight in the text or an important meaning in a specific context. Based on the context keyword search, determine the text pairs of several first keywords (attributes) in the formatted text content and associate the second keyword (value).

[0106] In this embodiment, the screening conditions are set: text length: limit the text length and exclude content that is too long or too short; keyword weight: weight evaluation is performed according to the importance of the keyword in the text to screen high-weight keywords; contextual relationship: consider the contextual relationship between keywords to ensure that there is a logical connection between keyword text pairs. Filtering is performed based on the screening conditions to obtain valid keyword text pairs.

[0107] In this embodiment, the valid keyword text pairs are converted into JSON key-value pairs: for each valid keyword text pair, the keywords and text are represented as JSON key-value pairs respectively to form a data structure in JSON format. The JSON library or tool in the programming language is used to easily convert the text information into JSON format. Multi-layer JSON key-value pairs are assembled: according to the hierarchical relationship of the text content, a multi-layer JSON key-value pair structure is constructed to ensure that the information of each data category is fully reflected. The parent-child relationship between the layers can be determined according to the logical relationship between the text content to form a complete multi-layer JSON data structure. Determine the structure objects and structure attributes included in each data category: traverse the entire JSON data structure, determine the structure attributes contained in each data category (object), and further refine the data structure. Confirm the attribute content of each structure object and the association relationship between the attributes. Paginate and process the JSON key-value pairs in full text: display the JSON key-value pairs in pages as needed to facilitate viewing and navigating the data. For full-text data, all JSON key-value pairs can be integrated and displayed to form a complete data set. Comprehensive summary of data structure: Comprehensively summarize the structure objects and attributes in each data category to form a detailed data structure document, including the attributes, data types, descriptions and other information of each object.

[0108] The beneficial effect of the above technical solution is that it is easy to accurately determine the structure objects and structure attributes included in each data category.

[0109] According to some embodiments of the present invention, the extracted structure information is classified into corresponding document title levels, and a structure object tree table is generated, including:

[0110] Based on the BERT deep learning model, the extracted structural information is classified into the corresponding document title level in the preset structure; the document title level is a multi-level structure, including business part and technical part;

[0111] Generate a tree table of structure objects based on the processed document title hierarchy.

[0112] The working principle and beneficial effects of the above technical solution: Document title hierarchy classification: Use the BERT model to process the document content and classify the extracted structural information into the corresponding document title hierarchy, including the business part, technical part, etc. The document vector or feature representation generated by BERT is combined with the association between the document title and the structural information to classify the structural information. Generate a tree table of structural objects: Integrate the structural information of each part according to the classification results of the document title hierarchy. According to the hierarchical relationship of the document title hierarchy, construct a tree table of structural objects to form a parent-child structural relationship to show the hierarchy and organizational relationship of the structural information. Use a data structure represented by a tree structure to characterize the structural information of each part.

[0113] According to some embodiments of the present invention, the method further includes: sending the structure object tree table to a user terminal;

[0114] or

[0115] An information sharing connection is established with the demand terminal based on the API interface, and the structure object tree table is sent to the demand terminal.

[0116] The working principle and beneficial effects of the above technical solution: The demand terminal includes various business system terminals that use structured data, which is a shared system. The structure object tree table can be sent to each business demand terminal, and the user can view the extraction results through the business system interface, and can also call the system function through the API interface to achieve integration with other systems.

[0117] The API interface is also used to:

[0118] Transmit the structure information to the demand terminal;

[0119] The cut unstructured data elements are synchronously transmitted to the demand terminal in the form of objects; the unstructured data elements include original picture elements and original chart elements.

[0120] According to some embodiments of the present invention, before determining the structure objects and structure attributes included in each data category based on the structure extraction model, the method further includes:

[0121] Get the test sample set and the corresponding sample structure information;

[0122] Input the test sample set into the structure extraction model to determine the preset structure information;

[0123] Compare the sample structure information with the preset structure information;

[0124] Calculate the The first structure parameter of the sample structure information of the test sample :

[0125]

[0126] in, For the The number of structure objects extracted from the sample structure information of the test sample; For the The number of structure attributes contained in a structure object; For the The normalized value of the i-th structure attribute included in the structure object; For the The normalized value of the jth structure attribute included in the structure object;

[0127] According to the first structural parameter of the sample structure information of each test sample, the matching degree with the second structural parameter of the preset structural information is calculated. :

[0128]

[0129] in, is the number of test samples included in the test sample set; The preset structure information output by the structure extraction model includes the second structure parameter of the a-th test sample;

[0130] The matching degree is compared with the preset matching degree. When it is determined that the matching degree is greater than the preset matching degree, it indicates that the structure extraction model is qualified; otherwise, it indicates that the structure extraction model is unqualified and the model parameters need to be adjusted.

[0131] The working principle and beneficial effects of the above technical solution: The normalized value represents the proportion of the i-th structure attribute to the The weight (importance) of all the structure attributes of a structure object. Before determining the structure objects and structure attributes included in each data category based on the structure extraction model, verify whether the structure extraction model is qualified. The specific method is: obtain the test sample set and the corresponding sample structure information; input the test sample set into the structure extraction model to determine the preset structure information; compare the sample structure information with the preset structure information; calculate the first The first structural parameter of the sample structure information of each test sample is used to represent the real structural feature of the test sample. The second structural parameter is the predicted structural feature of the test sample based on the output of the model. According to the first structural parameter of the sample structure information of each test sample, the matching degree with the second structural parameter of the preset structure information is calculated, and the matching degree is compared with the preset matching degree. When it is determined that the matching degree is greater than the preset matching degree, it indicates that the structure extraction model is qualified; otherwise, it indicates that the structure extraction model is unqualified and the model parameters need to be adjusted. In this way, the structure extraction model can be effectively adjusted to improve the accuracy of the model, thereby improving the accuracy of determining the structure objects and structure attributes included in each data category.

[0132] According to some embodiments of the present invention, an algorithm model for extracting structural information and classifying document elements is constructed based on optical character recognition technology and natural language processing technology to determine the data category included in the preprocessing file, including:

[0133] Character recognition and layout analysis based on optical character recognition technology. Character recognition converts characters in an image into editable text. Layout analysis identifies structural elements such as tables, paragraphs, and titles in a document.

[0134] Perform named entity recognition, syntactic analysis, and semantic analysis based on natural language processing technology. Named entity recognition: Identify entities with specific meanings from text, such as names of people, places, and institutions. Syntactic analysis: Analyze the structure of sentences and identify components such as subjects, predicates, and objects. Semantic analysis: Understand the meaning of text and identify themes and emotions in text.

[0135] Based on the information extracted by optical character recognition technology and natural language processing technology, classification algorithms are used to classify document elements. Classification algorithms include decision trees, random forests, support vector machines, neural networks, etc. Classification is based on the structural features of the document (such as titles, paragraphs, tables, etc.), text content features (such as keywords, themes, emotions, etc.), or a combination of the two.

[0136] The beneficial effect of the above technical solution is that it is easy to accurately determine the data category included in the preprocessing file.

[0137] like Figure 4 As shown, the second embodiment of the present invention provides a system for extracting structured information, including:

[0138] A processing module is used to perform data preprocessing on the tender document to obtain a preprocessed file;

[0139] Parsing module, used to parse the document structure of the preprocessed file;

[0140] A recognition module, which is used to recognize the structural information in the preprocessed file after the document structure is parsed based on optical character recognition technology and natural language processing technology;

[0141] The generation module is used to classify the extracted structure information into the corresponding document title level and generate a tree table of structure objects.

[0142] The beneficial effects of the above technical solution are as follows: an unstructured document element classification algorithm model is constructed through artificial intelligence and machine learning technologies, and structured information extraction of key document information is performed through optical character recognition technology to achieve extraction and classification of key information in tender documents, wherein the image recognition model can convert image information, information extraction can intelligently identify structural objects and attributes in tender documents, and element classification can intelligently classify structural objects. Through unstructured tender document information data, structured information entity objects and attributes in document data are extracted, and output according to the structural information classification and grading template. It can automatically and intelligently extract structured information from unstructured documents, improve the efficiency of information extraction and utilization, and also improve the accuracy of information extraction.

[0143] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for extracting structured information, characterized in that: include: Perform data preprocessing on the tender document to obtain a preprocessed file; Parse the document structure of the preprocessed file; Based on optical character recognition technology and natural language processing technology, identify the structural information in the pre-processed file after document structure analysis; Classify the extracted structure information into the corresponding document title level and generate a tree table of structure objects; Based on optical character recognition technology and natural language processing technology, the structural information in the preprocessed file after document structure analysis is identified, including: Based on optical character recognition technology and natural language processing technology, an algorithm model for extracting structural information and classifying document elements is constructed to determine the data category included in the preprocessed file; the data category includes at least one of text elements, picture elements and form elements; Based on the structure extraction model, determine the structure objects and structure attributes included in each data category; Determine the structure information in the preprocessing file according to the structure objects and structure attributes included in each data category; Based on the structure extraction model, determine the structure objects and structure attributes included in each data category, including: Convert the text elements into plain text and perform text preprocessing to obtain plain text segments; perform keyword retrieval and entity detection on the plain text segments, and determine a number of text pairs of the first keyword and the associated second keyword based on the context and the position information of the text elements in the document structure; Based on the optical character recognition technology, the image elements are converted into text content, and the original image content is sliced ​​for object file storage to generate the object file address; based on the context keyword search, several first keywords in the text content are determined and associated with the text pairs of the second keyword; For form elements, the recognized text content slices are formatted based on optical character recognition technology, and the original table slices are stored in object files to generate object file addresses; based on contextual keyword retrieval, several first keywords in the formatted text content are determined and associated with text pairs of the second keywords; Filtering and semantically analyzing the first keyword and the second keyword to obtain a valid keyword text pair; Convert the format of valid keyword text pairs to obtain JSON key-value pairs; Assemble and summarize JSON key-value pairs to obtain multi-layer JSON key-value pair pages and full text, and determine the structure objects and structure attributes included in each data category; Before determining the structure objects and structure attributes included in each data category based on the structure extraction model, it also includes: Get the test sample set and the corresponding sample structure information; Input the test sample set into the structure extraction model to determine the preset structure information; Compare the sample structure information with the preset structure information; Calculate the The first structure parameter of the sample structure information of the test sample : in, For the The number of structure objects extracted from the sample structure information of the test sample; For the The number of structure attributes contained in a structure object; For the The normalized value of the i-th structure attribute included in the structure object; For the The normalized value of the jth structure attribute included in the structure object; According to the first structural parameter of the sample structure information of each test sample, the matching degree with the second structural parameter of the preset structural information is calculated. : in, is the number of test samples included in the test sample set; The preset structure information output by the structure extraction model includes the second structure parameter of the a-th test sample; The matching degree is compared with the preset matching degree. When it is determined that the matching degree is greater than the preset matching degree, it indicates that the structure extraction model is qualified; otherwise, it indicates that the structure extraction model is unqualified and the model parameters need to be adjusted.

2. The method for extracting structured information according to claim 1, characterized in that: Perform data preprocessing on the tender document to obtain a preprocessed file, including: The tender document is preprocessed, including removing invalid data, garbled data, element parsing, word segmentation, part-of-speech tagging, and named entity recognition to obtain a preprocessed file.

3. The method for extracting structured information according to claim 1, characterized in that: The extracted structure information is classified into the corresponding document title level, and a structure object tree table is generated, including: Based on the BERT deep learning model, the extracted structural information is classified into the corresponding document title level in the preset structure; the document title level is a multi-level structure, including business part and technical part; Generate a tree table of structure objects based on the processed document title hierarchy.

4. The method for extracting structured information according to claim 1, characterized in that: Also includes: Sending the structure object tree table to the user terminal; or An information sharing connection is established with the demand terminal based on the API interface, and the structure object tree table is sent to the demand terminal.

5. The method for extracting structured information according to claim 4, characterized in that: The API interface is also used to: Transmit the structure information to the demand terminal; The cut unstructured data elements are synchronously transmitted to the demand terminal in the form of objects; the unstructured data elements include original picture elements and original chart elements.

6. A system for extracting structured information, characterized in that: include: A processing module is used to perform data preprocessing on the tender document to obtain a preprocessed file; Parsing module, used to parse the document structure of the preprocessed file; A recognition module, which is used to recognize the structural information in the preprocessed file after the document structure is parsed based on optical character recognition technology and natural language processing technology; A generation module is used to classify the extracted structure information into corresponding document title levels and generate a tree table of structure objects; Based on optical character recognition technology and natural language processing technology, the structural information in the preprocessed file after document structure analysis is identified, including: Based on optical character recognition technology and natural language processing technology, an algorithm model for extracting structural information and classifying document elements is constructed to determine the data category included in the preprocessed file; the data category includes at least one of text elements, picture elements and form elements; Based on the structure extraction model, determine the structure objects and structure attributes included in each data category; Determine the structure information in the preprocessing file according to the structure objects and structure attributes included in each data category; Based on the structure extraction model, determine the structure objects and structure attributes included in each data category, including: Convert the text elements into plain text and perform text preprocessing to obtain plain text segments; perform keyword retrieval and entity detection on the plain text segments, and determine a number of text pairs of the first keyword and the associated second keyword based on the context and the position information of the text elements in the document structure; Based on the optical character recognition technology, the image elements are converted into text content, and the original image content is sliced ​​for object file storage to generate the object file address; based on the context keyword search, several first keywords in the text content are determined and associated with the text pairs of the second keyword; For form elements, the recognized text content slices are formatted based on optical character recognition technology, and the original table slices are stored in object files to generate object file addresses; based on contextual keyword retrieval, several first keywords in the formatted text content are determined and associated with text pairs of the second keywords; Filtering and semantically analyzing the first keyword and the second keyword to obtain a valid keyword text pair; Convert the format of valid keyword text pairs to obtain JSON key-value pairs; Assemble and summarize JSON key-value pairs to obtain multi-layer JSON key-value pair pages and full text, and determine the structure objects and structure attributes included in each data category; Before determining the structure objects and structure attributes included in each data category based on the structure extraction model, it also includes: Get the test sample set and the corresponding sample structure information; Input the test sample set into the structure extraction model to determine the preset structure information; Compare the sample structure information with the preset structure information; Calculate the The first structure parameter of the sample structure information of the test sample : in, For the The number of structure objects extracted from the sample structure information of the test sample; For the The number of structure attributes contained in a structure object; For the The normalized value of the i-th structure attribute included in the structure object; For the The normalized value of the jth structure attribute included in the structure object; According to the first structural parameter of the sample structure information of each test sample, the matching degree with the second structural parameter of the preset structural information is calculated. : in, is the number of test samples included in the test sample set; The preset structure information output by the structure extraction model includes the second structure parameter of the a-th test sample; The matching degree is compared with the preset matching degree. When it is determined that the matching degree is greater than the preset matching degree, it indicates that the structure extraction model is qualified; otherwise, it indicates that the structure extraction model is unqualified and the model parameters need to be adjusted.

Images