Artificial intelligence-based document structured extraction method and system

Abstract

The present invention provides a document structured extraction method and system based on artificial intelligence, aiming to solve the problem of incomplete and inaccurate information extraction in existing document processing technologies. First, an original document data set containing multiple document units is obtained and preprocessed to obtain a standardized document data set. Then, a pre-trained document feature extraction model is called to process the standardized document data to extract the content features and structural features of the document units. Based on the content features and structural features of the document units, structured parsing processing is performed to generate a structured element set containing subject elements, logical relationship elements and key information elements. Finally, the structured element set is integrated and optimized to generate a final structured document result containing element association relationships, thereby realizing the automated and structured extraction of document information, improving the efficiency and accuracy of document processing, and providing a new approach for the intelligent management and application of documents.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based document structured extraction method and system. Background Art

[0002] In today's information age, the volume of literature in fields such as academic research, technical reports, and business analysis is exploding. These documents not only cover a wide range of topics but also possess increasingly complex content structures, including chapter divisions, paragraph organization, embedded charts, and citation annotations. For researchers, business decision-makers, and the general public, extracting valuable information from this vast amount of literature quickly and accurately, while also understanding its inherent logical structure and key takeaways, has become a pressing challenge.

[0003] Traditional document processing methods rely primarily on manual reading and organization. This approach is not only time-consuming, labor-intensive, and inefficient, but is also susceptible to subjective factors such as personal knowledge background and reading habits, making it difficult to guarantee the accuracy and comprehensiveness of information extracted. Although natural language processing and machine learning technologies have made progress in document processing in recent years, existing document processing techniques mostly focus on surface-level extraction of document content, such as keyword extraction and abstract generation, while failing to deeply explore the structured information within the document. This structured information, such as chapter titles, logical relationships between paragraphs, and the location and association of key information points, is crucial for in-depth understanding of document content, building knowledge graphs, and realizing intelligent document applications. Summary of the Invention

[0004] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides a method for extracting structured documents based on artificial intelligence, the method comprising:

[0005] Acquire an original document data set, wherein the original document data set includes a plurality of document units, each document unit consisting of text content and meta information;

[0006] Preprocessing the original document data set to obtain a standardized document data set, wherein the standardized document data set includes text paragraphs and meta-information items in a unified format;

[0007] Calling a pre-trained document feature extraction model to process the standardized document data set to obtain content features and structural features of the document unit;

[0008] Performing structured parsing based on the content features and the structural features to generate a structured element set of the document unit, wherein the structured element set includes a subject element, a logical relationship element, and a key information element;

[0009] The structured element set is integrated and optimized to generate a final structured document result including element association relationships.

[0010] On the other hand, an embodiment of the present invention also provides an artificial intelligence-based document structured extraction system, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0011] Based on the above aspects, the embodiments of the present invention achieve in-depth mining and efficient utilization of original document data by comprehensively applying advanced technical means such as preprocessing, feature extraction, structured analysis and integrated optimization. It can automatically identify and accurately extract the subject elements in the document to ensure accurate grasp of the core content of the document; at the same time, through in-depth analysis of the logical structure of the document, it extracts logical relationship elements and reveals the logical context and argumentation process within the document; in addition, it can also effectively capture the key information elements in the document, and more importantly, it can construct complex correlations between these key information elements to form a complete structured document result, so that users can intuitively and comprehensively understand the content of the document, improve the efficiency and quality of document reading, and thus significantly improve the automation and intelligence level of document processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of the execution flow of the artificial intelligence-based document structured extraction method provided by an embodiment of the present invention.

[0013] Figure 2 Schematic diagram of exemplary hardware and software components of an artificial intelligence-based document structured extraction system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 This is a flow chart of an artificial intelligence-based document structured extraction method provided by an embodiment of the present invention. The artificial intelligence-based document structured extraction method is introduced in detail below.

[0015] Step S110: obtaining an original document data set, where the original document data set includes multiple document units, each of which consists of text content and meta-information.

[0016] In this embodiment, in order to obtain a collection of original document data, a diversified data collection strategy can be adopted. For example, documents can be collected from multiple authoritative and widely used academic resource channels, which include but are not limited to well-known comprehensive academic databases, specific databases in professional fields, open access academic platforms, and internal knowledge bases of universities and research institutions. For different data sources, adaptive collection methods can be used. For example, for databases that support API interfaces, specific program codes can be written to send requests to obtain document data based on the parameters and format requirements specified in the API documents. This method can efficiently and accurately obtain the required document information, including text content and meta-information. For websites that do not have API interfaces, web crawler technology can be used. The web crawler program will locate the location of the document data in the web page according to preset rules, parse the HTML or XML structure, and extract the text content and meta-information. During the collection process, in order to ensure the legality and compliance of the data, the terms of use and relevant laws and regulations of each data source will be followed.

[0017] When acquiring document data, documents in various formats are usually involved, such as PDF, DOC, TXT, etc. Documents in different formats have different storage structures and encoding methods. For documents in PDF format, since they usually contain complex typesetting information and possible encryption protection, a professional PDF parsing library can be used to convert their content into a processable text form. For documents in DOC format, the text content and meta-information can be extracted with the help of the automated interface of office software or specialized document parsing tools. During the collection process, the documents can be preliminarily screened to eliminate those that are incomplete, damaged, or do not meet the requirements of the research field, ensuring that the document units in the original document data set have a certain quality and relevance.

[0018] The text content of each document unit is the core information carrier of the document, and may include sections such as research background, experimental methods, results analysis, and conclusions. Metadata, on the other hand, provides additional descriptions and identifiers for the document, such as author name, institution affiliation, publication date, keywords, and document type (e.g., journal article, conference paper, research report, etc.). This meta information facilitates subsequent classification, retrieval, and analysis of the document. Through this comprehensive and meticulous collection and organization process, document data from various sources and formats is integrated to form a raw document data collection containing multiple document units.

[0019] Step S120: pre-processing the original document data set to obtain a standardized document data set, which contains text paragraphs and meta-information items in a unified format.

[0020] In this embodiment, the original document dataset contains documents with diverse sources and formats, resulting in inconsistent formats, redundant content, missing or scattered metadata, and other issues. Therefore, preprocessing is required to obtain a standardized document dataset. The preprocessing process primarily includes format conversion, denoising, segmentation, metadata field alignment, missing value filling, and association binding.

[0021] Step S121: performing format conversion processing on each document unit in the original document data set, converting text contents from different sources into a unified plain text format, and obtaining converted text contents.

[0022] In this embodiment, documents in different formats have significant differences in storage and presentation methods. In order to facilitate subsequent processing and analysis, they need to be converted into a unified plain text format. For documents in PDF format, a special PDF text extraction library is used. Its working principle is to parse the internal structure of the PDF file, identify the text objects therein, and convert them into plain text. During the parsing process, possible character encoding issues will be handled to ensure that the extracted text content is accurate. For documents in DOC format, the COM interface of office software or an open source document parsing library can be used to extract its content as plain text. During the extraction process, the formatting settings in the document, such as font, font size, color, paragraph format, etc., can be removed, retaining only the text information. For documents in other formats, corresponding conversion methods will also be used to ensure that the text content of all documents is converted into a unified plain text format. The converted text content will serve as the basis for subsequent processing, facilitating operations such as denoising and segmentation.

[0023] Step S122: De-noising the converted text content to remove redundant symbols, repeated paragraphs, and additional information irrelevant to the core content of the document, thereby obtaining de-noised text content.

[0024] In this embodiment, the converted text content may contain a large amount of redundant information, such as non-text symbols, repeated paragraphs, reference lists, appendix descriptions, copyright notices, etc. This information will interfere with subsequent analysis and processing, so denoising is required. The denoising process includes the following sub-steps:

[0025] Step S1221: Identify the symbol types in the converted text content and filter out non-text symbols and hyperlink tags.

[0026] In this embodiment, regular expressions are used to identify and filter non-text symbols and hyperlink tags. Regular expressions are a powerful text matching tool that can accurately identify various symbols and tags in the text by defining specific patterns. For example, a regular expression pattern is defined to match non-text symbols such as common punctuation marks, special characters, and HTML tags. For hyperlink tags, a pattern starting with "http" or "https" can be defined for matching. When traversing the text content, once a matching symbol or tag is found, it is deleted from the text. In order to ensure the accuracy and completeness of the filtering, the regular expression pattern can be tested and optimized for different types of documents to adapt to various possible situations.

[0027] Step S1222: extracting repeated character sequences from the text content, and calculating the frequency of occurrence of the repeated character sequences in the continuous text.

[0028] In this embodiment, a sliding window method is used to extract repeated character sequences from text content. An appropriate window size is set, which can be adjusted according to the characteristics of the document and processing requirements. The window slides from left to right in the text, one character at a time, intercepting the character sequence within the window. For each intercepted character sequence, the entire text can be searched and its number of occurrences counted. Then, based on the total length of the text and the number of times the character sequence appears, its frequency of occurrence is calculated. To improve computational efficiency, a data structure such as a hash table can be used to store the counted character sequences and their occurrence counts to avoid repeated calculations.

[0029] Step S1223: Redundancy determination is performed on repeated character sequences based on the frequency of occurrence, and repeated paragraphs whose frequency of occurrence exceeds a preset threshold are deleted.

[0030] In this embodiment, a frequency threshold is preset, which is determined based on a large amount of statistical analysis of literature data and actual processing experience. The calculated frequency of occurrence of each repeated character sequence is compared with the preset threshold. If the frequency of occurrence of a repeated character sequence exceeds the threshold, the paragraph containing the character sequence is considered redundant and is deleted from the text. In the judgment process, the length and contextual information of the character sequence can be considered to avoid accidentally deleting important content. For example, if a shorter character sequence appears frequently in the text, but it is a key expression of an important concept, the paragraph containing it will not be deleted.

[0031] Step S1224: Detect the additional information area in the text content, where the additional information area includes a reference list, appendix description, and copyright statement.

[0032] In this embodiment, the additional information area is detected by analyzing the structure and features of the text. For a reference list, there is usually a specific title identifier, such as "References". Using regular expressions or string matching algorithms, these title identifiers are searched in the text to determine the starting position of the reference list. For appendix descriptions and copyright statements, corresponding landmark words or format features are also searched, such as "Appendix", "Copyright", etc. Once these landmark information are found, the scope of the additional information area can be determined.

[0033] Step S1225: extracting the starting position and the ending position of the additional information area, and truncating the additional information located before the starting position and after the ending position in the text content.

[0034] In this embodiment, after determining the starting and ending positions of the additional information area, the additional information located before the starting position and after the ending position in the text content is truncated through string truncation. For example, string slicing operations in programming languages are used to extract the core text content to be retained based on the starting and ending positions, and the additional information is discarded. This truncation process ensures the integrity and coherence of the retained text content, avoiding the loss of important information.

[0035] Step S1226: performing space normalization processing on the truncated text content, replacing multiple consecutive space characters with a single space character, and generating denoised text content.

[0036] In this embodiment, the truncated text content is normalized using a regular expression or a simple string replacement algorithm. A regular expression can define a pattern that matches multiple consecutive space characters. When matching consecutive spaces are found in the text, they are replaced with a single space character. This process can make the text content more standardized and neat, facilitating subsequent text analysis and processing.

[0037] Step S123: Segment the denoised text content according to semantic delimiters to generate logically coherent text paragraphs.

[0038] In this embodiment, the semantic delimiter is determined based on the semantics and logical structure of the text. Common semantic delimiters include punctuation marks such as periods, exclamation marks, question marks, and some specific paragraph separators. The position of the semantic delimiter is identified by traversing the denoised text content. During the recognition process, the context of the punctuation marks and the integrity of the sentence can be considered to avoid mis-segmentation. For example, in some abbreviations or specific expressions, a period may not indicate the end of a sentence and requires special processing. According to the position of the semantic delimiter, the text content is divided into multiple paragraphs. After segmentation, each paragraph can be checked to ensure that the content within the paragraph is logically coherent to avoid paragraphs that are too long or too short or logically confusing. If a problem is found in a paragraph, it can be adjusted and merged based on the context information.

[0039] Step S124: Perform field alignment processing on the metadata of the document unit, and map the dispersedly stored metadata items to the preset standard metadata fields.

[0040] In this embodiment, the metadata of different document units may be stored in different field names and formats. In order to facilitate subsequent unified processing and analysis, field alignment processing is required. First, a set of preset standard metadata fields are defined. These fields include author, publication date, keywords, document type, etc. For the metadata of each document unit, its field name and content are analyzed and mapped to the standard metadata fields. For example, if the metadata of a document unit contains the "Author" field, and the standard metadata field is "Author", the value of the "Author" field is mapped to the "Author" field. During the mapping process, the semantic similarity of the field name and the consistency of the content can be considered. For fields that cannot be directly mapped, manual intervention or classification and mapping can be performed through machine learning algorithms.

[0041] Step S125: Perform missing value filling processing on the metadata after field alignment, and complete the missing metadata entries based on the statistical rules of metadata of document units of the same type.

[0042] In this embodiment, the meta-information after field alignment may contain missing values. In order to ensure the integrity of the meta-information, missing value filling processing needs to be performed. First, the document units are classified according to the type of document, such as journal articles, conference papers, research reports, etc. For each type of document unit, the value distribution and statistical laws of each field in its meta-information are counted. For example, the common value range and distribution frequency of the publication date in a certain type of document unit are counted. For missing meta-information items, they are filled in according to the statistical laws of this type of document unit. If the publication date of a certain document unit is missing, it can be estimated and filled in according to the average year or common year range of the publication date of this type of document unit. For some missing values that cannot be filled in by statistical laws, further information retrieval or manual supplementation can be performed.

[0043] Step S126: Associating and binding the text paragraphs and meta-information items that have logical coherence to generate a standardized document data set including a text paragraph set and a meta-information item set.

[0044] In this embodiment, by establishing a unique identifier, text paragraphs and meta-information items with logical coherence are associated and bound. A unique identifier is assigned to each document unit, which can be the document number, DOI (digital object identifier), etc. For a set of text paragraphs and a set of meta-information items, each text paragraph and the corresponding meta-information item are associated with the unique identifier. For example, the set of text paragraphs and the set of meta-information items of a certain document unit are stored in a data structure, with the unique identifier as the key and the set of text paragraphs and the set of meta-information items as the value. Through the above-mentioned association and binding process, a standardized document data set containing a set of text paragraphs and a set of meta-information items is formed.

[0045] Step S130: calling a pre-trained document feature extraction model to process the standardized document data set to obtain content features and structural features of the document unit.

[0046] In this embodiment, the pre-trained document feature extraction model is trained based on a large amount of annotated document data, which can effectively extract the content features and structural features of document units from the standardized document data set. The processing process includes the following steps:

[0047] Step S131: Input the text paragraphs in the standardized document data set into the text encoding module of the document feature extraction model, and generate paragraph semantic vectors through word embedding and context encoding operations.

[0048] In this embodiment, the text encoding module is a crucial component of the document feature extraction model, primarily responsible for converting text paragraphs into vector representations. First, a word embedding operation is performed to map each word in the text paragraph into a low-dimensional vector space. There are various word embedding methods, such as Word2Vec and GloVe. These methods, by learning from large amounts of text data, can capture semantic relationships between words, placing semantically similar words closer together in the vector space. During the word embedding process, each word is converted into a corresponding vector based on the word sequence in the text paragraph. Next, a context encoding operation is performed to further enrich the word vector representation by taking into account the contextual information of the word within the paragraph. Context encoding can be performed using models based on recurrent neural networks (RNNs), long short-term memory networks (LSTMs), or gated recurrent units (GRUs), or models based on the Transformer architecture, such as BERT. These models can capture long-range dependencies between words and generate word vectors that incorporate contextual information. Finally, the vectors of all words in the paragraph are aggregated to generate a paragraph semantic vector. The aggregation method can be average, maximum, or weighted sum, depending on the specific task and model requirements.

[0049] Step S132: input the meta-information items in the standardized document data set into the meta-information encoding module of the document feature extraction model, and generate a meta-information feature vector through category feature conversion and numerical feature normalization operations.

[0050] In this embodiment, the metadata entry contains multiple types of information, such as categorical features (document type, author affiliation, etc.) and numerical features (publication date, number of citations, etc.). The metadata encoding module processes different types of metadata differently. For categorical features, methods such as one-hot encoding or label encoding are used to convert them into vector representations. One-hot encoding creates a binary vector for each category, with only the corresponding category position set to 1 and the remaining positions set to 0. Label encoding assigns a unique integer label to each category. For numerical features, normalization is performed to eliminate dimensional differences between different features. Normalization methods can include min-max normalization or z-score normalization. Min-max normalization maps the values of numerical features to a fixed interval, such as [0, 1]. Z-score normalization converts the feature to a standard normal distribution based on its mean and standard deviation. Through categorical feature conversion and numerical feature normalization, the metadata entry is converted into a metadata feature vector.

[0051] Step S133: performing cross-dimensional correlation analysis on the paragraph semantic vector and the meta-information feature vector to generate a correlation feature vector containing the correlation relationship between the text and the meta-information.

[0052] In the present embodiment, paragraph semantic vector and meta information feature vector have represented the text content and meta information of document respectively, but there may be potential correlation relationship between them.In order to dig this correlation relationship, cross-dimensional correlation analysis is carried out. Attention mechanism can be used to calculate the attention weight between paragraph semantic vector and meta information feature vector, and attention weight has represented the importance of each vector element in the correlation relationship.By carrying out weighted summation to attention weight, paragraph semantic vector and meta information feature vector are merged, generate the correlation feature vector that comprises text and meta information correlation relationship, it can better reflect the overall characteristics of document.

[0053] Step S134: input the associated feature vector into the content feature extraction layer of the document feature extraction model, and extract content features reflecting the core content of the document through feature screening and dimension compression operations.

[0054] In this embodiment, the main task of the content feature extraction layer is to extract features that reflect the core content of the document from the associated feature vectors. First, a feature screening operation is performed. By calculating the importance score of each element in the associated feature vector, elements with higher importance scores are selected as candidate features. The importance score can be calculated using feature selection algorithms such as the chi-square test and information gain. Then, a dimensionality reduction operation is performed to compress the high-dimensional candidate feature vectors to a lower dimension to reduce computational complexity and storage costs. Dimensionality reduction methods include principal component analysis (PCA) and linear discriminant analysis (LDA). Principal component analysis achieves dimensionality reduction by finding the principal component directions of the data and projecting the data onto these principal components. Through feature screening and dimensionality reduction operations, content features that reflect the core content of the document are extracted, thereby summarizing the main themes and key information of the document.

[0055] Step S135: Input the paragraph semantic vector into the structural feature extraction layer of the document feature extraction model, and extract the structural features reflecting the document organization through paragraph sequence modeling and logical relationship recognition operations.

[0056] In this embodiment, the structural feature extraction layer is mainly responsible for extracting structural features that reflect the document organization from the paragraph semantic vector. The processing process includes the following sub-steps:

[0057] Step S1351: constructing a sequential index of paragraph semantic vectors, which is used to represent the original arrangement order of text paragraphs in the document.

[0058] In this embodiment, in order to retain the original arrangement order information of the text paragraphs, a sequential index of the paragraph semantic vectors is constructed. In the standardized document data set, each text paragraph has its specific position in the document. By recording the paragraph position corresponding to each paragraph semantic vector, a sequential index is constructed. The sequential index can be a simple sequence of integers, each integer corresponding to a paragraph semantic vector, indicating the order of the paragraph semantic vector in the document.

[0059] Step S1352: perform joint encoding processing on the paragraph semantic vector and the sequence index to generate a temporal feature vector containing sequence information.

[0060] In the present embodiment, in order to integrate the sequential information of the paragraph into the paragraph semantic vector, a joint encoding process is performed. A method based on a sequence model, such as LSTM or GRU, can be used to encode the paragraph semantic vector and the sequential index as input. During the encoding process, the model will consider the sequential information of the paragraph and learn the temporal relationship between the paragraphs. Through the joint encoding process, a temporal feature vector containing sequential information is generated, which can better reflect the structural characteristics of the document.

[0061] Step S1353: extracting the cosine similarity values between the semantic vectors of adjacent paragraphs, and calculating the change gradient of the cosine similarity values in consecutive paragraphs.

[0062] In the present embodiment, cosine similarity is a common indicator for measuring the similarity between two vectors. For adjacent paragraph semantic vectors, the cosine similarity value between them is calculated. The calculation of the cosine similarity value is based on the inner product of the vector and the module length of the vector, which reflects the similarity degree of the two vectors in the direction. After calculating the cosine similarity values of the adjacent paragraph semantic vectors, the change gradient of these cosine similarity values in the continuous paragraphs is further calculated. The change gradient represents the change speed and direction of the cosine similarity value in the paragraph sequence. By analyzing the change gradient, the change of the semantic cohesion between the paragraphs can be captured.

[0063] Specifically, to calculate the gradient of change in cosine similarity values, we traverse the sequence of cosine similarity values for adjacent paragraphs. For each cosine similarity value in the sequence, we calculate the difference between it and the previous cosine similarity value. This difference represents the gradient of change at that position. Calculating the gradient of change can help identify sudden changes in semantic cohesion between paragraphs, which may correspond to chapter transitions or topic changes in the document.

[0064] Step S1354: Identify logical transition points between paragraphs based on the change gradient. The logical transition points are used to divide the chapter boundary information in the document.

[0065] In this embodiment, logical transition points between paragraphs are identified by analyzing the gradient of change. A gradient threshold is set. When the gradient of a certain location exceeds the threshold, the location is considered a logical transition point. The setting of this gradient threshold is based on extensive literature data statistics and practical application experience. It can ensure the accuracy of chapter division while avoiding over-division or under-division.

[0066] Identifying logical transition points is crucial for dividing document chapter boundaries. Different chapters in a document typically have distinct themes and content structures. Transitions between chapters often involve significant semantic shifts, which are reflected in the gradient of cosine similarity values. By identifying logical transition points, documents can be divided into different chapters.

[0067] Step S1355: Perform attention mechanism processing on the temporal feature vector to generate an attention weight distribution reflecting the importance of the paragraph.

[0068] In this embodiment, the attention mechanism is a technology that automatically focuses on important parts of a sequence. The time series feature vector is input into the attention mechanism module, which calculates the attention weight for each element of the time series feature vector. The attention weight indicates the importance of each paragraph to the overall document structure and content.

[0069] The attention mechanism typically calculates the similarity between the query vector, key vector, and value vector. This similarity is then converted into a probability distribution using the softmax function. This probability distribution serves as the attention weight. By applying the attention mechanism to time series feature vectors, important paragraphs can be highlighted and unimportant ones suppressed, generating an attention weight distribution that reflects paragraph importance.

[0070] Step S1356: Construct a document structure tree based on the attention weight distribution and chapter boundary information, and the node features of the document structure tree serve as structural features reflecting the document organization method.

[0071] In the present embodiment, a document structure tree is constructed by combining attention weight distribution and chapter boundary information. First, the document is divided into different chapters according to the chapter boundary information, and each chapter serves as a node of the document structure tree. Then, a corresponding weight is assigned to each node according to the attention weight distribution, and the weight represents the importance of the chapter in the document.

[0072] When constructing the document structure tree, the hierarchical relationships between chapters are also considered. For subordinate chapters, they are structured as parent-child nodes, thus forming a tree-like structure. Node features in the document structure tree include node weights and the semantic characteristics of the chapters they represent. These node features serve as structural features reflecting the organization of the document and are used for subsequent structured parsing and analysis.

[0073] Step S136: Standardize the content features and structural features to generate a document feature set with a unified dimensional representation.

[0074] In this embodiment, since the content features and structural features are extracted through different methods and modules, in order to facilitate subsequent processing and analysis, the content features and structural features need to be standardized.

[0075] Through standardization, a document feature set with a unified dimensional representation is generated. This document feature set includes the content features and structural features of the document, and can more comprehensively reflect the overall characteristics of the document.

[0076] Step S140: Performing structured parsing processing based on content features and structural features to generate a structured element set of the document unit, which includes subject elements, logical relationship elements, and key information elements.

[0077] In this embodiment, the purpose of structured parsing is to extract valuable structured information from the content and structural features of the document to form a set of structured elements. Specifically, the following steps are included:

[0078] Step S141: Input the content features into the topic identification module, and generate topic elements through topic word extraction and topic distribution probability calculation operations. The topic elements include core topic words and topic coverage description.

[0079] In this embodiment, the topic identification module receives content features as input and generates topic elements through a series of operations. First, topic word extraction is performed. This can be done using statistical methods, such as the term frequency-inverse document frequency (TF-IDF) algorithm. This algorithm calculates the frequency of each word in a document and its inverse frequency across the entire document collection to determine word importance. Words with higher importance are selected as topic words. Deep learning-based methods, such as using a pre-trained language model, can also be used for topic word extraction.

[0080] After extracting the subject terms, we further calculate the topic distribution probability. This represents the probability of each topic appearing in the document and can be calculated through cluster analysis of content features or using a probabilistic topic model. Through the process of subject term extraction and topic distribution probability calculation, we generate a topic element consisting of core subject terms and a topic coverage description. The core subject terms summarize the main themes of the document, while the topic coverage description describes the distribution of each topic within the document.

[0081] Step S142: Input the structural features into the logic analysis module, and generate logical relationship elements through paragraph association analysis and chapter level identification operations. The logical relationship elements include the deductive relationship between paragraphs and the subordinate relationship between chapters.

[0082] In this embodiment, the logic parsing module receives structural features as input and generates logical relationship elements through paragraph association analysis and chapter level recognition operations. The specific steps are as follows:

[0083] Step S1421: extracting paragraph association features from the structural features, where the paragraph association features include semantic cohesion strength and content continuity index of adjacent paragraphs.

[0084] In this embodiment, the structural features include the association information between paragraphs, and paragraph association features are extracted by analyzing this information. Paragraph association features can be obtained by calculating the semantic similarity of adjacent paragraphs, the reference relationship between paragraphs, etc. The semantic cohesion strength reflects the degree of semantic closeness between adjacent paragraphs, and the content continuity index represents the coherence of the content between adjacent paragraphs. By extracting paragraph association features, the logical relationship between paragraphs can be better understood.

[0085] Step S1422: constructing a paragraph relationship graph based on the paragraph association features, wherein the edge weights of the paragraph relationship graph represent the degree of logical association between paragraphs.

[0086] In this embodiment, a paragraph relationship graph is constructed based on the extracted paragraph association features. The paragraph relationship graph is a graph structure in which nodes represent paragraphs and edges represent the logical relationships between paragraphs. The edge weights are determined by factors such as the semantic cohesion strength and content continuity index in the paragraph association features. A higher edge weight indicates a higher degree of logical connection between paragraphs.

[0087] Graph theory can be used to construct a paragraph relationship graph. For example, an adjacency matrix can be used to represent the structure of a graph. The paragraph relationship graph can intuitively display the logical relationship between paragraphs.

[0088] Step S1423: Perform community discovery processing on the paragraph relationship graph to identify paragraph communities with significant association relationships as document chapters.

[0089] In this embodiment, community discovery is a technique for identifying closely connected sets of nodes in a graph structure. A paragraph relationship graph is processed for community discovery, and by calculating the connection strength between nodes and the cohesion within the community, paragraph communities with significant associations are identified. These paragraph communities can be thought of as chapters in a document, as the paragraphs within them are closely logically connected, while the logical connections with paragraphs in other communities are relatively weak.

[0090] Community discovery can use a variety of algorithms, such as the Louvain algorithm, spectral clustering algorithm, etc. These algorithms can divide paragraphs into different communities based on the structure and edge weights of the paragraph relationship graph, thereby identifying the chapters of the document.

[0091] Step S1424: extracting chapter-level features from the structural features, where the chapter-level features include font size information and indentation depth information of the chapter title.

[0092] In this embodiment, the structural features also include chapter level information, and chapter level features can be obtained by extracting this information. Chapter level features can be obtained by analyzing the text structure and formatting information of the document, such as the font size and indentation depth of the chapter title. The font size and indentation depth are generally related to the level of the chapter, with larger fonts and smaller indentation depths generally indicating a higher-level chapter.

[0093] By extracting chapter-level features, we can understand the chapter structure and hierarchical relationship of the document, and provide a basis for determining the subordinate relationship between chapters.

[0094] Step S1425: Determine the subordinate relationship between chapters based on the chapter hierarchy feature, where the subordinate relationship includes the correspondence between the parent chapter and the child chapter.

[0095] In this embodiment, the subordinate relationship between chapters is determined based on the extracted chapter hierarchy features. The hierarchical relationship between chapters can be determined by comparing the font size and indentation depth of the chapter titles. If a chapter title has a smaller font size and a larger indentation depth, it is likely to be a sub-chapter of another chapter.

[0096] Determining the subordinate relationships between chapters allows us to construct a chapter hierarchy tree, where nodes represent chapters and edges represent the subordinate relationships between chapters. This tree clearly displays the chapter structure and hierarchical relationships of a document.

[0097] Step S1426: Perform matching verification processing on the paragraph community and chapter-level features in the paragraph relationship graph to generate logical relationship elements including the deductive relationship between paragraphs and the subordinate relationship between chapters.

[0098] In this embodiment, in order to ensure the consistency of paragraph communities and chapter-level features, a matching verification process is performed on them. The paragraph communities identified in the paragraph relationship graph are compared with the chapters determined based on the chapter-level features to check whether they correspond to each other.

[0099] If the characteristics of paragraph communities and chapter hierarchies are inconsistent, adjustments and corrections are necessary. For example, if a paragraph community contains content from multiple chapters at different levels, further analysis of the logical relationships between paragraphs is required to re-divide the paragraph community. Through matching and verification processing, logical relationship elements are generated that include deductive relationships between paragraphs and subordinate relationships between chapters. These logical relationship elements can accurately reflect the logical structure of the document.

[0100] Step S143: Input the content features and structural features into the key information extraction module, and generate key information elements through entity recognition and key sentence screening operations. The key information elements include core research conclusions and key experimental conditions.

[0101] In this embodiment, the key information extraction module receives content features and structural features as input and generates key information elements through entity recognition and key sentence screening. Entity recognition can utilize named entity recognition (NER) technology, which can identify entities in text, such as names of people, places, organizations, and experimental equipment. Key sentence screening can utilize machine learning-based methods such as support vector machines (SVMs) and decision trees, or deep learning-based methods such as long short-term memory networks (LSTMs) and convolutional neural networks (CNNs).

[0102] When screening key sentences, consider the importance of the sentence and its relevance to the topic. Sentences with high importance and strong relevance to the topic are selected as key sentences. The information contained in key sentences can serve as key information elements, such as core research conclusions and key experimental conditions.

[0103] Step S144: Conflict detection is performed on the subject elements, logical relationship elements, and key information elements to identify semantic contradictions between different elements.

[0104] In this embodiment, since the subject elements, logical relationship elements and key information elements are generated by different methods and modules, there may be semantic contradictions between them. In order to ensure the consistency of the structured element set, conflict detection processing is required.

[0105] Conflict detection can be achieved by comparing the semantic information between different elements. For example, if the topic mentioned in the subject element is inconsistent with the core research conclusion mentioned in the key information element, or if the chapter relationship described in the logical relationship element does not match the distribution of key experimental conditions in the key information element, then a semantic contradiction is considered to exist.

[0106] Step S145: Based on the semantic contradictions, the subject elements, logical relationship elements and key information elements are modified to generate a structured element set that passes the consistency check.

[0107] In this embodiment, after identifying semantic contradictions, the subject elements, logical relationship elements, and key information elements need to be revised. The revision process can be adjusted based on the specific contradiction. For example, if there is a contradiction between the subject elements and the key information elements, the results of the subject word extraction and key information extraction can be re-evaluated, and the content of the subject elements and key information elements can be adjusted.

[0108] If there are inconsistencies between logical relationship elements and key information elements, the logical relationships between paragraphs and the subordinate relationships between chapters can be reanalyzed and the content of the logical relationship elements adjusted. This correction ensures that there are no semantic inconsistencies between the subject elements, logical relationship elements, and key information elements in the structured element set, generating a structured element set that passes the consistency check.

[0109] Step S150: integrating and optimizing the structured element set to generate a final structured document result including element association relationships.

[0110] In this embodiment, the purpose of the integration and optimization process is to associate and optimize the elements in the structured element set to form a final structured document result containing the element association relationship. Specifically, the following steps are included:

[0111] Step S151: constructing a feature index of each element in the structured element set, wherein the feature index is used to record the unique identifier of the subject element, the logical relationship element and the key information element.

[0112] In this embodiment, to facilitate the management and association of each element in the structured element set, a feature index for each element is constructed. The feature index can use a data structure such as a hash table, with the unique identifiers of the subject element, logical relationship element, and key information element as the key and the specific content of the element as the value.

[0113] The unique identifier can be the name or number of the feature, and the feature index can be used to quickly locate and access each feature in the structured feature set.

[0114] Step S152: performing association analysis on the feature index to identify the semantic mapping relationship between the subject elements and the key information elements, where the semantic mapping relationship includes the correspondence between the subject words and the key experimental conditions.

[0115] In this embodiment, the semantic mapping relationship between the subject elements and the key information elements is identified by performing association analysis on the feature indexes. The corresponding relationship between them can be found by comparing the core keywords in the subject elements with the key experimental conditions and other information in the key information elements.

[0116] For example, if a subject word in the subject element is semantically related to a key experimental condition in the key information element, a mapping relationship can be established between them. Semantic mapping relationships can help understand the inherent connection between the subject and key information of a document.

[0117] Step S153: Identify the structural dependency between the logical relationship elements and the key information elements, where the structural dependency includes the distribution position relationship between the chapter affiliation and the core research conclusions.

[0118] In this embodiment, the structural dependency between the logical relationship elements and the key information elements is analyzed, and the relationship between them is found by observing the chapter subordination in the logical relationship elements and the distribution position of the core research conclusions in the key information elements.

[0119] For example, if a core research conclusion always appears in the child chapters under a parent chapter, then the relationship between chapter subordination and the distribution position of the core research conclusion can be established. Structural dependency can reflect the logical structure of the document and the distribution pattern of key information.

[0120] Step S154: constructing an element association graph based on the semantic mapping relationship and the structural dependency relationship, wherein the nodes of the element association graph are structured elements and the edges are the association relationships between elements.

[0121] In this embodiment, a feature association graph is constructed based on the identified semantic mapping relationships and structural dependency relationships. The feature association graph is a graph structure in which nodes represent structural elements, such as topic elements, logical relationship elements, and key information elements, and edges represent associations between elements, such as semantic mapping relationships and structural dependency relationships.

[0122] When constructing a feature association graph, you also need to set weight attributes for edges. Weight attributes represent the strength of the association between features. Weight attributes can be calculated based on the importance and stability of the association. For example, the weight of a semantic mapping relationship can be determined by calculating the co-occurrence frequency of subject terms and key information elements, and the weight of a structural dependency relationship can be determined by the consistency of the occurrence of logical relationship elements in different document units.

[0123] Step S1541: extracting the association strength value in the semantic mapping relationship, where the association strength value is calculated by the co-occurrence frequency of the subject word and the key information element.

[0124] In this embodiment, to determine the association strength of the semantic mapping relationship, the association strength value in the semantic mapping relationship is extracted. The association strength value can be obtained by calculating the co-occurrence frequency of the subject word and the key information element. The co-occurrence frequency represents the ratio of the number of times the subject word and the key information element appear together in the document to the number of times each appears separately.

[0125] By counting the co-occurrence of subject words and key information elements in the literature, the association strength value can be calculated. The higher the association strength value, the closer the association between the subject words and key information elements.

[0126] Step S1542: extracting the association stability value in the structural dependency relationship, which is calculated by the consistency of the appearance of logical relationship elements in different document units.

[0127] In this embodiment, in order to determine the association stability of the structural dependency relationship, the association stability value in the structural dependency relationship is extracted. The association stability value can be obtained by calculating the appearance consistency of logical relationship elements in different document units.

[0128] Specifically, we can analyze multiple document units and count the occurrences of logical relationship elements within these units. If the occurrence patterns of logical relationship elements across different document units are relatively consistent, then the association stability value is high; otherwise, it is low. The association stability value can reflect the reliability of structural dependency relationships.

[0129] Step S1543: setting a weight attribute for the edge of the element association graph, where the weight attribute is obtained by weighted summation of the association strength value and the association stability value.

[0130] In this embodiment, in order to comprehensively consider the association strength of the semantic mapping relationship and the association stability of the structural dependency relationship, a weight attribute is set for the edge of the element association graph. The weight attribute can be obtained by weighted summing the association strength value and the association stability value.

[0131] The weight of the weighted summation can be adjusted according to the specific application scenario and requirements. For example, if more emphasis is placed on the association strength of the semantic mapping relationship, the weight of the association strength value can be appropriately increased; if more emphasis is placed on the association stability of the structural dependency relationship, the weight of the association stability value can be appropriately increased.

[0132] Step S1544: merge the repeated elements in the structured element set, and merge the subject elements, logical relationship elements or key information elements with the same semantics into a single node.

[0133] In this embodiment, in order to simplify the structure of the element association graph, repeated elements in the structured element set are merged. If semantically identical subject elements, logical relationship elements, or key information elements are found, they are merged into a single node.

[0134] During the merging process, all attribute information of the features can be retained and integrated into a single node after the merger. By merging duplicate features, the number of nodes in the feature association graph can be reduced, improving the readability of the graph and processing efficiency.

[0135] Step S1545: Perform attribute supplementation processing on the merged element node, and integrate element attribute information from different sources into the same node.

[0136] In this embodiment, after merging element nodes, there may be a situation where the element attribute information is scattered. In order to make the attribute information of the element node more complete, the merged element node is subjected to attribute supplementation processing. The element attribute information from different sources is integrated into the same node. For example, if a certain theme element obtains different attribute descriptions in different analysis processes, such as determining the coverage of the theme in the theme identification module and discovering the associated attributes with other elements in the conflict detection process, then it is necessary to merge the attribute information from these different sources into the node corresponding to the theme element.

[0137] When supplementing attributes, consider compatibility and consistency. For attributes of the same type, if multiple values exist, merge them or select the most appropriate value. For example, if two sources both provide slightly different descriptions of a topic's coverage, they can be merged into a more accurate and complete description through methods such as semantic analysis. Attributes of different types are added directly to the node's attribute list.

[0138] Step S1546: Generate a complete element association graph based on the weight attributes and the attribute information of the element nodes. The element association graph is used to represent the multi-dimensional association relationship between structured elements.

[0139] In this embodiment, after setting edge weights, merging element nodes, and supplementing attributes, a complete element association graph is generated based on this information. The element association graph is a graph structure that comprehensively represents the multi-dimensional association relationships between structured elements. It not only includes the semantic mapping relationships and structural dependencies between elements, but also reflects the strength and stability of the association through edge weights, and describes the characteristics of each element in detail through node attribute information.

[0140] The process of generating a feature association graph involves determining the location of nodes and the way edges connect. Nodes are arranged based on the closeness of the associations between features, with closely related nodes placed closer together to more intuitively demonstrate their relationships. Edge connections are determined based on previously identified semantic mappings and structural dependencies, ensuring that the edges in the graph accurately represent the associations between features.

[0141] Step S155: topologically sort the element association graph to generate a structured arrangement scheme that reflects the order of element importance.

[0142] In this embodiment, to further clarify the order of importance between structured elements, a topological sorting process is performed on the element association graph. Topological sorting is an algorithm for sorting directed acyclic graphs. It can arrange the nodes in the graph into a linear sequence based on the dependencies between the nodes, so that for any directed edge, the starting node always appears before the ending node in the sequence.

[0143] In a feature-association graph, the dependencies between nodes reflect the logical order and importance of features. Topological sorting can generate a structured arrangement that reflects the order of feature importance. Edge weights can be taken into account during the sorting process. Nodes connected by edges with larger weights are prioritized in the sorting because they indicate a closer relationship and higher importance.

[0144] Step S156: reorganize the structured element set according to the structured arrangement scheme to generate a final structured document result including element association relationships.

[0145] In this embodiment, the structured element set is reorganized according to the generated structured arrangement scheme. The structured elements are rearranged according to the order of the arrangement scheme, and the association information between the elements is also incorporated into the final result based on the association relationships in the element association graph.

[0146] During the reorganization process, the detailed information and attributes of each element can be retained, while the relationships between them can be presented in a clear and easy-to-read manner. For example, the subject elements, logical relationship elements, and key information elements can be listed in order of importance, and the other elements associated with each element can be annotated after each element. The final structured document generated in this way can comprehensively and accurately reflect the document's theme, logical structure, key information, and the relationships between them, facilitating subsequent applications such as document analysis, knowledge mining, and information retrieval.

[0147] Furthermore, the pre-trained document feature extraction model is constructed through the following steps:

[0148] In this embodiment, building a pre-trained document feature extraction model is one of the key steps in the entire document structured extraction method. By learning from a large amount of annotated document data, the model can automatically extract valuable content features and structural features from the documents. The construction process includes the following steps:

[0149] Step S211: obtaining a training data set containing annotated documents, wherein the annotated documents contain manually annotated content feature labels and structure feature labels.

[0150] In this embodiment, to train the document feature extraction model, a training dataset containing annotated documents is required. These annotated documents have been carefully analyzed and annotated by humans and contain both content feature tags and structural feature tags. Content feature tags can include the core theme and key concepts of a document, while structural feature tags can include the document's chapter structure and the logical relationships between paragraphs.

[0151] Training data sets can be obtained by collecting documents from multiple academic resource platforms and organizing professional annotators to perform the annotation. During the annotation process, detailed annotation standards and guidelines need to be developed to ensure accuracy and consistency. For example, for content feature labeling, clear regulations should be set for determining core themes and key concepts; for structural feature labeling, regulations should be set for dividing chapters and identifying logical relationships between paragraphs.

[0152] Step S212: Input the training data set into the initial feature extraction model, and generate predicted content features and predicted structural features through a forward propagation operation.

[0153] In this embodiment, the acquired training data set is input into an initial feature extraction model. The initial feature extraction model is a pre-designed neural network model that includes multiple modules and layers, such as a text encoding module, a meta-information encoding module, a content feature extraction layer, and a structural feature extraction layer.

[0154] After inputting training data, the model performs a forward propagation operation. Forward propagation refers to the process by which data, starting from the input layer, is processed through various modules and layers, ultimately generating predicted content features and predicted structural features. Within each module and layer, the data undergoes a series of calculations and transformations, such as word embedding, context encoding, feature filtering, and dimensionality compression. These operations are all aimed at extracting meaningful features from the input document data.

[0155] Step S213: Calculate a first loss value between the predicted content feature and the content feature label, where the first loss value is calculated using a cross entropy loss function.

[0156] In this embodiment, a first loss value is calculated to measure the difference between the predicted content features generated by the model and the manually annotated content feature labels. The cross-entropy loss function is a commonly used loss function that measures the difference between two probability distributions. In this model, the predicted content features and content feature labels can be viewed as two probability distributions, and the cross-entropy loss function is used to calculate the difference between them.

[0157] Specifically, the cross-entropy loss function compares the predicted content features with the content feature labels. For each feature element, it calculates the difference between its predicted value and the true value and then takes a weighted sum of these differences. The weights of the weighted sum can be set based on the importance of the feature, with more important features taking a greater proportion in the loss calculation.

[0158] Step S214: Calculate a second loss value between the predicted structural feature and the structural feature label, where the second loss value is calculated using a mean square error loss function.

[0159] In this embodiment, a second loss value is similarly calculated to measure the difference between the predicted structural features generated by the model and the manually annotated structural feature labels. The mean squared error loss function is a function used to measure the difference between two numerical sequences. It calculates the square of the difference between each corresponding element and then sums and averages these squared values.

[0160] When calculating the second loss between the predicted structural features and the structural feature labels, the mean squared error loss function compares the predicted and true values of each structural feature element, calculates the square of the difference between them, and then sums and averages the squared differences across all elements. The mean squared error loss function effectively reflects the overall difference between the predicted and true structural features.

[0161] Step S215: The first loss value and the second loss value are integrated to generate a joint loss function, and the network parameters of the initial feature extraction model are optimized through a back-propagation operation.

[0162] In this embodiment, to comprehensively consider the accuracy of predicted content features and predicted structural features, the first loss value and the second loss value are combined to generate a joint loss function. The joint loss function combines the first and second loss values through a weighted summation, with the weights adjusted based on specific application requirements. For example, if the accuracy of content features is more important, the weight of the first loss value can be appropriately increased; if the accuracy of structural features is more important, the weight of the second loss value can be appropriately increased.

[0163] After obtaining the joint loss function, the backpropagation algorithm is used to optimize the network parameters of the initial feature extraction model. Backpropagation is a gradient descent-based optimization algorithm that calculates the gradient of the joint loss function with respect to each parameter in the model and then adjusts the parameter values based on the direction and magnitude of the gradient, gradually reducing the value of the joint loss function. During each backpropagation step, the model updates its parameters to improve its fit to the training data.

[0164] Step S216: When the convergence value of the joint loss function meets the preset threshold, the training is stopped and the corresponding initial feature extraction model is determined as the pre-trained document feature extraction model.

[0165] In this embodiment, during the training process, the convergence of the joint loss function can be continuously monitored. A convergence threshold is preset. When the value of the joint loss function drops below the threshold, the model is considered to have converged and the training process can be stopped. At this point, the current initial feature extraction model is determined to be the pre-trained document feature extraction model.

[0166] In actual training, the joint loss function may not decrease for a certain period of time or decrease very slowly. This can also be used as a basis for judging model convergence. By continuously adjusting the model parameters and training strategy, we ensure that the model can accurately extract content and structural features from the literature when the convergence conditions are met.

[0167] Throughout the structured document extraction process, attention must be paid to data privacy protection and leakage prevention. During the data collection phase, documents containing privacy-sensitive data, such as personal identity information and commercial secrets, require rigorous screening and processing. Data desensitization techniques can be employed to replace, delete, or encrypt sensitive information to ensure data security during collection and storage.

[0168] Appropriate privacy protection measures should also be implemented during model training and use. For example, model training data can be encrypted and stored, making it accessible only to authorized personnel. During the model inference phase, privacy protection should also be applied to input document data to prevent the leakage of sensitive information. Furthermore, a comprehensive security management system should be established, with regular security checks and vulnerability fixes for data and models to ensure the security and reliability of the entire document structured extraction system.

[0169] Figure 2 The following is a schematic diagram illustrating exemplary hardware and software components of an artificial intelligence-based document structured extraction system 100 that can implement the concepts of the present application, as provided in some embodiments of the present application. For example, the processor 120 can be used in the artificial intelligence-based document structured extraction system 100 and used to perform the functions of the present application.

[0170] The artificial intelligence-based document structured extraction system 100 can be a general-purpose server or a special-purpose server, both of which can be used to implement the artificial intelligence-based document structured extraction method of the present application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0171] For example, the document structured extraction system 100 based on artificial intelligence may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and storage media 140 in different forms, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the document structured extraction system 100 based on artificial intelligence may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The document structured extraction system 100 based on artificial intelligence also includes an input / output I / O interface 150 between the computer and other input and output devices.

[0172] For ease of explanation, only one processor is described in the artificial intelligence-based document structured extraction system 100. However, it should be noted that the artificial intelligence-based document structured extraction system 100 in the present application may also include multiple processors, so the steps performed by one processor described in the present application may also be performed jointly or individually by multiple processors. For example, if the processor of the artificial intelligence-based document structured extraction system 100 executes step A and step B, it should be understood that step A and step B may also be performed jointly by two different processors or individually in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor execute steps A and B together.

[0173] In addition, an embodiment of the present invention further provides a readable storage medium, in which computer-executable instructions are preset. When a processor executes the computer-executable instructions, the above-mentioned artificial intelligence-based document structured extraction method is implemented.

[0174] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, multiple features are sometimes combined into one embodiment, figure or description thereof.

Claims

1. A method for extracting structured documents based on artificial intelligence, characterized in that: The method comprises: Acquire an original document data set, wherein the original document data set includes a plurality of document units, each document unit consisting of text content and meta information; Preprocessing the original document data set to obtain a standardized document data set, wherein the standardized document data set includes text paragraphs and meta-information items in a unified format; Calling a pre-trained document feature extraction model to process the standardized document data set to obtain content features and structural features of the document unit; Performing structured parsing based on the content features and the structural features to generate a structured element set of the document unit, wherein the structured element set includes a subject element, a logical relationship element, and a key information element; Performing integration and optimization processing on the structured element set to generate a final structured document result including element association relationships; The structured parsing process is performed based on the content features and the structural features to generate a structured element set of the document unit, wherein the structured element set includes a subject element, a logical relationship element, and a key information element, including: Input the content features into a topic identification module, and generate topic elements through topic word extraction and topic distribution probability calculation operations, wherein the topic elements include core topic words and topic coverage description; Inputting the structural features into a logic parsing module, generating logical relationship elements through paragraph association analysis and chapter level identification operations, wherein the logical relationship elements include deductive relationships between paragraphs and subordinate relationships between chapters; Inputting the content features and the structural features into a key information extraction module, generating key information elements through entity recognition and key sentence screening operations, wherein the key information elements include core research conclusions and key experimental conditions; Performing conflict detection on the subject elements, the logical relationship elements, and the key information elements to identify semantic contradictions between different elements; Based on the semantic contradictions, the subject elements, the logical relationship elements and the key information elements are modified to generate a structured element set that passes the consistency check.

2. The method for extracting structured documents based on artificial intelligence according to claim 1, characterized in that: The preprocessing of the original document data set to obtain a standardized document data set, wherein the standardized document data set contains text paragraphs and meta-information items in a unified format, including: Performing format conversion processing on each document unit in the original document data set, converting text contents from different sources into a unified plain text format, and obtaining converted text contents; De-noising is performed on the converted text content to remove redundant symbols, repeated paragraphs, and additional information irrelevant to the core content of the document to obtain the denoised text content; The denoised text content is segmented according to semantic delimiters to generate logically coherent text paragraphs; Performing field alignment processing on the metadata of the document unit, mapping the dispersedly stored metadata entries to preset standard metadata fields; Perform missing value filling on the metadata after field alignment, and complete the missing metadata entries based on the statistical rules of metadata of document units of the same type; The text paragraphs and meta-information items with logical coherence are associated and bound to generate a standardized document data set including a text paragraph set and a meta-information item set.

3. The method for extracting structured documents based on artificial intelligence according to claim 2, characterized in that: The converted text content is subjected to denoising processing to remove redundant symbols, repeated paragraphs, and additional information irrelevant to the core content of the document to obtain denoised text content, including: Identifying the symbol types in the converted text content and filtering non-text symbols and hyperlink tags; Extracting repeated character sequences from the text content, and calculating the frequency of occurrence of the repeated character sequences in the continuous text; Performing redundancy determination on repeated character sequences based on the occurrence frequency, and deleting repeated paragraphs whose occurrence frequency exceeds a preset threshold; detecting an additional information area in the text content, wherein the additional information area includes a reference list, an appendix description, and a copyright statement; Extracting the starting position and the ending position of the additional information area, and truncating the additional information located before the starting position and after the ending position in the text content; The truncated text content is normalized by replacing multiple consecutive space characters with a single space character to generate the denoised text content.

4. The method for extracting structured documents based on artificial intelligence according to claim 1, characterized in that: The calling of the pre-trained document feature extraction model to process the standardized document data set to obtain the content features and structural features of the document unit includes: Inputting the text paragraphs in the standardized document dataset into the text encoding module of the document feature extraction model, and generating paragraph semantic vectors through word embedding and context encoding operations; Inputting the meta-information items in the standardized document data set into the meta-information encoding module of the document feature extraction model, and generating a meta-information feature vector through category feature conversion and numerical feature normalization operations; Performing cross-dimensional correlation analysis on the paragraph semantic vector and the meta-information feature vector to generate a correlation feature vector containing a correlation relationship between the text and the meta-information; Inputting the associated feature vector into the content feature extraction layer of the document feature extraction model, and extracting content features reflecting the core content of the document through feature screening and dimension compression operations; Inputting the paragraph semantic vector into the structural feature extraction layer of the document feature extraction model, and extracting structural features reflecting the document organization through paragraph sequence modeling and logical relationship recognition operations; The content features and the structural features are standardized to generate a document feature set with a unified dimensional representation.

5. The method for extracting structured documents based on artificial intelligence according to claim 4, characterized in that: Inputting the paragraph semantic vector into the structural feature extraction layer of the document feature extraction model, and extracting structural features reflecting the document organization through paragraph sequence modeling and logical relationship recognition operations, includes: Constructing a sequential index of the paragraph semantic vector, wherein the sequential index is used to represent the original arrangement order of the text paragraphs in the document; Performing joint encoding processing on the paragraph semantic vector and the sequence index to generate a time series feature vector containing sequence information; Extracting cosine similarity values between semantic vectors of adjacent paragraphs, and calculating the change gradient of the cosine similarity values in consecutive paragraphs; Identifying logical transition points between paragraphs based on the change gradient, wherein the logical transition points are used to demarcate chapter boundaries in the document; Performing attention mechanism processing on the time series feature vector to generate an attention weight distribution reflecting the importance of the paragraph; A document structure tree is constructed according to the attention weight distribution and the chapter boundary information, and the node features of the document structure tree serve as structural features reflecting the document organization method.

6. The method for extracting structured documents based on artificial intelligence according to claim 5, characterized in that: The structural features are input into a logic parsing module, and logical relationship elements are generated through paragraph association analysis and chapter level identification operations. The logical relationship elements include derivation relationships between paragraphs and subordination relationships between chapters, including: Extracting paragraph association features from the structural features, wherein the paragraph association features include semantic cohesion strength and content continuity index of adjacent paragraphs; Building a paragraph relationship graph based on the paragraph association features, wherein the edge weights of the paragraph relationship graph represent the degree of logical association between paragraphs; Performing community discovery processing on the paragraph relationship graph to identify paragraph communities with significant correlation as document chapters; Extracting chapter-level features from the structural features, the chapter-level features including font size information and indentation depth information of chapter titles; Determining a subordinate relationship between chapters based on the chapter hierarchical features, wherein the subordinate relationship includes a correspondence between a parent chapter and a child chapter; Matching and verification processing is performed on the paragraph community in the paragraph relationship graph and the chapter-level features to generate logical relationship elements including deductive relationships between paragraphs and subordinate relationships between chapters.

7. The method for extracting structured documents based on artificial intelligence according to claim 1, characterized in that: The integration and optimization processing of the structured element set to generate a final structured document result containing element association relationships includes: Constructing a feature index for each element in the structured element set, wherein the feature index is used to record the unique identifier of the subject element, the logical relationship element, and the key information element; Performing association analysis on the feature index to identify semantic mapping relationships between subject elements and key information elements, wherein the semantic mapping relationships include corresponding relationships between subject words and key experimental conditions; Identify the structural dependencies between logical relationship elements and key information elements, including the distribution relationship between chapter affiliation and core research conclusions; Constructing an element association graph based on the semantic mapping relationship and the structural dependency relationship, wherein the nodes of the element association graph are structured elements and the edges are association relationships between elements; Performing topological sorting on the element association graph to generate a structured arrangement scheme reflecting the order of importance of the elements; Reorganizing the structured element set according to the structured arrangement scheme to generate a final structured document result including element association relationships; The element association graph is constructed based on the semantic mapping relationship and the structural dependency relationship, wherein the nodes of the element association graph are structured elements and the edges are association relationships between elements, including: Extracting an association strength value from the semantic mapping relationship, wherein the association strength value is calculated by calculating the co-occurrence frequency of the subject word and the key information element; Extracting an association stability value in the structural dependency relationship, wherein the association stability value is calculated based on the consistency of occurrence of logical relationship elements in different document units; Setting a weight attribute for an edge of the element association graph, wherein the weight attribute is obtained by weighted summation of the association strength value and the association stability value; Merging repeated elements in the structured element set, and merging semantically identical subject elements, logical relationship elements, or key information elements into a single node; Perform attribute supplementation processing on the merged feature nodes to integrate feature attribute information from different sources into the same node; A complete element association graph is generated based on the weight attributes and the attribute information of the element nodes, and the element association graph is used to represent the multi-dimensional association relationship between structured elements.

8. The method for extracting structured documents based on artificial intelligence according to claim 1, characterized in that: The pre-trained document feature extraction model is constructed by the following steps: Obtaining a training data set comprising annotated documents, wherein the annotated documents comprise manually annotated content feature labels and structure feature labels; Inputting the training data set into an initial feature extraction model to generate predicted content features and predicted structural features through a forward propagation operation; Calculating a first loss value between the predicted content feature and the content feature label, where the first loss value is calculated using a cross entropy loss function; Calculating a second loss value between the predicted structural feature and the structural feature label, wherein the second loss value is calculated using a mean square error loss function; fusing the first loss value and the second loss value to generate a joint loss function, and optimizing the network parameters of the initial feature extraction model through a back-propagation operation; When the convergence value of the joint loss function meets a preset threshold, the training is stopped and the corresponding initial feature extraction model is determined as the pre-trained document feature extraction model.

9. A document structured extraction system based on artificial intelligence, characterized in that: It includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the artificial intelligence-based document structured extraction method described in any one of claims 1 to 8.

Images