Document title hierarchy extraction method and device, equipment and medium

Abstract

The invention discloses a document title hierarchy extraction method and device, equipment and a medium, and relates to the field of document information extraction, and the method comprises the steps: taking a large title of a target document as a root node, determining each text paragraph in the target document as each node, and determining a current node from each node; obtaining a current title ordinal number identification mode matched with the current node by utilizing a regular expression list constructed based on the target document; determining the title hierarchy of the current node based on a preset hierarchy matching logic, the current title ordinal number identification mode and the previous title ordinal number identification mode; wherein the preset hierarchy matching logic comprises that the title ordinal number identification modes of different hierarchies are the same or different, and the title ordinal number identification modes of the same hierarchies are the same; and tracing the complete hierarchical title information of each node to the root node based on the title hierarchy of each node, and obtaining a complete hierarchical structure of the target document. The document content is reasonably segmented at low cost, and meanwhile, the integrity and definition of a logic structure and semantic information are kept.

Description

Technical Field

[0001] The present invention relates to the field of document information extraction, and in particular to a document title level extraction method, device, equipment and medium. Background Art

[0002] In the electronic document segmentation scheme, one of the simplest methods is to give a maximum length l, and divide the document content into n non-overlapping blocks of length l according to the number of words, and each block is regarded as a piece of knowledge. This scheme is very simple and easy to implement. However, this method does not take into account the semantics and structure of the document. It only mechanically divides the text according to the number of words. This may cause the segmented knowledge to be semantically incomplete, and may even divide a complete sentence or paragraph into several parts, thereby losing the structural information and contextual association of the document.

[0003] For electronic documents, it is a more natural solution to segment the document according to its subheadings (chapter titles, paragraph titles, etc.), because the subheadings of the document naturally divide the text content into different parts semantically. There are two types of methods to achieve automatic extraction of document-level titles, one is based on specific rules, and the other is based on deep learning. The main problem with the specific rule-based method is that the rules are narrowly applicable. When it is necessary to process a large number of documents with varying content organization forms, it takes a lot of manpower to maintain a correct rule for each specific type of document or even each special document, which is difficult to achieve in practical applications. Although the deep learning-based method can learn the features in the data more automatically, this learning is based on a large amount of high-quality annotated data with rich forms. The time cost and computing resource cost required for early data annotation and model iteration optimization are relatively high.

[0004] From the above, it can be seen that how to reasonably segment the content in a document at a low cost while maintaining the integrity and clarity of the logical structure and semantic information is a problem to be solved in this field. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a document title level extraction method, device, equipment and medium, which can reasonably segment the content in the document at a low cost while maintaining the integrity and clarity of the logical structure and semantic information. The specific scheme is as follows:

[0006] In a first aspect, the present application discloses a document title level extraction method, comprising:

[0007] The main title of the target document is set as a root node, each text paragraph in the target document is determined as a node with the root node as the starting point, and the current node is determined from each of the nodes;

[0008] Matching the current node with a list of title ordinal identification patterns using a regular expression list constructed based on the target document to obtain a current title ordinal identification pattern that matches the current node;

[0009] Determine the title level of the current node based on a preset level matching logic, the current title ordinal identification mode and the previous title ordinal identification mode; wherein the preset level matching logic includes that the title ordinal identification modes of different levels are the same or different and the title ordinal identification modes of the same level are the same, and the previous title ordinal identification mode is the title ordinal identification mode of the previous node of the current node;

[0010] Based on the title level of each of the nodes, the complete hierarchical title information of each of the nodes is traced back to the root node to obtain the complete hierarchical structure of the target document.

[0011] Optionally, matching the current node with a list of title ordinal identification patterns using a regular expression list constructed based on the target document to obtain a current title ordinal identification pattern matching the current node includes:

[0012] Using a regular expression list constructed based on the target document, the current node is matched with a title ordinal identification pattern list to obtain a matching result;

[0013] If the matching result indicates that there is no title ordinal identification pattern matching the current node in the title ordinal identification pattern list, it is determined that the category of the current node is not a title category, and the process jumps again to the step of determining the current node from each of the nodes;

[0014] If the matching result indicates that there is a title ordinal identification pattern matching the current node in the title ordinal identification pattern list, then the current title ordinal identification pattern matching the current node is acquired.

[0015] Optionally, matching the current node with a list of title ordinal identification patterns using a regular expression list constructed based on the target document includes:

[0016] A regular expression list is constructed based on the target document, and the title ordinal identification pattern list and the current node are input into a regular expression title matching function to match the current node with the title ordinal identification pattern list.

[0017] Optionally, the determining the title level of the current node based on a preset level matching logic, the current title ordinal identification mode, and the previous title ordinal identification mode includes:

[0018] Determining whether the current title ordinal identification mode is the same as the previous title ordinal identification mode;

[0019] If the current title ordinal identification mode is the same as the previous title ordinal identification mode, determining that the title level of the current node is the same as the title level of the previous node;

[0020] If the current title ordinal identification mode is different from the previous title ordinal identification mode, the title level of the current node is determined according to the previous title ordinal identification mode.

[0021] Optionally, determining the title level of the current node according to the previous title ordinal identification mode includes:

[0022] If the previous title ordinal identification mode is the root node, determining the title level of the current node to be the first subtitle in the target document;

[0023] If the ordinal identification mode of the previous title is a first-level title mode, the title level of the current node is determined to be a sub-level of the previous node.

[0024] Optionally, determining the title level of the current node according to the previous title ordinal identification mode includes:

[0025] If the previous title ordinal identification pattern is not the root node or the first-level title pattern, then the title ordinal identification pattern of the ancestor node of the previous node is traced back to find a target node with the same title ordinal identification pattern as the current title ordinal identification pattern from the ancestor node of the previous node, and the target node is determined as the parent node of the current node, so as to determine the sub-level of the title level of the parent node as the title level of the current node.

[0026] Optionally, after determining the title level of the current node based on the preset level matching logic, the current title ordinal identification mode and the previous title ordinal identification mode, the method further includes:

[0027] Recording the attribute information of the current node in the current content list to update the current content list; wherein the attribute information includes the name, the index of the current title ordinal identification mode in the title ordinal identification mode list, the index of the current node, and the parent node;

[0028] Accordingly, tracing back the complete hierarchical title information of each node to the root node based on the title hierarchy of each node to obtain the complete hierarchical structure of the target document includes:

[0029] Based on the title level of each node and the parent node in the attribute information, the complete hierarchical title information of each node is obtained, and the complete hierarchical structure of the target document is obtained based on the complete hierarchical title information of each node.

[0030] In a second aspect, the present application discloses a document title level extraction device, comprising:

[0031] A node determination module, used to set the main title of the target document as a root node, determine each text paragraph in the target document as a node with the root node as a starting point, and determine the current node from each of the nodes;

[0032] A pattern matching module, used to match the current node with a list of title ordinal identification patterns using a regular expression list constructed based on the target document, so as to obtain a current title ordinal identification pattern that matches the current node;

[0033] A level determination module, used to determine the title level of the current node based on a preset level matching logic, the current title ordinal identification mode and a previous title ordinal identification mode; wherein the preset level matching logic includes that the title ordinal identification modes of different levels are the same or different and the title ordinal identification modes of the same level are the same, and the previous title ordinal identification mode is the title ordinal identification mode of the previous node of the current node;

[0034] The structure acquisition module is used to trace back the complete hierarchical title information of each node to the root node based on the title hierarchy of each node, so as to obtain the complete hierarchical structure of the target document.

[0035] In a third aspect, the present application discloses an electronic device, including:

[0036] Memory, used to store computer programs;

[0037] The processor is used to execute the computer program to implement the steps of the document title level extraction method disclosed above.

[0038] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the document title hierarchy extraction method disclosed above are implemented.

[0039] The beneficial effects of the present application are as follows: the present application sets the main title of the target document as the root node, determines each text paragraph in the target document as each node with the root node as the starting point, and determines the current node from each of the nodes; matches the current node with a list of title ordinal identification patterns using a regular expression list constructed based on the target document to obtain a current title ordinal identification pattern that matches the current node; determines the title level of the current node based on a preset hierarchical matching logic, the current title ordinal identification pattern, and a previous title ordinal identification pattern; wherein the preset hierarchical matching logic includes that the title ordinal identification patterns of different levels are the same or different and the title ordinal identification patterns of the same level are the same, and the previous title ordinal identification pattern is the title ordinal identification pattern of the previous node of the current node; traces back the complete hierarchical title information of each node to the root node based on the title level of each node to obtain the complete hierarchical structure of the target document. It can be seen that the present application uses the main title of the target document as the root node, and traverses each text paragraph from the root node, that is, determines each paragraph as the current node in turn, and then matches the title ordinal identification for each node, so as to determine whether the current node is a title. If it is a title, the title level of the current node is determined based on the preset level matching logic, the current title ordinal identification mode and the previous title ordinal identification mode, because the preset level matching logic includes whether the title ordinal identification modes of different levels are the same or different and the title ordinal identification modes of the same level are the same, that is, in the preset level matching logic, it does not limit whether the title ordinal identification modes at different levels are the same, that is, the title ordinal identification modes at different levels can be the same. It can also be different, as long as the title ordinal identification patterns at the same level are the same. Compared with traditional solutions based on specific rules, there is no need to add new rules due to the different title ordinal identification patterns at different levels. The present application can automatically discover the title ordinal identification pattern for any document, dynamically extract the hierarchical title of the document according to the title ordinal identification pattern, and then trace the complete hierarchical title information of each node to the root node based on the title hierarchy of each node, so as to obtain the complete hierarchical structure of the target document. Compared with the cost of large model training, the present application does not require model training, and the cost of title hierarchy extraction is lower, so as to achieve low-cost and reasonable segmentation of the content in the document while maintaining the integrity and clarity of the logical structure and semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0041] Figure 1 A flowchart of document title level extraction disclosed in this application;

[0042] Figure 2 A specific target document schematic diagram disclosed in this application;

[0043] Figure 3 This is a schematic diagram of the structure of a document title level extraction device disclosed in this application;

[0044] Figure 4 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0046] With the rapid development of big model technology in the field of artificial intelligence, downstream tasks have put forward higher requirements on the timeliness and authenticity of model knowledge. To solve this problem, the current mainstream technical solution is to use Retrieval Augmented Generation (RAG), the core of which is to maintain a knowledge base, retrieve the knowledge most relevant to the user's question from the knowledge base, and organize the knowledge and the question into a prompt to input the big model, so as to get a more accurate answer. Under the framework of RAG, the knowledge base is an effective supplement to the knowledge of the big model, and the quality of the knowledge it contains will directly affect the final performance of the model. In practical applications, a large amount of knowledge is carried by electronic documents such as PDF and Word, and its content is usually lengthy. However, due to the number of tokens visible to the big model, each piece of knowledge in the knowledge base has a certain length limit. So, how to efficiently and reasonably segment the content in the document while maintaining the integrity and clarity of the logical structure and semantic information has become a key issue that needs to be solved urgently.

[0047] In the electronic document segmentation scheme, one of the simplest methods is to give a maximum length l and divide the document content into n non-overlapping blocks of length l according to the number of words, with each block as a piece of knowledge. This scheme is very simple and easy to implement. However, this method does not take into account the semantics and structure of the document. It only mechanically divides the text according to the number of words, which may cause the segmented knowledge to be semantically incomplete, and may even divide a complete sentence or paragraph into several parts, thereby losing the structural information and contextual association of the document.

[0048] For electronic documents, it is a more natural solution to segment the document according to its subheadings (chapter titles, paragraph titles, etc.), because the subheadings of the document naturally divide the content of the text into different parts semantically. In this solution, it is usually relatively easy to obtain the first-level subheading information closest to a certain piece of knowledge, but it is difficult to obtain its complete hierarchical structure in the document. In some scenarios, this will still cause incomplete knowledge. To give a simple example: "Content 1" and "Content 2" are both from the same annual report document, and their hierarchical titles are as follows:

[0049] “Annual report (level 1 title) → Section 10 Financial report (level 2 title) → VII. Notes to consolidated financial statements (level 3 title) → 4. Accounts receivable (level 4 title) → 1) Disclosure by age (level 5 title) → Content 1”;

[0050] “Annual report (level 1 title) → Section 10 Financial Report (level 2 title) → 10. Notes to major items in the financial statements (level 3 title) → 1. Accounts receivable (level 4 title) → 1) Disclosure by age of accounts (level 5 title) → Content 2”.

[0051] Judging from the level 5 title, "Content 1" and "Content 2" have the same subheadings. If they are divided only according to the subheadings at this level, there will be two conflicting pieces of knowledge in the knowledge base. At this time, it is necessary to combine the higher-level title information to clarify the connotations of these two pieces of knowledge.

[0052] For electronic documents, it is a more natural solution to segment the document according to its subheadings (chapter titles, paragraph titles, etc.), because the subheadings of the document naturally divide the text content into different parts semantically. There are two types of methods to achieve automatic extraction of document-level titles, one is based on specific rules, and the other is based on deep learning. The main problem with the specific rule-based method is that the rules are narrowly applicable. When it is necessary to process a large number of documents with varying content organization forms, it takes a lot of manpower to maintain a correct rule for each specific type of document or even each special document, which is difficult to achieve in practical applications. Although the deep learning-based method can learn the features in the data more automatically, this learning is based on a large amount of high-quality annotated data with rich forms. The time cost and computing resource cost required for early data annotation and model iteration optimization are relatively high.

[0053] To this end, the present application provides a document title level extraction solution, which can reasonably segment the content in the document at a low cost while maintaining the integrity and clarity of the logical structure and semantic information.

[0054] See also Figure 1 As shown, the embodiment of the present application discloses a document title level extraction method, comprising:

[0055] Step S11: setting the main title of the target document as a root node, determining each text paragraph in the target document as a node with the root node as a starting point, and determining the current node from each of the nodes.

[0056] Read the target document with the help of Apache POI open source tool Content, for the target document Preprocess the target document The target document will be The main title is set as the root node, and each text paragraph in the target document is determined as a node with the root node as the starting point. It can be understood that when determining each node, the order of each node is also determined. According to this order, each node is taken as the current node in turn, and the previous node of the current node is also determined according to this order.

[0057] Define a class TitleNode with attributes name, pattern, index, and father to record any The information of the node, where name records the content of the current node; pattern records the title ordinal identification pattern of the current node. The index in; index is the index of the current node; father records the parent node of the current node, that is, the index of its upper level title, and the subtitle ordinal number identifies the mode list , Cover as many ordinal recognition patterns as possible, table of contents ,in, yes A paragraph of text in The word count is greater than 0.

[0058] Before determining the specific algorithm for the title level of the current node, initialization is required, specifically: 1) Determine the main title , build the root node: root=TitleNode( , "root", len(nodes), None); 2) Record the root node information into the nodes list: nodes.append(root); 3) Define the variable section_pattern for recording mode, initialized to an empty string; define a local variable cur_node to record the information of the current node, with the same initialization value as root; define a local variable pre_node to record the information of the previous node of the current node, initialized to pre_node=TitleNode(None, None, None, None); 4) Traverse the text of each paragraph and dynamically add the subtitle node to nodes.

[0059] Step S12: Match the current node with the title ordinal identification pattern list using the regular expression list constructed based on the target document to obtain a current title ordinal identification pattern matching the current node.

[0060] Target Document Includes multiple nodes; express No. indivual Level 1 title is the main title of the document. There is only one level 1 title, which is the main title. The level heading is a subheading; list , where each Represents a subheading title ordinal identification mode. The title ordinal identification mode refers to a subheading, such as "Section 1 Background Introduction", where "Section 1" is the "title ordinal identification" and "background introduction" is the title content. Title ordinal identification mode It can be described by regular expressions, for example, , , etc., and specifically can be set as follows:

[0061] Set P = ["\\d\\.?(\\d\\.){0,6}\\d?[\\s]*[\\u4e00-\\u9fa5]+[::]?",

[0062] "Chapter.*[\\s、.]+[\\u4e00-\\u9fa5]+",

[0063] "Article.*[\\s、.]+[\\u4e00-\\u9fa5]+",

[0064] "[一二三四五六七八九十百]{1,3}[\\s、.]",

[0065] "[0-9]{1,3}[\\s、.]+[\\u4e00-\\u9fa5]+",

[0066] "[\(\(][一二三四五六七八九十]{1,2}[\)\)][\\s\、]*[\\u4e00-\\u9fa5]+",

[0067] "[\(\(][0-9]{1,2}[\)\)][\\s\、]*[\\u4e00-\\u9fa5]+",

[0068] "\\d{1,3}[\))]",

[0069] "Section.*",

[0070] "%\d\.%\d\.%\d\.%\d\.",

[0071] "%\d\.%\d\.%\d\.|%\d\、%\d\、%\d\、",

[0072] "%\d\.%\d\.|%\d\、%\d\、",

[0073] "%\d\.|%\d\、"];

[0074] This list P of title ordinal identification patterns defines multiple regular expressions for matching different title ordinal identification patterns. The matched patterns are as follows line by line:

[0075] 1) First, "[\u4e00-\u9fa5]" represents matching Chinese characters, which is the character range of Chinese utf-8 encoded symbols. This regular expression matches a sequence of numbers that may contain a decimal point, followed by zero or more whitespace characters, then a block of Chinese text, and there may be a colon after this block of Chinese text. For example, it can match titles like "1.1 Background", "2.3.4 Technical Features:", etc.;

[0076] 2) Match patterns similar to "Chapter X...";

[0077] 3) Match patterns similar to "Article X...";

[0078] 4) Match serial numbers composed of Chinese numerals (from one to ten and hundred), followed by a whitespace character, a comma, or a period;

[0079] 5) Used to match serial numbers composed of 1 to 3 Arabic numerals, immediately followed by one or more whitespace characters (\s), a comma (,), or a period (.), and then one or more blocks of Chinese characters; for example, "1. Clause Content", "12. Stipulated Matters", "123, Chapter Title";

[0080] 6) Used to match text in a specific format, especially cases where the number is represented by Chinese numerals and enclosed in parentheses, followed by Chinese text, such as "(One) Clause Content", "(One)Stipulated Matters", "(Twelve) Chapter Title";

[0081] 7) Match numbers represented by Arabic numerals, which are surrounded by parentheses, followed by Chinese text, such as "(2)Stipulated Matters", "(12) Chapter Title";

[0082] 8) Match a sequence composed of 1 to 3 Arabic numerals, followed by a right parenthesis (which can be full-width or half-width);

[0083] 9) Match text that starts with "Chapter" and ends with "Section";

[0084] 10) Match text like "%1.%1.%1.%1.";

[0085] 11) Match text like "%1.%1.%1." or "%1, %1, %1, ";

[0086] 12) Match text like "%1.%1." or "%1, %1, ";

[0087] 13) Match text like "%1." or "%1, ";

[0088] Furthermore, "\" means marking the next character as a special character, a literal character, a backreference, or an octal escape character; "^" means matching the beginning of an input word line; "$" means matching the end of an input line; "*" means matching the preceding subexpression any number of times; "+" means matching the preceding subexpression one or more times (greater than or equal to 1 time); "?" means matching the preceding subexpression zero or one time; "." means matching any single character except "\n" and "\r"; "[az]" means a character range, matching any character in the specified range, for example, "[az]" can match any lowercase alphabetic character in the range from "a" to "z"; "[^az]" means a negative character range; "\d" means matching a numeric character; equivalent to "[0-9]"; "\s" matches any invisible character, including space, tab, form feed, etc.

[0089] In this embodiment, the use of a regular expression list constructed based on the target document to match the current node with a list of title ordinal identification patterns to obtain a current title ordinal identification pattern that matches the current node includes: using a regular expression list constructed based on the target document to match the current node with a list of title ordinal identification patterns to obtain a matching result; if the matching result indicates that there is no title ordinal identification pattern matching the current node in the list of title ordinal identification patterns, then it is determined that the category of the current node is not a title category, and the step of determining the current node from each of the nodes is jumped again; if the matching result indicates that there is a title ordinal identification pattern matching the current node in the list of title ordinal identification patterns, then the current title ordinal identification pattern matching the current node is obtained. Title ordinal identification pattern list Contains various title ordinal identification patterns, and uses a regular expression list built based on the target document to match the current node with the title ordinal identification pattern list to obtain a matching result. It can be understood that if no title ordinal identification pattern is matched, it is considered that the current node is not a title of any level, then the operation on the current node is stopped, and the title ordinal identification pattern list matching operation for the next node is performed.

[0090] In this embodiment, the use of a regular expression list constructed based on the target document to match the current node with a title ordinal identification pattern list includes: constructing a regular expression list based on the target document, inputting the title ordinal identification pattern list and the current node into a regular expression title matching function to match the current node with the title ordinal identification pattern list. The input of the regular expression title matching function RexTitleMatch() is the current node and a list of title ordinal identifier patterns , the output of the expression title matching function RexTitleMatch() is the matching result, which is a Boolean type and / or a string type (i.e. is_match and / or p_match). When it is a Boolean type, it indicates whether the title ordinal identification pattern matching the current node is matched; when it is a string type, it indicates the index number of the title ordinal identification pattern matching the current node. For example, the title ordinal identification pattern matching the current node is , then p_match=str(k).

[0091] Step S13: Determine the title level of the current node based on a preset level matching logic, the current title ordinal identification pattern and the previous title ordinal identification pattern; wherein the preset level matching logic includes that the title ordinal identification patterns of different levels are the same or different and the title ordinal identification patterns of the same level are the same, and the previous title ordinal identification pattern is the title ordinal identification pattern of the previous node of the current node.

[0092] This method is based on a preset hierarchical matching logic: the ordinal identification patterns of titles at different levels are the same or different, and the ordinal identification patterns of titles at the same level are the same, that is, for the target document , in any Within the jurisdiction, all All have the same ordinal identification pattern, that is, the ordinal identification pattern of titles at the same level is the same. For example, yes The big title, all Level 2 headings are within its jurisdiction, and The ordinal identification patterns are consistent, for example, they all conform to Similarly, under each level 2 heading, there may be level 3 headings, but All within the jurisdiction All have the same ordinal identification pattern , All within the jurisdiction All have the same ordinal identification pattern It is worth noting that the preset level matching logic does not require , that is, the title ordinal identification patterns of different levels are the same or different, while the traditional method based on specific rules has the limitation of assuming that there is a document collection , any of which All of them satisfy the same ordinal identification pattern, otherwise a new set of rules is needed. The preset hierarchical matching logic proposes a general document-level title extraction rule. , automatic discovery mode, and according to The hierarchical title of the document is dynamically extracted using the pattern. The ordinal identification patterns of titles at different levels can be the same or different.

[0093] In this embodiment, the title level of the current node is determined based on the preset level matching logic, the current title ordinal identification pattern, and the previous title ordinal identification pattern, including: judging whether the current title ordinal identification pattern is the same as the previous title ordinal identification pattern; if the current title ordinal identification pattern is the same as the previous title ordinal identification pattern, then determining that the title level of the current node is the same as the title level of the previous node; if the current title ordinal identification pattern is not the same as the previous title ordinal identification pattern, then determining the title level of the current node according to the previous title ordinal identification pattern. After obtaining the current title ordinal identification pattern that matches the current node, the title level of the current node is determined based on the preset level matching logic, the current title ordinal identification pattern, and the previous title ordinal identification pattern, the previous title ordinal identification pattern is the title ordinal identification pattern of the previous node of the current node, and the process of determining the title level of the current node is:

[0094] 1) Determine whether the current title ordinal identification mode is the same as the previous title ordinal identification mode;

[0095] 1.1) If the current title ordinal identification mode is the same as the previous title ordinal identification mode, it is determined that the title level of the current node is the same as the title level of the previous node;

[0096] 1.2) If the current title ordinal identification mode is different from the previous title ordinal identification mode, the title level of the current node is determined according to the previous title ordinal identification mode.

[0097] In this embodiment, the determining of the title level of the current node according to the previous title ordinal identification mode includes: if the previous title ordinal identification mode is the root node, determining the title level of the current node as the first sub-title in the target document; if the previous title ordinal identification mode is a first-level title mode, determining the title level of the current node as a sub-level of the previous node. If the current title ordinal identification mode is different from the previous title ordinal identification mode, determining the title level of the current node according to the previous title ordinal identification mode, the specific process is:

[0098] 1.2.1) If the previous title ordinal identification mode is the root node, the title level of the current node is determined to be the first sub-title in the target document;

[0099] 1.2.2) If the ordinal identification mode of the previous title is the first-level title mode, the title level of the current node is determined to be the sublevel of the previous node.

[0100] In this embodiment, the determining of the title level of the current node according to the previous title ordinal identification pattern includes: if the previous title ordinal identification pattern is not the root node or the first-level title pattern, tracing back to the title ordinal identification pattern of the ancestor node of the previous node to find a target node with the same title ordinal identification pattern as the current title from the ancestor node of the previous node, and determining the target node as the parent node of the current node, so as to determine the sub-level of the title level of the parent node as the title level of the current node. Furthermore, if the previous title ordinal identification pattern is neither the root node nor the first-level title pattern, then the title ordinal identification pattern of the ancestor node of the previous node is traced back to find the target node with the same title ordinal identification pattern as the current title from the ancestor node of the previous node. The ancestor node refers to the parent node of a node, the parent node of the parent node, and so on in a tree data structure, until the root node. That is to say, starting from the previous node, traversing upward along the path of the tree, all the nodes passed are the ancestor nodes of the node, so as to find the target node with the same title ordinal identification pattern as the current node, and determine the target node as the parent node of the current node, so as to determine the child level of the parent node's title level as the title level of the current node.

[0101] In this embodiment, after determining the title level of the current node based on the preset level matching logic, the current title ordinal identification pattern and the previous title ordinal identification pattern, it also includes: recording the attribute information of the current node in the current content list to update the current content list; wherein the attribute information includes the name, the index of the current title ordinal identification pattern in the title ordinal identification pattern list, the index of the current node, and the parent node. It should be noted that after determining the title level of the current node, the attribute information of the current node should also be recorded in the current content list to update the current content list, and it is necessary to record the name (name) of the current node, the index (pattern) of the current title ordinal identification pattern in the title ordinal identification pattern list, the index (index) of the current node, the parent node (father), etc., wherein the index (index) of the current node indicates the title ordinal of the current node in the target document, that is, the number of the title in the target document when the current node is present; further, the attribute information may also include the length len(nodes) of the current node.

[0102] Step S14: Tracing back the complete hierarchical title information of each node to the root node based on the title hierarchy of each node to obtain the complete hierarchical structure of the target document.

[0103] In this embodiment, the complete hierarchical title information of each node is traced back to the root node based on the title hierarchy of each node to obtain the complete hierarchical structure of the target document, including: based on the title hierarchy of each node and the parent node in the attribute information, tracing back to the root node layer by layer to obtain the complete hierarchical title information of each node, and obtaining the complete hierarchical structure of the target document based on the complete hierarchical title information of each node.

[0104] After each paragraph is taken as the current node and the title level of the current node is determined, the title level of each node is obtained. Then, for any of the nodes, all the hierarchical titles of the node can be found by tracing back to the root node through the index of its parent node, thereby obtaining the complete hierarchical structure of the target document based on the complete hierarchical title information of the node.

[0105] The beneficial effects of the present application are as follows: the present application sets the main title of the target document as the root node, determines each text paragraph in the target document as each node with the root node as the starting point, and determines the current node from each of the nodes; matches the current node with a list of title ordinal identification patterns using a regular expression list constructed based on the target document to obtain a current title ordinal identification pattern that matches the current node; determines the title level of the current node based on a preset hierarchical matching logic, the current title ordinal identification pattern, and a previous title ordinal identification pattern; wherein the preset hierarchical matching logic includes that the title ordinal identification patterns of different levels are the same or different and the title ordinal identification patterns of the same level are the same, and the previous title ordinal identification pattern is the title ordinal identification pattern of the previous node of the current node; traces back the complete hierarchical title information of each node to the root node based on the title level of each node to obtain the complete hierarchical structure of the target document. It can be seen that the present application uses the main title of the target document as the root node, and traverses each text paragraph from the root node, that is, determines each paragraph as the current node in turn, and then matches the title ordinal identification for each node, so as to determine whether the current node is a title. If it is a title, the title level of the current node is determined based on the preset level matching logic, the current title ordinal identification mode and the previous title ordinal identification mode, because the preset level matching logic includes whether the title ordinal identification modes of different levels are the same or different and the title ordinal identification modes of the same level are the same, that is, in the preset level matching logic, it does not limit whether the title ordinal identification modes at different levels are the same, that is, the title ordinal identification modes at different levels can be the same. It can also be different, as long as the title ordinal identification patterns at the same level are the same. Compared with traditional solutions based on specific rules, there is no need to add new rules due to the different title ordinal identification patterns at different levels. The present application can automatically discover the title ordinal identification pattern for any document, dynamically extract the hierarchical title of the document according to the title ordinal identification pattern, and then trace the complete hierarchical title information of each node to the root node based on the title hierarchy of each node, so as to obtain the complete hierarchical structure of the target document. Compared with the cost of large model training, the present application does not require model training, and the cost of title hierarchy extraction is lower, so as to achieve low-cost and reasonable segmentation of the content in the document while maintaining the integrity and clarity of the logical structure and semantic information.

[0106] Below Figure 2 The target document is preprocessed to obtain the main title and each text paragraph (i.e. , …, ), set the main title of the target document as the root node, and use the root node as the starting point to determine each text paragraph in the target document as a node (i.e. , …, ), and input each node as the current node into the document title level extraction algorithm in turn, so as to obtain the title level of each node, wherein the document title level extraction algorithm is specifically as follows:

[0107] #For each text c starting from the second element in the content list C, perform the following operations as the current node;

[0108] FOR c in :

[0109] # Input c and the title ordinal identification pattern list P into the RexTitleMatch function, and output whether it matches (is_match) and the ordinal string of the matched pattern in the list (p_match);

[0110] is_match, p_match=RexTitleMatch(,c);

[0111] #If no pattern is matched, the current node is considered not to be a title of any level, then subsequent operations on the current node are stopped and the pattern matching operation on the next node is performed;

[0112] IF is_match==False: CONTINUE;

[0113] #If the current title ordinal identification mode is the same as the previous title ordinal identification mode, then determine that the title level of the current node is the same as the title level of the previous node:

[0114] IF p_match==pre_node.pattern:

[0115] #Then you need to create a new node cur_node, which is of the TitleNode data type. Its attribute information includes name, pattern, index, len(nodes) and father;

[0116] cur_node=TitleNode(c, p_match, len(nodes), pre_node.father);

[0117] #If the current title ordinal identification mode is different from the previous title ordinal identification mode, perform the following operations:

[0118] ELSE:

[0119] #If the previous title ordinal identification mode is the root node "root", that is, the father of the current node is the root node:

[0120] IF pre_node.pattern == "root":

[0121] #Then the current node is the first subheading of the full text, that is, the title level of the current node is determined to be the first subheading in the target document, then a record is made, recording the pattern sequence number of the current node as the index number of the first-level title pattern, and assigning the value to section_pattern;

[0122] section_pattern = p_match;

[0123] #And you need to create a new node cur_node, which is of the TitleNode data type. Its attribute information includes name, pattern, index, len(nodes) and father. The value of the father attribute is the index of the root node.

[0124] cur_node=TitleNode(c, p_match, len(nodes), root.index)

[0125] #If the previous title ordinal identification pattern is not the root node "root", but the index number section_pattern of the first-level title pattern:

[0126] ELIF pre_node.pattern==section_pattern:

[0127] #Then determine that the title level of the current node is the sub-level of the previous node, that is, the current node is the first sub-title of a secondary title. You need to create a new node cur_node to record the current node information, and the value of the father attribute of the current node is the index of the previous node;

[0128] cur_node=TitleNode(c, p_match, len(nodes), pre_node.index);

[0129] #If the previous title ordinal identification pattern is neither the root node "root" nor the first-level title pattern section_pattern, then it is necessary to determine whether the current title ordinal identification pattern is the same as a certain level of ancestor node of the previous node. If not, the current node is considered to be a child node of the previous node:

[0130] ELSE:

[0131] #In order to make this judgment, first use a temporary variable temp_pre_node to record the value of the local variable pre_node;

[0132] temp_pre_node = pre_node;

[0133] #candidate_father_id is the index of the candidate parent node, initialized to the parent node index of temp_pre_node;

[0134] candidate_father_id=temp_pre_node.father;

[0135] #From the nodes list, use the index of the candidate parent node to retrieve (index to) the candidate parent node from the nodes list;

[0136] candidate_father=nodes[candidate_father_id];

[0137] #The following WHILE loop does the following: traverse the ancestor nodes of the previous node, that is, look for the parent node layer by layer, and determine whether the current title ordinal identification mode is consistent with a certain ancestor node;

[0138] #When the candidate parent node pattern is inconsistent with the current title ordinal identification pattern p_match, this loop will continue:

[0139] WHILE candidate_father.pattern! =p_match:

[0140] #If the pattern of the candidate parent node is already the pattern of the first-level title, stop the loop (because the root node is the previous one, and if it has not appeared here, it proves that the head has been found, so the loop can be stopped);

[0141] IF candidate_father.pattern==section_pattern: BREAK;

[0142] #Update the value of temp_pre_node and assign it to the candidate parent node;

[0143] temp_pre_node = candidate_father;

[0144] #Update the index value of the candidate parent node and assign it to the parent node of temp_pre_node;

[0145] candidate_father_id=temp_pre_node.father;

[0146] #Update the value of the candidate parent node by using the index of the candidate parent node to retrieve (index to) the candidate parent node from the nodes list;

[0147] candidate_father=nodes[candidate_father_id];

[0148] #After completing this loop, it should be noted that the loop stops when the pattern of the candidate parent node is found to be the same as the ordinal identification pattern of the current title, or the current pattern is not found after indexing forward to the first-level title. Further processing is performed for these two stop situations respectively;

[0149] #The current title ordinal identification mode is the same as a certain level of ancestors of the previous node:

[0150] IF candidate_father.pattern==p_match:

[0151] #Then the current node is considered to be at the same level as the ancestor, and a new node cur_node is created, whose father attribute value is the father attribute value of the candidate parent node;

[0152] cur_node=TitleNode(c, p_match, len(nodes), candidate_father.father);

[0153] #Otherwise, that is to say, if the current mode is not found even after going back to the first-level title, then this mode is a completely new mode:

[0154] ELSE:

[0155] #At this point, we can assume that the current node is the first child node of the previous node, so we create a new node cur_node, whose father attribute value is the index of the previous node;

[0156] cur_node=TitleNode(c, p_match, len(nodes), pre_node.index);

[0157] #Add a new element to the nodes list to record the current node;

[0158] nodes.append(cur_node);

[0159] #Update the value of pre_node and assign it to the current node;

[0160] pre_node=cur_node.

[0161] Through the above algorithm, the specific nodes can be output ( , …, ), as shown in Table 1:

[0162] Table 1

[0163]

[0164] Furthermore, based on the title level of each node, the complete hierarchical title information of each node is traced back to the root node to obtain the complete hierarchical structure of the target document. The specific operation is: based on the title level and attribute information of each node, the parent node is traced back to the root node layer by layer to obtain the complete hierarchical title information of each node, and the complete hierarchical structure of the target document is obtained based on the complete hierarchical title information of each node.

[0165] See also Figure 3 As shown, the embodiment of the present application discloses a document title level extraction device, comprising:

[0166] A node determination module 11 is used to set the main title of the target document as a root node, determine each text paragraph in the target document as a node with the root node as the starting point, and determine the current node from each of the nodes;

[0167] A pattern matching module 12, configured to match the current node with a list of title ordinal identification patterns using a regular expression list constructed based on the target document, so as to obtain a current title ordinal identification pattern that matches the current node;

[0168] The level determination module 13 is used to determine the title level of the current node based on a preset level matching logic, the current title ordinal identification mode and the previous title ordinal identification mode; wherein the preset level matching logic includes that the title ordinal identification modes of different levels are the same or different and the title ordinal identification modes of the same level are the same, and the previous title ordinal identification mode is the title ordinal identification mode of the previous node of the current node;

[0169] The structure acquisition module 14 is used to trace back the complete hierarchical title information of each node to the root node based on the title hierarchy of each node, so as to acquire the complete hierarchical structure of the target document.

[0170] The beneficial effects of the present application are as follows: the present application sets the main title of the target document as the root node, determines each text paragraph in the target document as each node with the root node as the starting point, and determines the current node from each of the nodes; matches the current node with a list of title ordinal identification patterns using a regular expression list constructed based on the target document to obtain a current title ordinal identification pattern that matches the current node; determines the title level of the current node based on a preset hierarchical matching logic, the current title ordinal identification pattern, and a previous title ordinal identification pattern; wherein the preset hierarchical matching logic includes that the title ordinal identification patterns of different levels are the same or different and the title ordinal identification patterns of the same level are the same, and the previous title ordinal identification pattern is the title ordinal identification pattern of the previous node of the current node; traces back the complete hierarchical title information of each node to the root node based on the title level of each node to obtain the complete hierarchical structure of the target document. It can be seen that the present application uses the main title of the target document as the root node, and traverses each text paragraph from the root node, that is, determines each paragraph as the current node in turn, and then matches the title ordinal identification for each node, so as to determine whether the current node is a title. If it is a title, the title level of the current node is determined based on the preset level matching logic, the current title ordinal identification mode and the previous title ordinal identification mode, because the preset level matching logic includes whether the title ordinal identification modes of different levels are the same or different and the title ordinal identification modes of the same level are the same, that is, in the preset level matching logic, it does not limit whether the title ordinal identification modes at different levels are the same, that is, the title ordinal identification modes at different levels can be the same. It can also be different, as long as the title ordinal identification patterns at the same level are the same. Compared with traditional solutions based on specific rules, there is no need to add new rules due to the different title ordinal identification patterns at different levels. The present application can automatically discover the title ordinal identification pattern for any document, dynamically extract the hierarchical title of the document according to the title ordinal identification pattern, and then trace the complete hierarchical title information of each node to the root node based on the title hierarchy of each node, so as to obtain the complete hierarchical structure of the target document. Compared with the cost of large model training, the present application does not require model training, and the cost of title hierarchy extraction is lower, so as to achieve low-cost and reasonable segmentation of the content in the document while maintaining the integrity and clarity of the logical structure and semantic information.

[0171] Furthermore, an embodiment of the present application also provides an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0172] Figure 4A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the document title level extraction method performed by the electronic device disclosed in any of the aforementioned embodiments.

[0173] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0174] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0175] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0176] Among them, the operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, which can be Windows, Unix, Linux, etc. In addition to including a computer program that can be used to complete the document title level extraction method performed by the electronic device disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to data received by the electronic device and transmitted from an external device, the data 223 can also include data collected by its own input and output interface 25, etc.

[0177] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the document title level extraction method disclosed above. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the above embodiments, and no further description will be given here.

[0178] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0179] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be implemented directly with hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable EPROM (Erasable Programmable Read Only Memory), electrically erasable programmable EEPROM (Electrically Erasable Programmable read only memory), register, hard disk, removable disk, CD-ROM (Compact Disc Read-Only Memory), or any other form of storage medium known in the technical field.

[0180] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0181] The above is a detailed introduction to a document title hierarchy extraction method, device, equipment and medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in the field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A document title level extraction method, characterized in that: include: The main title of the target document is set as a root node, each text paragraph in the target document is determined as a node with the root node as the starting point, and the current node is determined from each of the nodes; Matching the current node with a list of title ordinal identification patterns using a regular expression list constructed based on the target document to obtain a current title ordinal identification pattern that matches the current node; Determine the title level of the current node based on a preset level matching logic, the current title ordinal identification mode and the previous title ordinal identification mode; wherein the preset level matching logic includes that the title ordinal identification modes of different levels are the same or different and the title ordinal identification modes of the same level are the same, and the previous title ordinal identification mode is the title ordinal identification mode of the previous node of the current node; Based on the title level of each of the nodes, the complete hierarchical title information of each of the nodes is traced back to the root node to obtain the complete hierarchical structure of the target document.

2. The document title level extraction method according to claim 1, characterized in that: The step of matching the current node with a list of title ordinal identification patterns using a regular expression list constructed based on the target document to obtain a current title ordinal identification pattern matching the current node includes: Using a regular expression list constructed based on the target document, the current node is matched with a title ordinal identification pattern list to obtain a matching result; If the matching result indicates that there is no title ordinal identification pattern matching the current node in the title ordinal identification pattern list, it is determined that the category of the current node is not a title category, and the process jumps again to the step of determining the current node from each of the nodes; If the matching result indicates that there is a title ordinal identification pattern matching the current node in the title ordinal identification pattern list, then the current title ordinal identification pattern matching the current node is acquired.

3. The document title level extraction method according to claim 2, characterized in that: The matching of the current node with the title ordinal identification pattern list by using the regular expression list constructed based on the target document includes: A regular expression list is constructed based on the target document, and the title ordinal identification pattern list and the current node are input into a regular expression title matching function to match the current node with the title ordinal identification pattern list.

4. The document title level extraction method according to claim 1, characterized in that: The determining the title level of the current node based on the preset level matching logic, the current title ordinal identification mode and the previous title ordinal identification mode includes: Determining whether the current title ordinal identification mode is the same as the previous title ordinal identification mode; If the current title ordinal identification mode is the same as the previous title ordinal identification mode, determining that the title level of the current node is the same as the title level of the previous node; If the current title ordinal identification mode is different from the previous title ordinal identification mode, the title level of the current node is determined according to the previous title ordinal identification mode.

5. The document title level extraction method according to claim 4, characterized in that: The determining the title level of the current node according to the previous title ordinal identification mode includes: If the previous title ordinal identification mode is the root node, determining the title level of the current node to be the first subtitle in the target document; If the ordinal identification mode of the previous title is a first-level title mode, the title level of the current node is determined to be a sub-level of the previous node.

6. The document title level extraction method according to claim 5, characterized in that: The determining the title level of the current node according to the previous title ordinal identification mode includes: If the previous title ordinal identification pattern is not the root node or the first-level title pattern, then the title ordinal identification pattern of the ancestor node of the previous node is traced back to find a target node with the same title ordinal identification pattern as the current title ordinal identification pattern from the ancestor node of the previous node, and the target node is determined as the parent node of the current node, so as to determine the sub-level of the title level of the parent node as the title level of the current node.

7. The document title level extraction method according to any one of claims 1 to 6, characterized in that: After determining the title level of the current node based on the preset level matching logic, the current title ordinal identification mode and the previous title ordinal identification mode, the method further includes: Recording the attribute information of the current node in the current content list to update the current content list; wherein the attribute information includes the name, the index of the current title ordinal identification mode in the title ordinal identification mode list, the index of the current node, and the parent node; Accordingly, tracing back the complete hierarchical title information of each node to the root node based on the title hierarchy of each node to obtain the complete hierarchical structure of the target document includes: Based on the title level of each node and the parent node in the attribute information, the complete hierarchical title information of each node is obtained, and the complete hierarchical structure of the target document is obtained based on the complete hierarchical title information of each node.

8. A document title level extraction device, characterized in that: include: A node determination module, used to set the main title of the target document as a root node, determine each text paragraph in the target document as a node with the root node as a starting point, and determine the current node from each of the nodes; A pattern matching module, used to match the current node with a list of title ordinal identification patterns using a regular expression list constructed based on the target document, so as to obtain a current title ordinal identification pattern that matches the current node; A level determination module, used to determine the title level of the current node based on a preset level matching logic, the current title ordinal identification mode and a previous title ordinal identification mode; wherein the preset level matching logic includes that the title ordinal identification modes of different levels are the same or different and the title ordinal identification modes of the same level are the same, and the previous title ordinal identification mode is the title ordinal identification mode of the previous node of the current node; The structure acquisition module is used to trace back the complete hierarchical title information of each node to the root node based on the title hierarchy of each node, so as to obtain the complete hierarchical structure of the target document.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the steps of the document title hierarchy extraction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, the steps of the document title hierarchy extraction method according to any one of claims 1 to 7 are implemented.

Images