Document processing method and device, storage medium, terminal, and computer program product

Abstract

A document processing method and device, storage medium, terminal, and computer program product, the method comprising: determining a document to be processed; parsing each page of the document to be processed to determine a content list and a directory list, wherein the content list contains multiple paragraph items, each paragraph item contains a single paragraph, and the directory list contains multiple title items, each title item contains a single title; constructing a directory tree based on the directory list; traversing the directory tree layer by layer, and performing text matching on the title indicated by the traversed current node and the title indicated by the adjacent node at the same level, respectively, with at least a part of the paragraphs in the content list to determine two matching paragraphs, and then splicing all the paragraphs between the two matching paragraphs in the content list, and using the splicing result as the associated text block of the current node. By adopting the above scheme, the document can be accurately divided into text blocks and an association relationship can be established between each title to obtain a tree-structured document.

Description

Technical Field

[0001] The present invention relates to the field of document processing technology, and in particular to a document processing method and device, a storage medium, a terminal, and a computer program product. Background Art

[0002] In the question-answering system based on knowledge base retrieval (Retrieval-Augmented Generation, RAG), regular documents (for example, unstructured flat documents, or chain-structured, sequential-structured documents, etc.) are parsed and converted into tree-structured documents. The tree-structured documents are then used to construct the knowledge base to be retrieved, which helps to quickly locate the text blocks (or knowledge blocks, knowledge units) most relevant to the question, thereby greatly improving the retrieval efficiency and the accuracy of the answer.

[0003] Current document parsing technologies are mainly limited to basic text extraction and format analysis. For example, by analyzing the style information (such as font) of the text in a word document, the title and body of the document are distinguished to generate the table of contents page of the word document. However, these technologies are not accurate enough in parsing the structured information contained in the document, and are unable to effectively perform intelligent text block division according to the directory structure of the document, and are unable to accurately associate the divided text blocks with titles of different levels and multiple titles of the same level with a sequential order. These shortcomings seriously limit the ability to conduct in-depth analysis of the document content and subsequent processing. In addition, existing document parsing technologies also have problems such as low automation, low processing efficiency, and excessive reliance on manual intervention. Summary of the invention

[0004] The technical problem solved by the embodiment of the present invention is how to accurately divide a document into text blocks and establish association relationships with each title to obtain a document with a tree structure in which each title has an associated text block.

[0005] To solve the above technical problems, an embodiment of the present invention provides a document processing method, comprising the following steps: determining a document to be processed, wherein the document to be processed includes at least one page, and each page includes at least a title and / or a body text; parsing each page of the document to be processed to determine a content list and a directory list, wherein the content list includes multiple paragraph items, each paragraph item includes a single paragraph, and the directory list includes multiple title items, each title item includes a single title, wherein at least a part of the paragraphs in the content list is consistent with at least a part of the titles in the directory list; constructing a directory tree based on the directory list, wherein The nodes correspond to the titles in the directory list one by one, the hierarchy of the directory tree indicates the level of the title, and the child nodes of the same layer under the same parent node sequentially indicate the titles of the same level with a chronological order; the directory tree is traversed layer by layer, and the title indicated by the traversed current node and the titles indicated by the adjacent nodes of the same layer are respectively matched with at least a part of the paragraphs in the content list to determine two matching paragraphs, and then all the paragraphs in the content list between the two matching paragraphs are spliced, and the splicing result is used as the associated text block of the current node, so as to obtain a tree-structured document after the traversal of the directory tree is completed.

[0006] Optionally, parsing each page of the document to be processed to determine the directory list includes: traversing each page of the document to be processed, and whenever the current page traversed is a directory page, converting the current page into HTML format, and adding the conversion result to the HTML directory document; traversing each page of the HTML directory document Label, for each tags, at least based on the One or more titles indicated by the tag determine the corresponding single list item; based on all Each list item corresponding to the tag determines a plurality of title items and their arrangement order to form the directory list.

[0007] Optional, each The single list item corresponding to the tag also contains: The first distance between the starting position of each title indicated by the tag and the left edge of the page to which it belongs; the first distance between the starting position of each title indicated by the tag and the left edge of the page to which it belongs; The method comprises: providing multiple distance intervals, counting the situations where each first distance contained in each list item falls into the distance interval; if all the first distances fall into a single distance interval, confirming that the directory page only contains a single-column directory; determining a corresponding single title item based on the title in each list item to obtain the multiple title items, and arranging each of the HTML directory document into a single column; The order of traversal of the tags is used as the order of arrangement of the title items to form the directory list.

[0008] Optional, each The single list item corresponding to the tag also contains: The second distance between the starting position of each title indicated by the tag and the top edge of the page to which it belongs; the method also includes: if the first distances contained in each list item fall into at least two distance intervals, confirming that the directory page contains at least two columns of directories; for each distance interval, based on the titles to which each first distance falling into the distance interval belongs, determining a single title item, thereby obtaining one or more title items corresponding to the distance interval; sorting the second distances of each title falling into the distance interval from small to large, and using the sorting as the arrangement order of each title item corresponding to the distance interval, so as to obtain a sub-list of title items corresponding to the distance interval; sorting the sub-list of title items corresponding to each distance interval according to the lower limit values ​​or upper limit values ​​of the multiple distance intervals in ascending order, so as to form the directory list.

[0009] Optionally, each page of the document to be processed is parsed to determine the content list and the directory list, including: traversing each page of the document to be processed, and whenever the current page traversed is a non-directory page, converting the current page into HTML format, and adding the conversion result to the HTML content document; traversing each page of the HTML content document Label, for each tags, at least based on the The paragraph indicated by the tag determines the corresponding single paragraph item; each of the HTML content document The traversal order of the tags is used as the arrangement order of the corresponding paragraph items to form the content list.

[0010] Optional, each The single paragraph item corresponding to the tag also contains: The method further comprises: if it is confirmed that each page of the document to be processed does not include a directory page, then based on at least the style information of each paragraph item in the content list, filtering the paragraph items in the content list to determine the directory list.

[0011] Optionally, the style information includes the font size of the paragraph to which it belongs; and the screening of the paragraph items in the content list based at least on the style information of each paragraph item in the content list to determine the directory list includes: counting the frequency of occurrence of the font size of each paragraph in the content list, and taking the font size with the highest frequency of occurrence as the main text size; screening out font sizes that are larger than the main text size and whose frequency of occurrence is greater than a preset frequency threshold as target font sizes; and determining the directory list based on the paragraph items to which each target font size belongs.

[0012] Optionally, the style information also includes the character color of the paragraph to which it belongs, and the third distance between the starting position of the paragraph and the left edge of the page to which it belongs; the directory list is determined based on the paragraph items to which each target font size belongs, including: selecting paragraph items whose character color is consistent with any one or more candidate character colors from the paragraph items to which each target font size belongs as re-selected paragraph items; summing the third distance of each re-selected paragraph item and a preset error to obtain a corresponding fourth distance; selecting paragraph items whose fourth distance falls within a preset target distance interval from the re-selected paragraph items as final-selected paragraph items; using each final-selected paragraph item as a title item, and using the arrangement order of each final-selected paragraph item in the content list as the arrangement order of each title item to form the directory list.

[0013] Optionally, constructing a directory tree based on the directory list includes: for each title in the directory list, text matching the title number in the title with a plurality of preset title number sets, wherein each title number set has its own title level, and each title number set contains a plurality of title numbers in a sequential order; if the title number in the title is consistent with one of the title numbers in any title number set, then taking the title level to which the title number set belongs as the level of the title; using each title as a node of the tree, taking the level of each title as the level to which the node belongs, and taking the sequential order of the title numbers of each title at the same level as the sequential order of the corresponding child nodes at the same level, so as to construct the directory tree.

[0014] Optionally, the at least part of the paragraphs for text matching are determined in the following manner: the title indicated by the parent node of the current node is taken as the first title, and the title indicated by the node at the same level and adjacent to the parent node is taken as the second title; in the content list, the paragraphs matching the first title and the second title are determined respectively, and recorded as the first paragraph and the second paragraph; all the paragraphs between the first paragraph and the second paragraph are taken as the at least part of the paragraphs.

[0015] Optionally, determining the document to be processed includes: obtaining an original document, the original document containing at least one table to be processed; performing preliminary parsing on each table to be processed to obtain a preliminary table, the preliminary table containing multiple cells, at least some of the cells containing the same text; traversing the first row of cells of the preliminary table, and whenever the text of a current cell is consistent with that of a previous cell in the same row, merging the consistent texts; starting from the second row of the preliminary table, whenever the text of a current cell is consistent with that of a previous cell in the same row, and / or whenever the text content of a current cell is consistent with that of a previous cell in the same column, merging the consistent texts; based on the text that has not been merged and the text that has been merged in the preliminary table, determining the corresponding converted text, and using the converted text to replace the table to be processed to which it belongs, to obtain the document to be processed.

[0016] The embodiment of the present invention further provides a document processing device, comprising: a to-be-processed document determination module, used to determine a to-be-processed document, wherein the to-be-processed document comprises at least one page, and each page comprises at least a title and / or a body text; a list determination module, used to parse each page of the to-be-processed document to determine a content list and a directory list, wherein the content list comprises a plurality of paragraph items, each paragraph item comprises a single paragraph, and the directory list comprises a plurality of title items, each title item comprises a single title, wherein at least a portion of the paragraphs in the content list is consistent with at least a portion of the titles in the directory list; and a directory tree construction module, used to construct a directory tree based on the directory list, wherein the directory tree The nodes of the directory tree correspond to the titles in the directory list one by one, the hierarchy of the directory tree indicates the level of the title, and the child nodes of the same layer under the same parent node sequentially indicate the titles of the same level with a chronological order; the tree structure document determination module is used to traverse the directory tree layer by layer, and perform text matching on the title indicated by the traversed current node and the titles indicated by the adjacent nodes of the same layer with at least a part of the paragraphs in the content list respectively to determine two matching paragraphs, and then splice all the paragraphs in the content list between the two matching paragraphs, and use the splicing result as the associated text block of the current node, so as to obtain the tree structure document after the traversal of the directory tree is completed.

[0017] An embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above document processing method are executed.

[0018] An embodiment of the present invention further provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of the above document processing method when running the computer program.

[0019] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the steps of the above document processing method when executed by a processor.

[0020] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:

[0021] In an embodiment of the present invention, structured information in a document is extracted from two dimensions, namely, directory and content, and converted into a tree-structured document. Specifically, the document to be processed is converted into a content list and a directory list, wherein the content list includes multiple paragraph items, each paragraph item includes a single paragraph, and the directory list includes multiple title items, each title item includes a single title; then the directory list is converted into a directory tree structure; each layer of nodes in the directory tree is traversed, and text matching is performed between the titles (two titles) indicated by the traversed current node and its adjacent nodes at the same layer and the paragraphs in the content list to find two matching paragraphs; then all paragraphs in the content list between the two matching paragraphs are spliced, and the splicing result is a text block between the two titles, and an association relationship can be established between the text block and the title indicated by the traversed current node, so as to obtain a tree-structured document after the traversal is completed.

[0022] From the above, this implementation scheme utilizes the characteristics that each paragraph in the content list contains at least a part of the title, and the order of each paragraph corresponds to the order of the position of the paragraph in the document, and the node hierarchy of the directory tree and the arrangement order of the nodes at the same level correspond to the title level and the order of different titles at the same level. The paragraphs between every two adjacent titles at the same level are spliced ​​to obtain the associated text block of each title, thereby realizing the precise division of the document content into multiple text blocks and establishing the association relationship between the text blocks and the titles, thereby obtaining a document with a tree structure in which each title has an associated text block.

[0023] Furthermore, in an embodiment of the present invention, in the process of matching the titles indicated by two adjacent nodes at the same level with the content list, only the title indicated by the parent node of the two adjacent nodes and the paragraphs between the titles indicated by the adjacent node at the same level of the parent node (i.e., corresponding to the two paragraphs in the content list) are used as paragraphs to be matched. Compared with matching all the paragraphs in the content list each time, this helps to significantly narrow the scope of matching, reduce computing overhead, and improve the efficiency of generating data structure documents.

[0024] Furthermore, in an embodiment of the present invention, in the case where the document to be processed does not include a directory page, the title items are filtered from the content list based on the font size and font size frequency, and also based on multiple different parameters such as font color, the distance between the starting position of the paragraph and the left edge of the page (and considering appropriate errors). Compared with the prior art that relies on a single style information in the document (e.g., font or font size) to distinguish between the title and the body, the above scheme helps to avoid missing the title or mistakenly using other content types as titles, and can accurately filter out the title items that actually indicate the title from the content list, and obtain an accurate and reliable directory list. Furthermore, the accuracy of subsequent directory tree construction and obtaining tree-structured documents can be improved.

[0025] Furthermore, the table of contents pages included in the document are divided into two situations: a single-column table of contents and a multi-column table of contents. Specifically, based on the characteristic that the first distances of various titles included in the same column of the table of contents are relatively close to the left edge of the table of contents page, while the first distances of titles in different columns of the table of contents are relatively different from the left edge of the table of contents page, it is possible to accurately determine whether the table of contents page contains a single-column table of contents or a multi-column table of contents through the number of distance intervals within which the first distances of all the titles on the table of contents page fall, and adopt different methods according to different situations to obtain the title items corresponding to all the titles, thereby forming an accurate and reliable table of contents list. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of a document processing method in an embodiment of the present invention;

[0027] Figure 2 yes Figure 1 Flow chart of a first specific implementation of step S12;

[0028] Figure 3 yes Figure 1 Flow chart of a second specific implementation of step S12;

[0029] Figure 4 yes Figure 1 Partial flow chart of a third specific implementation of step S12;

[0030] Figure 5 yes Figure 1 A flowchart of a specific implementation of step S13;

[0031] Figure 6 is a schematic diagram of building a directory tree based on a directory list in an embodiment of the present invention;

[0032] Figure 7 It is a structural schematic diagram of a document processing device in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the above-mentioned objects, features and beneficial effects of the present invention more obvious and understandable, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0034] Reference Figure 1 , Figure 1 This is a flowchart of a document processing method in an embodiment of the present invention. The method can be applied to a terminal device with a document processing function, such as but not limited to: a mobile phone, a computer, a tablet computer, a smart watch, an automatic question-answering robot, a vehicle-mounted terminal, etc., and can also be a server, a cloud platform, etc.

[0035] Specifically, taking the automatic question-answering robot as an example, it can be configured with a knowledge base to be searched, and the knowledge base can contain several tree-structured documents. After receiving the user's question, the automatic question-answering robot can determine the answer to the user's question and output it by searching the tree-structured documents.

[0036] The tree-structured document can be Figure 1 The document processing method shown is obtained, and the document processing method may specifically include the following steps S11 to S14:

[0037] Step S11: determining a document to be processed, wherein the document to be processed includes at least one page, and each page includes at least a title and / or a body text;

[0038] Step S12: parsing each page of the document to be processed to determine a content list and a directory list, wherein the content list includes a plurality of paragraph items, each paragraph item includes a single paragraph, and the directory list includes a plurality of title items, each title item includes a single title, wherein at least a portion of the paragraphs in the content list is consistent with at least a portion of the titles in the directory list;

[0039] Step S13: constructing a directory tree based on the directory list, wherein the nodes of the directory tree correspond one-to-one to the titles in the directory list, the hierarchy of the directory tree indicates the level of the title, and the same-layer child nodes under the same parent node indicate the titles of the same level in order;

[0040] Step S14: traversing the directory tree layer by layer, performing text matching on the title indicated by the current node being traversed and the title indicated by the adjacent node at the same layer with at least a part of the paragraphs in the content list, respectively, to determine two matching paragraphs, then splicing all the paragraphs in the content list between the two matching paragraphs, and using the splicing result as the associated text block of the current node, so as to obtain a tree-structured document after traversing the directory tree.

[0041] In the specific implementation of step S11, in actual applications, each page of the document to be processed may include a directory page, or may not include a directory page (that is, each page is a content page). The directory page specifically refers to a page that includes a title (and may also include a "directory" logo and information such as the page number where the title is located), but does not include text information; the content page (also referred to as a "non-directory page") may refer to a page that includes a title and text information, or only includes text information. As some non-limiting embodiments, the document to be processed may be selected from, but not limited to, a pdf document, a ppt document, a word document, etc.

[0042] Specifically, for a document to be processed that includes multiple pages, the multiple pages usually have a preset page sequence. In addition, each title included in the document to be processed has a title level to which it belongs, and multiple titles belonging to the same level and having the same upper title have a preset sequence, which can be used to indicate the position sequence of the multiple titles in the document to be processed. Each title has its own title number.

[0043] Furthermore, the document to be processed may be a document obtained by preprocessing the original document. It is understandable that, in practical applications, various types of original documents may contain not only text but also other types of information such as tables and pictures. Taking table-type information as an example, after obtaining the original document, text information of the table therein may be extracted to convert it into corresponding text, and the converted text may be used to replace the original table, thereby obtaining the document to be processed.

[0044] In a specific embodiment, a method for preprocessing an original document may include: obtaining an original document, wherein the original document includes at least one table to be processed; performing preliminary parsing on each table to be processed to obtain a preliminary table, wherein the preliminary table includes multiple cells, and at least a portion of the cells contain the same text; traversing the first row of cells of the preliminary table, and whenever the text of a current cell is consistent with that of a previous cell in the same row, merging the consistent texts; starting from the second row of the preliminary table, whenever the text of a current cell is consistent with that of a previous cell in the same row, and / or whenever the text content of a current cell is consistent with that of a previous cell in the same column, merging the consistent texts; based on the text that has not been merged and the text that has been merged in the preliminary table, determining the corresponding conversion text, and using the conversion text to replace the table to be processed to which it belongs, to obtain the document to be processed.

[0045] Among them, performing preliminary analysis on each table to be processed to obtain a preliminary table can be implemented using existing table processing tools. In the case of a merged cell, a single merged cell of the table to be processed corresponds to multiple cells in the preliminary table, and the multiple cells contain the same text. The multiple cells are actually multiple original cells before the merged cell is merged.

[0046] Wherein, determining the corresponding conversion text based on the text that has not been merged and the text that has been merged may specifically include: keeping the positions of the text that has not been merged and the text that has been merged in the preliminary table unchanged, splicing all the texts to obtain a spliced ​​text; and using the spliced ​​text as the conversion text. Alternatively, in another implementation, the storage path of the spliced ​​text may be used as the conversion text.

[0047] In an embodiment of the present invention, by adopting the above-mentioned method of traversing cells and merging text row by row, and using different text merging methods for the first row of cells and the cells in subsequent rows, it helps to avoid missing cell merges and improve the accuracy of identifying merged cells and text merging.

[0048] In the specific implementation of step S12, each page of the document to be processed is parsed to determine a content list and a directory list. In particular, different methods are used to determine the directory list for the case where the document to be processed includes a directory page and does not include a directory page. Specifically, for the case where the directory page is included, the directory page can be converted into an HTML directory document, and the directory list is obtained by parsing the HTML directory document; for the case where the directory page is not included, all non-directory pages (also called content pages) can be converted into an HTML content document, and the content list is obtained by parsing the HTML content document, and then the directory list is determined according to a part of the paragraph items in the content list.

[0049] Reference Figure 2 , Figure 2 yes Figure 1 Flow chart of the first specific implementation of step S12 in the embodiment. In this implementation, the method for determining the directory list is aimed at the case where the document to be processed contains a directory page. Specifically, in the step S12, the method for determining the directory list may include the following steps S21 to S23.

[0050] In step S21, each page of the document to be processed is traversed, and whenever the current page being traversed is a directory page, the current page is converted into HTML format, and the conversion result is added to the HTML directory document.

[0051] In a specific implementation, an HTML directory document with empty content can be pre-established, and during the traversal process, the directory information after HTML conversion (i.e., the conversion result) can be added to the HTML directory document according to the order of traversing pages. The order of traversing pages can be consistent with the order of pages in the document to be processed (e.g., the order of page numbers).

[0052] In a specific implementation, whether the current page is a directory page can be determined based on whether the current page contains a "directory" logo.

[0053] In step S22, traverse each of the HTML directory documents Label, for each tags, at least based on the One or more titles indicated by the tag determine the corresponding single list item.

[0054] For documents in HTML format, The tag is used to indicate a paragraph in the document to be processed.

[0055] In a specific embodiment, the The tag indicates one or more titles, as the In another specific implementation, in addition to using the The tag indicates one or more titles, and can also The tag extracts other appropriate information (for example, the position information of the title (left value, top value) and / or style information (font-family value, font-size value and color value of the style attribute) to form the The single list item corresponding to the label.

[0056] Among them, the left value represents The first distance between the starting position of the title (or paragraph) indicated by the tag and the left edge of the page to which the title (or paragraph) belongs. The top value indicates The second distance between the start position of the title (or paragraph) indicated by the tag and the top edge of the page to which the title (or paragraph) belongs. The font-family value, font-size value, and color value represent The font, font size, and font color of the title (or paragraph) indicated by the tag.

[0057] In step S23, based on all Each list item corresponding to the tag determines a plurality of title items and their arrangement order to form the directory list.

[0058] For documents in HTML format, Tags are usually used to define paragraphs. In regular documents, a title usually occupies a paragraph (or a line). Traversal and parsing of tags to accurately distinguish multiple titles in different paragraphs (or different lines).

[0059] It is understandable that a directory page may contain either a single-column directory or a multi-column directory. For single-column directories and multi-column directories, the way to form the directory list is also different. Specifically, for a directory page containing only a single-column directory, each The tag usually indicates a single heading located in the same paragraph (or line) of the directory page. Accordingly, each The single list item corresponding to the tag also contains only a single title, that is, each list item can correspond to a single title item; for a directory page containing multiple columns, each The tag usually indicates multiple headings in different columns of the same paragraph (or the same line) of the directory page. A single list item corresponding to a tag also includes multiple titles, that is, each list item can correspond to multiple title items.

[0060] Furthermore, each The single list item corresponding to the tag also contains: The first distance between the starting position of each title indicated by the tag and the left edge of the corresponding page (i.e., the left value); the step S23 may include: providing multiple distance intervals, and counting the situations in which each first distance contained in each list item falls into the distance interval; if all the first distances fall into a single distance interval, confirming that the directory page only contains a single-column directory; determining a corresponding single title item based on the title in each list item to obtain the multiple title items, and The order of traversal of the tags is used as the order of arrangement of the title items to form the directory list.

[0061] Specifically, for the case where the above-mentioned directory page only contains a single-column directory, since each list item only contains a single title, each list item can be directly used as a single title item; or, the title and other appropriate information can be extracted from each list item to form a corresponding single title item.

[0062] Furthermore, each The single list item corresponding to the tag also contains: The second distance (i.e., top value) between the starting position of each title indicated by the tag and the top edge of the page to which it belongs; the method further includes: if each first distance contained in each list item falls into at least two distance intervals, confirming that the directory page contains at least two columns of directories; for each distance interval, determining a single title item based on the title to which each first distance falling into the distance interval belongs, thereby obtaining one or more title items corresponding to the distance interval; sorting the second distances of each title falling into the distance interval from small to large, and using the sorting as the arrangement order of each title item corresponding to the distance interval, so as to obtain a sub-list of title items corresponding to the distance interval; sorting the sub-list of title items corresponding to each distance interval according to the lower limit values ​​or upper limit values ​​of the multiple distance intervals from small to large, so as to form the directory list.

[0063] In an embodiment of the present invention, based on the characteristic that the first distances of the titles included in the same column of the directory are relatively close to the left edge of the directory page, while the first distances of the titles of different columns of the directory are relatively different from the left edge of the directory page, it is possible to accurately determine whether the directory page contains a single-column directory or a multi-column directory through the number of distance intervals into which the first distances of all the titles of the directory page fall, and adopt different methods to obtain the title items corresponding to all the titles according to different situations, thereby forming an accurate and reliable directory list.

[0064] Reference Figure 3 , Figure 3 yes Figure 1 This embodiment provides a method for determining a content list. In the step S12, the method for determining a content list may specifically include the following steps S31 to S33.

[0065] In step S31, the pages of the document to be processed are traversed, and whenever the current page being traversed is a non-directory page, the current page is converted into HTML format, and the conversion result is added to the HTML content document.

[0066] Since the steps of traversing the page and adding the conversion results do not differ depending on whether the page is a non-directory page or a directory page, for the method of traversing the page and adding the conversion results to the HTML content document, please refer to Figure 2 The description of the specific implementation in step S21 is not repeated here.

[0067] In step S32, traverse each of the HTML content documents Label, for each tags, at least based on the The paragraph indicated by the tag determines the corresponding single paragraph item.

[0068] Among them, based on The method for determining the corresponding single paragraph item for the paragraph indicated by the tag can refer to the previous article on The method of determining the corresponding single list item for one or more titles indicated by the tag is not described in detail here.

[0069] As mentioned above, in HTML documents Tags are usually used to define paragraphs. It should be noted that since non-table of contents pages (also called "content pages") can contain titles and text, and in conventional documents, titles usually occupy a paragraph (or line), each The paragraph indicated by the tag can be a paragraph as the main text or a paragraph as the title. That is, the paragraph information contained in a single paragraph item can be title information or main text information.

[0070] In step S33, each of the HTML content documents is The traversal order of the tags is used as the arrangement order of the corresponding paragraph items to form the content list.

[0071] It should be pointed out that in addition to the above-mentioned method for determining the content list, other appropriate methods can also be used. For example, the text style information of the document to be processed can be analyzed, such as by analyzing the font size, taking the font size greater than or equal to the preset font size threshold or the font size in the first preset font size range as the title, taking the font size less than the preset font size threshold or the font size in the second preset range as the text, and combining the distance between the font and the left edge of the page to determine the different paragraphs where the title and the text are located, so as to determine the content list containing the title and the text, and the directory list containing only the title. Among them, the font size in the first preset font size range is larger than the font size in the second preset range.

[0072] Reference Figure 4 , Figure 4 yes Figure 1 Partial flow chart of the third specific implementation of step S12 in the embodiment. In this implementation, the method for determining the directory list is aimed at the case where the document to be processed does not contain a directory page (that is, all pages are non-directory pages).

[0073] Specifically, in this embodiment, each The single paragraph item corresponding to the tag also contains: The style information of the paragraph indicated by the tag, in step S12, the method for determining the directory list may specifically include Figure 3 The steps S31 to S33 shown in the figure may further include the following steps S41 to S42, wherein step S42 is executed based on the content list obtained in step S33, and there is no particular order in which step S41 and steps S31 to S33 are executed. Figure 3 The different contents in are explained.

[0074] In step S41, it is confirmed that the pages of the document to be processed do not include a directory page.

[0075] In step S42, the paragraph items in the content list are screened based on at least the style information of each paragraph item in the content list to determine the directory list.

[0076] In a specific implementation, the paragraph items in the content list can be screened using the font size (i.e., font-size value) in the style information of each paragraph item, which can specifically include: counting the frequency of occurrence of the font size of each paragraph in the content list, and taking the font size with the highest frequency of occurrence as the main text size; screening out the font size that is larger than the main text size and whose frequency of occurrence is larger than a preset frequency threshold as the target font size; and determining the directory list based on the paragraph items to which each target font size belongs. Among them, the paragraph items to which each target font size belongs can be used as the primary paragraph items.

[0077] It is understandable that if one only relies on font size to distinguish between titles and other content such as text, other content with larger font sizes may be mistakenly identified as titles (for example, the name of a chart or explanatory content, etc.). Therefore, in an embodiment of the present invention, not only the font size parameter is considered, but also the frequency of font size occurrence is counted, which helps to more accurately identify titles from a variety of different content types.

[0078] In another specific implementation, on the basis of determining the preliminary selected paragraph items, the preliminary selected paragraph items may be further screened in two rounds (that is, this embodiment includes three rounds of screening operations in total) by combining the font color (i.e., color value) in the style information of each preliminary selected paragraph item and the third distance between the start position of the paragraph and the left edge of the corresponding page (i.e., left value), and the preliminary selected paragraph items may be further screened in two rounds, which specifically includes the following steps:

[0079] The method comprises the following steps: selecting paragraph items whose character colors are consistent with any one or more candidate character colors from paragraph items to which each target font size belongs (i.e., preliminary paragraph items) as reselected paragraph items; summing the third distance of each reselected paragraph item with a preset error to obtain a corresponding fourth distance; selecting paragraph items whose fourth distances fall within a preset target distance interval from the reselected paragraph items as final selected paragraph items; using each final selected paragraph item as a title item, and using the arrangement order of each final selected paragraph item in the content list as the arrangement order of each title item to form the directory list.

[0080] In the embodiment of the present invention, the situation that the document contains a directory page and a non-directory page is distinguished, and different methods are adopted accordingly to determine the title in the document to form a directory list. Figure 2 The solution for determining the directory list in the illustrated embodiment is for the case where the document contains a directory page. Figure 4 The solution for determining a directory list in the illustrated embodiment is for a situation where the document does not contain a directory page.

[0081] Furthermore, for the case where the directory page is not included, this implementation scheme not only combines the font size and font size frequency, but also further combines multiple different parameters such as font color, the distance between the starting position of the paragraph and the left edge of the page (and taking into account appropriate errors). Compared with the prior art that relies on a single style information in the document (e.g., font or font size) to distinguish between the title and the text, the above scheme helps to avoid missing the title or mistakenly using other content types as titles, and can accurately filter out the title items that actually indicate the title from the content list, and obtain an accurate and reliable directory list. Furthermore, the accuracy of subsequent directory tree construction and tree-structured documents can be improved.

[0082] Continue to refer to Figure 1 In the specific implementation of step S13, a directory tree is constructed based on the directory list. The same-layer child nodes under the same parent node indicate the titles with a sequence at the same level in sequence, which specifically means that the arrangement order of the same-layer child nodes under the same parent node (for example, from left to right) is consistent with the sequence of multiple titles belonging to the same level and having the same upper-level title, and the upper-level title corresponds to the parent node of the same-layer child node. The sequence of the multiple titles can correspond to the sequence of the positions of the multiple titles in the document to be processed.

[0083] Reference Figure 5 , Figure 5 yes Figure 1 A flowchart of a specific implementation of step S13 in the embodiment of the present invention. In the step S13, the method of constructing a directory tree based on the directory list may include the following steps S51 to S53.

[0084] In step S51, for each title in the directory list, the title number in the title is text-matched with multiple preset title number sets, wherein each title number set has its own title level, and each title number set contains multiple title numbers in a sequential order.

[0085] In step S52, if the title number in the title is consistent with one of the title numbers in any title number set, the title level to which the title number set belongs is used as the level of the title.

[0086] In step S53, each title is used as a node of the tree, the level of each title is used as the layer to which the node belongs, and the sequence of the title numbers of the titles at the same level is used as the sequence of the corresponding child nodes at the same layer to construct the directory tree.

[0087] In an embodiment of the present invention, by adopting the above-mentioned method of matching the numbers in the titles using multiple preset title number sets, the level to which each title belongs and the order between multiple titles under the same title can be determined to construct an accurate directory tree structure.

[0088] It should be pointed out that in addition to using the above method to construct a directory tree, the level relationship of the title can also be determined based on the distance between the starting position of the title (i.e., the position of the first word of the title from the left to the back in the page) and the left edge of the page, thereby constructing a directory tree.

[0089] The following combination Figure 6 And an example explains the constructed directory tree structure in detail.

[0090] Reference Figure 6 , Figure 6 It is a schematic diagram of constructing a directory tree based on a directory list in an embodiment of the present invention.

[0091] As a non-limiting example, the first-level titles of a document to be processed about nutrition may include "Chapter 1 Vitamins", "Chapter 2 Minerals", "Chapter 3 Proteins", "Chapter 4 Fats", etc.; further, the second-level titles under the first-level title "Chapter 1 Vitamins" may include: "Section 1 Introduction to Vitamins", "Section 2 Benefits of Vitamins", "Section 3 Side Effects of Vitamins", etc.; further, the third-level titles under the second-level title "Section 1 Introduction to Vitamins" may include: "1.1 Origin of Vitamins", "1.2 Categories of Vitamins", etc.

[0092] The directory list obtained by parsing the document to be processed is as follows Figure 6 As shown on the left (only some of the title items are shown in the figure), the directory tree constructed based on the directory list shown on the left is as follows Figure 6 As shown on the right (only some nodes are shown as examples in the figure), the root node of the directory tree can be set to an empty node, and the nodes in the first layer from left to right under the root node correspond to the first-level titles with a sequence (i.e., "Chapter 1 Vitamins", "Chapter 2 Minerals", "Chapter 3 Proteins", "Chapter 4 Fat"), and the child nodes of the next layer under the "Chapter 1 Vitamins" node correspond to the second-level titles with a sequence from left to right (i.e., "Section 1 Introduction to Vitamins", "Section 2 Benefits of Vitamins", "Section 3 Side Effects of Vitamins", etc.), and the child nodes of the next layer under the "Section 1 Introduction to Vitamins" node correspond to the third-level titles with a sequence from left to right "1.1 Origin of Vitamins", "1.2 Categories of Vitamins", etc.

[0093] Continue to refer to Figure 1 In the specific implementation of step S14, it should be noted that the paragraph items in the content list are arranged in order, and the arrangement order of the paragraph items is consistent with the position order of the corresponding paragraphs in the document to be processed. In addition, since the arrangement order of the same-layer child nodes of the same parent node is consistent with the order of multiple titles belonging to the same level and with the same upper-level title, the titles (two titles) indicated by each two adjacent nodes with the same parent node and the same layer can be text-matched with the paragraph items in the content list, and the two paragraphs that are consistent with the two title texts (or the text overlap degree is greater than the target threshold) are regarded as the two matching paragraphs.

[0094] Specifically, all the paragraphs between the two matching paragraphs in the content list are all the paragraphs between the positions of the two titles in the to-be-processed document. Then, by splicing all the paragraphs between the positions of the two titles, and using the splicing result as the associated text block of the current node (that is, establishing an association relationship between the spliced ​​text block and the title), a tree-structured document is obtained in which each title has an associated text block.

[0095] From the above, this implementation scheme utilizes the characteristics that each paragraph in the content list contains at least a part of the title, and the order of each paragraph corresponds to the order of the position of the paragraph in the document, and the node hierarchy of the directory tree and the arrangement order of the nodes at the same level correspond to the title level and the order of different titles at the same level. The paragraphs between every two adjacent titles at the same level are spliced ​​to obtain the associated text blocks of each title, thereby achieving the precise division of the document content into multiple text blocks and obtaining a document with a tree structure in which each title has an associated text block.

[0096] Furthermore, the at least a portion of the paragraphs for text matching (i.e., the paragraphs to be matched) can be determined in the following manner: the title indicated by the parent node of the current node is taken as the first title, and the title indicated by the node at the same level and adjacent to the parent node is taken as the second title; in the content list, the paragraphs matching the first title and the second title are determined respectively, and recorded as the first paragraph and the second paragraph; all the paragraphs between the first paragraph and the second paragraph are taken as the at least a portion of the paragraphs.

[0097] For example, the first title indicated by the parent node of the current node is "Chapter 1 xxx" (the first title is also the first paragraph in the content list), and the second title indicated by the node at the same level and adjacent to the parent node is "Chapter 2 xxx" (the second title is also the second paragraph in the content list). Therefore, the at least part of the paragraphs for text matching (i.e., the paragraphs to be matched) can be all the paragraphs between "Chapter 1 xxx" and "Chapter 2 xxx".

[0098] In the embodiment of the present invention, in the process of matching the titles indicated by two adjacent nodes at the same level with the content list, only the title indicated by the parent node of the two adjacent nodes and the paragraphs between the titles indicated by the adjacent node at the same level of the parent node (i.e., corresponding to the two paragraphs in the content list) are used as paragraphs to be matched. Compared with matching with all paragraphs in the content list every time, this helps to significantly narrow the scope of matching, reduce computing overhead, and improve the generation efficiency of data structure documents.

[0099] Reference Figure 7 , Figure 7 1 is a schematic diagram of the structure of a document processing device in an embodiment of the present invention. The document processing device may include:

[0100] A to-be-processed document determination module 71 is used to determine a to-be-processed document, wherein the to-be-processed document comprises at least one page, and each page comprises at least a title and / or a body text;

[0101] A list determination module 72 is used to parse each page of the document to be processed to determine a content list and a directory list, wherein the content list includes a plurality of paragraph items, each paragraph item includes a single paragraph, and the directory list includes a plurality of title items, each title item includes a single title, wherein at least a portion of the paragraphs in the content list is consistent with at least a portion of the titles in the directory list;

[0102] A directory tree construction module 73 is used to construct a directory tree based on the directory list, wherein the nodes of the directory tree correspond to the titles in the directory list one by one, the hierarchy of the directory tree indicates the level of the title, and the same-layer child nodes under the same parent node indicate the titles of the same level in order;

[0103] The tree-structured document determination module 74 is used to traverse the directory tree layer by layer, perform text matching on the title indicated by the current node being traversed and the title indicated by the adjacent node at the same layer with at least a part of the paragraphs in the content list, respectively, to determine two matching paragraphs, and then splice all the paragraphs in the content list that are between the two matching paragraphs, and use the splicing result as the associated text block of the current node, so as to obtain a tree-structured document after the directory tree is traversed.

[0104] For the principle, specific implementation and beneficial effects of the document processing device, please refer to the previous article and Figures 1 to 5 The relevant description about the document processing method shown in any embodiment will not be repeated here.

[0105] The embodiment of the present invention further provides a storage medium, such as a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor. Figures 1 to 5 The steps of the document processing method shown in any embodiment. The computer-readable storage medium may include a non-volatile memory or a non-transitory memory, and may also include an optical disk, a mechanical hard disk, a solid-state hard disk, etc.

[0106] Specifically, in the embodiment of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0107] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).

[0108] The embodiment of the present invention further provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the above-mentioned Figures 1 to 5 The steps of the document processing method shown in any embodiment.

[0109] The embodiment of the present invention further provides a computer program product, including a computer program, wherein the computer program is executed by a processor to implement the above-mentioned Figures 1 to 5 The steps of the document processing method shown in any embodiment.

[0110] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0111] The "multiple" appearing in the embodiments of the present application refers to two or more.

[0112] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0113] It should be pointed out that the serial numbers of the steps in this embodiment do not limit the execution order of the steps.

[0114] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A document processing method, characterized in that: include: Determine a document to be processed, wherein the document to be processed includes at least one page, and each page includes at least a title and / or a body text; Parsing each page of the document to be processed to determine a content list and a directory list, wherein the content list includes a plurality of paragraph items, each paragraph item includes a single paragraph, and the directory list includes a plurality of title items, each title item includes a single title, wherein at least a portion of the paragraphs in the content list is consistent with at least a portion of the titles in the directory list; Building a directory tree based on the directory list, wherein the nodes of the directory tree correspond to the titles in the directory list one by one, the hierarchy of the directory tree indicates the level of the title, and the child nodes of the same layer under the same parent node indicate the titles of the same level in order; Traversing the directory tree layer by layer, performing text matching on the title indicated by the current node and the title indicated by the adjacent node at the same layer with at least a part of the paragraphs in the content list, respectively, to determine two matching paragraphs, then splicing all the paragraphs between the two matching paragraphs in the content list, and using the splicing result as the associated text block of the current node, so as to obtain a tree-structured document after traversing the directory tree; The at least one portion of paragraphs for text matching is determined in the following manner: The title indicated by the parent node of the current node is used as the first title, and the title indicated by the node at the same level and adjacent to the parent node is used as the second title; In the content list, paragraphs matching the first title and the second title are determined respectively, and recorded as the first paragraph and the second paragraph; All paragraphs between the first paragraph and the second paragraph are regarded as the at least one part of the paragraphs.

2. The method according to claim 1, characterized in that Each page of the document to be processed is parsed to determine that the directory list includes: Traversing each page of the document to be processed, and whenever the current page being traversed is a directory page, converting the current page into HTML format, and adding the conversion result to the HTML directory document; Traverse each of the HTML directory documents Label, for each tags, at least based on the One or more titles indicated by the tag determine the corresponding single list item; Based on all Each list item corresponding to the tag determines a plurality of title items and their arrangement order to form the directory list.

3. The method according to claim 2, characterized in that Each The single list item corresponding to the tag also contains: The first distance between the start position of each heading indicated by the label and the left edge of the page to which it belongs; Based on all Each list item corresponding to the tag determines multiple title items and their arrangement order to form the directory list, including: Providing multiple distance intervals, and counting situations where each first distance included in each list item falls within the distance interval; If all the first distances fall within a single distance interval, then confirming that the directory page only contains a single-column directory; Determine a corresponding single title item based on the title in each list item to obtain the plurality of title items, and The order of traversal of the tags is used as the order of arrangement of the title items to form the directory list.

4. The method according to claim 3, characterized in that: Each The single list item corresponding to the tag also contains: The second distance between the start of each heading indicated by the label and the top edge of the page to which it belongs; The method further comprises: If each first distance included in each list item falls into at least two distance intervals, then confirming that the directory page includes at least two columns of directories; For each distance interval, a single title item is determined based on the titles to which each first distance falling within the distance interval belongs, thereby obtaining one or more title items corresponding to the distance interval; The second distances of the titles falling within the distance interval are sorted from small to large, and the sorting is used as the arrangement order of the title items corresponding to the distance interval, so as to obtain a sub-list of title items corresponding to the distance interval; According to the descending order of the lower limit values ​​or the upper limit values ​​of the multiple distance intervals, the title item sub-lists corresponding to the respective distance intervals are sorted to form the directory list.

5. The method according to claim 1, characterized in that Each page of the document to be processed is parsed to determine a content list and a directory list, including: Traversing each page of the document to be processed, and whenever the current page being traversed is a non-directory page, converting the current page into HTML format, and adding the conversion result to the HTML content document; Traverse each of the HTML content documents Label, for each tags, at least based on the The paragraph indicated by the tag determines the corresponding single paragraph item; The HTML content document The traversal order of the tags is used as the arrangement order of the corresponding paragraph items to form the content list.

6. The method according to claim 5, characterized in that Each The single paragraph item corresponding to the tag also contains: The style information of the paragraph indicated by the tag; Parsing each page of the document to be processed to determine a content list and a directory list, further comprising: If it is confirmed that each page of the to-be-processed document does not include a directory page, then based on at least the style information of each paragraph item in the content list, the paragraph items in the content list are screened to determine the directory list.

7. The method according to claim 6, characterized in that The style information includes the font size of the corresponding paragraph; The filtering of the paragraph items in the content list based at least on the style information of each paragraph item in the content list to determine the directory list includes: Counting the frequency of occurrence of the font size of each paragraph in the content list, and taking the font size with the highest frequency of occurrence as the main text size; Filter out a font size that is larger than the main text size and whose occurrence frequency is greater than a preset frequency threshold as a target font size; The directory list is determined based on the paragraph items to which each target font size belongs.

8. The method according to claim 7, characterized in that The style information also includes the color of the words in the paragraph, and the third distance between the start position of the paragraph and the left edge of the page to which it belongs; Determining the directory list based on the paragraph items to which each target font size belongs includes: Filter out paragraph items whose font colors are consistent with any one or more candidate font colors from among the paragraph items to which each target font size belongs, as the reselected paragraph items; The third distance of each reselected paragraph item is summed with the preset error to obtain the corresponding fourth distance; Filter out paragraph items whose fourth distance falls within a preset target distance interval from the reselected paragraph items as final selected paragraph items; The final selected paragraph items are respectively used as title items, and the arrangement order of the final selected paragraph items in the content list is used as the arrangement order of the title items to form the directory list.

9. The method according to claim 1, characterized in that: Building a directory tree based on the directory list includes: For each title in the directory list, text matching is performed between the title number in the title and a plurality of preset title number sets, wherein each title number set has a respective title level, and each title number set contains a plurality of title numbers in a sequential order; If the heading number in the title is the same as one of the heading numbers in any heading number set, the heading level of the heading number set shall be used as the level of the title; The directory tree is constructed by using each title as a node of the tree, the level of each title as the layer to which the node belongs, and the sequence of the title numbers of the titles at the same level as the sequence of the corresponding child nodes at the same layer.

10. The method according to claim 1, characterized in that The step of determining the documents to be processed includes: Acquire an original document, wherein the original document includes at least one form to be processed; Performing preliminary parsing on each table to be processed to obtain a preliminary table, wherein the preliminary table includes a plurality of cells, and at least a portion of the cells include the same text; Traversing the cells in the first row of the preliminary table, whenever the text of the current cell is consistent with the text of the previous cell in the same row, merging the consistent texts; Starting from the second row of the preliminary table, whenever the text of the current cell is consistent with that of the previous cell in the same row, and / or whenever the text content of the current cell is consistent with that of the previous cell in the same column, the consistent texts are merged; Based on the text that has not been merged and the text that has been merged in the preliminary table, the corresponding converted text is determined, and the table to be processed is replaced with the converted text to obtain the document to be processed.

11. A document processing device, characterized in that: include: A module for determining a document to be processed, used for determining a document to be processed, wherein the document to be processed comprises at least one page, and each page comprises at least a title and / or a body text; a list determination module, used to parse each page of the to-be-processed document to determine a content list and a directory list, wherein the content list includes a plurality of paragraph items, each paragraph item includes a single paragraph, and the directory list includes a plurality of title items, each title item includes a single title, wherein at least a portion of the paragraphs in the content list is consistent with at least a portion of the titles in the directory list; a directory tree construction module, used to construct a directory tree based on the directory list, wherein the nodes of the directory tree correspond one-to-one to the titles in the directory list, the hierarchy of the directory tree indicates the level of the title, and the same-layer child nodes under the same parent node sequentially indicate the titles at the same level with a sequential order; A tree-structured document determination module is used to traverse the directory tree layer by layer, perform text matching on the title indicated by the current node and the title indicated by the adjacent node at the same layer with at least a part of the paragraphs in the content list, respectively, to determine two matching paragraphs, and then splice all the paragraphs in the content list between the two matching paragraphs, and use the splicing result as the associated text block of the current node, so as to obtain a tree-structured document after traversing the directory tree; The at least one portion of paragraphs for text matching is determined in the following manner: The title indicated by the parent node of the current node is used as the first title, and the title indicated by the node at the same level and adjacent to the parent node is used as the second title; In the content list, paragraphs matching the first title and the second title are determined respectively, and recorded as the first paragraph and the second paragraph; All paragraphs between the first paragraph and the second paragraph are regarded as the at least one part of the paragraphs.

12. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the document processing method according to any one of claims 1 to 10 are executed.

13. A terminal comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the steps of the document processing method according to any one of claims 1 to 10 are performed.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the document processing method according to any one of claims 1 to 10 are implemented.

Images