Document processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a document processing method and device, electronic equipment and a storage medium, and relates to the fields of NLP (Natural Language Processing), intelligent search and the like. According to the specific implementation scheme, title extraction processing is conducted on a to-be-processed document, and multiple pieces of suspected title data are obtained; performing title head extraction processing on the plurality of pieces of suspected title data to obtain a plurality of title heads corresponding to the plurality of pieces of suspected title data; performing coding processing on the plurality of title heads to obtain a plurality of codes corresponding to the plurality of title heads; pairing processing is conducted on the multiple codes, hierarchical relation information is determined according to the obtained at least one pair of paired codes, and the hierarchical relation information comprises at least one hierarchy and a code corresponding to the at least one hierarchy; and processing the plurality of pieces of suspected title data according to the hierarchical relationship information to obtain titles and title hierarchies in the to-be-processed document. According to the embodiment of the invention, the document title and the title level can be recalled and accurately extracted.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, in particular to the fields of natural language processing (NLP), intelligent search, and more particularly to a document processing method, device, electronic device, and storage medium. Background Art

[0002] In the preprocessing stage of Retrieval-augmented Generation (RAG), the accuracy of document parsing is crucial. If the document parsing is inaccurate, this will directly affect the subsequent retrieval and generation stages, resulting in a decrease in the performance of the entire RAG system. Among them, the extraction result of the document title hierarchy information is an important step in document parsing. Accurately extracting the title hierarchy not only directly affects the quality of the document segmentation, but also can provide more accurate meta information (meta information) for the segmentation, which has a significant impact on the effect of RAG. Therefore, how to recall and accurately extract document titles and title hierarchies has become a problem that needs to be solved urgently. Summary of the invention

[0003] The present disclosure provides a document processing method, device, electronic device and storage medium.

[0004] According to a first aspect of the present disclosure, there is provided a document processing method, comprising: performing title extraction processing on a document to be processed to obtain a plurality of suspected title data;

[0005] Performing title header extraction processing on the plurality of suspected title data respectively to obtain a plurality of title headers corresponding to the plurality of suspected title data;

[0006] Performing encoding processing on the multiple title headers respectively to obtain multiple codes corresponding to the multiple title headers; wherein the title headers of titles belonging to the same level correspond to the same code;

[0007] Performing pairing processing on the multiple codes corresponding to the multiple title headers, and determining hierarchical relationship information according to at least one pair of paired codes obtained, the hierarchical relationship information including at least one level and a code corresponding to the at least one level;

[0008] The plurality of suspected title data are processed according to the hierarchical relationship information to obtain titles and title hierarchies in the document to be processed.

[0009] According to a second aspect of the present disclosure, there is provided a document processing device, comprising:

[0010] The first extraction module is used to perform title extraction processing on the document to be processed to obtain a plurality of suspected title data;

[0011] A second extraction module is used to perform title header extraction processing on the plurality of suspected title data respectively to obtain a plurality of title headers corresponding to the plurality of suspected title data;

[0012] An encoding module, used for encoding the multiple title headers respectively to obtain multiple codes corresponding to the multiple title headers; wherein the title headers of the titles belonging to the same level correspond to the same code;

[0013] a determination module, configured to perform pairing processing on the multiple codes corresponding to the multiple title headers, and determine hierarchical relationship information according to at least one pair of paired codes obtained, wherein the hierarchical relationship information includes at least one level and a code corresponding to the at least one level;

[0014] A processing module is used to process the multiple suspected title data according to the hierarchical relationship information to obtain the titles and title hierarchies in the document to be processed.

[0015] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0019] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0020] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the steps of the method described in the first aspect when executed by a processor.

[0021] According to the technical solution disclosed in the present invention, the chapter titles and subtitles of a document can be automatically extracted without defining a large number of regular matching rules and hierarchical regular correspondences in advance. The matching and extraction of document titles and title hierarchies can be automatically completed with high recall and greater flexibility. Data that does not meet the title hierarchy order can be cleaned, thereby improving the accuracy. The problems of low title recognition recall and accuracy that exist in the prior art when identifying document title hierarchy information through models, as well as the need to write a large number of regular expressions and high labor costs when using regular matching rules to match titles and title hierarchies can be solved.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0024] Figure 1 is a flowchart of a document processing method provided by an embodiment of the present disclosure;

[0025] Figure 2 is a flowchart of a document processing method provided by an embodiment of the present disclosure;

[0026] Figure 3a is a flow chart for determining hierarchical relationship information provided by an embodiment of the present disclosure;

[0027] Figure 3b is an example diagram of the coding pairing process provided by an embodiment of the present disclosure;

[0028] Figure 4 is a flow chart of suspected title data cleaning processing provided by an embodiment of the present disclosure;

[0029] Figure 5 is a block diagram of a document processing device provided by an embodiment of the present disclosure;

[0030] Figure 6 is a block diagram of a document processing device provided by an embodiment of the present disclosure;

[0031] Figure 7 It is a block diagram of an electronic device used to implement the document processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0034] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0035] It should also be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0036] It is worth noting that in the preprocessing stage of Retrieval-augmented Generation (RAG), the accuracy of document parsing is crucial. If the document parsing is inaccurate, this will directly affect the subsequent retrieval and generation stages, resulting in a decrease in the performance of the entire RAG system. Among them, the extraction of document title level information is an important step in document parsing. Accurately extracting the title level not only directly affects the quality of document segmentation, but also can provide more accurate meta information for segmentation, which has a significant impact on the effect of RAG.

[0037] In the related art, there are two main methods to extract title level information. In the first method, a neural network model performs layout recognition to extract title level information. In the second method, regular matching rules are used to match titles and title levels. For example, many regular matching rules are written to identify the regular expression of the title and the corresponding title level. However, the above two methods have the following problems:

[0038] In the first method, the title recognition recall rate and accuracy are not high, and there are cases of missed or false positives; it is impossible to determine the subtitle and subtitle level; it is not applicable to plain text data (such as TXT documents).

[0039] The second method is that titles come in various forms. If you want to accurately extract all titles, you need a large number of regular expressions, which has high labor costs.

[0040] Based on this, the embodiments of the present disclosure provide a document processing method, device, electronic device and storage medium, which can automatically extract the chapter titles and subtitles of a document. It is not necessary to define a large number of regular matching rules and hierarchical regular correspondences in advance, and the matching and extraction of document titles and title hierarchies are automatically completed, and data that does not meet the title hierarchy order can be cleaned, so that the document title and title hierarchy can be recalled and accurately extracted.

[0041] The document processing method, device, electronic device and storage medium of the embodiments of the present disclosure are described below with reference to the accompanying drawings. It should be noted that the document processing scheme in the embodiments of the present disclosure can be applied to the following scenarios: document parsing scenario, document level extraction scenario, large model RAG document preprocessing scenario, etc.

[0042] Figure 1 is a flowchart of a document processing method provided by an embodiment of the present disclosure. Figure 1 As shown, the document processing method may include but is not limited to the following steps.

[0043] In step 101, a title extraction process is performed on the document to be processed to obtain a plurality of suspected title data.

[0044] In some embodiments, a neural network model can be used to perform title extraction processing on the document to be processed to obtain multiple suspected title data. Exemplarily, the neural network model has been pre-learned to obtain document title extraction capabilities, and the neural network model can be used to perform title extraction processing on the document to be processed to obtain all suspected title data in the document to be processed.

[0045] In some embodiments, the text data of the document to be processed can be obtained, and the title extraction processing is performed on the text data based on the regular expression to obtain multiple suspected title data. In some embodiments, the format of the document to be processed can be any of the following: plain text TXT format; portable document format PDF; document DOC format; document DOCX format, etc. Exemplarily, the text data of the document to be processed can be obtained by using a corresponding processing method according to the format of the document to be processed. For example, for a document to be processed in plain text TXT format, there is no need to convert the document in TXT format, and the text part in the document to be processed can be directly extracted, so that the text data of the document to be processed can be obtained. For a document to be processed in DOC format or DOCX format, the text part in the document to be processed can be extracted, so that the text data of the document to be processed can be obtained. For a document to be processed in PDF format, the text part in the document to be processed can be extracted by OCR (Optical Character Recognition, optical character recognition) technology, so that the text data of the document to be processed can be obtained. It can be seen that the present disclosure can support multiple document formats, such as TXT, DOCX, PDF, etc.

[0046] Exemplarily, after obtaining the text data of the document to be processed, all suspected title data can be extracted from the text data using a pre-set regular expression. The regular expression can be unified, or it can also be related to the title header style of the document. For example, different title header styles correspond to different regular expressions. Since the number of title header styles is limited, the number of corresponding regular expressions will also be limited, that is, there is no need to write a lot of regular matching rules. For example, taking a paper title as an example, the regular expression for the paper title can be: r"^[((一二三四五六七八九十0-9]+[).、]+[.0-9]*[\u4e00-\u9fa5a-zA-Z]+"

[0047] It should be noted that in some embodiments, the above title header can be understood as the numbered information at the beginning of the title and does not include other content. For example, the title header of "1INTRODUCTION" is 1; the title header of "2.1Generative Language Models" is 2.1; the title header of "第一章介绍" is 第一章

[0048] In step 102, title header extraction processing is respectively performed on multiple suspected title data to obtain multiple title headers corresponding to the multiple suspected title data.

[0049] In some embodiments, a character list allowed to be included in the title header is determined, and this list can be used to respectively perform title header extraction processing on multiple suspected title data to obtain multiple title headers corresponding to the multiple suspected title data. Exemplarily, the character list allowed to be included in the title header can be pre-defined. Exemplarily, this list can have an associated relationship with the regular expression used to extract the suspected title data. For example, the characters included in this list are included in the regular expression. The title header can be extracted from multiple suspected title data respectively according to this list to obtain the title headers of the multiple suspected title data, and the title header can be the numbered information at the beginning of the corresponding suspected title data.

[0050] For example, assume that the character list allowed to be included in the title header is as follows: '()().0123456789一二三四五六七八九十'. For each of the above multiple suspected title data, it can be truncated before the characters not included in this list. That is to say, the characters before the characters not in this list are extracted as the title header of this suspected title data. For example, taking the suspected title data "(一)XXXX" as an example, the characters before the characters not in this list are extracted, that is, (一) is used as the title header of this suspected title data.

[0051] It should be noted that the above-mentioned extraction of the title header in the title by defining the list of characters allowed to be included in the title header is only an example given to facilitate technical personnel in the field to understand the present application. That is to say, other technical means can also be used to extract the title header in the title, such as by identifying the space character to extract the title header in the title, such as taking the characters before the space character as the title header in the title; for example, the title header in the title can be identified through a neural network model, and the present disclosure does not limit this and will not elaborate on it.

[0052] In step 103, multiple title headers are respectively encoded to obtain multiple codes corresponding to the multiple title headers. The title headers of titles belonging to the same level correspond to the same code. In some embodiments, the terms "level", "level", "level" and the like can be interchangeable.

[0053] In some embodiments, for each of the multiple title headers, the title header is encoded to obtain the code of the title header. The title headers with the same code belong to the same level. For example, the title headers are "Chapter 1", "Chapter 2", and "Chapter 3". The three title headers have the same code, and the titles corresponding to the three title headers belong to the same level, such as level 1.

[0054] In some embodiments, there are many ways to encode the title header. For example, the title header can be encoded using multiple Arabic numerals, or a bitmap (including multiple bits) can be used to encode the title header, or other means can be used to encode the title header. The present disclosure does not limit this and will not elaborate on it.

[0055] Exemplarily, taking the use of a bitmap to encode the title header as an example, the bitmap may include multiple bits, including a first bit, a second bit, and at least one third bit. The first bit of the multiple bits can be used to indicate whether the title header contains Chinese numerals, wherein the value of the first bit is set to 0, which can indicate that the title header does not contain Chinese numerals (such as one, two, three, four, five, six, seven, eight, nine, ten...), and the value of the first bit is set to 1, which can indicate that the title header contains Chinese numerals. The second bit of the multiple bits can be used to indicate whether the title header contains Arabic numerals, wherein the value of the second bit is set to 1, which can indicate that the title header contains Arabic numerals, and the value of the second bit is set to 0, which can indicate that the title header does not contain Arabic numerals.

[0056] Exemplarily, the third bit of the plurality of bits can be used to indicate whether the title header contains a key character, wherein the value of the third bit is set to 1, which can indicate that the title header contains a key character, and the value of the third bit is set to 0, which can indicate that the title header does not contain a key character. Exemplarily, the number of the at least one third bit can be associated with the number of key characters in the title header, for example, one third bit corresponds to one key character, and each third bit can be used to indicate whether the corresponding key character is contained in the title header.

[0057] In step 104, a plurality of codes corresponding to the plurality of title headers are paired, and hierarchical relationship information is determined based on at least one pair of paired codes obtained.

[0058] In some embodiments, multiple codes corresponding to multiple headers may be paired in pairs, for example, two adjacent different codes may be paired to obtain at least one pair of paired codes, and the obtained paired codes may be hierarchically analyzed to determine hierarchical relationship information. In some embodiments, the above hierarchical relationship information may include, but is not limited to: at least one level and the code corresponding to the at least one level. Exemplarily, the code corresponding to the at least one level may refer to the code corresponding to each level in the at least one level, for example, each level corresponds to a code; or, for example, one or more levels (such as two levels) correspond to a code, etc.

[0059] Exemplarily, the title headers of titles belonging to the same level correspond to the same code, and the title level order is as follows: the number of high-level title pairing combinations must be greater than or equal to the number of low-level title pairing combinations. For example, the number of high-level title pairing combinations (level 1, level 2) is greater than or equal to the number of second-high-level title pairing combinations (level 2, level 3), which is greater than or equal to the number of low-level title pairing combinations (level 3, level 4), and so on. Based on this, the pairing codes can be sorted in descending order of the number of the same pairing combinations, and the hierarchical relationship information can be determined using the front-to-back relationship of the codes. For example, assuming that the sorted paired codes are as follows: (code 1, code 2), (code 2, code 3), (code 3, code 4), from the relationship between the codes, we can get that code 1 is the code for the title header of the 1st level title, code 2 is the code for the title header of the 2nd level title, code 3 is the code for the title header of the 3rd level title, and code 4 is the code for the title header of the 4th level title.

[0060] In step 105, multiple suspected title data are processed according to the hierarchical relationship information to obtain the titles and title hierarchies in the document to be processed.

[0061] In some embodiments, the hierarchical relationship information can be used to clean the above-mentioned suspected title data, for example, removing the suspected title data that is not in the hierarchical relationship information, or removing the suspected title data that is non-compliant (not in compliance with the rules) in each level. The suspected title data that is finally retained can be used as the title in the document to be processed, and the level of the title can also be obtained.

[0062] By implementing the embodiments of the present disclosure, all suspected title data can be extracted from the document first, and then the title header can be extracted from the suspected title data, the title header can be encoded to obtain the title header code, the title header code can be paired, the hierarchical relationship information can be determined according to the obtained pairing code, and the suspected title data can be cleaned according to the hierarchical relationship, and finally the title and title hierarchy of the document can be obtained. It can be seen that the present disclosure can automatically extract the chapter titles and subtitles of the document without defining a large number of regular matching rules and hierarchical regular correspondences in advance, and can automatically complete the matching and extraction of document titles and title hierarchies, with high recall and more flexibility, and can clean data that does not meet the title hierarchy order, improve the accuracy, and can solve the problems of low title recognition recall and accuracy in the prior art that exist in identifying document title hierarchy information through models, as well as the problem that a large number of regular expressions need to be written when using regular matching rules to match titles and title hierarchies, and the labor cost is high.

[0063] Figure 2 is a flowchart of a document processing method provided by an embodiment of the present disclosure. Figure 2 As shown, the document processing method may include but is not limited to the following steps.

[0064] In step 201, a title extraction process is performed on the document to be processed to obtain a plurality of suspected title data.

[0065] In the embodiments of the present disclosure, step 201 may be implemented in any manner in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0066] In step 202, title header extraction processing is performed on the plurality of suspected title data respectively to obtain a plurality of title headers corresponding to the plurality of suspected title data.

[0067] In the embodiments of the present disclosure, step 202 may be implemented in any manner in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0068] In step 203, multiple title headers are respectively encoded using multiple Arabic numerals to obtain multiple codes corresponding to the multiple title headers.

[0069] In some embodiments, title headers of titles belonging to the same level correspond to the same encoding.

[0070] In an embodiment of the present disclosure, for each of the multiple title headers, a multi-digit Arabic numeral may be used to encode the title header to obtain the code corresponding to the title header. In some embodiments, the digits of the multi-digit Arabic numeral may include a first digit, a second digit, and at least one third digit; wherein the first digit may be used to indicate whether the title header contains Chinese numerals; the second digit may be used to indicate whether the title header contains Arabic numerals; at least one third digit is associated with at least one key character, and the third digit may be used to indicate the number of key characters associated with the third digit contained in the title header. Exemplarily, the at least one third digit being associated with at least one key character may mean that each third digit in the at least one third digit is associated with a key character, and each third digit may be used to indicate the number of key characters associated with the third digit contained in the title header. Alternatively, exemplarily, the number of the at least one third digit may be multiple, the multiple third digits may be associated with a key character, and the multiple third digits may represent the number of key characters associated with the third digit contained in the title header through binary numbers. Alternatively, the association relationship between the at least one third digit and the at least one key character may also be realized in other ways, which is not specifically limited in the present disclosure and will not be repeated. Exemplarily, the key character may be “.”, or “(”, or “)”, or “,”, etc.

[0071] Exemplarily, the digits of multiple Arabic numerals include a first digit, a second digit, and two third digits, wherein the first digit can point to whether the title header contains Chinese numerals, the value of the first digit is set to 0, indicating that the title header does not contain Chinese numerals, and the value of the first digit is set to 1, indicating that the title header contains Chinese numerals. The second digit can point to whether the title header contains Arabic numerals, the value of the second digit is set to 0, indicating that the title header does not contain Arabic numerals, and the value of the second digit is set to 1, indicating that the title header contains Arabic numerals. The first third digit of the above two third digits is associated with the key character ".", pointing to the number of times the key character "." is contained in the title header, if the value of the first third digit is set to 0, it means that the title header does not contain the key character ".", the value of the first third digit is set to 1, indicating that the number of times the key character "." is contained in the title header is 1, the value of the first third digit is set to 2, indicating that the number of times the key character "." is contained in the title header is 2, and so on. The second third bit of the above two third bits is associated with the key character "(", pointing to the number of times the key character "(" is contained in the title header. If the value of the second third bit is set to 0, it means that the title header does not contain the key character "(". The value of the second third bit is set to 1, indicating that the number of times the key character "(" is contained in the title header is 1. The value of the second third bit is set to 2, indicating that the number of times the key character "(." is contained in the title header is 2, and so on.

[0072] In the embodiments of the present disclosure, step 203 may be implemented in any manner in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0073] In step 204, a plurality of codes corresponding to the plurality of title headers are paired, and hierarchical relationship information is determined based on at least one pair of paired codes obtained.

[0074] In the embodiments of the present disclosure, step 204 may be implemented in any manner in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0075] In some embodiments, the hierarchical relationship information includes at least one hierarchy and at least one encoding corresponding to the hierarchy.

[0076] In step 205, the plurality of suspected title data are processed according to the hierarchical relationship information to obtain the titles and title hierarchies in the document to be processed.

[0077] In the embodiments of the present disclosure, step 205 may be implemented in any manner in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0078] In some embodiments, Figure 3aAs shown, the above step 204 may include but is not limited to the following steps.

[0079] In step 301, two adjacent different codes in a plurality of codes are paired according to the order of the headers in the plurality of headers to obtain at least one pair of paired codes.

[0080] In some embodiments, the at least one pair of pairing codes may be one pair of pairing codes, or may be multiple pairs of pairing codes. When multiple pairs of pairing codes are obtained, the multiple pairs of pairing codes are different pairing codes.

[0081] For example, suppose there are six title headers, namely: Chapter 1, 1., 2., Chapter 2, 1., Chapter 3. The codes corresponding to these six title headers are: 1000, 0110, 0110, 1000, 0110, 1000. Figure 3b As shown, two adjacent different codes among the six codes are paired to obtain two pairs of paired codes: (1000, 0110), (0110, 1000).

[0082] In step 302, the number of occurrences of at least one pair of pairing codes in the pairing process is determined.

[0083] Exemplarily, in the process of pairing two adjacent different codes among the multiple codes, the number of the paired codes appearing in the pairing process is recorded, for example, the number of each pair of paired codes appearing in the pairing process is recorded.

[0084] For example, Figure 3b Taking the code pairing shown as an example, it can be determined that the number of occurrences of the pairing code (1000,0110) is 2, and the number of occurrences of the pairing code (0110,1000) is 2.

[0085] In step 303, at least one pair of pairing codes is sorted according to the number of pairing codes that appear in the pairing process, and hierarchical relationship information is determined according to the sorted at least one pair of pairing codes.

[0086] In some embodiments, at least one pair of pairing codes is sorted according to the number of pairing codes that appear in the pairing process, and the hierarchical relationship information is determined based on the positional relationship of the codes in the sorted at least one pair of pairing codes.

[0087] Exemplarily, the pairing codes are sorted from largest to smallest according to the number of pairing codes that appear in the pairing process, and the hierarchical relationship information is determined based on the position relationship of the codes in the sorted pairing codes combined with the title hierarchy order. Figure 3bTaking the code pairing shown as an example, the sorted pairing codes are: (1000, 0110), (0110, 1000). The high-level titles must be placed before the low-level titles. In other words, the number of high-level title pairing combinations is greater than or equal to the number of low-level title pairing combinations. Based on this, it can be determined that the code "1000" is the code corresponding to level 1, and the code "0110" is the code corresponding to level 2.

[0088] In some embodiments, Figure 4 As shown, the above step 205 may include but is not limited to the following steps.

[0089] In step 401, the title data that are not in the hierarchical relationship information among the plurality of suspected title data are filtered.

[0090] Exemplarily, the titles corresponding to the codes in the hierarchical relationship information are determined, and the suspected title data that are not in the plurality of suspected title data are filtered, that is, the suspected title data that are not in the plurality of suspected title data are removed.

[0091] In step 402, the title headers of each level in the recursive hierarchical relationship information are cleaned for suspected title data in a single level, and the titles that meet the rules are retained to obtain the titles and title levels in the document to be processed.

[0092] Exemplarily, the title headers of each level are recursively compared, the single-level title headers are cleaned, the titles that meet the rules are retained, and the titles that do not meet the rules are removed, and finally the titles and title levels in the document to be processed are obtained.

[0093] In some embodiments, for suspected title data within level 1, the suspected title data is cleaned using rules, and titles that meet the rules are retained; the index corresponding to the retained level 1 title is obtained, and the index is the index of the retained title among multiple suspected title data; based on the index corresponding to the level 1 title, the suspected title data within level 2 is determined; the suspected title data within level 2 is cleaned, and so on, until all suspected title data within all levels are cleaned, and the titles and title levels in the document to be processed are obtained.

[0094] Exemplarily, the headers of each level in the above recursive hierarchical relationship information can be cleaned using a single-level header cleaning algorithm, and its optional implementation method includes the following steps 1 to 3:

[0095] Step 1, clean the titles to be cleaned in level 1 of the suspected title data. Exemplarily, the preset rules are used for cleaning, which may be as follows: the titles in each level need to meet the following requirements: a) the title header numbers are incremented (for example, similar to 1, 2, 3, ...); b) the title header numbers are incremented by no more than a certain value (such as 2) each time. For example, when the title headers are: 1, 2023, 2, 3, the 2023 in this case is an increment that exceeds the standard and should be removed.

[0096] Step 2, obtain the index corresponding to the reserved title, such as the index of title header 1 is 1, the index of title header 2 is 5, the index of title header 3 is 9..., and the index may refer to the index in the multiple suspected title data.

[0097] Step 3, according to the index of the reserved 1-level title, determine the title to be cleaned in the 2-level in the suspected title data, such as the 2-level title between (index 1, index 5), and the title in the 2-level suspected title data between (index 5, index 9), that is, the title between the indexes of the title in the reserved 1-level suspected title data is the title to be cleaned in the 2-level;

[0098] After cleaning the 2-level titles, obtain the index of the retained 2-level titles, use the index of the 2-level titles to determine the 3-level titles to be cleaned, and clean the 3-level titles to be cleaned, and so on, until all levels of titles are cleaned. Thus, by recursively checking whether the titles in each level meet the compliance requirements to achieve title cleaning, the misidentified chapter titles can be automatically cleaned, which can further improve the accuracy.

[0099] In order to facilitate those skilled in the art to more clearly understand the present disclosure, an example is given below for description.

[0100] For example, the text data of the document to be processed is obtained, and the suspected title data in the text data is extracted. The title header in the suspected title data is extracted, and the obtained title header is encoded respectively. For example, the following Table 1 gives an example of the suspected title data extracted from a document, the title header of the suspected title data, and the title header encoding.

[0101] Table 1

[0102]

[0103]

[0104] In the order of the headers, two adjacent different codes in the header codes in the above table can be paired to obtain paired codes and the number of occurrences of each pair of paired codes, wherein the characters '' in the following curly brackets {} represent paired codes, and the character ':' represents the number of occurrences of paired codes in the encoding process: {'0100,1000':2},{'1000,0100':1},{'1000,1001':4},{'1001,0110':4},{'0110,0100':4},{'0100,0110':2},{'0110,1001':1},{'0100,1001':4},{'0110,1000':1},{'1001,0100':3},{'1001,1000':2}.

[0105] The above pairing codes are sorted according to the number of occurrences to obtain the sorted pairing codes: '1000,1001', '1001,0110', '0110,0100', '0100,1001', '1001,0100', '0100,1000', '0100,0110', '1001,1000', '1000,0100', '0110,1001', '0110,1000'. According to the sorted pairing codes combined with the title hierarchy order (such as the number of high-level title pairing combinations must be greater than or equal to the number of low-level title pairing combinations), the hierarchical relationship information is confirmed in turn, that is: '1000,1001', '1001,0110', and the final hierarchical relationship information can be obtained: {'1000':1,'1001':2,'0110':3}, that is, the title coded as 1000 corresponds to the 1st level, the title coded as 1001 corresponds to the 2nd level, and the title coded as 0110 corresponds to the 3rd level. Among them, the title coded as 0100 is not determined as a hierarchical title ( Figure 4 Hierarchical title) is because: assuming the hierarchical relationship corresponding to the paired combinations '1000,1001', '1001,0110', '0110,0100', '0100,1001', '1001,0100' is: '1 level, 2 levels', '2 levels, 3 levels', '3 levels, 4 levels', '4 levels, 2 levels', '2 levels, 4 levels', according to the title hierarchy order, the paired combinations '4 levels, 2 levels', '2 levels, 4 levels' do not conform to the title hierarchy order, that is, it is inappropriate to determine the title encoded as 0100 as the hierarchical title, so the hierarchical relationship information can be finally obtained: {'1000':1,'1001':2,'0110':3}.

[0106] After obtaining the hierarchical relationship information, the hierarchical relationship information can be used to clean the suspected title data, that is: remove the suspected title data that is not encoded in the hierarchical relationship information; determine whether the titles in each level meet the rules, that is: the titles in each level need to meet the following requirements: a) the title header number is incremented (for example, similar to 1, 2, 3, ...); b) the title header number does not increase by more than a certain value (such as 2) each time. For example, when the title header is: 1, 2023, 2, 3, 2023 in this case is an excessively incremented title and should be removed. During the cleaning process, a single-level title header cleaning algorithm can be used for title cleaning. The optional implementation method can be found in the description of the above embodiment and will not be repeated here, so that the title and title hierarchy of the document can be obtained.

[0107] In summary, the disclosed embodiment can automatically extract the chapter titles and subtitles of a document without having to define a large number of regular matching rules and hierarchical regular correspondences in advance, automatically complete the matching and extraction of the document title hierarchy, and clean up data that does not meet the title hierarchy order.

[0108] Figure 5 is a block diagram of a document processing device provided by an embodiment of the present disclosure. Figure 5 As shown, the document processing device may include: a first extraction module 501 , a second extraction module 502 , an encoding module 503 , a determination module 504 and a processing module 505 .

[0109] The first extraction module 501 is used to extract the title of the document to be processed to obtain multiple suspected title data. In some embodiments, the format of the document to be processed is any one of the following: plain text TXT format; portable document format PDF format; document DOC format; document DOCX format.

[0110] In some embodiments, the first extraction module 501 is used to: obtain text data of a document to be processed; and perform title extraction processing on the text data based on a regular expression to obtain a plurality of suspected title data.

[0111] The second extraction module 502 is used to perform title header extraction processing on the multiple suspected title data respectively to obtain multiple title headers corresponding to the multiple suspected title data.

[0112] In some embodiments, the second extraction module 502 is used to: determine a list of characters allowed to be included in the title header; extract the title headers from multiple suspected title data according to the list to obtain the title headers of each of the multiple suspected title data, where the title header is the numbering information at the beginning of the corresponding suspected title data.

[0113] The encoding module 503 is used to encode the multiple title headers respectively to obtain multiple codes corresponding to the multiple title headers; wherein the title headers of the titles belonging to the same level correspond to the same code.

[0114] In some embodiments, the encoding module 503 is used to: respectively encode multiple title headers using multiple Arabic numerals to obtain multiple codes corresponding to the multiple title headers. In some embodiments, the digits of the multiple Arabic numerals include a first digit, a second digit, and at least one third digit; wherein the first digit is used to indicate whether the title header contains Chinese numerals; the second digit is used to indicate whether the title header contains Arabic numerals; at least one third digit is associated with at least one key character, and the third digit is used to indicate the number of key characters contained in the title header and associated with the third digit.

[0115] The determination module 504 is used to perform pairing processing on multiple codes corresponding to multiple title headers, and determine hierarchical relationship information according to at least one pair of paired codes obtained, where the hierarchical relationship information includes at least one level and at least one code corresponding to the level.

[0116] The processing module 505 is used to process the multiple suspected title data according to the hierarchical relationship information to obtain the title and title hierarchy in the document to be processed.

[0117] In some embodiments, the processing module 505 is used to: filter out the title data that is not in the hierarchical relationship information among multiple suspected title data; recursively clean the title headers of each level in the hierarchical relationship information, clean the suspected title data within a single level, retain the titles that meet the rules, and obtain the titles and title levels in the document to be processed.

[0118] In some embodiments, processing module 505 is used to: for suspected title data within level 1, clean the suspected title data using rules, and retain the titles that meet the rules; obtain the index corresponding to the retained level 1 title, where the index is the index of the retained title among multiple suspected title data; determine the suspected title data within level 2 according to the index corresponding to the level 1 title; clean the suspected title data within level 2, and so on, until all suspected title data within all levels are cleaned, and the titles and title levels in the document to be processed are obtained.

[0119] In some embodiments, Figure 6As shown, the determination module 604 may include: a pairing unit 604a, a first determination unit 604b, and a second determination unit 604c. The pairing unit 604a is used to pair two adjacent different codes in the multiple codes according to the order of the headers in the multiple headers to obtain at least one pair of paired codes; the first determination unit 604b is used to determine the number of at least one pair of paired codes that appear in the pairing process; the second determination unit 604c is used to sort the at least one pair of paired codes according to the number of appearances, and determine the hierarchical relationship information according to the sorted at least one pair of paired codes. Figure 6 601-605 and Figure 5 501-505 have the same function and structure.

[0120] In some embodiments, the second determining unit 604c is used to: sort the at least one pair of paired codes according to the number of occurrences; and determine the hierarchical relationship information based on the positional relationship of the codes in the sorted at least one pair of paired codes.

[0121] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0122] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0123] like Figure 7 , is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0124] like Figure 7As shown, the electronic device includes: one or more processors 701, a memory 702, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 701 is taken as an example.

[0125] The memory 702 is a non-transitory computer-readable storage medium provided by the present disclosure. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the document processing method provided by the present disclosure. The non-transitory computer-readable storage medium of the present disclosure stores computer instructions, which are used to enable a computer to perform the document processing method provided by the present disclosure.

[0126] The memory 702 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the document processing method in the embodiment of the present disclosure (for example, the attached Figure 5 The processor 701 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 702, that is, implements the document processing method in the above method embodiment.

[0127] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0128] The electronic device may further include: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0129] The input device 703 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 704 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0130] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0132] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0133] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0134] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0135] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in this disclosure can be achieved, and this document is not limited here.

[0136] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A document processing method, comprising: Perform title extraction on the document to be processed to obtain multiple suspected title data; Performing title header extraction processing on the plurality of suspected title data respectively to obtain a plurality of title headers corresponding to the plurality of suspected title data; Performing encoding processing on the multiple title headers respectively to obtain multiple codes corresponding to the multiple title headers; wherein the title headers of titles belonging to the same level correspond to the same code; Performing pairing processing on the multiple codes corresponding to the multiple title headers, and determining hierarchical relationship information according to at least one pair of paired codes obtained, the hierarchical relationship information including at least one level and a code corresponding to the at least one level; The plurality of suspected title data are processed according to the hierarchical relationship information to obtain titles and title hierarchies in the document to be processed.

2. The method of claim 1, wherein: The title extraction process is performed on the document to be processed to obtain multiple suspected title data, including: Acquire text data of the document to be processed; The text data is subjected to title extraction processing based on a regular expression to obtain the plurality of suspected title data.

3. The method of claim 1, wherein: The step of performing title header extraction processing on the plurality of suspected title data to obtain a plurality of title headers corresponding to the plurality of suspected title data comprises: Determine the list of characters allowed in the header; According to the list, title headers are respectively extracted from the plurality of suspected title data to obtain respective title headers of the plurality of suspected title data, wherein the title header is the numbering information at the beginning of the corresponding suspected title data.

4. The method of claim 1, wherein: The encoding process is performed on the multiple title headers respectively to obtain multiple codes corresponding to the multiple title headers, including: The plurality of title headers are respectively encoded using multi-digit Arabic numerals to obtain a plurality of codes corresponding to the plurality of title headers.

5. The method of claim 4, wherein: The number of digits of the multiple Arabic numerals includes a first digit, a second digit and at least a third digit; wherein, The first digit is used to indicate whether the title contains Chinese numerals; The second digit is used to indicate whether the title contains Arabic numerals; The at least one third digit is associated with at least one key character, and the third digit is used to indicate the number of key characters associated with the third digit contained in the title header.

6. The method of claim 1, wherein: The performing pairing processing on the multiple codes corresponding to the multiple title headers and determining the hierarchical relationship information according to at least one pair of paired codes obtained includes: Pairing two adjacent different codes among the multiple codes according to the order of the title headers in the multiple title headers to obtain at least one pair of paired codes; Determining the number of occurrences of the at least one pair of pairing codes during the pairing process; The at least one pair of paired codes is sorted according to the number of occurrences, and the hierarchical relationship information is determined according to the sorted at least one pair of paired codes.

7. The method of claim 6, wherein: The step of sorting the at least one pair of paired codes according to the number of occurrences, and determining the hierarchical relationship information according to the sorted at least one pair of paired codes includes: Sort the at least one pair of paired codes according to the number of occurrences; The hierarchical relationship information is determined based on the positional relationship of the codes in the at least one pair of paired codes after sorting.

8. The method of claim 1, wherein: The processing of the plurality of suspected title data according to the hierarchical relationship information to obtain titles and title hierarchies in the to-be-processed document includes: filtering the title data that are not in the hierarchical relationship information among the plurality of suspected title data; Recursively analyze the title headers of each level in the hierarchical relationship information, clean the suspected title data in a single level, retain the titles that meet the rules, and obtain the titles and title levels in the document to be processed.

9. The method of claim 8, wherein: The recursive title headers of each level in the hierarchical relationship information are cleaned for suspected title data in a single level, and titles that meet the rules are retained to obtain titles and title levels in the document to be processed, including: For suspected title data in level 1, the suspected title data is cleaned using the rules, and titles that meet the rules are retained; Obtaining an index corresponding to the reserved level 1 title, where the index is an index of the reserved title among the plurality of suspected title data; Determine suspected title data in level 2 according to the index corresponding to the title in level 1; The suspected title data in the two levels are cleaned, and so on, until the suspected title data in all levels are cleaned, and the titles and title levels in the document to be processed are obtained.

10. The method according to any one of claims 1 to 9, wherein: The format of the document to be processed is any of the following: Plain text TXT format; Portable Document Format PDF; Document DOC format; The document is in DOCX format.

11. A document processing device, comprising: The first extraction module is used to perform title extraction processing on the document to be processed to obtain a plurality of suspected title data; A second extraction module is used to perform title header extraction processing on the plurality of suspected title data respectively to obtain a plurality of title headers corresponding to the plurality of suspected title data; An encoding module, used for encoding the multiple title headers respectively to obtain multiple codes corresponding to the multiple title headers; wherein the title headers of the titles belonging to the same level correspond to the same code; a determination module, configured to perform pairing processing on the multiple codes corresponding to the multiple title headers, and determine hierarchical relationship information according to at least one pair of paired codes obtained, wherein the hierarchical relationship information includes at least one level and a code corresponding to the at least one level; A processing module is used to process the multiple suspected title data according to the hierarchical relationship information to obtain the titles and title hierarchies in the document to be processed.

12. The device of claim 11, wherein: The first extraction module is used for: Acquire text data of the document to be processed; The text data is subjected to title extraction processing based on a regular expression to obtain the plurality of suspected title data.

13. The device of claim 11, wherein: The second extraction module is used for: Determine the list of characters allowed in the header; According to the list, title headers are respectively extracted from the plurality of suspected title data to obtain respective title headers of the plurality of suspected title data, wherein the title header is the numbering information at the beginning of the corresponding suspected title data.

14. The device of claim 11, wherein: The encoding module is used for: The plurality of title headers are respectively encoded using multi-digit Arabic numerals to obtain a plurality of codes corresponding to the plurality of title headers.

15. The device of claim 14, wherein: The number of digits of the multiple Arabic numerals includes a first digit, a second digit and at least a third digit; wherein, The first digit is used to indicate whether the title contains Chinese numerals; The second digit is used to indicate whether the title contains Arabic numerals; The at least one third digit is associated with at least one key character, and the third digit is used to indicate the number of key characters associated with the third digit contained in the title header.

16. The device of claim 11, wherein: The determination module comprises: a pairing unit, configured to pair two adjacent different codes in the multiple codes according to the order of the headers in the multiple headers to obtain at least one pair of paired codes; A first determining unit, used to determine the number of occurrences of the at least one pair of pairing codes in the pairing process; The second determining unit is configured to sort the at least one pair of pairing codes according to the number of occurrences, and determine the hierarchical relationship information according to the sorted at least one pair of pairing codes.

17. The device of claim 16, wherein: The second determining unit is used for: Sort the at least one pair of paired codes according to the number of occurrences; The hierarchical relationship information is determined based on the positional relationship of the codes in the at least one pair of paired codes after sorting.

18. The device of claim 11, wherein: The processing module is used for: filtering the title data that are not in the hierarchical relationship information among the plurality of suspected title data; Recursively analyze the title headers of each level in the hierarchical relationship information, clean the suspected title data in a single level, retain the titles that meet the rules, and obtain the titles and title levels in the document to be processed.

19. The device of claim 18, wherein: The processing module is used for: For suspected title data in level 1, the suspected title data is cleaned using the rules, and titles that meet the rules are retained; Obtaining an index corresponding to the reserved level 1 title, where the index is an index of the reserved title among the plurality of suspected title data; Determine suspected title data in level 2 according to the index corresponding to the title in level 1; The suspected title data in the two levels are cleaned, and so on, until the suspected title data in all levels are cleaned, and the titles and title levels in the document to be processed are obtained.

20. The device according to any one of claims 11 to 19, wherein: The format of the document to be processed is any of the following: Plain text TXT format; Portable Document Format PDF format; Document DOC format; The document is in DOCX format.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 10.

Images