Information extraction method and device, electronic equipment and storage medium

Abstract

This disclosure provides an information extraction method, apparatus, electronic device, and storage medium, relating to the field of data processing, and particularly to the field of data extraction. The specific implementation scheme is as follows: The document to be extracted is converted into text without spaces; the index positions of each specified field in the text without spaces are found; for each specified field's corresponding index position, a first length of text before and a first length of text after that index position are extracted from the text without spaces to obtain the matching text corresponding to that specified field; the matching text is then matched using a preset regular expression corresponding to that specified field to obtain the start and end index intervals of the matching portion in the matching text; based on these start and end index intervals, the field value corresponding to that specified field is extracted from the document to be extracted. This saves manpower and time costs.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more particularly to the field of data extraction technology. Background Technology

[0002] As businesses grow, some enterprises and institutions accumulate a large number of similar historical documents, such as PDF documents, containing similar information. To facilitate the storage and retrieval of information from these historical documents, information can be extracted from these similar historical documents, and the extracted information can be stored in a structured manner. Summary of the Invention

[0003] This disclosure provides an information extraction method, apparatus, electronic device, and storage medium.

[0004] According to a first aspect of this disclosure, an information extraction method is provided, comprising:

[0005] Convert the document to be extracted into text without spaces;

[0006] Find the index position of each specified field in the text without spaces;

[0007] For each specified field, extract the first length of text before and the first length of text after the specified index position from the text without spaces to obtain the hit text corresponding to the specified field.

[0008] The matched text is obtained by matching the specified regular expression with the specified field to obtain the start and end index range of the part of the matched text that matches the specified regular expression.

[0009] Based on the start and end index range, extract the field value corresponding to the specified field from the document to be extracted.

[0010] According to a second aspect of this disclosure, an information extraction apparatus is provided, comprising:

[0011] The first conversion module is used to convert the document to be extracted into text without spaces.

[0012] The search module is used to find the index position of each specified field in the text without spaces;

[0013] The truncation module is used to extract a first length of text before and a first length of text after the index position corresponding to each specified field from the text without spaces, so as to obtain the hit text corresponding to the specified field.

[0014] The matching module is used to match the hit text using a preset regular expression corresponding to the specified field, and to obtain the start and end index range of the part of the hit text that matches the preset regular expression;

[0015] The extraction module is used to extract the field value corresponding to the specified field from the document to be extracted based on the start and end index range.

[0016] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0020] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect.

[0021] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in the first aspect.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0023] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0024] Figure 1 This is a flowchart of an information extraction method provided in an embodiment of this disclosure;

[0025] Figure 2 This is a flowchart of a method for generating regular expressions provided in an embodiment of this disclosure;

[0026] Figure 3 This is a flowchart of an information extraction method provided in an embodiment of this disclosure;

[0027] Figure 4 This is an exemplary flowchart of a method for generating regular expressions provided in an embodiment of this disclosure;

[0028] Figure 5 This is an exemplary flowchart of an information extraction method provided in an embodiment of this disclosure;

[0029] Figure 6 This is a schematic diagram of the structure of an information extraction device provided in an embodiment of this disclosure;

[0030] Figure 7 This is a block diagram of an electronic device used to implement the information extraction method of the embodiments of this disclosure. Detailed Implementation

[0031] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0032] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0033] The documents involved in this embodiment are from a public dataset.

[0034] In related technologies, semi-structured data is a data organization form that lies between structured data (such as relational data, key-value databases, etc.) and unstructured data (such as sound, images, documents, etc.). Examples include relevant standard documents and notices within enterprises. These documents are characterized by implicitly incomplete and non-standardized structures. For instance, documents related to "geographic resource exploration" may describe geographic resources in different locations, but they all contain similar information, such as latitude and longitude coordinates, exploration time, and exploration methods.

[0035] This information is hidden across different pages of the document. Staff must open each document and search using keywords to extract and structure the information. This extraction method relies on staff experience. Because the main fields being extracted lack a strict structure and format within the document, keyword searches may hit multiple areas. Staff must then filter through these hit areas to find the desired information, resulting in a significant amount of repetitive and inefficient work and wasted manpower.

[0036] Currently, extraction rules can be summarized manually and then programmed into extraction templates using computer programming languages. These templates can then be used to extract information from similar documents. However, this requires staff to read a large number of related documents to summarize and deduce extraction rules. Different types of documents also have different extraction rules, which consumes a lot of manpower and time.

[0037] Documents of the same type are those with common content, but the structure of this common content within the document is not standardized; these documents can be called semi-structured documents. For example, documents of the same type could all be documents about geographic resource exploration.

[0038] The documents in this embodiment can be documents of various formats, such as PDF documents, Word documents, etc., and this embodiment does not specifically limit them.

[0039] The information extraction method provided in this disclosure can be executed by an electronic device, such as a smartphone, tablet, desktop computer, or server.

[0040] The information extraction method provided in the embodiments of this disclosure will be described in detail below.

[0041] like Figure 1 As shown in the embodiments of this disclosure, an information extraction method is provided, the method comprising:

[0042] S101. Convert the document to be extracted into text without spaces.

[0043] In this embodiment, information can be extracted from multiple documents to be extracted, which may contain similar content. For example, multiple documents to be extracted may all be related to "geographic resource exploration." For instance, each document to be extracted may contain the latitude, longitude, altitude, and other information of a region. As another example, the multiple documents to be extracted may be resume documents.

[0044] S102. Find the index position of each specified field in the text without spaces.

[0045] S103. For each specified field, extract the first length of text before and the first length of text after the specified index position from the text without spaces to obtain the hit text corresponding to the specified field.

[0046] The index position can represent the approximate position of the field value corresponding to the specified field in the text without spaces. The text before and after the first length of the index position can be extracted to obtain the hit text. Then, in subsequent steps, the field value corresponding to the specified field can be matched from the hit text.

[0047] The hit text specifically includes: the text of the first length before the index position, the text at the index position, and the text of the first length after the index position.

[0048] S104. Match the hit text using the preset regular expression corresponding to the specified field to obtain the start and end index range of the part of the hit text that matches the preset regular expression.

[0049] The preset regular expression is a regular expression that can match the sample label value from a preset number of sample substrings corresponding to the specified field. Specifically, the preset regular expression is generated in advance based on the sample substrings, which include text of a second length before the sample label value corresponding to the specified field in the sample text without spaces, the sample label value corresponding to the specified field, and text of a second length after the sample label value corresponding to the specified field. The method for generating this preset regular expression will be described in subsequent embodiments.

[0050] The start and end index range is the interval formed by the position index of the first character and the position index of the last character in the matched text that matches the preset regular expression.

[0051] It is understandable that the field values ​​of the same specified field in different documents have commonalities, and the context of the same specified field in different documents also has commonalities. Therefore, the pre-generated regular expression that can match the sample value from the sample substring has universality. Using this preset regular expression, the field value corresponding to the same field in other documents of the same type to be extracted can also be matched.

[0052] S105. Extract the field value corresponding to the specified field from the document to be extracted based on the start and end index range.

[0053] In this embodiment of the disclosure, after extracting the field corresponding to each specified field from multiple documents to be extracted, the extracted field values ​​can be stored in a structured manner.

[0054] By converting the document to be extracted into blank text, the influence of spaces and other formatting elements on the extraction position can be avoided. The index position of each specified field in the blank text is found, thus determining the approximate position of the field value. Then, a first-length section of text before and after the index position is extracted to obtain the target text. This essentially determines the possible range of the field value. A preset regular expression is then used to match the start and end index intervals of the field value from the target text. Based on these intervals, the corresponding field value can be extracted from the document to be extracted. Therefore, this embodiment can extract field values ​​in batches using preset regular expressions, eliminating the need for manually writing extraction templates and reducing manpower and time costs.

[0055] In another embodiment of this disclosure, after converting the document to be extracted into text without spaces, the method further includes:

[0056] Establish a first character index mapping from the text without spaces to the document to be extracted (nospace_to_raw_dict).

[0057] The first character index mapping is a one-to-one mapping relationship between the position of a character in the text without spaces and the position of the corresponding character in the document to be extracted. When the position of a certain character in the text without spaces is obtained, the position of the character in the corresponding document to be extracted can be obtained through the character index mapping.

[0058] Therefore, the above-mentioned S105, extracting the field value corresponding to the specified field from the text to be extracted based on the start and end index range, can be implemented as follows:

[0059] Based on character index mapping, the start and end index ranges are mapped to the documents to be extracted, thus obtaining the start and end index positions of the start and end index ranges in the documents to be extracted.

[0060] Extract the text from the starting index position to the ending index position in the document to be extracted, and obtain the field value corresponding to the specified field.

[0061] Optionally, after determining the start and end index positions of the start and end index ranges in the text without spaces in the document to be extracted, the field value of the specified field can be highlighted in the document to be extracted, that is, the extraction results determined from the text without spaces can be displayed in the original document.

[0062] Using the embodiments of this disclosure, a field value corresponding to a specified field can be determined in text without spaces by using a preset regular expression. Furthermore, by establishing a character index mapping from the text without spaces to the document to be extracted, the field value matched in the text without spaces can be found in the document to be extracted, facilitating the extraction of field values ​​from the document.

[0063] The regular expressions in the embodiments of this disclosure can be obtained as follows: Figure 2 The process shown is generated, and the method includes:

[0064] S201. Convert the preset number of sample documents into sample text without spaces.

[0065] The preset number of sample documents are those with common content. For example, they are all documents related to "geographic resource exploration".

[0066] As an example, the preset quantity can be 20.

[0067] S202. For each specified field, obtain the sample substring corresponding to that specified field from the sample text without spaces.

[0068] The sample substring includes text of a second length before the sample label value corresponding to the specified field, the sample label value corresponding to the specified field, and text of a second length after the sample label value corresponding to the specified field.

[0069] The second length is a preset empirical value, which can be 15 characters.

[0070] The specified fields can be predefined. For example, in a document about geographic resource exploration, specified fields could include mining date, latitude and longitude, and the rock stratum to which it belongs. The sample annotation value is the field value corresponding to the specified field in the sample text without spaces. For example, if the specified field is named altitude, the corresponding annotation field value could be 8848.13m.

[0071] S203. Generate a regular expression that can match the sample label value from all sample substrings corresponding to the specified field.

[0072] In this embodiment of the disclosure, for each specified field, a regular expression that can match the sample label value from these sample substrings can be generated using the sample substrings corresponding to the specified field and a regular expression generator.

[0073] By employing the embodiments of this disclosure, a preset number of sample documents are converted into sample text without spaces. Sample substrings corresponding to specified fields are then extracted from the text without spaces, avoiding the presence of meaningless spaces in the sample substrings. Based on the extracted sample substrings, a regular expression can be generated that can match sample annotation values ​​from all sample substrings corresponding to the specified field. Furthermore, when extracting information from multiple documents, this regular expression can be used to batch extract field values ​​for specified fields, eliminating the need for manually writing extraction rules and reducing manpower and time costs.

[0074] In another embodiment of this disclosure, after obtaining the sample substring corresponding to each specified field from each sample text without spaces for each specified field, the method further includes:

[0075] Convert each sample substring corresponding to the specified field into dictionary format, and set the first and second longest texts of each sample substring to not need matching.

[0076] For example, assuming the sample document is a tender document, part of the sample document's content is as follows:

[0077] Bidding Entity: Dream Co., Ltd.

[0078] Legal Representative: Zhang San

[0079] Company Address: Dream Building, Chaoyang District, Beijing

[0080] Postal code: 10000

[0081] Contact number: 123456789

[0082] Fax: 123467

[0083] After converting the sample document into blank text, the content of this section in the blank text is: "Bidding unit: Dream Co., Ltd. Legal representative: Zhang San Address: Dream Building, Chaoyang District, Beijing Postcode: 10000 Contact number: 123456789 Fax: 123467".

[0084] Taking a second length of 15 as an example, if the labeled sample value corresponding to the specified field is "Dream Building, Chaoyang District, Beijing", and the sample substring corresponding to the specified field obtained from the text without spaces is "Legal Representative of the Company: Zhang San; Company Address: Dream Building, Chaoyang District, Beijing; Postal Code: 10000; Contact Number:", this sample substring can be converted to the following format:

[0085]

[0086]

[0087] That is, the sample substring includes characters 0 to 39, a total of 40 characters. Among them, the characters to be matched start from character 15 and end at character 25, that is, it is necessary to match characters 15 to 24 (Dream Building, Chaoyang District, Beijing). The first 15 characters and the last 15 characters in the sample substring do not need to be matched. The first 15 characters start from character 0 and end at character 15, that is, it is not necessary to match characters 0 to 14 (Legal representative of the company: Zhang San Company address:). The last 15 characters start from character 25 and end at character 39, that is, it is not necessary to match characters 25 to 39 (Postal code: 10000 Contact phone number:).

[0088] Each specified field corresponds to multiple sample substrings in dictionary format, and the training set for each specified field can be stored in json format. After obtaining the training set corresponding to each specified field, a regular expression generator (RegexGenerator) can be used to read the training set for each specified field, and then train and generate the regular expression corresponding to each specified field. Optionally, the regular expression generator can be a jar package.

[0089] By adopting the embodiment of the present disclosure, the sample substring is processed into a dictionary form, and the text with the starting second length and the ending second length of each sample substring is set to not need to be matched. Furthermore, when generating a regular expression, a regular expression that can match the sample annotation value from the sample substring can be generated, so that the generated regular expression can more accurately extract the field value corresponding to the specified field.

[0090] In another embodiment of the present disclosure, after obtaining the sample substring corresponding to the specified field from each sample non-space text for each specified field, the method further includes:

[0091] Record the maximum length of the sample substring corresponding to the specified field, and use half of the recorded maximum length as the first length used when intercepting the hit text of the specified field, that is, the first length in S103 above.

[0092] Since the lengths of the field values of each specified field can be different, the first length corresponding to each specified field can also be different.

[0093] By adopting this method, half of the maximum length of the preset number of sample substrings corresponding to the specified field is used as the first length used when intercepting the hit text of the specified field, so that the length of the intercepted hit text is close to the length of the sample substring, which can maximize the guarantee of intercepting the field value corresponding to the specified field, and avoid too large interception range, resulting in a large amount of calculation for matching the preset regular expression, or too small interception range, resulting in the inability to completely extract the field value of the specified field.

[0094] Optionally, the first length can also be set based on empirical values. Alternatively, the first length can be the average length of a preset number of sample substrings corresponding to a specified field.

[0095] In another embodiment of this disclosure, such as Figure 3 As shown, after step S201, which converts a preset number of sample documents into sample text without spaces, the method further includes:

[0096] S301. Establish a second character index mapping from each sample document to the sample text without spaces.

[0097] The second character index mapping is a one-to-one mapping relationship between the position of a character in the text without spaces and the position of the corresponding character in the document to be extracted. When the position of a certain character in the sample document is obtained, the position of that character in the text without spaces can be obtained through the second character index mapping.

[0098] S302. For each specified field, based on the start and end indexes and character index mapping of the specified field's annotation value in the sample document, determine the starting position of the specified field's annotation value in each sample text without spaces.

[0099] The start and end indices of the specified field's annotation value in the document are obtained through the following steps:

[0100] The system displays a preset number of sample documents. Users can choose to annotate each sample document. Accordingly, the document annotation platform can identify the annotation field values ​​that the user annotates in each sample document, as well as the specified fields that the user selects for the annotated field values, and obtain the start and end indices of the preset number of annotation field values ​​corresponding to each specified field in the sample document.

[0101] Specifically, during the sample annotation process, sample documents can be uploaded to the annotation platform. Users can then select the field values ​​corresponding to specified fields through the platform, and the selected field values ​​will be highlighted. If a user selects a field value to annotate (e.g., by clicking on the selected field value), the annotation platform can display multiple predefined field names, from which the user can choose the field name as the selected field value's name. Furthermore, the annotation platform records the start and end indices of the user-selected field value within the sample document, as well as the selected field name, thus completing the annotation of a field value.

[0102] S303. Extract the third length of text before the determined starting position from the text without spaces in each sample, and determine the maximum common subsequence of the extracted text.

[0103] The third length is a preset empirical value, such as 25 characters.

[0104] For a given field, if the given field has a label value in each sample text without spaces, then a third length of text can be extracted from before the label value of each text without spaces. This will extract a preset number of texts of the third length, hereinafter referred to as the preceding text.

[0105] Furthermore, the longest consecutive common character combination can be determined from a preset number of preceding texts as the maximum common subsequence. In this embodiment, the n-gram algorithm can be used to traverse all character combinations from n=3 to n=25 in a preset number of preceding texts, record the character combinations that appear in each preceding text, and then select the consecutive common character combination with the largest n as the maximum common subsequence.

[0106] Understandably, the text preceding the labeled value is usually the field name. For example, if the field name is altitude, the text without spaces would typically display as "Altitude of Mountain Range A: 8848.13m" or "Altitude of Mountain Range B: 5000m". It is evident that the text preceding the labeled value of the same specified field has commonalities, so the largest common subsequence for each specified field can be predetermined.

[0107] Using the embodiments of this disclosure, the starting position of the annotation value can be determined by mapping the second character index to the text without spaces. Then, the text of the third length before the starting position can be extracted from the text without spaces, so as to avoid interference from spaces and other factors when directly extracting text from the sample document. By taking advantage of the common characteristics of the text before the annotation value of the same specified field, the maximum common subsequence corresponding to each specified field can be accurately obtained, so as to facilitate batch information extraction from the document to be extracted.

[0108] Given that the longest common subsequence corresponding to each specified field is predetermined, the above step S102, finding the index position of each specified field in the text without spaces, can be implemented as follows:

[0109] For each specified field, the index position of the longest common subsequence corresponding to that specified field in the text without spaces is used as the index position of that specified field.

[0110] Optionally, the index position corresponding to the specified field can be the index position of any character in the maximum common subsequence in the text without spaces.

[0111] To ensure that the first length of text following the index position includes the field value of the specified field, the index position of the last character of the greatest common subsequence in the text without spaces can be used as the index position of the specified field.

[0112] Using this method, the index position of a specified field can be quickly located from text without spaces by using a pre-determined maximum common subsequence, avoiding the need for manual input of search keywords and improving information extraction efficiency.

[0113] The following section introduces methods for generating regular expressions and extracting information using specific examples.

[0114] See Figure 4 , Figure 4 This is an exemplary flowchart of a regular expression generation method provided in an embodiment of this disclosure. The following is a description of the method in conjunction with... Figure 4 Please provide an explanation.

[0115] S401. Upload sample PDF document.

[0116] The operators uploaded multiple similar sample PDF documents to the annotation platform.

[0117] S402. The annotation platform receives the field names defined by the operator and identifies the operator's annotation of the fields.

[0118] On the annotation platform, operators define multiple field names, annotate each uploaded sample PDF document to obtain the annotated field values, and specify the field names corresponding to the annotated field values. A field ID can be set for each field to distinguish different fields.

[0119] S403, Record the start and end indexes.

[0120] Record the start and end indices and page numbers of the labeled field values ​​in the sample PDF document.

[0121] S404, Parse PDF documents.

[0122] Convert multiple sample PDF documents into text without spaces.

[0123] S405. Establish a text index mapping.

[0124] Establish a character index mapping from the original text to the text without spaces.

[0125] S406, Obtain the maximum common subsequence.

[0126] Based on the n-gram algorithm, obtain the longest common subsequence of the preceding text for multiple labeled field values ​​corresponding to each field ID in text without spaces.

[0127] The label field values ​​corresponding to each field ID can be grouped together, and the maximum common subsequence corresponding to each field ID can be determined separately.

[0128] For each field ID corresponding to multiple annotation field values, based on the character index mapping established in S405 and the start and end indices of the annotation field values ​​in the sample PDF document recorded in S403, find each annotation field value and its starting index in the text without spaces. Extract the preceding text containing the first 25 characters of each annotation field value's starting index from the text without spaces. Use the n-gram algorithm to traverse all character combinations with n ranging from 3 to 25 in the extracted preceding text, and record the common character combinations in each preceding text. Determine the largest consecutive character combination with n as the maximum common subsequence corresponding to that field ID.

[0129] S407, Extract the context text of the annotation value.

[0130] For each field ID, extract the first 15 characters of the starting index and the last 15 characters of the ending index of each field value from the text without spaces. Combine each field value with the text before and after it to form a sample substring. Process the sample substrings formed by the multiple field values ​​and the text before and after each field ID into a dictionary and store it in JSON format as a training set.

[0131] S408, record the maximum length of the sample substring, max-window.

[0132] Specifically, the maximum length of the multiple sample substrings corresponding to each field ID can be recorded as max-window.

[0133] S409. Generate regular expressions.

[0134] The training set corresponding to each field ID is read using a regular expression generator, and the generator is trained to produce a regular expression corresponding to each field ID. Then, each field ID and its corresponding regular expression are stored for later use in information extraction using regular expressions.

[0135] S410, save the regular expression, the maximum common subsequence, and the maximum length of the sample substring (max-window).

[0136] Save each field ID generated in S409 along with its corresponding regular expression, the maximum common subsequence corresponding to each field ID obtained in S406, and the maximum length of the sample substring corresponding to each field ID recorded in S408 into the database.

[0137] After completing Figure 4 After the preprocessing step, information can be extracted from the document to be extracted based on the regular expression corresponding to each saved field ID, the longest common subsequence, and the maximum length of the sample substring. Taking a PDF document as an example, see [link to relevant documentation]. Figure 5 , Figure 5 This is an exemplary flowchart of the information extraction method provided in the embodiments of this disclosure, which is described below in conjunction with... Figure 5 Please provide an explanation.

[0138] S501. Upload PDF document.

[0139] This is the PDF document to be extracted.

[0140] S502, Parse PDF documents.

[0141] Electronic devices can parse PDF documents into text without spaces.

[0142] S503. Establish index mapping.

[0143] This index mapping is a character index mapping from the original text to the text without spaces.

[0144] S504, the maximum common subsequence hits the full-text index position idx.

[0145] Find the longest common subsequence (LCS) for each field ID in the text without spaces, and record the index position idx of the LCS in the text without spaces. The index position idx is the index position of the last character in the LCS in the text without spaces.

[0146] S505. Extract the short text of the window [idx-max-window / 2:idx+max-window / 2].

[0147] That is, extract the text before and after the index position idx by a length of max-window / 2.

[0148] S506. Use regular expressions to match the extracted short text and return the start and end index range.

[0149] In this process, the regular expression corresponding to each field ID can be used to match the short text corresponding to that field ID, thereby obtaining the start and end index range of the field value corresponding to each field ID in the text without spaces.

[0150] S507. Restore the start and end index range in the PDF document through index mapping.

[0151] This index mapping is a character index mapping from the text without spaces to the PDF document. Based on this character index mapping and the start and end index range of the field value in the text without spaces, the index position of the field value in the PDF document can be obtained.

[0152] S508. Returns the start and end index range of the field value for each ID field in the PDF document.

[0153] The method provided in this disclosure utilizes implicit text structure patterns, requiring only a small amount of labeled data to generate regular expressions capable of automatically extracting similar documents, thus saving manpower. Because these documents share commonalities, this disclosure can summarize the commonalities in the context of areas where each specified field may appear. For example, if the field to be extracted is the first drilling date, typically "first drilling date" appears before the field value in multiple documents. By analyzing the field values ​​corresponding to such fields, such as "October 10, 2019 to February 10, 2020," it can be determined that the field values ​​of the same field also have strong commonalities. Therefore, during the learning phase, the common text preceding the field value to be extracted can be summarized in multiple sample documents, and the regular expression for the field value to be extracted can be learned using these commonalities.

[0154] During the extraction phase, regions containing common text are identified in the documents to be extracted. The strings within `max-window / 2` before and after these regions are considered the "hit text." Regular expressions are then used to match this hit text, extracting the field values ​​within it. Field IDs and values ​​can be returned as key-value pairs for structured storage.

[0155] Through practice, for the same type of documents, by annotating 20 documents, regular expressions for each specified field were generated. During the extraction phase, these regular expressions can then be used to quickly and accurately extract documents in batches.

[0156] In other words, the embodiments of this disclosure can automatically extract a large number of documents with a small amount of annotation. Compared with manually summarizing rule templates, this can greatly reduce the operational threshold for operators and save a lot of manpower and time.

[0157] Corresponding to the above method embodiments, this disclosure also provides an information extraction device, such as... Figure 6 As shown, the device includes:

[0158] The first conversion module 601 is used to convert the document to be extracted into text without spaces.

[0159] Search module 602 finds the index position of each specified field in text without spaces;

[0160] The truncation module 603 is used to extract the first length of text before and the first length of text after the index position corresponding to each specified field from the text without spaces, so as to obtain the hit text corresponding to the specified field.

[0161] Matching module 604 is used to match the hit text using the preset regular expression corresponding to the specified field, and obtain the start and end index range of the part of the hit text that matches the preset regular expression.

[0162] Extraction module 605 is used to extract the field value corresponding to the specified field from the document to be extracted based on the start and end index range.

[0163] In another embodiment of this disclosure, the device further includes:

[0164] The first mapping establishment module is used to establish the first character index mapping from text without spaces to the document to be extracted.

[0165] Extraction module 605 is specifically used for:

[0166] Based on character index mapping, the start and end index intervals are mapped to the document to be extracted, thus obtaining the start and end index positions of the first start and end index intervals in the document to be extracted;

[0167] Extract the text from the starting index position to the ending index position in the document to be extracted, and obtain the field value corresponding to the specified field.

[0168] In another embodiment of this disclosure, the device further includes:

[0169] The second conversion module is used to convert a preset number of sample documents into sample text without spaces.

[0170] The acquisition module is used to acquire the sample substring corresponding to each specified field from each sample text without spaces for each specified field; wherein, the sample substring includes the text of the second length before the sample label value corresponding to the specified field, the sample label value corresponding to the specified field, and the text of the second length after the sample label value corresponding to the specified field.

[0171] The generation module is used to generate a regular expression that can match all sample substrings corresponding to the specified field except for the sample label value. The regular expression is used to extract the field value of the specified field in documents of the same type as the sample document.

[0172] In another embodiment of this disclosure, the device further includes:

[0173] A recording module is used to record the maximum length of the sample substring corresponding to the specified field, and to use half of the recorded maximum length as the first length used when extracting the hit text of the specified field. In another embodiment of this disclosure, the apparatus further includes:

[0174] The second mapping establishment module is used to establish a second character index mapping from each sample document to the sample text without spaces.

[0175] The determination module is used to determine the starting position of the annotation value of the specified field in each sample text without spaces, based on the start and end indexes and the second character index mapping of the annotation value of the specified field in the sample document; and to extract a third length of text before the determined starting position from each sample text without spaces, and to determine the maximum common subsequence of the extracted text.

[0176] In another embodiment of this disclosure, the lookup module 602 is specifically used for:

[0177] For each specified field, the index position of the longest common subsequence corresponding to the specified field in the text without spaces is used as the index position of the specified field.

[0178] In another embodiment of this disclosure, the device further includes:

[0179] The format conversion module is used to convert each sample substring corresponding to the specified field into dictionary format, and set the first and last second length texts of each sample substring to not need to be matched.

[0180] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0181] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0182] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0183] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0184] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as information extraction methods. For example, in some embodiments, the information extraction method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the information extraction method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the information extraction method by any other suitable means (e.g., by means of firmware).

[0185] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0186] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0187] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0188] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0189] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0190] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0191] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0192] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An information extraction method, comprising: Convert the document to be extracted into text without spaces; Find the index position of each specified field in the text without spaces; For each specified field, extract the first length of text before and the first length of text after the specified index position from the text without spaces to obtain the hit text corresponding to the specified field. The matched text is obtained by matching the specified regular expression with the specified field to obtain the start and end index range of the part of the matched text that matches the specified regular expression. Based on the start and end index range, extract the field value corresponding to the specified field from the document to be extracted; After converting the document to be extracted into text without spaces, the method further includes: Establish a first character index mapping from the text without spaces to the document to be extracted; The step of extracting the field value corresponding to the specified field from the document to be extracted based on the start and end index range includes: Based on the first character index mapping, the start and end index intervals are mapped to the document to be extracted, so as to obtain the start index position and end index position of the start and end index intervals in the document to be extracted; Extract the text from the starting index position to the ending index position in the document to be extracted, and obtain the field value corresponding to the specified field.

2. The method according to claim 1, wherein, The regular expression is generated through the following steps: Convert a preset number of sample documents into sample text without spaces. For each specified field, obtain the sample substring corresponding to the specified field from each sample text without spaces; wherein, the sample substring includes text of a second length before the sample label value corresponding to the specified field, the sample label value corresponding to the specified field, and text of a second length after the sample label value corresponding to the specified field; Generate a regular expression that can match the sample label value from all sample substrings corresponding to the specified field.

3. The method according to claim 2, after obtaining the sample substring corresponding to each specified field from each sample text without spaces for each specified field, the method further includes: Record the maximum length of the sample substring corresponding to the specified field, and use half of the recorded maximum length as the first length to extract the hit text of the specified field.

4. The method according to claim 2, after converting the preset number of sample documents into sample text without spaces, the method further includes: Establish a second character index mapping from each sample document to the sample text without spaces; For each specified field, based on the start and end indices of the specified field's annotation value in the sample document and the second character index mapping, the starting position of the specified field's annotation value in each sample text without spaces is determined; Extract the third length of text before the determined starting position from each sample's non-space-free text, and determine the maximum common subsequence of the extracted text.

5. The method according to claim 4, wherein, The process of finding the index position of each specified field in the text without spaces includes: For each specified field, the index position of the longest common subsequence corresponding to the specified field in the text without spaces is used as the index position of the specified field.

6. The method according to claim 2, after obtaining the sample substring corresponding to each specified field from each sample text without spaces for each specified field, the method further includes: Convert each sample substring corresponding to the specified field into dictionary format, and set the first and second longest texts of each sample substring to not need matching.

7. An information extraction device, comprising: The first conversion module is used to convert the document to be extracted into text without spaces. The search module is used to find the index position of each specified field in the text without spaces; The truncation module is used to extract a first length of text before and a first length of text after the index position corresponding to each specified field from the text without spaces, so as to obtain the hit text corresponding to the specified field. The matching module is used to match the hit text using a preset regular expression corresponding to the specified field, and to obtain the start and end index range of the part of the hit text that matches the preset regular expression; An extraction module is used to extract the field value corresponding to the specified field from the document to be extracted based on the start and end index range; The device further includes: The first mapping establishment module is used to establish a first character index mapping from the text without spaces to the document to be extracted; The extraction module is specifically used for: Based on the first character index mapping, the start and end index intervals are mapped to the document to be extracted, so as to obtain the start index position and end index position of the start and end index intervals in the document to be extracted; Extract the text from the starting index position to the ending index position in the document to be extracted, and obtain the field value corresponding to the specified field.

8. The apparatus according to claim 7, further comprising: The second conversion module is used to convert a preset number of sample documents into sample text without spaces. The acquisition module is used to acquire the sample substring corresponding to each specified field from each sample text without spaces for each specified field; wherein, the sample substring includes text of a second length before the sample label value corresponding to the specified field, the sample label value corresponding to the specified field, and text of a second length after the sample label value corresponding to the specified field. The generation module is used to generate regular expressions that can match sample label values ​​from all sample substrings corresponding to the specified field.

9. The apparatus according to claim 8, further comprising: The recording module is used to record the maximum length of the sample substring corresponding to the specified field, and half of the recorded maximum length is used as the first length when extracting the hit text of the specified field.

10. The apparatus of claim 8, further comprising: The second mapping establishment module is used to establish a second character index mapping from each sample document to the sample text without spaces. The determination module is used to determine the starting position of the annotation value of the specified field in each sample text without spaces, based on the start and end indexes of the annotation value of the specified field in the sample document and the second character index mapping. And from each sample of text without spaces, extract the text three lengths before the determined starting position, and determine the maximum common subsequence of the extracted text.

11. The apparatus according to claim 10, wherein, The search module is specifically used for: For each specified field, the index position of the longest common subsequence corresponding to the specified field in the text without spaces is used as the index position of the specified field.

12. The apparatus according to claim 8, further comprising: The format conversion module is used to convert each sample substring corresponding to the specified field into dictionary format, and set the first and last second length texts of each sample substring to not need to be matched.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images