Configuration word data extraction method based on credential environment

Abstract

The invention relates to the technical field of information processing, in particular to a configuration word data extraction method based on a credential environment, and the method comprises the following steps: S1, constructing an extraction content template and a query positioning rule; s2, constructing a target paragraph set; s3, analyzing and processing the text and the table; s4, post-processing an extraction result; and S5, outputting an extraction result. Compared with the prior art, the method has the advantages that an openjdk poi technology is adopted to analyze the Word document, through high configuration and flexibility, a user is allowed to customize an extraction rule according to needs, and data in the document is accurately positioned and processed through a regular expression and a synonym dictionary. Through integration of JSON and Kafka, standardized output of data and efficient data stream processing are realized. According to the technical scheme, the adaptability of the method in the Kylin environment is embodied in that the method can be compatible with a domestic operating system and a software environment, and the requirements of domestic users for data processing and information security are met.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a configurable word data extraction method based on a trusted computing environment. Background Art

[0002] The current industry mainly relies on GPU-based large model technology or x86-based Office toolkits when collecting structured data from announcements, but the compatibility of these technologies with domestic information innovation environments has not yet been achieved. In addition, the application of artificial intelligence technology has led to increased software and hardware costs, as well as low efficiency in processing long announcements. The structure of listed companies' announcements is complex and changeable, and the demand for data is updated rapidly, while the existing customized Word extraction tools are not scalable enough, and each time a new extraction requirement is added, a lot of development resources are required. At the same time, the existing technology is not ideal when parsing complex situations such as cross-page tables and frameless tables in announcements, resulting in uneven data quality, which requires manual verification and correction of each announcement and each content.

[0003] Therefore, there is an urgent need to develop a configurable word data extraction method based on the trusted computing environment to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a configurable word data extraction method based on the trusted innovation environment, so as to adapt to the trusted innovation environment to achieve compatibility, reduce software and hardware costs, improve the timeliness of processing long announcements, improve the scalability of word data extraction tools, and reduce the need for manual proofreading and correction.

[0005] In order to achieve the above purpose, a configurable word data extraction method based on the trusted innovation environment is designed, and the method is as follows: S1. Constructing extraction content templates and query positioning rules: the extracted content is divided into two types: text and table. The user fills in the regular expression of the paragraph where the extracted content is located in advance. If a keyword in a sentence is to be extracted, write the corresponding regular expression. If the table content is to be extracted, select the table type, including wide table, narrow table or information point table. The table header supports the alias function. The user standardizes the extracted header content by maintaining a synonym dictionary; S2. Constructing the target paragraph set: traverse the document elements, set the hth paragraph keyword hit, record it as , the style level is calculated by Poi built-in method, denoted as , continue to calculate the style level of subsequent paragraph elements ,when Character code less than , the paragraph is Sub-sections of Add to the result set until you reach Paragraphs of the same level , that is, calculate the style level of the jth paragraph Character encoding equals When , stop traversal, and the final target paragraph set Ω is expressed as ,in ; S3. Text and table parsing: S31. Text type field: Match in the paragraph set according to the pre-set regular expression list, stop traversal after hitting and return the matching result; S32. Table extraction: S321. Create a table entity based on the table element in the paragraph set, and use List <LinkedHashMap<> >Store the extracted content in a data structure so that the order of the original table entries is retained after json structuring; S322. Create a new one-dimensional header array to store the header information, and determine whether the cells are merged horizontally or vertically according to the CTTcPr style; S323. Look up the configured header synonym dictionary, and if there is a synonym, replace the header column name; S324. Assemble data according to the configured table type: If it is a wide table, the first row of the table is used as a fixed key, and the content of the cell below the same column is the value, forming r LinkedHashMaps of size c; If it is a narrow table, press The columns form a LinkedHashMap, that is, the value of the first column of each row is used as a fixed key, and the content of the right cell of the same row is used as the value, forming c LinkedHashMaps of size r; the information point table is suitable for long tables with many disclosed subjects. The subject keywords are entered in advance. After matching the cells containing the keywords, a LinkedHashMap containing the row name and list is generated, and the existence of sub-subjects is determined based on the multi-level numbering and style indentation. According to the above operation, the Map of the sub-entry is also added to the result list. After parsing to the subject of the same level, the table traversal is stopped.

[0006] Preferably, the method described in the present invention also includes other technical features, wherein the S322 is specifically as follows: if the CTTcPr style determines that the cell is merged vertically, the array index remains unchanged and the table index is increased by 1; if it is merged horizontally, the array fills the same value index increased by 1, and the table index is added to the number of cells merged horizontally.

[0007] Preferably, the method described in the present invention also includes other technical features, wherein the method also includes S4. Post-processing of extraction results: standardizing the format of the extracted values ​​and dates; S5. Output of extraction results: after the extraction is completed, the extraction results will be associated with document information, assembled into json, and pushed to the downstream system in the form of kafka message push.

[0008] Compared with the prior art, the present invention has the following advantages: The present invention adopts OpenJDK POI technology to parse Word documents. Through high configuration and flexibility, users are allowed to customize extraction rules as needed, and regular expressions and synonym dictionaries are used to accurately locate and process data in documents. At the same time, the complexity of table data is also considered, and different types of table structures are adapted through different processing logics. Finally, through the integration of JSON and Kafka, standardized data output and efficient data stream processing are achieved. The adaptability of the technical solution of the present invention in the information creation environment is reflected in the completion of word parsing by customizing and developing the underlying XML. Only using the CPU has good execution performance and supports multi-platform operation, solves the dependence on existing Microsoft Office integrated tools, Windows systems and GPUs, and enables it to be compatible with domestic operating systems and software environments, meeting the needs of domestic users for data processing and information security. After extensive document testing, compared with conventional announcement extraction systems, the Word extraction process and method designed by us can automatically extract the key elements of announcements more efficiently and accurately, and quickly identify models that fail to extract, thereby reducing the workload of manual review. In addition, the system has good scalability, and the extraction requirements of new elements can be achieved by simply creating a new model configuration, which significantly reduces the workload of data development. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 , is the overall flow chart of the present invention; Figure 2 , is a schematic diagram of table extraction. DETAILED DESCRIPTION

[0010] In order to make the purpose, principle and structure of the present invention more clear, it is further described below in conjunction with the drawings and specific embodiments.

[0011] Embodiment 1: See also Figure 1 , 2 , this embodiment provides a configurable word data extraction method based on the trust creation environment. The inventor first designed the fields that need to be extracted from different types of announcements according to the needs of market supervision and information providers, but different disclosure elements of the announcement are in different positions in the text (tables, paragraphs, sentences, etc.), and the extraction methods are also different. The table also includes wide tables and narrow tables, as well as frameless tables without headers, etc. In order to solve the problem of developing a new extraction module for each new extraction demand, we designed a set of general configuration templates based on the characteristics of information elements to complete the automatic extraction of new elements of the announcement. The main steps are: S1: Construct extraction content templates and query positioning rules: The extracted content is divided into two types: text and table. The user fills in the regular expression of the paragraph where the extracted content is located in advance. If you want to extract a keyword in a sentence, you need to write the corresponding regular expression. If you want to extract table content, you need to select the table type (wide table, narrow table, information point table). The table header supports the alias function. Users can standardize the extracted header content by maintaining a synonym dictionary.

[0012] S2: Construct the target paragraph set: traverse the document elements, set the hth paragraph keyword hit, record it as We calculate the style level through built-in methods such as getStyle() and getOutlineLvl()poi, denoted as , continue to calculate the style level of subsequent elements ,when Character code less than , the paragraph is Sub-sections of Add to the result set until you reach Paragraphs of the same level , that is, calculate the style level of the jth paragraph Character encoding equals When , stop traversal. The final target paragraph set Ω can be expressed as ,in ,The paragraph set generated by this method only retains the required chapter content, which is more efficient than full-text traversal matching and removes ambiguous text in non-target chapters, making the extraction more accurate.

[0013] S3: Text and table parsing 3.1 The text type field is matched in the paragraph set according to the pre-set regular expression list. After a match is found, the traversal stops and the matching result is returned.

[0014] 3.2 The steps for table extraction are as follows: Step 1: Create a table entity based on the table element in the paragraph set. The entity contains the table name, unit, currency, and uses List <LinkedHashMap<> >The data structure stores the extracted content so that the order of the original table entries is retained after JSON structuring.

[0015] Step 2: Create a new one-dimensional header array to store header information. Determine whether the cells are merged horizontally or vertically based on the CTTcPr style. If it is a vertical merge, the array index remains unchanged and the table index increases by 1. If it is a horizontal merge, the array is filled with the same value index increased by 1, and the table index is added to the number of cells merged horizontally.

[0016] Step 3: Search the configured header synonym dictionary. If there is a synonym, replace the header column name. Step 4: Assemble data according to the configured table type. Assume that the table has c columns and r rows. If it is a wide table, the first row of the table is used as a fixed key, and the content of the cells below the same column is the value, forming r LinkedHashMaps of size c; if it is a narrow table, LinkedHashMap is formed by column, that is, the value of the first column of each row is used as a fixed key, and the content of the cells on the right side of the same row is used as the value, forming c LinkedHashMaps of size r; Information point tables are generally suitable for long tables with many disclosed subjects. Subject keywords need to be entered in advance. After matching the cells containing keywords, a LinkedHashMap containing row names and lists is generated, and the existence of sub-subjects is determined based on multi-level numbering and style indentation. According to the above operation, the Map of the sub-entry is also added to the result list. After parsing to the same level subject, the table traversal is stopped. This method can more flexibly complete the extraction of established table elements and better provide services to different demand parties.

[0017] S4: Post-processing of extraction results In order to facilitate the storage of structured data and subsequent quantitative analysis, the extracted values ​​and dates need to be standardized in format. By default, the program multiplies the number by the extracted unit to convert it into a value in "yuan" and converts the extracted date into the format of yyyy-MM-dd. In addition, if the post-processing method is pre-configured (for example, the date is output as "yyyy-MM-dd", and the value is converted into "million yuan" for output, etc.), the program will give priority to using the custom method for conversion output.

[0018] S5: Extraction result output After the extraction is completed, the extraction results will be associated with the document information, assembled into JSON, and pushed to the downstream system in the form of Kafka message push.

[0019] Embodiment 2: This embodiment specifically provides a core specific implementation scheme for implementing the technical solution of Embodiment 1: 1. Specific implementation plan for paragraph positioning: Provides a public method named `fliterParaByTile`, which is used to filter out paragraphs containing specific heading keywords from a Word document.

[0020] Returns a list of `IBodyElement` type, receiving an `XWPFDocument` object, a `IBodyElement` list, a title regular expression `headingRex`, and an exclude title regular expression `headingExclueRex` as parameters.

[0021] Create an empty `IBodyElement` list to store the filtered paragraphs. Initialize an empty string `titleLvl` to store the level of the current title.

[0022] Iterate over each element in the passed `elements` list. Check if the current element is of type `XWPFParagraph` (i.e., a paragraph). Get the text content of the current paragraph. Convert the current element to type `XWPFParagraph`.

[0023] If the paragraph text is empty, skip the current loop. If the current paragraph is the beginning of a title at the same level, reset titleLvl. If the paragraph text contains the title keyword, the title level is not empty, and does not contain tabs, it is considered a matching title.

[0024] If the paragraph text contains the exclusion keyword, terminate the loop. Add the matching paragraph to the result list. Update the current heading level.

[0025] If the current title level is not empty, but the paragraph level is empty, add the paragraph to the result list. If the current title level is less than the paragraph level, add the paragraph to the result list. If `titleLvl` is not empty, and the current element is not a paragraph, it is also added to the result list.

[0026] End the loop and return the filtered paragraph list.

[0027] 2. Specific implementation plan for header generation: The function of this specific implementation scheme is to extract header information from a table in a Word document.

[0028] Provide a public method that returns a map with keys as integers and values ​​as string arrays, and receives an `XWPFTable` object as a parameter. Call the `calMaxCol` method to calculate the maximum number of columns in the table and return a map.

[0029] Get the key set iterator of the map. Get the index of the first row of the table header. Get the table header array corresponding to the index. Initialize all elements of the table header array to empty strings. Get all rows of the table. Traverse the rows of the table until the last row of the table header.

[0030] Get all cells in the current row. Initialize cell index. Initialize column index. Traverse cells in the current row. Get the current cell. Get cell properties. Get the text content of the cell. Get the horizontal merge property of the cell.

[0031] If the cell has horizontal merges and the text is not empty. Initialize the number of horizontal merged columns. Get the number of merged cells. If the merge count is not empty, print and set the number of merged columns. For each merged cell, fill the header array and increase the column index.

[0032] Increase the cell index if the cell has vertical merge and the cell text is empty, but there is content in the array.

[0033] Increase the cell index and column index. If the cell text is not empty. Fill the header array and increase the column index and cell index. If the cell text is empty, increase the cell index and column index.

[0034] End the loop.

[0035] Create an empty map to store the results. Put the header array into the map, using the row index + 1 as the key.

[0036] Returns the header map.

[0037] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent substitutions or changes according to the technical solutions and novel concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A configuration-based word data extraction method based on a trusted environment, characterized in that The method is as follows: S1. Constructing extraction content template and query positioning rules: The extracted content is divided into two types: text and table. Users pre-fill in the regular expression of the paragraph where the extracted content is located. If you want to extract a keyword in a sentence, write the corresponding regular expression. If you want to extract table content, select the table type, including wide table, narrow table or information point table. The table header supports the alias function. Users can standardize the extracted header content by maintaining a synonym dictionary. S2. Construct the target paragraph set: Traverse the document elements, set the hth paragraph keyword hit, record it as , the style level is calculated by Poi built-in method, denoted as , continue to calculate the style level of subsequent paragraph elements ,when Character code less than , the paragraph is Sub-sections of Add to the result set until you reach Paragraphs of the same level , that is, calculate the style level of the jth paragraph Character encoding equals When , stop traversal, and the final target paragraph set Ω is expressed as ,in ; S3. Text and table parsing: S31. Text type field: matches in the paragraph set according to the pre-set regular expression list, stops traversal after a hit, and returns the matching result; S32. Table extraction: S321. Create a table entity based on the table element in the paragraph set, and use List <LinkedHashMap<> >The data structure stores the extracted content so that the order of the original table entries is retained after JSON structuring; S322. Create a new one-dimensional header array to store header information, and determine whether the cells are merged horizontally or vertically according to the CTTcPr style; S323. Search the configured table header synonym dictionary, and if there is a synonym, replace the table header column name; S324. Assemble data according to the configured table type: If it is a wide table, the first row of the table is used as a fixed key, and the content of the cells below the same column is the value, forming r LinkedHashMaps of size c; If it is a narrow table, LinkedHashMap is formed by columns. That is, the value of the first column of each row is used as a fixed key, and the content of the cell on the right side of the same row is used as the value, forming c LinkedHashMaps of size r; The information point table is suitable for long tables with many disclosed subjects. Enter the subject keyword in advance, match the cell containing the keyword, generate a LinkedHashMap containing the row name and list, and determine whether there are sub-subjects based on the multi-level numbering and style indentation. According to the above operation, the Map of the sub-entry is also added to the result list. After parsing to the subject of the same level, stop traversing the table.

2. A method for extracting word data based on a configurable information creation environment as described in claim 1, characterized in that The S322 is specifically as follows: if the CTTcPr style determines that the cell is merged vertically, the array index remains unchanged and the table index increases by 1; if it is merged horizontally, the array fills the same value index and increases by 1, and the table index increases by the number of cells merged horizontally.

3. A method for extracting word data based on a configurable information creation environment as described in claim 1, characterized in that The method further comprises S4. Post-processing of extraction results: Standardize the format of the extracted values ​​and dates; S5. Extraction result output: After the extraction is completed, the extraction results will be associated with the document information, assembled into JSON, and pushed to the downstream system in the form of Kafka message push.

Images