Method for extracting PDF inspection report data based on character feature template matching

Abstract

The present invention discloses a method for extracting PDF test report data based on character feature template matching. The method converts the location information of key-value pairs, the data types that meet the requirements, and the location of tables into a computer-readable layout character feature template through encoding. The table character feature template provides auxiliary functions for table extraction, and the table is restored in three different ways. The present invention is particularly suitable for native PDF test reports containing semi-structured data in the fields of biochemical and environmental materials. The method can extract the data of interest specified in the PDF test report, realize structured storage, efficiently and accurately obtain relevant information about the test substances, and facilitate subsequent data analysis.

Description

Technical Field

[0001] The present invention relates to the field of data governance, and in particular to a method for extracting PDF detection report data based on character feature template matching. Background Art

[0002] Portable Document Format (PDF) is a data carrier widely used in electronic file exchanges in various industries due to its cross-platform versatility and unique advantages of complete reproduction of document originals, and has become one of the important data transmission methods.

[0003] Many manufacturers purchase testing equipment that exports numerous PDF reports via templates after completing inspections, without providing interfaces for data integration. This creates inconvenience for future data storage and analysis. Faced with massive amounts of PDF data to be processed, manual review and data entry is clearly not the optimal solution. Leveraging powerful processing capabilities, computers can replace manual data entry, offering a breakthrough in improving data processing efficiency and reducing costs.

[0004] For example, the technical solution of the patent application number "CN202211208393.6", the patent name "Method and Storage Medium for Extracting Information from Large-Scale Multi-Format Files", and the patent applicant "Fujian Zhengfu Software Co., Ltd." is as follows: obtain files in a large-scale multi-format file collection one by one; if the obtained file is a PDF file or a PPT file, determine its form and content; if the PDF file or PPT file is plain text information, extract the text information therein; if the PDF file or PPT file is a scanned copy or a picture, first process the picture and recognize the picture, and then use OCR recognition to extract the information therein. For this solution, we only focus on extracting the information of the PDF file, and do not care how the extracted information is output or whether data integration is performed.

[0005] For example, the technical solution with application number "CN202111113749.3", patent name "A method and device for extracting structured information of PDF files", and patent applicant "Chengdu Sifang Weiye Software Co., Ltd." discloses the following content: S1. Filter out editable PDF documents and read editable PDF documents; S2. Extract the text content of the editable PDF document in step S1, and then segment the text content to obtain a string group; S3. Traverse the string group of step S2, add prefixes to form a discriminant index; S4. According to the discriminant index of step S3, extract its structured information; S5. Write the structured information conversion format of step S4 into the database. This technical solution can process PDF files in batches and form a complete method for converting unstructured data to structured data from PDF documents. This technical solution has the following disadvantages: 1. This technical solution has high skill requirements for users, requiring them to have certain knowledge of character matching principles, especially the use of regular expressions. This may present a certain learning curve for ordinary users. 2. Before use, users must summarize and generalize the types of PDF documents to be processed in order to create character matching templates for automated information extraction. This process can be relatively tedious, increasing the complexity of use. 3. Current technical solutions may encounter problems when processing complex tables. For example, the accuracy of information extraction may be affected when encountering cross-column or cross-row situations. 4. This technical solution is built on the principle of PDF code recognition, but in some cases, the content within the PDF code may differ from the actual PDF display content, which will affect the accuracy of information extraction.

[0006] The PDF test reports for biochemical and environmental materials (such as those provided by instruments from manufacturers such as Thermo Fisher Scientific and Agilent) have their own particularities:

[0007] 1. PDF test reports for biochemical and environmental materials are primarily native PDFs. PDFs are divided into native PDFs and scanned PDFs. Native PDFs, also known as editable PDFs, are typically exported from Word or Excel documents and created using PDF software, allowing for modification and editing. Scanned PDFs, on the other hand, are generated from images scanned and are generally unmodifiable and non-editable. Their primary purpose is to allow users to open and view the contents.

[0008] 2. In the field of biochemical and environmental materials, there is a lot of data that needs statistical analysis in the PDF inspection reports, and the demand for extracting, reusing and re-editing PDF table information is increasing.

[0009] 3. Most importantly, the internal data in PDF inspection reports in the biochemical, environmental, and materials fields can be broadly divided into key-value pairs and tabular data. Key-value pairs generally use delimiters like ":" to form what is known as a key and value in the computer field. This is an implementation of the mathematical concept of mapping relationships, with the "key" serving as an index to an element and the "value" representing the stored data to be retrieved. Key-value pairs themselves do not have a complex structure and are relatively simple to extract using string matching. Tables are regular blocks of text or areas divided by regular lines, with certain relationships and constraints between the text blocks. However, the PDF standard lacks the concept of tables or rigorous logical declarations. PDFs simply use the positional arrangement of text blocks, often using graphic lines to demarcate cells, to visually assist in forming tables. From an encoding perspective, PDF tables lack the structural features similar to HTML tables. This means that it is impossible to directly perceive tables from the encoding and obtain their two-dimensional logical structure. Therefore, extracting this data is much more complex. Unstructured digital documents, such as tabular data in PDF, face difficulties in parsing into a machine-readable structured format due to their complex and diverse structure and format. This poses new challenges for PDF table recovery and extraction.

[0010] 4. PDF test reports in the biochemical, environmental, and materials fields cannot be simply viewed as unstructured data. They include some structured data, as well as a large amount of unstructured data. Their data organization is not strictly in the form of tables and fields, but rather uses tags, attributes, or encapsulated elements to express the relationships and organization between elements. PDF test reports in the biochemical, environmental, and materials fields typically contain a large number of key-value pairs and tables and should be considered semi-structured data, a data format that lies between structured and unstructured data.

[0011] Compared to structured data, semi-structured data is indeed more difficult to process. This is primarily because semi-structured data possesses both structured and unstructured data elements, requiring its processing to consider the characteristics of both types of data. Specifically, semi-structured data often contains numerous tags, attributes, or encapsulated elements. The relationships and organization between these elements are often complex and do not always adhere to a fixed pattern. This makes the parsing, understanding, and processing of semi-structured data more complex than that of structured data. Summary of the Invention

[0012] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and provide a method for extracting PDF test report data based on character feature template matching. The method is particularly suitable for native PDF test reports containing semi-structured data in the fields of biochemical and environmental materials. The method can extract the data of interest specified in the PDF test report, realize structured storage, efficiently and accurately grasp the relevant information of the test substance, and facilitate subsequent data analysis.

[0013] The purpose of the present invention is achieved through the following technical solutions:

[0014] The method for extracting PDF inspection report data based on character feature template matching includes the following steps:

[0015] S1. Convert the same type of native PDF detection report into character txt text;

[0016] S2. Configuring a corresponding feature template and a table experience distribution configuration file for this type of PDF test report, wherein the feature template includes a layout character feature template and a table character feature template;

[0017] S3. Match the character txt text corresponding to each native PDF detection report with the surrounding layout character feature template line by line: if it does not match, no information extraction is performed; if it matches, key-value pair data is extracted and the process goes to step S4;

[0018] S4. Locate the table header and end position of the native PDF detection report, return the upper and lower bounds of the table, and the text content of the table body, to pave the way for restoring the table using the embedded table character feature template;

[0019] S5. Determine whether the table type of the native PDF test report is a table with complete and regular table lines:

[0020] If so, we use the PDF table line detection and restoration method to restore all lines in the PDF, guess the lines that can form a table, and form multiple possible tables. We use the upper and lower bounds of the table to locate the target table. All lines of the table intersect to form intersections, and every four adjacent intersections form a cell. Based on the location of the cell, we query the PDF for the characters belonging to each cell, and finally restore the table to obtain a unique and reasonable solution for each row.

[0021] If not, jump to step S6;

[0022] S6. Continue to determine whether the table type is the type where the table body is aligned with the header:

[0023] If so, extract all character blocks in all PDFs, with each string accompanied by the position and size parameters of its bounding rectangle; filter out the character block set belonging to the target table based on the upper and lower bounds of the table and sort them; cluster the character set into several rows by row; traverse all character blocks in the character block set, and use the position of the character block in the row to make a horizontal projection intersection with the position of the table header. If the intersection is greater than a certain threshold, the character block is considered to belong to the value of the table header; when all operations are completed, the two-dimensional logical structure of the table is restored, and a unique reasonable solution is obtained for each row;

[0024] If not, jump to step S7;

[0025] S7. Use the txt content of the table body to split the characters according to the spaces in each line of the table body: Assume that it is divided into m parts. For the number of table headers, plus the maximum number of spaces that each header value can contain, the number of m parts is n. The values ​​of m parts are combined into n spaces in a relative order. All possible distributions are For each type of table value distribution, compare the data type of the table header with the value it is assigned to. If they all match, it is considered a reasonable solution. Process each row to restore the table's logical structure.

[0026] The relationship between the txt content of the table body and the table experience distribution configuration file is:

[0027] When restoring a table row, first split the row corresponding to the table body txt content into cells according to the space. Suppose it is divided into m cells, and then match these m cells with the type of each column of the row in the table character feature template. Suppose there are n columns, n is generally greater than m, so all cases have species, but only a few species can match the type of each column, and then compare the distribution of these m matching cells on n columns with the distribution recorded in the table's empirical distribution configuration file. If this distribution exists in the distribution recorded in the table's empirical distribution configuration file, then the accuracy index returned by the algorithm will be higher, otherwise the accuracy index returned will be reduced to a certain extent.

[0028] S8. For each row of reasonable solutions, first set the initial accuracy based on the matching of the corresponding header type and the table empirical distribution profile, and finally update the row accuracy according to the formula; return the reasonable solution with the highest accuracy among all reasonable solutions for each row and its accuracy, and output the remaining solutions as alternative suboptimal solutions.

[0029] The layout character feature template is generated by converting the positioning information of the key-value pair, the data type that is satisfied, and the positioning of the table into a form that can be interpreted by a computer through encoding; the layout character feature template describes the page header and footer, the restoration method used by the table, the pattern that the extracted character string should meet, and other strict positioning information.

[0030] The layout character feature template includes the key in the key-value pair, the corresponding pattern satisfied by the value, and the positioning anchor, which is used to locate various information; the positioning anchor includes separators, headers, and table positioning information; the layout character feature template is an abstract expression of the PDF layout, and the arrangement of features matches the layout method in the PDF.

[0031] The parameters of the table character feature template include a table header, a value type corresponding to each table header, and a maximum number of spaces that a value corresponding to each table header can contain.

[0032] In step S8, the initial accuracy is set as follows: if the type match is successful and belongs to one of the previous empirical distributions, the initial accuracy is set to 100%; if the type match is successful but does not belong to one of the previous empirical distributions, the initial accuracy is X, 0%<X<100%; in other cases, the initial accuracy is 0%.

[0033] In step S8, the updated row accuracy is new acc , the calculation formula is as follows:

[0034]

[0035] Among them, num is the number of multiple reasonable solutions obtained for a row, old acc is the initial accuracy of the row, old accs Represents the initial accuracy vector of all optimal solutions in this row.

[0036] When the table type of the native PDF test report is a table with complete and regular table lines, the extraction method is as follows:

[0037] Let the original PDF page be P, and convert it into an image, represented as:

[0038] I = f(P);

[0039] Where I represents the transformed image; for the set of several horizontal lines L that may exist in the predicted PDF h and some vertical line sets L v ,in:

[0040]

[0041] The detection process is described as:

[0042] g(I)={L h , L v};

[0043] According to the detected lines, the horizontal and vertical lines intersect with each other to form intersection points, and the intersection points form the text block set B of the table:

[0044] B=h(L h , L v );

[0045] The text block set determines the row area of ​​the table through clustering and other methods Column Area

[0046]

[0047] Fill the cell contents based on the row and column areas and the text in the image to restore the original structure of the table:

[0048]

[0049] Where T represents the restored table, and the entire table restoration process can be represented by a composite function:

[0050] T = m(f(P), k(h(g(f(P))));

[0051] This algorithm locates the beginning and end of the table through the matching results of the layout character feature template. After restoring the table parts scattered on each page through the table restoration algorithm, it merges all the partial tables located at the beginning and end of the table to adapt to the merging of paged tables.

[0052] The table type is a table body aligned with the header, and the extraction method is as follows:

[0053] Extract word fragments formed after word segmentation in the pdfplumber library i , the word fragment set formed is textblock; among them:

[0054] textblock={tb1,tb2,...,tb n};

[0055] And return the bounding rectangle parameters of each word fragment together, indicating the position of the word fragment in the PDF page; the word fragment tb i The bounding rectangle parameters include the horizontal coordinate of the upper left corner of the bounding rectangle vertical axis And the horizontal coordinate of the lower right corner of the circumscribed rectangle vertical axis

[0056] For two adjacent word fragments tb i and tb i+1 ,if:

[0057] and

[0058] Then tb i and tb i+1 Combine into a complete word:

[0059] w=merge(tb i , tb i+1 , t w);

[0060] in, For word fragment tb i+1 The horizontal coordinate of the upper left corner of the circumscribed rectangle, For word fragment tb i+1 The vertical coordinate of the upper left corner of the circumscribed rectangle; t w is a parameter obtained through mathematical statistical analysis;

[0061] Select multiple samples to find the mean μ and standard deviation σ, and calculate t w :

[0062] t w =μ+4σ;

[0063] If the horizontal distance between adjacent words is small, less than t g (t g Using the same calculation method of mean and standard deviation), we consider that there is a space between two adjacent words and classify them as the same phrase:

[0064] ph=merge(w1,w2,t g );

[0065] The phrase blocks are clustered by position rows to complete the division of table columns:

[0066] {r1, r2, ..., r l}=cluster(ph1,ph2,...,ph k );

[0067] Among them, r represents the column of the table, l is the number of clustering result classes, that is, the number of table rows, and k is the total number of phrases; for two phrases ph i and ph i+1 , the intersection interval S of its horizontal projection interval can be calculated as follows:

[0068]

[0069] in, It is a word fragment tb i+1 The horizontal coordinate of the lower right corner of the circumscribed rectangle;

[0070] If S satisfies one of the following three conditions:

[0071] S>T P ;

[0072]

[0073] It is believed that the two phrases should belong to the same column of the table, where T Pis a constant threshold value, which is an empirical parameter. Dividing the columns of the table by this operation can be regarded as a disguised clustering process:

[0074] {c1,c2,...,c0}=cluster(ph1,ph2,...,ph k );

[0075] Get the structured table T:

[0076] T={(r1,c1,v 1,1 )(r1,c2,v 1,2 ),...,(r l , c o ,v l，o )}

[0077] =structural({r1, r2, .., r l},{c1,c2,..,c o});

[0078] Among them, C represents the column of the table, o is the number of clustering result classes, that is, the number of columns in the table; v is the cell of the table, and its two subscripts represent the row and column of the cell respectively.

[0079] The table type does not belong to the table body alignment table header, and its extraction method is as follows:

[0080] By matching the layout character feature template, we can get the rough text rows [S1, S2, ..., S k ], first, each row S i Split into columns For the table header row [H1, H2, ..., H n ], observe the multiple tables of this PDF, and abstractly define a data type Ty1, Ty2, ..., Ty based on semantic experience. n , in order Assign to the table header n columns, generally m≤n, according to the principles of combinatorial mathematics distribution of species;

[0081] A certain allocation method M = {(j, q) | j ≤ m, q ≤ n, f (j) = q} represents Assigned to the qth header, check its type and the assigned header type Ty q Are they consistent? If they are consistent, they will get points, otherwise no points:

[0082]

[0083] For this allocation, the total score is:

[0084]

[0085] in, The jth column into which the i-th row of bold text is split;

[0086] A score of a combined allocation method score=m indicates a complete match of the table header type. This matching method can be returned as a restored table result.

[0087] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0088] 1. The present invention can effectively extract key-value pairs and table data from semi-structured or unstructured PDF documents, thereby achieving accurate data acquisition.

[0089] 2. For multiple test report PDFs under the same template, the present invention has the function of quickly extracting data in batches, which greatly improves data processing efficiency.

[0090] 3. Whether it is a single or batch PDF test report, the present invention can accurately extract the information therein to meet the needs of various usage scenarios.

[0091] 4. For PDF test reports with special requirements, the present invention also provides the function of customizing special templates to extract the required data more accurately.

[0092] 5. In order to help users understand the extraction effect, the present invention also provides feedback data so that users can have a preliminary understanding of the possible extraction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 A PDF sample image.

[0094] Figure 2 It is a txt text image.

[0095] Figure 3 It is the layout character feature template diagram.

[0096] Figure 4 It is a template diagram of table character features.

[0097] Figure 5 This is the empirical distribution diagram.

[0098] Figure 6 This is a flow chart of the method for extracting PDF detection report data based on character feature template matching described in the present invention.

[0099] Figure 7 This is a PDF sample diagram of a table with complete and neat table lines.

[0100] Figure 8 This is a PDF sample image of a table with the table body aligned with the header.

[0101] Figure 9 This is a PDF sample image of a table with a non-body-aligned header. DETAILED DESCRIPTION

[0102] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0103] like Figure 6 The method for extracting PDF inspection report data based on character feature template matching includes the following steps:

[0104] Step 1: Collect a large number of various PDF test report samples and initially eliminate scanned PDF files;

[0105] Step 2: To avoid the impact of Chinese and English characters on the configuration template, first convert all PDF text information in the class into character txt text files using the PDF Plumber third-party library parsing tool for subsequent matching and extraction, such as Figure 2 Shown Figure 1 The corresponding txt text image.

[0106] Step 3. For a certain type of PDF, although there are differences in data filling and the relative position of data is offset, the consistency of the output template can ensure that each report has the same structure and information display method, ensuring the stability of the PDF; observe the information arrangement and data type pattern in multiple PDFs within the class, summarize the abstract data type and layout method, and combine the txt file corresponding to the PDF to complete the feature template required by the configuration algorithm for each PDF class. The feature template includes layout character feature template and table character feature template, as shown below. Figure 3 、 Figure 4 shown.

[0107] A layout character feature template includes the key in a key-value pair, the corresponding pattern that the value satisfies, and positioning anchors. Positioning anchors are used to locate various information, including delimiters, table headers, and table positioning information. A layout feature template is an abstract expression of the PDF layout, and the arrangement of features matches the layout in the PDF.

[0108] The table character feature template parameters include the table header, the value type corresponding to each table header, and the maximum number of spaces that the value corresponding to each table header can contain, such as Figure 4 As shown;

[0109] Step 4: Observe the pattern of missing values ​​in the target table rows and form a table experience distribution profile, such as Figure 5As shown, each row indicates the possible distribution of a row of data in the table in which columns, 1 indicates that the column has a value, and 0 indicates that the column has no value; the empirical distribution profile needs to record all possible distributions of the table in order to calculate the accuracy;

[0110] Step 5: Use the txt file corresponding to each PDF to match the peripheral layout character feature template line by line. If a PDF is found to not meet the set peripheral layout character feature template during line-by-line matching, exit the program directly without performing any extraction; otherwise, extract key information key-value pairs and check whether the values ​​meet the specified pattern. If they meet, return the value result and 100% accuracy; otherwise, return the value result and 0% accuracy. The accuracy is used to alert users that the extracted results may not meet the previous rules;

[0111] Step 6: Match the outer layout character feature template to locate the table header and end position, return the table's upper and lower bounds (beg_loc and end_loc), and the txt content of the table body, to pave the way for restoring the table using the embedded table character feature template.

[0112] Step 7: Determine whether the table type is a table with complete and regular lines:

[0113] If so, use the PDF table line detection and restoration method to restore all lines in the PDF, guess the lines that can form a table, and form possible multiple tables; use the table's upper and lower bounds beg_loc and end_loc to locate the target table. All lines of the table intersect to form intersections, and every four adjacent intersections form a cell. Based on the cell location, query the PDF for the characters belonging to each cell, and finally restore the table to obtain a unique and reasonable solution for each row, and jump to step 10;

[0114] like Figure 7 When the table type of the native PDF test report is a table with complete and regular table lines, the extraction method is as follows:

[0115] Let the original PDF page be P, and convert it into an image, represented as:

[0116] I = f(P);

[0117] Where I represents the transformed image; for the set of several horizontal lines L that may exist in the predicted PDF h and some vertical line sets L v ,in:

[0118]

[0119] The detection process is described as:

[0120] g(I)={Lh , L v};

[0121] According to the detected lines, the horizontal and vertical lines intersect with each other to form intersection points, and the intersection points form the text block set B of the table:

[0122] B=h(L b , L v );

[0123] The text block set determines the row area of ​​the table through clustering and other methods Column Area

[0124]

[0125] Fill the cell contents based on the row and column areas and the text in the image to restore the original structure of the table:

[0126]

[0127] Where T represents the restored table, and the entire table restoration process can be represented by a composite function:

[0128] TG=m(f(P),k(h(g(f(P)))));

[0129] This algorithm locates the start and end of the table by matching the layout character feature template. After restoring the table parts scattered on each page using the table restoration algorithm, it merges all the parts of the table located between the start and end of the table to adapt to the merging of paged tables.

[0130] Otherwise, skip to step 8.

[0131] Step 8: Determine whether the table type is a table body aligned with the header. If so, extract all character blocks in the PDF, with each string accompanied by the position and size parameters of its bounding rectangle. Filter the character blocks belonging to the target table based on the table's upper and lower bounds (beg_loc and end_loc). Use a sorting algorithm to sort the character set from top to bottom and left to right.

[0132] Cluster the character set by row into several rows; traverse all character blocks in the character block set, and use the position of the character block in the row to make a horizontal projection intersection with the position of the table header. If the intersection is greater than a certain threshold, the character block is considered to belong to the value of the table header. When all operations are completed, the two-dimensional logical structure of the table is restored, and a unique reasonable solution is obtained for each row, and jump to step 10;

[0133] like Figure 8The table type belongs to the table body aligned with the header, and its extraction method is as follows:

[0134] The PDF Plumber library provides a method called extract_words that can parse and extract characters from PDF stream files. However, the extracted characters are not complete words or phrases that humans understand, but fragments of words formed after the words are split. i , the word fragment set formed is textblock; among them:

[0135] textblock={tb1,tb2,…,tb n};

[0136] And return the bounding rectangle parameters of each word fragment together, indicating the position of the word fragment in the PDF page; the word fragment tb i The bounding rectangle parameters include the horizontal coordinate of the upper left corner of the bounding rectangle vertical axis And the horizontal coordinate of the lower right corner of the circumscribed rectangle vertical axis

[0137] For two adjacent word fragments tb i and tb i+1 ,if:

[0138] and

[0139] Then tb i and tb i+1 Combine into a complete word:

[0140] w=merge(tb i , tb i+1 , t w );

[0141] in, For word fragment tb i+1 The horizontal coordinate of the upper left corner of the circumscribed rectangle, For word fragment tb i+1 The vertical coordinate of the upper left corner of the circumscribed rectangle; T W is a parameter obtained through mathematical statistical analysis;

[0142] Select multiple samples to find the mean μ and standard deviation σ, and calculate t w :

[0143] t w =μ+4σ;

[0144] If the horizontal distance between adjacent words is small, less than tg (t g Using the same calculation method of mean and standard deviation), we consider that there is a space between two adjacent words and classify them as the same phrase:

[0145] ph=merge(w1,w2,t g );

[0146] The phrase blocks are clustered by position rows to complete the division of table columns:

[0147] {r1, r2, ..., r l}=cluster(ph1,ph2,...,ph k );

[0148] Among them, r represents the column of the table, l is the number of clustering result classes, that is, the number of table rows, and k is the total number of phrases; for two phrases ph i and ph i+1 , the intersection interval S of its horizontal projection interval can be calculated as follows:

[0149]

[0150] in, It is a word fragment tb i+1 The horizontal coordinate of the lower right corner of the circumscribed rectangle;

[0151] If S satisfies one of the following three conditions:

[0152] S>T P ;

[0153]

[0154] It is believed that the two phrases should belong to the same column of the table, where T P is a constant threshold value, which is an empirical parameter. Dividing the columns of the table by this operation can be regarded as a disguised clustering process:

[0155] {c1,c2,...,c O}=cluster(ph1,ph2,...,ph k ); get the structured table T:

[0156] T={(r1,c1,v 11，1 ),(r1,c2,v 1, 2),...,(r l ,c0,v L,o )}

[0157] =structural({r1, r2,...,r l},{c1,c2,....,c o});

[0158] Among them, C represents the column of the table, O is the number of clustering result classes, that is, the number of columns in the table; v is the cell of the table, and its two subscripts represent the row and column of the cell respectively.

[0159] Otherwise, skip to step 9.

[0160] Step 9. Use the table body txt content extracted in step 4 to split each line in the table body by spaces. Assuming it is split into m parts, the number of table headers plus the maximum number of spaces that each header value can contain is n. The m parts are combined into n spaces in a relative order. All possible distributions are: For each type of table value distribution, compare the data type of the table header and the value assigned to it to see if they match. If they all match, it is considered a reasonable solution. Process each row to restore the table's logical structure. Jump to step 10.

[0161] like Figure 9 The table type does not belong to the table body alignment table header, and its extraction method is as follows:

[0162] By matching the layout character feature template, we can get the rough text rows [S1, S2, ..., S k ], first, each row S i Split into columns For the table header row [H1, H2, ..., H n ], observe the multiple tables of this PDF, and abstractly define a data type Ty1, Ty2, ..., Ty based on semantic experience. n , in order Assign to the table header n columns, generally m≤n, according to the principles of combinatorial mathematics distribution of species;

[0163] A certain allocation method M = {(j, q) | j ≤ m, q ≤ n, f (j) = q} represents Assigned to the qth header, check its type and the assigned header type Ty q Are they consistent? If they are consistent, they will get points, otherwise no points:

[0164]

[0165] For this allocation, the total score is:

[0166]

[0167] in, The jth column into which the i-th row of bold text is split;

[0168] A score of a combined allocation method score=m indicates a complete match of the table header type. This matching method can be returned as a restored table result.

[0169] Step 10: First, match each row of logical solutions with the corresponding header type. If the type matches successfully and belongs to one of the previous empirical distributions, the initial accuracy is set to 100%. If the type matches successfully but does not belong to one of the previous empirical distributions, the initial accuracy is 60%. In other cases, the initial accuracy is 0%.

[0170] Step 11: Since multiple solutions may be extracted, multiple solutions will affect the row accuracy. The number of multiple reasonable solutions obtained for a row is num, and the initial row accuracy is old acc ,old accs Represents the initial accuracy vector of all optimal solutions for this row, and the updated row accuracy is:

[0171]

[0172] Step 12: Return the most accurate reasonable solution and its accuracy among all reasonable solutions in each row. The remaining solutions are also output as alternative suboptimal solutions.

[0173] At the same time, this embodiment provides:

[0174] A server includes a processor and a memory, wherein the memory stores at least one program, and the program is loaded and executed by the processor to implement the above-mentioned method for extracting PDF detection report data based on character feature template matching.

[0175] A computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the above-mentioned method for extracting PDF detection report data based on character feature template matching.

[0176] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for extracting PDF test report data based on character feature template matching, characterized in that: The following steps are involved: S1. Convert the same type of native PDF detection report into character txt text; S2. Configuring a corresponding feature template and a table experience distribution configuration file for this type of PDF test report, wherein the feature template includes a layout character feature template and a table character feature template; S3. Match the character txt text corresponding to each native PDF detection report with the surrounding layout character feature template line by line: if it does not match, no information extraction is performed; if it matches, key-value pair data is extracted and the process goes to step S4; S4. Locate the table header and end position of the native PDF detection report, return the upper and lower bounds of the table, and the text content of the table body, to pave the way for restoring the table using the embedded table character feature template; S5. Determine whether the table type of the native PDF test report is a table with complete and regular table lines: If so, we use the PDF table line detection and restoration method to restore all lines in the PDF, guess the lines that can form a table, and form multiple possible tables. We use the upper and lower bounds of the table to locate the target table. All lines of the table intersect to form intersections, and every four adjacent intersections form a cell. Based on the location of the cell, we query the PDF for the characters belonging to each cell, and finally restore the table to obtain a unique and reasonable solution for each row. If not, jump to step S6; S6. Continue to determine whether the table type is the type where the table body is aligned with the header: If so, extract all character blocks in all PDFs, with each string accompanied by the position and size parameters of its bounding rectangle; filter out the character blocks belonging to the target table based on the upper and lower bounds of the table and sort them; Cluster the character set into several rows by row; traverse all character blocks in the character block set, and use the position of the character block in the row and the position of the table header to make a horizontal projection intersection. If the intersection is greater than a certain threshold, the character block is considered to belong to the value of the table header; when all operations are completed, the two-dimensional logical structure of the table is restored, and a unique reasonable solution is obtained for each row; If not, jump to step S7; S7. Use the txt content of the table body to split the characters according to the spaces in each line of the table body: Assume that it is divided into m parts. For the number of table headers, plus the maximum number of spaces that each header value can contain, the number of m parts is n. The values ​​of m parts are combined into n spaces in a relative order. All possible distributions are For each type of table value distribution, compare the data type of the table header with the value it is assigned to. If they all match, it is considered a reasonable solution. Process each row to restore the table's logical structure. S8. For each row of reasonable solutions, first set the initial accuracy based on the matching with the corresponding header type and the table experience distribution profile, and update the row accuracy; return the reasonable solution with the highest accuracy among all reasonable solutions for each row and its accuracy, and output the remaining solutions as alternative suboptimal solutions.

2. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: The layout character feature template is generated by converting the positioning information of the key-value pair, the data type that is satisfied, and the positioning of the table into a form that can be interpreted by a computer through encoding; the layout character feature template describes the page header and footer, the restoration method used by the table, the pattern that the extracted character string should meet, and other strict positioning information.

3. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: The layout character feature template includes the key in the key-value pair, the corresponding pattern satisfied by the value, and the positioning anchor, which is used to locate various information; the positioning anchor includes separators, headers, and table positioning information; the layout character feature template is an abstract expression of the PDF layout, and the arrangement of features matches the layout method in the PDF.

4. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: The parameters of the table character feature template include a table header, a value type corresponding to each table header, and a maximum number of spaces that a value corresponding to each table header can contain.

5. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: In step S8, the initial accuracy is set as follows: if the type match is successful and belongs to one of the previous empirical distributions, the initial accuracy is set to 100%; if the type match is successful but does not belong to one of the previous empirical distributions, the initial accuracy is X, 0%<X<100%; in other cases, the initial accuracy is 0%.

6. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: In step S8, the updated row accuracy is new acc , the calculation formula is as follows: Among them, num is the number of multiple reasonable solutions obtained for a row, old acc is the initial accuracy of the row, old accs Represents the initial accuracy vector of all optimal solutions in this row.

7. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: When the table type of the native PDF test report is a table with complete and regular table lines, the extraction method is as follows: Let the original PDF page be P, and convert it into an image, represented as: I = f(P); Where I represents the transformed image; for the set of several horizontal lines L that may exist in the predicted PDF h and some vertical line sets L v ,in: The detection process is described as: g(I)={L h ,L v }; According to the detected lines, the horizontal and vertical lines intersect with each other to form intersection points, and the intersection points form the text block set B of the table: B=h(L h ,L v ); The text block set determines the row area of ​​the table through clustering and other methods Column Area Fill the cell contents based on the row and column areas and the text in the image to restore the original structure of the table: Where T represents the restored table, and the entire table restoration process is represented by a composite function: T = m(f(P), k(h(g(f(P))))); This algorithm locates the beginning and end of the table through the matching results of the layout character feature template. After restoring the table parts scattered on each page through the table restoration algorithm, it merges all the partial tables located at the beginning and end of the table to adapt to the merging of paged tables.

8. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: The table type is a table body aligned with the header, and the extraction method is as follows: Extract the word fragments formed after word segmentation in the PDF plumber library i , the word fragment set formed is textblock; among them: textblock={tb1,tb2,...,tb n }; And return the bounding rectangle parameters of each word fragment together, indicating the position of the word fragment in the PDF page; the word fragment tb i The bounding rectangle parameters include the horizontal coordinate of the upper left corner of the bounding rectangle vertical axis and the horizontal coordinate of the lower right corner of the circumscribed rectangle vertical axis For two adjacent word fragments tb i and tb i+1 ,if: and Then tb i and tb i+1 Merge into a complete word w: w=merge(tb i ,tb i+1 ,t w ); in, For word fragment tb i+1 The horizontal coordinate of the upper left corner of the circumscribed rectangle, For word fragment tb i+1 The vertical coordinate of the upper left corner of the circumscribed rectangle; t w is a parameter obtained through mathematical statistical analysis; Select multiple samples to find the mean μ and standard deviation σ, and calculate t w : t w =μ+4σ; If the horizontal distance between adjacent words is small, less than the parameter t g , it is considered that there is a space between two adjacent words and they are classified as the same phrase ph: ph=merge(w1,w2,t g ); The phrase blocks are clustered by position rows to complete the division of table columns: {r1,r2,…r l }=cluster(ph1,ph2,…,ph k ); Among them, r represents the column of the table, l is the number of clustering result classes, that is, the number of table rows, and k is the total number of phrases; for two phrases ph i and ph i+1 , the intersection interval S of its horizontal projection interval can be calculated as follows: in, It is a word fragment tb i+1 The horizontal coordinate of the lower right corner of the circumscribed rectangle; If S satisfies one of the following three conditions: S>T P ; It is believed that the two phrases should belong to the same column of the table, where T P is a constant threshold value, which is an empirical parameter. Dividing the columns of the table by this operation can be regarded as a disguised clustering process: {c1,c2,...,c o }=cluster(ph1,ph2,...,ph k ); Get the structured table T: T={(r1,c1,v 1,1 ),(r1,c2,v 1,2 ),...,(r l ,c0,v l，0 )} =structural({r1,r2,...,r l },{c1,c2,...,c o }); Among them, c represents the column of the table, o is the number of clustering result classes, that is, the number of columns in the table; v is the cell of the table, and its two subscripts represent the row and column of the cell respectively.

9. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: The table type does not belong to the table body alignment table header, and its extraction method is as follows: By matching the layout character feature template, we can get the rough text rows [S1, S2, ..., S k ], first, each row S i Split into columns For the table header row [H1, H2, ..., H n ], observe the multiple tables of this PDF, and abstractly define a data type Ty1, Ty2, ..., Ty based on semantic experience. n , in order Assign to the table header n columns, generally m≤n, according to the principles of combinatorial mathematics distribution of species; A certain allocation method M = {(j, q) | j ≤ m, q ≤ n, f (j) = q} represents Assigned to the qth header, check its type and the assigned header type Ty q Are they consistent? If they are consistent, they will get points, otherwise no points: For this allocation, the total score is: in, The jth column into which the i-th row of bold text is split; A score of a combined allocation method score=m indicates a complete match of the table header type. This matching method can be returned as a restored table result.

10. A computer-readable storage medium, characterized in that The storage medium stores at least one program, which is loaded and executed by the processor to implement the method for extracting PDF detection report data based on character feature template matching as claimed in any one of claims 1 to 9.

Images