Method for extracting PDF detection report data based on character feature template matching

Abstract

The invention discloses a method for extracting PDF detection report data based on character feature template matching, and the method comprises the steps: converting the positioning information of a key value pair, a satisfied data type and the positioning of a table into a layout character feature template which can be interpreted by a computer in a coding manner, and enabling a table character feature template to provide an auxiliary effect for table extraction. And the tables under different conditions are restored in three modes. The method is particularly suitable for a native PDF detection report containing semi-structured data in the field of biochemical ring materials, the specified interested data in the PDF detection report can be extracted, structured storage is achieved, related information of detected substances is efficiently and accurately mastered, and subsequent data analysis is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of data management, and in particular to a method for extracting PDF detection report data based on character feature template matching. Background Art

[0002] Portable Document Format (PDF), due to its cross-platform versatility and unique advantages of complete reproduction of document originals, is widely used as a data carrier in electronic file interaction in all walks of life and has become one of the important data transmission methods.

[0003] Many manufacturers purchase testing equipment that exports a large number of PDF test reports through templates after completing the test task, but do not provide data integration interfaces, which brings inconvenience to future data storage and analysis. Faced with a large number of PDFs to be processed, manual review and input is obviously not the best choice. With powerful processing capabilities, computers replace manual data entry to improve data processing efficiency and reduce costs.

[0004] For example, the technical solution of the patent application number "CN202211208393.6", the patent name "Large-scale multi-format file information extraction method and storage medium", and the patent applicant "Fujian Zhengfu Software Co., Ltd." is as follows: obtain the files in the large-scale multi-format file collection one by one; if the obtained file is a pdf file or a ppt file, determine its form and content; if the pdf file or ppt file is pure text information, extract the text information therein; if the pdf file or ppt file is a scanned copy or a picture, first process the picture and recognize the picture, and then extract the information therein through OCR recognition. For this solution, we only focus on extracting the information of the pdf file, and do not care how the extracted information is output or whether data integration is performed.

[0005] For example, the technical solution with application number "CN202111113749.3", patent name "A method and device for extracting structured information of PDF files", and patent applicant "Chengdu Sifang Weiye Software Co., Ltd." discloses the following contents: S1. Filter out editable PDF documents and read editable PDF documents; S2. Extract the text content of the editable PDF document in step S1, and then segment the text content to obtain a string group; S3. Traverse the string group of step S2, add prefixes to form a discriminant index; S4. According to the discriminant index of step S3, extract its structured information; S5. Write the structured information conversion format of step S4 into the database. This technical solution can process PDF files in batches and form a complete conversion method from unstructured data to structured data in PDF documents. This technical solution has the following disadvantages: 1. This technical solution has high skill requirements for users, requiring them to have certain knowledge of character matching principles, especially the use of regular expressions. This may be a learning curve for ordinary users; 2. Before use, users must summarize and generalize the types of PDF documents to be processed to create character matching templates to achieve automatic information extraction. This process may be relatively cumbersome, increasing the complexity of use; 3. The current technical solution may have problems when processing complex tables, such as when encountering cross-column and cross-row situations, the accuracy of information extraction may be affected; 4. This technical solution is built on the recognition principle of PDF code, but in some cases, there may be differences between the content inside the PDF code and the actual PDF display content, which will affect the accuracy of information extraction.

[0006] The PDF test reports of biochemical environmental materials (such as the PDF test reports provided by instruments from manufacturers such as Thermo Fisher and Agilent) have their own particularities:

[0007] 1. The PDF test report of biochemical and environmental materials is mainly native PDF. PDF is divided into native PDF and scanned PDF. Native PDF is also called editable PDF, which is usually exported through word documents and excel documents and made by PDF software. It can be modified and edited. Scanned PDF is generated from a picture through a scanner. It is generally not modifiable and is a non-editable PDF. It is mainly for the purpose of users being able to open and view the content.

[0008] 2. There is a lot of data that needs to be statistically analyzed in the PDF test reports in the field of biochemical and environmental materials, and the demand for extracting, reusing and re-editing PDF table information is increasing.

[0009] 3. Most importantly, the internal data of PDF test reports in the field of biochemical and environmental materials are roughly divided into key-value data and table data. Key-value data generally refers to the data called key and value in the computer field formed by separators like ":". It is the realization of the mapping relationship in mathematical concepts. "Key" is used as the index of the element, and "value" represents the data that needs to be read and stored. The key-value pair itself does not have too many complex structures, and it is relatively simple to extract using string matching. A table is a region with regular text blocks or regularly divided by line segments. There are certain associations and constraints between text blocks. However, in the PDF standard, there is no concept of table and rigorous logical declaration. PDF only arranges the position of text blocks, usually using graphic segments to complete the cell division work, so as to visually assist in forming a table. From the perspective of encoding characteristics, PDF tables do not have structural features similar to HTML tables, that is, it is impossible to directly perceive the table from the encoding and obtain the two-dimensional logical structure of the table. Therefore, the task of extracting data in this aspect will be much more complicated. Unstructured digital documents such as table data in PDF, due to the complexity and diversity of their structure and format, bring difficulties to parsing into machine-readable structured formats, which poses new challenges to the recovery and extraction of PDF tables.

[0010] 4. PDF test reports in the field of biochemical and environmental materials cannot be simply regarded as unstructured data. They include certain structured data and a large amount of unstructured data. The data organization form is not a strict table, field, etc., but through tags, attributes or encapsulated elements to express the relationship and organization between elements. PDF test reports in the field of biochemical and environmental materials usually contain a large amount of key-value pairs and tables, which should be regarded as semi-structured data. Semi-structured data is a data form between structured data and unstructured data.

[0011] Compared with structured data, semi-structured data is indeed more difficult to process. This is mainly because semi-structured data has both the characteristics of structured data and elements of unstructured data, so its processing needs to consider the characteristics of both types of data at the same time. Specifically, semi-structured data often contains a large number of tags, attributes or encapsulated elements. The relationship and organization between these elements are often complex and do not always follow a fixed pattern. This makes the parsing, understanding and processing of semi-structured data more complicated than structured data. Summary of the invention

[0012] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and provide a method for extracting PDF test report data based on character feature template matching, which is particularly suitable for native PDF test reports containing semi-structured data in the field of biochemical and environmental materials. The method can extract the data of interest specified in the PDF test report, realize structured storage, efficiently and accurately grasp the relevant information of the test substance, and facilitate subsequent data analysis.

[0013] The purpose of the present invention is achieved through the following technical solutions:

[0014] The method for extracting PDF inspection report data based on character feature template matching includes the following steps:

[0015] S1. Convert the same type of native PDF detection report into character txt text;

[0016] S2, configuring a corresponding feature template and a table experience distribution configuration file for this type of PDF detection report, wherein the feature template includes a layout character feature template and a table character feature template;

[0017] S3, the character txt text corresponding to each native PDF detection report is matched with the peripheral layout character feature template line by line: if it does not match, no information extraction is performed; if it matches, the key-value pair data is extracted and the process goes to step S4;

[0018] S4. Locate the table header and table end positions of the native PDF detection report, return the upper and lower bounds of the table, and the txt content of the table body in the table, to pave the way for restoring the table using the embedded table character feature template;

[0019] S5. Determine whether the table type of the native PDF test report is a table with complete and regular table lines:

[0020] If so, use the detection and restoration method of table lines in PDF to restore all lines in PDF, guess the lines that can form a table, and form possible multiple tables; use the upper and lower boundaries of the table to locate the target table, all lines of the table intersect to form intersections, and every four adjacent intersections form a cell. According to the position of the cell, query the characters belonging to each cell from PDF, and finally restore the table to obtain a unique and reasonable solution for each row;

[0021] If not, jump to step S6;

[0022] S6. Continue to determine whether the table type is the type where the table body is aligned with the header:

[0023] If so, extract all character blocks in all PDFs, and each character string is accompanied by the position and size parameters of its external rectangular box; filter out the character block set belonging to the target table according to the upper and lower bounds of the table and sort them; cluster the character set into several rows according to the rows; traverse all character blocks in all character block sets, and use the position of the character block in the row and the position of the table header to make a horizontal projection intersection. If the intersection is greater than a certain threshold, it is determined that the character block belongs to the value of the table header; when all operations are completed, the two-dimensional logical structure of the table is restored, and a unique reasonable solution for each row is obtained;

[0024] If not, jump to step S7;

[0025] S7. Use the txt content of the table body to split the characters according to the spaces in each line of the table body: Assume that it is split into m parts. For the number of table headers, plus the maximum number of spaces that each header value can contain, the m values ​​are combined into n spaces in a relative order. All possible distributions are For each type of table value distribution, compare the data type of the table header with the value it is assigned to. If they all match, it is considered a reasonable solution. Process each row to restore the table's logical structure.

[0026] The relationship between the txt content of the table body and the table experience distribution configuration file is:

[0027] When restoring the rows of a table, first split the row corresponding to the table body txt content into cells according to the spaces. Suppose it is divided into m cells, and then match these m cells with the type of each column of the row in the table character feature template. Suppose there are n columns, n is generally greater than m, so there are species, but only a few species can match the type of each column, and then compare the distribution of these m matching cells on n columns with the distribution recorded in the table's empirical distribution configuration file. If this distribution exists in the distribution recorded in the table's empirical distribution configuration file, then the accuracy index returned by the algorithm will be higher, otherwise it will be lowered to a certain extent.

[0028] S8. For each row of reasonable solutions, first set the initial accuracy according to the matching with the corresponding header type and the table experience distribution configuration file, and finally update the row accuracy according to the formula; return the reasonable solution with the highest accuracy among all reasonable solutions for each row and its accuracy, and the remaining solutions are also output as alternative suboptimal solutions.

[0029] The layout character feature template is generated by converting the positioning information of the key-value pair, the data type that is satisfied, and the positioning of the table into a form that can be interpreted by a computer through encoding; the layout character feature template describes the page header and footer, the restoration method used by the table, the pattern that the extracted character string should meet, and other strict positioning information.

[0030] The layout character feature template includes the key in the key-value pair, the corresponding pattern satisfied by the value, and the positioning anchor point, and the positioning anchor point is used to locate various information; the positioning anchor point includes separators, headers, and table positioning information; the layout character feature template is an abstract expression of the PDF layout, and the arrangement of the features matches the layout method in the PDF.

[0031] The parameters of the table character feature template include table headers, the value type corresponding to each table header, and the maximum number of spaces that the value corresponding to each table header can contain.

[0032] In step S8, the initial accuracy is set as follows: if the type match is successful and belongs to one of the previous empirical distributions, the initial accuracy is set to 100%; if the type match is successful but does not belong to one of the previous empirical distributions, the initial accuracy is X, 0%<X<100%; in other cases, the initial accuracy is 0%.

[0033] In step S8, the updated row accuracy is new acc , the calculation formula is as follows:

[0034]

[0035] Among them, num is the number of multiple reasonable solutions for a row, old acc is the initial accuracy of the row, old accs Represents the initial accuracy vector of all optimal solutions in this row.

[0036] When the table type of the native PDF detection report is a table with complete and regular table lines, the extraction method is as follows:

[0037] Let the original PDF page be P, and convert it into an image, represented as:

[0038] I = f(P);

[0039] Where I represents the transformed image; for the set of several horizontal lines L that may exist in the predicted PDF h and some vertical line set L v ,in:

[0040]

[0041] The detection process is described as:

[0042] g(I)={L h , L v};

[0043] According to the detected lines, the horizontal and vertical lines intersect with each other to form intersections, and the intersections form a text block set B of the table:

[0044] B=h(L h , L v );

[0045] The text block collection determines the area of ​​each row of the table through clustering and other methods Column Area

[0046]

[0047] Fill the cell contents according to the row and column areas and the text in the image to restore the original structure of the table:

[0048]

[0049] Where T represents the restored table, and the entire table restoration process can be represented by a composite function:

[0050] T = m(f(P), k(h(g(f(P))));

[0051] This algorithm locates the beginning and end of the table by matching the layout character feature template. After restoring the table parts scattered on each page through the table restoration algorithm, it merges all the partial tables located at the beginning and end of the table to adapt to the merging of paged tables.

[0052] The table type belongs to the table body alignment table header, and its extraction method is as follows:

[0053] Extract the word fragments formed after word segmentation in the pdfplumber library i , the word fragment set formed is textblock; where:

[0054] textblock={tb1,tb2,...,tb n};

[0055] And return the bounding rectangle parameters of each word fragment together, indicating the position of the word fragment in the PDF page; the word fragment tb i The bounding rectangle parameters include the horizontal coordinate of the upper left corner of the bounding rectangle Vertical axis And the horizontal coordinate of the lower right corner of the bounding rectangle Vertical axis

[0056] For two adjacent word fragments tb i and tb i+1 ,if:

[0057] and

[0058] Then tb i and tb i+1 Combine into a complete word:

[0059] w=merge(tb i ,tb i+1 , t w);

[0060] in, For word fragment tb i+1 The horizontal coordinate of the upper left corner of the bounding rectangle, For word fragment tb i+1 The vertical coordinate of the upper left corner of the circumscribed rectangle; t w is the parameter obtained through mathematical statistical analysis;

[0061] Select multiple samples to find the mean μ and standard deviation σ, and calculate t w :

[0062] t w =μ+4σ;

[0063] If the horizontal distance between adjacent words is small, less than t g (t g The same calculation method of mean and standard deviation is used), and spaces between two adjacent words are considered to be the same phrase:

[0064] ph = merge(w1, w2, t g );

[0065] The word blocks are clustered by position row to complete the division of table columns:

[0066] {r1, r2, ..., r l}=cluster(ph1,ph2,...,ph k );

[0067] Among them, r represents the column of the table, l is the number of clustering result classes, that is, the number of table rows, and k is the total number of phrases; for two phrases ph i and ph i+1 , the intersection interval S of its horizontal projection interval can be calculated as follows:

[0068]

[0069] in, It is a word fragment tb i+1 The horizontal coordinate of the lower right corner of the circumscribed rectangle;

[0070] If S satisfies one of the following three conditions:

[0071] S>T P ;

[0072]

[0073] It is considered that the two phrases should belong to the same column of the table, where T Pis a constant threshold value, which is an empirical parameter. Dividing the columns of the table by this operation can be regarded as a disguised clustering process:

[0074] {c1,c2,...,c0}=cluster(ph1,ph2,...,ph k );

[0075] Get the structured table T:

[0076] T={(r1,c1,v 1,1 )(r1,c2,v 1,2 ),...,(r l , c o ,v l，o )}

[0077] =structural({r1, r2, .., r l}, {c1,c2,..,c o});

[0078] Among them, C represents the column of the table, o is the number of clustering result classes, that is, the number of table columns; v is the cell of the table, and its two subscripts represent the row and column of the cell respectively.

[0079] The table type does not belong to the table body alignment table header, and its extraction method is as follows:

[0080] By matching the layout character feature template, we can get the rough text rows [S1, S2, ..., S k ], firstly, each row S i Split into columns For the table header row [H1, H2, ..., H n ], observe multiple tables of this PDF, and abstractly define a data type Ty1, Ty2, ..., Ty based on semantic experience. n , in order Assign to the table header n columns, generally m ≤ n, according to the principles of combinatorial mathematics kind of distribution;

[0081] A certain allocation method M = {(j, q) | j ≤ m, q ≤ n, f (j) = q} represents Assigned to the qth header, check its type With the assigned header type Ty q Are they consistent? If they are consistent, they will get points, otherwise they will not get points:

[0082]

[0083] For this allocation, the total score is:

[0084]

[0085] in, The jth column into which the i-th row of bold text is split;

[0086] A score of a combined allocation method score=m indicates a complete match of the table header type. This matching method can be returned as a restored table result.

[0087] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0088] 1. The present invention can effectively extract key-value pairs and table data from semi-structured or unstructured PDF documents to achieve accurate data acquisition.

[0089] 2. For multiple test report PDFs under the same template, the present invention has the function of quickly extracting data in batches, which greatly improves the data processing efficiency.

[0090] 3. Whether it is a single or batch PDF test report, the present invention can accurately extract the information therein to meet the needs of various usage scenarios.

[0091] 4. For PDF test reports with special requirements, the present invention also provides the function of customizing special templates to extract the required data more accurately.

[0092] 5. In order to help users understand the extraction effect, the present invention also provides feedback data so that users can have a preliminary understanding of the possible extraction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 A PDF sample image.

[0094] Figure 2 It is a txt text image.

[0095] Figure 3 It is the layout character feature template diagram.

[0096] Figure 4 It is a template diagram of table character features.

[0097] Figure 5 This is an empirical distribution diagram.

[0098] Figure 6 The present invention is a flowchart of the method for extracting PDF detection report data based on character feature template matching.

[0099] Figure 7 This is a PDF sample image of a table with complete and neat table lines.

[0100] Figure 8 This is a PDF sample image of a table with body-aligned header type.

[0101] Fig. 9 This is a PDF sample image of a table type that does not align the header with the body. DETAILED DESCRIPTION

[0102] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0103] like Figure 6 , a method for extracting PDF detection report data based on character feature template matching, comprising the following steps:

[0104] Step 1: Collect a large number of various PDF test report samples and initially remove scanned PDF files;

[0105] Step 2: To avoid the impact of Chinese and English characters on the configuration template, first convert all PDF text information in the class into character txt text files using the pdf plumber third-party library parsing tool for subsequent matching and extraction, such as Figure 2 Shown Figure 1 The corresponding txt text image.

[0106] Step 3. For a certain type of PDF, although the data filling is different and the relative position of the data is offset, the consistency of the output template can ensure that each report has the same structure and information display method, ensuring the stability of the PDF; observe the information arrangement and data type mode in multiple PDFs within the class, summarize the abstract data type and layout method, and combine the txt file corresponding to the PDF to complete the feature template required by the configuration algorithm for each PDF class. The feature template includes layout character feature template and table character feature template, as shown below: Figure 3 , Figure 4 shown.

[0107] The layout character feature template includes the key in the key-value pair, the corresponding pattern satisfied by the value, and the positioning anchor. The positioning anchor is used to locate various information, including separators, headers, table positioning information, etc. The layout feature template is an abstract expression of the PDF layout, and the arrangement of features matches the layout in the PDF.

[0108] The table character feature template parameters include the table header, the value type corresponding to each table header, and the maximum number of spaces that each table header value can contain, such as Figure 4 As shown;

[0109] Step 4: Observe the pattern of missing values ​​in the target table rows and form a table experience distribution profile, such as Figure 5As shown in the figure, each row indicates the possible distribution of a row of data in the table in which columns, 1 indicates that the column has a value, and 0 indicates that the column has no value; the empirical distribution configuration file needs to record all possible distributions of the table in order to calculate the accuracy;

[0110] Step 5, using the txt file corresponding to each PDF and the peripheral layout character feature template, match the characters line by line. If a PDF is found to not meet the set peripheral layout character feature template during the line-by-line matching, exit the program directly without any extraction; otherwise, extract the key information key-value pair, and check whether the value meets the specified pattern. If it meets the pattern, return the value result and 100% accuracy, otherwise return the value result and 0% accuracy. The accuracy is used to alert the user that the extracted result may not meet the previous rules;

[0111] Step 6: Match the outer layout character feature template, locate the table header and the table end, return the table upper and lower bounds beg_loc and end_loc, and the txt content belonging to the table body, to pave the way for the embedded table character feature template to restore the table;

[0112] Step 7: Determine whether the table type is a table with complete and regular lines:

[0113] If yes, then use the detection and restoration method of table lines in PDF to restore all lines in PDF, guess the lines that can form a table, and form possible multiple tables; use the upper and lower bounds of the table beg_loc and end_loc to locate the target table, all lines of the table intersect to form intersections, and every four adjacent intersections form a cell. According to the position of the cell, query the characters belonging to each cell from PDF, and finally restore the table to obtain a unique and reasonable solution for each row, and jump to step 10;

[0114] like Figure 7 When the table type of the native PDF test report is a table with complete and regular table lines, the extraction method is as follows:

[0115] Let the original PDF page be P, and convert it into an image, represented as:

[0116] I = f(P);

[0117] Where I represents the transformed image; for the set of several horizontal lines L that may exist in the predicted PDF h and some vertical line set L v ,in:

[0118]

[0119] The detection process is described as:

[0120] g(I)={Lh , L v};

[0121] According to the detected lines, the horizontal and vertical lines intersect with each other to form intersections, and the intersections form a text block set B of the table:

[0122] B=h(L b , L v );

[0123] The text block collection determines the area of ​​each row of the table through clustering and other methods Column Area

[0124]

[0125] Fill the cell contents according to the row and column areas and the text in the image to restore the original structure of the table:

[0126]

[0127] Where T represents the restored table, and the entire table restoration process can be represented by a composite function:

[0128] TG=m(f(P),k(h(g(f(P)))));

[0129] This algorithm locates the beginning and end of the table by matching the layout character feature template. After restoring the table parts scattered on each page by the table restoration algorithm, it merges all the partial tables located at the beginning and end of the table to adapt to the merging of paged tables.

[0130] Otherwise jump to step 8.

[0131] Step 8: Determine whether the table type is a table body aligned with the table header type. If so, extract all character blocks in the PDF. Each character string is accompanied by the position and size parameters of its external rectangular box. Filter out the character block set belonging to the target table according to the table upper and lower boundary positions beg_loc and end_loc; use the sorting algorithm to sort the character set from top to bottom and from left to right;

[0132] Cluster the character set by row into several rows; traverse all character blocks in the character block set, use the position of the character block in the row and the position of the table header to make a horizontal projection intersection, and if the intersection is greater than a certain threshold, the character block is considered to belong to the value of the table header. When all operations are completed, the two-dimensional logical structure of the table is restored, and a unique reasonable solution is obtained for each row, and jump to step 10;

[0133] like Figure 8, the table type belongs to the table body alignment table header, and its extraction method is as follows:

[0134] The PDF Plumber library provides a method called extract_words that can parse and extract characters from PDF stream files. However, the extracted characters are not complete words or phrases that humans understand, but fragments of words formed after the words are split. i , the word fragment set formed is textblock; where:

[0135] textblock={tb1,tb2,…,tb n};

[0136] And return the bounding rectangle parameters of each word fragment together, indicating the position of the word fragment in the PDF page; the word fragment tb i The bounding rectangle parameters include the horizontal coordinate of the upper left corner of the bounding rectangle Vertical axis And the horizontal coordinate of the lower right corner of the bounding rectangle Vertical axis

[0137] For two adjacent word fragments tb i and tb i+1 ,if:

[0138] and

[0139] Then tb i and tb i+1 Combine into a complete word:

[0140] w=merge(tb i ,tb i+1 , t w );

[0141] in, For word fragment tb i+1 The horizontal coordinate of the upper left corner of the bounding rectangle, For word fragment tb i+1 The vertical coordinate of the upper left corner of the circumscribed rectangle; T W is the parameter obtained through mathematical statistical analysis;

[0142] Select multiple samples to find the mean μ and standard deviation σ, and calculate t w :

[0143] t w =μ+4σ;

[0144] If the horizontal distance between adjacent words is small, less than tg (t g The same calculation method of mean and standard deviation is used), and spaces between two adjacent words are considered to be the same phrase:

[0145] ph=merge(w1,w2,t g );

[0146] The word blocks are clustered by position row to complete the division of table columns:

[0147] {r1, r2, ..., r l}=cluster(ph1,ph2,...,ph k );

[0148] Among them, r represents the column of the table, l is the number of clustering result classes, that is, the number of table rows, and k is the total number of phrases; for two phrases ph i and ph i+1 , the intersection interval S of its horizontal projection interval can be calculated as follows:

[0149]

[0150] in, It is a word fragment tb i+1 The horizontal coordinate of the lower right corner of the circumscribed rectangle;

[0151] If S satisfies one of the following three conditions:

[0152] S>T P ;

[0153]

[0154] It is considered that the two phrases should belong to the same column of the table, where T P is a constant threshold value, which is an empirical parameter. Dividing the columns of the table by this operation can be regarded as a disguised clustering process:

[0155] {c1,c2,...,c O}=cluster(ph1,ph2,...,ph k ); get the structured table T:

[0156] T={(r1,c1,v 11，1 ),(r1,c2,v 1, 2),...,(r l ,c0,v L,o )}

[0157] =structural({r1, r2,...,r l},{c1,c2,....,c o});

[0158] Among them, C represents the column of the table, O is the number of clustering result classes, that is, the number of table columns; v is the cell of the table, and its two subscripts represent the row and column of the cell respectively.

[0159] Otherwise jump to step 9.

[0160] Step 9. Use the table body txt content extracted in step 4 to split the characters according to the spaces in each line of the table body. Assuming that it is split into m parts, the number of table headers plus the maximum number of spaces that can be contained in the value corresponding to each table header is n. The m values ​​are combined into n spaces in a relative order. All possible distributions are: For each type of table value distribution, compare the data type of the table header and the value assigned to it to see if they match. If they all match, it is considered a reasonable solution. Process each row to restore the table's logical structure. Jump to step 10.

[0161] like Fig. 9 The table type does not belong to the table body alignment table header, and its extraction method is as follows:

[0162] By matching the layout character feature template, we can get the rough text rows [S1, S2, ..., S k ], firstly, each row S i Split into columns For the table header row [H1, H2, ..., H n ], observe multiple tables of this PDF, and abstractly define a data type Ty1, Ty2, ..., Ty based on semantic experience. n , in order Assign to the table header n columns, generally m ≤ n, according to the principles of combinatorial mathematics kind of distribution;

[0163] A certain allocation method M = {(j, q) | j ≤ m, q ≤ n, f (j) = q} represents Assigned to the qth header, check its type With the assigned header type Ty q Are they consistent? If they are consistent, they will get points, otherwise they will not get points:

[0164]

[0165] For this allocation, the total score is:

[0166]

[0167] in, The jth column into which the i-th row of bold text is split;

[0168] A score of a combined allocation method score=m indicates a complete match of the table header type. This matching method can be returned as a restored table result.

[0169] Step 10: First, match each row of reasonable solutions with the corresponding header type. If the type matches successfully and belongs to one of the previous empirical distributions, the initial accuracy is set to 100%. If the type matches successfully but does not belong to one of the previous empirical distributions, the initial accuracy is 60%. In other cases, the initial accuracy is 0%.

[0170] Step 11: Since multiple solutions may be extracted, multiple solutions will affect the row accuracy. For a row, the number of multiple reasonable solutions obtained is num, and the initial row accuracy is old acc ,old accs represents the initial accuracy vector of all optimal solutions for this row, then the updated row accuracy is:

[0171]

[0172] Step 12: Return the most accurate reasonable solution and its accuracy among all reasonable solutions in each row, and output the remaining solutions as alternative suboptimal solutions.

[0173] At the same time, this embodiment provides:

[0174] A server comprises a processor and a memory, wherein the memory stores at least one program, and the program is loaded and executed by the processor to implement the above-mentioned method for extracting PDF detection report data based on character feature template matching.

[0175] A computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the above-mentioned method for extracting PDF detection report data based on character feature template matching.

[0176] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for extracting PDF test report data based on character feature template matching, characterized in that: The following steps are involved: S1. Convert the same type of native PDF detection report into character txt text; S2, configuring a corresponding feature template and a table experience distribution configuration file for this type of PDF detection report, wherein the feature template includes a layout character feature template and a table character feature template; S3, the character txt text corresponding to each native PDF detection report is matched with the peripheral layout character feature template line by line: if it does not match, no information extraction is performed; if it matches, the key-value pair data is extracted and the process goes to step S4; S4. Locate the table header and table end positions of the native PDF detection report, return the upper and lower bounds of the table, and the txt content of the table body in the table, to pave the way for restoring the table using the embedded table character feature template; S5. Determine whether the table type of the native PDF test report is a table with complete and regular table lines: If so, use the detection and restoration method of table lines in PDF to restore all lines in PDF, guess the lines that can form a table, and form possible multiple tables; use the upper and lower boundaries of the table to locate the target table, all lines of the table intersect to form intersections, and every four adjacent intersections form a cell. According to the position of the cell, query the characters belonging to each cell from PDF, and finally restore the table to obtain a unique and reasonable solution for each row; If not, jump to step S6; S6. Continue to determine whether the table type is the type where the table body is aligned with the header: If yes, then extract all character blocks in all PDFs, each character string is accompanied by the position and size parameters of its external rectangular box; filter out the character block set belonging to the target table according to the upper and lower bounds of the table and sort them; Cluster the character set by row into several rows; traverse all character blocks in the character block set, use the position of the character block in the row and the position of the header to make a horizontal projection intersection, and if the intersection is greater than a certain threshold, it is determined that the character block belongs to the value of the header; when all operations are completed, the two-dimensional logical structure of the table is restored, and a unique reasonable solution for each row is obtained; If not, jump to step S7; S7. Use the txt content of the table body to split the characters according to the spaces in each line of the table body: Assume that it is split into m parts. For the number of table headers, plus the maximum number of spaces that each header value can contain, the m values ​​are combined into n spaces in a relative order. All possible distributions are For each type of table value distribution, compare the data type of the table header with the value it is assigned to. If they all match, it is considered a reasonable solution. Process each row to restore the table's logical structure. S8. For each row of reasonable solutions, first set the initial accuracy according to the matching with the corresponding header type and the table experience distribution configuration file, and update the row accuracy; return the reasonable solution with the highest accuracy among all reasonable solutions for each row and its accuracy, and the remaining solutions are also output as alternative suboptimal solutions.

2. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: The layout character feature template is generated by converting the positioning information of the key-value pair, the data type that is satisfied, and the positioning of the table into a form that can be interpreted by a computer through encoding; the layout character feature template describes the page header and footer, the restoration method used by the table, the pattern that the extracted character string should meet, and other strict positioning information.

3. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: The layout character feature template includes the key in the key-value pair, the corresponding pattern satisfied by the value, and the positioning anchor point, and the positioning anchor point is used to locate various information; the positioning anchor point includes separators, headers, and table positioning information; the layout character feature template is an abstract expression of the PDF layout, and the arrangement of the features matches the layout method in the PDF.

4. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: The parameters of the table character feature template include table headers, the value type corresponding to each table header, and the maximum number of spaces that the value corresponding to each table header can contain.

5. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: In step S8, the initial accuracy is set as follows: if the type match is successful and belongs to one of the previous empirical distributions, the initial accuracy is set to 100%; if the type match is successful but does not belong to one of the previous empirical distributions, the initial accuracy is X, 0%<X<100%; in other cases, the initial accuracy is 0%.

6. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: In step S8, the updated row accuracy is new acc , the calculation formula is as follows: Among them, num is the number of multiple reasonable solutions for a row, old acc is the initial accuracy of the row, old accs Represents the initial accuracy vector of all optimal solutions in this row.

7. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: When the table type of the native PDF detection report is a table with complete and regular table lines, the extraction method is as follows: Let the original PDF page be P, and convert it into an image, represented as: I = f(P); Where I represents the transformed image; for the set of several horizontal lines L that may exist in the predicted PDF h and some vertical line set L v ,in: The detection process is described as: g(I)={L h ,L v }; According to the detected lines, the horizontal and vertical lines intersect with each other to form intersections, and the intersections form a text block set B of the table: B=h(L h ,L v ); The text block collection determines the area of ​​each row of the table through clustering and other methods Column Area Fill the cell contents according to the row and column areas and the text in the image to restore the original structure of the table: Where T represents the restored table, and the entire table restoration process is represented by a composite function: T = m(f(P), k(h(g(f(P))))); This algorithm locates the beginning and end of the table by matching the layout character feature template. After restoring the table parts scattered on each page through the table restoration algorithm, it merges all the partial tables located at the beginning and end of the table to adapt to the merging of paged tables.

8. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: The table type belongs to the table body alignment table header, and its extraction method is as follows: Extract the word fragments formed after word segmentation in the PDF plumber library i , the word fragment set formed is textblock; where: textblock={tb1,tb2,...,tb n }; And return the bounding rectangle parameters of each word fragment together, indicating the position of the word fragment in the PDF page; the word fragment tb i The bounding rectangle parameters include the horizontal coordinate of the upper left corner of the bounding rectangle Vertical axis And the horizontal coordinate of the lower right corner of the bounding rectangle Vertical axis For two adjacent word fragments tb i and tb i+1 ,if: and Then tb i and tb i+1 Merge into a complete word w: w=merge(tb i ,tb i+1 ,t w ); in, For word fragment tb i+1 The horizontal coordinate of the upper left corner of the bounding rectangle, For word fragment tb i+1 The vertical coordinate of the upper left corner of the circumscribed rectangle; t w is the parameter obtained through mathematical statistical analysis; Select multiple samples to find the mean μ and standard deviation σ, and calculate t w : t w =μ+4σ; If the horizontal distance between adjacent words is small, less than the parameter t g , considering that there is a space between two adjacent words, they are classified as the same phrase ph: ph=merge(w1,w2,t g ); The word blocks are clustered by position row to complete the division of table columns: {r1,r2,…r l }=cluster(ph1,ph2,…,ph k ); Among them, r represents the column of the table, l is the number of clustering result classes, that is, the number of table rows, and k is the total number of phrases; for two phrases ph i and ph i+1 , the intersection interval S of its horizontal projection interval can be calculated as follows: in, It is a word fragment tb i+1 The horizontal coordinate of the lower right corner of the circumscribed rectangle; If S satisfies one of the following three conditions: S>T P ; It is considered that the two phrases should belong to the same column of the table, where T P is a constant threshold value, which is an empirical parameter. Dividing the columns of the table by this operation can be regarded as a disguised clustering process: {c1,c2,...,c o }=cluster(ph1,ph2,...,ph k ); Get the structured table T: T={(r1,c1,v 1,1 ),(r1,c2,v 1,2 ),...,(r l ,c0,v l，0 )}=structural({r1,r2,...,r l },{c1,c2,...,c o }); Among them, c represents the column of the table, o is the number of clustering result classes, that is, the number of table columns; v is the cell of the table, and its two subscripts represent the row and column of the cell respectively.

9. The method for extracting PDF test report data based on character feature template matching according to claim 1, characterized in that: The table type does not belong to the table body alignment table header, and its extraction method is as follows: By matching the layout character feature template, we can get the rough text rows [S1, S2, ..., S k ], firstly, each row S i Split into columns For the table header row [H1, H2, ..., H n ], observe multiple tables of this PDF, and abstractly define a data type Ty1, Ty2, ..., Ty based on semantic experience. n , in order Assign to the table header n columns, generally m ≤ n, according to the principles of combinatorial mathematics kind of distribution; A certain allocation method M = {(j, q) | j ≤ m, q ≤ n, f (j) = q} represents Assigned to the qth header, check its type The assigned header type Ty q Are they consistent? If they are consistent, they will get points, otherwise they will not get points: For this allocation, the total score is: in, The jth column into which the i-th row of bold text is split; A score of a combined allocation method score=m indicates a complete match of the table header type. This matching method can be returned as a restored table result.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, which is loaded and executed by the processor to implement the method for extracting PDF detection report data based on character feature template matching as claimed in any one of claims 1 to 9.

Images