A method for in-depth analysis of PDF research reports

Abstract

The invention discloses a method for deeply analyzing a PDF research report, which relates to the field of data processing technology, and comprises the following steps: step S1. basic analysis, reading PDF to obtain basic elements; the basic elements include character elements, curve elements, straight line elements, graphic elements and rectangular elements; the spatial range of each basic element is described by a rectangular range, a coordinate axis is established, and the coordinate value of the rectangular range of each basic element is determined; step S2. basic layout analysis, clustering all character elements into text lines, and then finding the main text block through text line clustering; step S3. complex clustering analysis, accurately locating the table range and the curve chart range in the main text block; step S4. extracting the main text content, filtering out the text lines in the main text block that are located in the table range and the curve chart range, and then sorting the remaining text lines in the main text block to obtain the main text content.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for in-depth analysis of PDF research reports. Background Art

[0002] The existing technology for parsing research reports generally has the problem of inaccurate table recognition. Complex tables are not standardized and have no separators, making them extremely difficult to parse. Since graphs contain multiple parts such as graph descriptions, legends, and figures, parsing is complicated. Inaccurate recognition of tables and graphs directly leads to incorrect boundaries for extracting the main text content, which is disorganized.

[0003] The existing patent "Research report parsing method, device, equipment and storage medium based on target detection" (patent application number: 202110380917.9) introduces target detection and processes research reports by targets. The processing effect is significantly improved for research reports with clear target boundaries and obvious features; however, the layout of research reports is very flexible, and research reports with unclear target boundaries, insignificant target features and possible changes may be processed incorrectly. If unrecognizable targets appear, the model needs to be retrained, and the process is relatively cumbersome. Therefore, there is an urgent need for a method to automatically identify research report targets to improve the accuracy of parsing research reports and improve parsing efficiency. Summary of the invention

[0004] In view of the defects existing in the prior art, the purpose of the present invention is to provide a method for in-depth analysis of PDF research reports, which utilizes the stable characteristics of tables and graphs in the research reports to perform cluster analysis, automatically locate the range of tables and graphs, and the precise positioning of tables and graphs ensures the orderliness and accuracy of the main text content.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is: a method for in-depth analysis of PDF research reports, characterized in that it includes the following steps:

[0006] Step S1. Basic parsing, reading PDF to obtain basic elements; the basic elements include character elements, curve elements, straight line elements, graphic elements and rectangular elements; the spatial range of each basic element is described by a rectangular range, a coordinate axis is established, and the coordinate value (x0, x1, y0, y1) of the rectangular range of each basic element is determined, where x0 represents the minimum value of the horizontal coordinate, x1 represents the maximum value of the horizontal coordinate, y0 represents the minimum value of the vertical coordinate, and y1 represents the maximum value of the vertical coordinate;

[0007] Step S2. Basic layout analysis: cluster all character elements into text lines, and then find the main text block through text line clustering;

[0008] Step S3. Complex cluster analysis, accurately locating the table range and the curve chart range in the main text block;

[0009] Step S4. extract the main text content, filter out the text lines in the main text block that are within the table range and the curve chart range, and then sort the remaining text lines in the main text block to obtain the main text content.

[0010] Based on the above technical solution, in step S2, the process of clustering character elements into text lines is as follows:

[0011] Step S2101. Select two character elements for cluster analysis. After the clustering condition is met, a text line is formed, and the coordinate value of the rectangular range corresponding to the text line is obtained;

[0012] Step S2102. Continue to perform cluster analysis on the text line and other character elements, and continuously expand the scope of the text line;

[0013] Step S2103: After character element clustering is completed, the final range of the text line is determined.

[0014] Based on the above technical solution, in step S2, the process of clustering text lines into main text blocks is as follows:

[0015] Step S2201. Sort all text lines by width, and select the text line with the largest width as the reference text line of the main text block;

[0016] Step S2202. Perform cluster analysis on the reference text line and other text lines. When the clustering condition is met, a main text block is formed, and the coordinate values ​​of the rectangular range corresponding to the main text block are obtained;

[0017] Step S2203. The main text block continues to perform cluster analysis with other text lines, and the main text block range is continuously updated;

[0018] Step S2204: After the text line clustering is completed, the final range of the main text block is determined.

[0019] Based on the above technical solution, the specific process of the cluster analysis is as follows:

[0020] Step S2301. Calculate the horizontal distance d between the two rectangular ranges A and B x,A,B , then

[0021] d x,A,B =min (|x 0,A -x 1,B |,|x 1,A -x 0,B |) (1);

[0022] Among them, |x 0,A -x 1,B | represents the absolute value of the difference between x0 of A and x1 of B. 1,A -x 0,B| represents the absolute value of the difference between x1 of A and x0 of B; min represents the minimum operation, |x 0,A -x 1,B |with|x 1,A -x 0,B The minimum value of | is the horizontal distance between A and B;

[0023] Step S2302. Calculate the vertical distance d between the two rectangular ranges A and B y,A,B , then

[0024] d y,A,B =min (|y 0,A -y 1,B |,|y 1,A -y 0,B |) (2);

[0025] Among them, |y 0,A -y 1,B | represents the absolute value of the difference between y0 of A and y1 of B. 1,A -y 0,B | represents the absolute value of the difference between y1 of A and y0 of B; |y 0,A -y 1,B |with|y 1,A -y 0,B The minimum value of | is the longitudinal distance between A and B;

[0026] Step S2303: Determine whether the two rectangular areas A and B overlap vertically. y,A,B , then

[0027]

[0028] Among them, when y0 of A is greater than y1 of B, or y1 of A is less than y0 of B, A and B do not overlap longitudinally; in other cases, A and B overlap longitudinally;

[0029] Step S2304: Determine whether the two rectangular areas A and B overlap horizontally. x,A,B , then

[0030]

[0031] Among them, when x0 of A is greater than x1 of B, or x1 of A is less than x0 of B, A and B do not overlap horizontally; in other cases, A and B overlap horizontally;

[0032] Step S2305. Determine whether the two rectangular ranges A and B meet the clustering condition j1 A,B , then

[0033]

[0034] Among them, when A and B overlap horizontally and the vertical distance is less than the threshold t l Then cluster and arrange vertically in rows; when A and B overlap vertically and the horizontal distance is less than the threshold t l In other cases, they are clustered and arranged in rows horizontally; in other cases, they are not clustered;

[0035] Step S2306. If the two rectangular ranges A and B meet the clustering condition, the combined rectangular coordinates (x 0,A,B ,x 1,A,B ,y 0,A,B ,y 1,A,B ) is calculated as follows:

[0036] (x 0,A,B ,x 1,A,B ,y 0,A,B ,y 1,A,B )=(min(x 0,A ,x 0,B ),max(x 1,A ,x 1,B ),min(y 0,A ,y 0,B ),max(y 1,A ,y 1,B ))(6);

[0037] Among them, the combined x 0,A,B is the minimum value of x0 of A and x0 of B; the combined x 1,A,B is the maximum value of x1 of A and x1 of B; the combined y 0,A,B is the minimum value of y0 of A and y0 of B; the combined y 1,A,B It is the maximum value of y1 of A and y1 of B.

[0038] Based on the above technical solution, it is determined whether the two rectangular ranges A and B are clustered into a block jb A,B Calculated as follows:

[0039]

[0040] Among them, when A and B overlap horizontally but do not overlap vertically, they belong to the same block; in other cases, they do not belong to the same block.

[0041] Based on the above technical solution, in step S3, the process of locating the table range is:

[0042] Step S3101. Sort the text lines of the main text block, using y reverse order first and x positive order first;

[0043] Step S3102. Traverse the text lines of the main text block as table header cells;

[0044] Step S3103. Check whether the header cell belongs to the table range: if yes, continue the traversal of step S3102 and check the next header cell; if no, continue the next table check;

[0045] Step S3104. Referring to the header cell, all other text lines are subjected to overlap analysis: text lines that meet the horizontal overlap condition form cells in the same column, text lines that meet the vertical overlap condition form cells in the same row, and text lines that meet the horizontal non-overlap condition form horizontal non-overlap cells;

[0046] Step S3105. Check the number of cells in the same row and the same column: if both are greater than 1, proceed to the next step of table check; otherwise, the table condition is not met, and the traversal of step S3102 is continued to check the next header cell;

[0047] Step S3106. Start searching for rows in the table: initialize the number of rows in the table to 0; traverse the cells in the same column and use them as the first cell in the row;

[0048] Step S3107. Refer to the first cell of the row and check the number of cells that overlap with the first cell of the row in the y direction among the horizontally non-overlapping cells, which represents the number of columns of this row. If it is greater than 1, a new table row is found and the number of table rows is increased by 1.

[0049] Step S3108. After the traversal of step S3106 is completed, check the number of table rows. If it is greater than 1, the table is found; otherwise, continue the traversal of step S3102 and check the next header cell;

[0050] Step S3109. After the table is found, the table range is calculated and updated.

[0051] Based on the above technical solution, in step S3109, the table coordinates (x 0,t ,x 1,t ,y 0,t ,y 1,t ) is as follows:

[0052] (x 0,t ,x 1,t ,y 0,t ,y 1,t )=(min(x 0,A ,x 0,B ,x 0,C ,…),max(x 1,A ,x 1,B ,x 1,C ,…),min(y 0,A ,y 0,B ,y 0,C ,…),max(y 1,A ,y1,B ,y 1,C ,…))(8);

[0053] Among them, the x of the table 0,t is the minimum value of x0 of all cells; 1,t is the maximum value of x1 for all cells; y of the table 0,t is the minimum value of y0 of all cells; the y 1,t is the maximum value of y1 for all cells.

[0054] Based on the above technical solution, in step S3109, updating the table range is performed as follows: first, traverse all text lines of the main text block to check whether the text line belongs to the table range, and the text lines that do not belong to the table range constitute non-table text lines; then traverse the non-table text lines and aggregate the text lines near the table;

[0055] Whether the rectangular range A corresponding to the text line is aggregated with the rectangular range B corresponding to the table A,B , calculated as follows:

[0056]

[0057] Among them, t t Represents the distance threshold of text near the table; when A and B overlap horizontally and the vertical distance is less than the threshold t t In other cases, no aggregation;

[0058] After the aggregation conditions are met, the table coordinates are calculated and the table range is updated.

[0059] Based on the above technical solution, in step S3, the process of locating the range of the curve graph is as follows:

[0060] Step S3201. After the table search is completed, for all text lines of the main text block, determine whether they are in any table, filter out the text lines in the table range, and obtain non-table text lines; traverse the text lines;

[0061] Step S3202. Check all the straight line elements to see if they form straight line related texts with the text line, and calculate the combined rectangular range for the straight line related texts that meet the conditions; check all the rectangular elements to see if they form rectangular related texts with the text line, and calculate the combined rectangular range for the rectangular related texts that meet the conditions; after the traversal is completed, obtain the straight line related texts and the rectangle related texts;

[0062] Check whether the rectangular range A corresponding to the rectangular element and the rectangular range B corresponding to the text line are combined into the legend jtl A,B The formula is as follows:

[0063]

[0064] Among them, t lt Indicates the distance threshold of the legend; the legend is characterized by the rectangular element being on the left side of the text line, and the range of the text line in the y direction includes the range of the rectangle in the y direction; so y1 of A is less than or equal to y1 of B, y0 of A is greater than or equal to y0 of B, x1 of A is less than x0 of B, and the distance between the text line and the rectangular element is x. 0,B -x 1,A , the distance is less than the threshold t lt ;

[0065] After the combined legend conditions are met, the combined coordinates are calculated;

[0066] Check whether the rectangular range A corresponding to the straight line element and the rectangular range B corresponding to the text line are combined into a curve diagram. A,B The formula is as follows:

[0067]

[0068] Description: sm1 Represents the horizontal distance threshold of the curve diagram, t sm2 Indicates the vertical distance threshold of the curve description; the characteristics of the curve description are that the straight line elements are above or below the text line, and the horizontal coordinates of A and B have constraints; A's x0 is less than B's x1, A's x1 is greater than B's x1, and the absolute value of the difference between A's x0 and B's x0 is less than t sm1 , and the absolute value of the difference between y0 of A and y1 of B is less than t sm2 , the absolute value of the difference between y1 of A and y0 of B is less than t sm2 ,When these conditions are met, A and B are combined into a curve graph; in other cases, A and B are not aggregated;

[0069] After the conditions for the combined curve graph are met, the combined coordinates are calculated;

[0070] Step S3203. Traverse the curve graph, which includes two basic elements: curve elements and graph elements;

[0071] Step S3204. First, perform rectangle-related text aggregation: refer to the curve graph, perform aggregation analysis on all rectangle-related texts, and update the range of the curve graph; then perform line-related text aggregation: refer to the curve graph, perform aggregation analysis on all line-related texts, and update the range of the curve graph;

[0072] Whether the rectangular range A corresponding to the line-related text or the rectangle-related text is aggregated with the rectangular range B corresponding to the curve element A,B Calculated as follows:

[0073]

[0074] Among them, t qx Represents the distance threshold between the curve element and the text associated with the nearby straight line or rectangle; A and B meet the horizontal overlap, and the vertical distance is less than the threshold t qx Aggregate in the case of; do not aggregate in other cases;

[0075] After the aggregation conditions of the line-related text or rectangle-related text and the curve are met, the range of the merged curve graph is calculated;

[0076] Whether the rectangular range A corresponding to the line-related text or rectangle-related text is aggregated with the rectangular range B corresponding to the graphic element A,B Calculated as follows:

[0077]

[0078] Among them, t tu Represents the distance threshold between the graphic element and the text associated with the line or rectangle nearby; A and B meet the horizontal overlap, and the vertical distance is less than the threshold t tu Aggregate in the case of; do not aggregate in other cases;

[0079] After the aggregation conditions of the line-related text or rectangle-related text and graph are met, the range of the merged curve graph is calculated;

[0080] Step S3205. Final graph and text aggregation: refer to the graph, perform aggregation analysis on all non-table text lines, and finally determine the range of the graph;

[0081] Whether the rectangular range A corresponding to the text line is aggregated with the rectangular range B corresponding to the curve graph A,B Calculated as follows:

[0082]

[0083] Among them, t w Represents the distance threshold of the text near the curve graph; A and B meet the horizontal overlap, and the vertical distance is less than the threshold t w Aggregate in the case of; do not aggregate in other cases;

[0084] After the aggregation conditions of the text line and the curve chart are met, the merged curve chart range is calculated.

[0085] On the basis of the above technical solution, in step S4, it is determined whether the rectangular range A corresponding to the text line is within the rectangular range B corresponding to the table or the curve graph. A,B The formula is as follows:

[0086]

[0087] Among them, when A and B overlap horizontally and vertically, A is within the area of ​​B; in other cases, A is not within the area of ​​B.

[0088] The beneficial effects of the present invention are:

[0089] The present invention utilizes the stable characteristics of tables and graphs in research reports to perform cluster analysis, automatically locates the range of tables and graphs, and the precise positioning of tables and graphs ensures the orderliness and accuracy of the main text content. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 A flowchart of the steps of the method for in-depth analysis of PDF research reports in the present invention;

[0091] Figure 2 It is the coordinate diagram of the basic elements in the present invention;

[0092] Figure 3 It is a flowchart for searching a table in the present invention;

[0093] Figure 4 A flow chart for searching a curve graph in the present invention;

[0094] Figure 5 This is an example diagram of a table search in the present invention;

[0095] Figure 6 An example diagram of a search curve diagram in the present invention;

[0096] Figure 7 This is a finished picture of the main text content after filtering the table in the present invention;

[0097] Figure 8 This is the finished image of the main text content after filtering the curve graph in the present invention. DETAILED DESCRIPTION

[0098] The following is a further description of the specific embodiments of the present invention in conjunction with the drawings of the specification, so that the technical solution and beneficial effects of the present invention are clearer and more explicit. The following description of the embodiments with reference to the drawings is exemplary and intended to explain the present invention, but cannot be understood as limiting the present invention.

[0099] See also Figure 1 As shown, an embodiment of the present invention provides a method for in-depth analysis of a PDF research report, comprising the following steps:

[0100] Step S1. See Figure 2As shown, basic parsing is performed by reading PDF to obtain basic elements; the basic elements include character elements, curve elements, straight line elements, graphic elements and rectangular elements; the spatial range of each basic element is described by a rectangular range, a coordinate axis is established, and the coordinate value (x0, x1, y0, y1) of the rectangular range of each basic element is determined, where x0 represents the minimum value of the horizontal coordinate, x1 represents the maximum value of the horizontal coordinate, y0 represents the minimum value of the vertical coordinate, and y1 represents the maximum value of the vertical coordinate;

[0101] Step S2. Basic layout analysis, clustering all character elements into text lines, and then finding the main text block through text line clustering to prepare for the search of tables and graphs; character clustering has two directions, horizontal rows and vertical columns, collectively referred to as text lines.

[0102] Step S3. Complex cluster analysis, accurately locating the table range and the curve chart range in the main text block;

[0103] Step S4. extract the main text content, filter out the text lines in the main text block that are within the table range and the curve chart range, and then sort the remaining text lines in the main text block to obtain the main text content.

[0104] Specifically, in step S2, the process of clustering character elements into text lines is as follows:

[0105] Step S2101. Select two character elements for cluster analysis. After the clustering condition is met, a text line is formed, and the coordinate value of the rectangular range corresponding to the text line is obtained;

[0106] Step S2102. Continue to perform cluster analysis on the text line and other character elements, and continuously expand the scope of the text line;

[0107] Step S2103: After character element clustering is completed, the final range of the text line is determined.

[0108] Specifically, in step S2, the process of clustering text lines into main text blocks is as follows:

[0109] Step S2201. Sort all text lines by width, and select the text line with the largest width as the reference text line of the main text block;

[0110] Step S2202. Perform cluster analysis on the reference text line and other text lines. When the clustering condition is met, a main text block is formed, and the coordinate values ​​of the rectangular range corresponding to the main text block are obtained;

[0111] Step S2203. The main text block continues to perform cluster analysis with other text lines, and the main text block range is continuously updated;

[0112] Step S2204: After the text line clustering is completed, the final range of the main text block is determined.

[0113] Specifically, the specific process of the cluster analysis is as follows:

[0114] Step S2301. Calculate the horizontal distance d between the two rectangular ranges A and B x,A,B , then

[0115] d x,A,B =min (|x 0,A -x 1,B |,|x 1,A -x 0,B |) (1);

[0116] Among them, |x 0,A -x 1,B | represents the absolute value of the difference between x0 of A and x1 of B. 1,A -x 0,B | represents the absolute value of the difference between x1 of A and x0 of B; min represents the minimum operation, |x 0,A -x 1,B |with|x 1,A -x 0,B The minimum value of | is the horizontal distance between A and B;

[0117] Step S2302. Calculate the vertical distance d between the two rectangular ranges A and B y,A,B , then

[0118] d y,A,B =min (|y 0,A -y 1,B |,|y 1,A -y 0,B |) (2);

[0119] Among them, |y 0,A -y 1,B | represents the absolute value of the difference between y0 of A and y1 of B. 1,A -y 0,B | represents the absolute value of the difference between y1 of A and y0 of B; |y 0,A -y 1,B |with|y 1,A -y 0,B The minimum value of | is the longitudinal distance between A and B;

[0120] Step S2303: Determine whether the two rectangular areas A and B overlap vertically. y,A,B , then

[0121]

[0122] Among them, when y0 of A is greater than y1 of B, or y1 of A is less than y0 of B, A and B do not overlap longitudinally; in other cases, A and B overlap longitudinally;

[0123] Step S2304: Determine whether the two rectangular areas A and B overlap horizontally. x,A,B , then

[0124]

[0125] Among them, when x0 of A is greater than x1 of B, or x1 of A is less than x0 of B, A and B do not overlap horizontally; in other cases, A and B overlap horizontally;

[0126] Step S2305. Determine whether the two rectangular ranges A and B meet the clustering condition j1 A,B , then

[0127]

[0128] Among them, when A and B overlap horizontally and the vertical distance is less than the threshold t l Then cluster and arrange vertically in rows; when A and B overlap vertically and the horizontal distance is less than the threshold t l In other cases, they are clustered and arranged in rows horizontally; in other cases, they are not clustered;

[0129] Step S2306. If the two rectangular ranges A and B meet the clustering condition, the combined rectangular coordinates (x 0,A,B ,x 1,A,B ,y 0,A,B ,y 1,A,B ) is calculated as follows:

[0130] (x 0,A,B ,x 1,A,B ,y 0,A,B ,y 1,A,B )=(min(x 0,A ,x 0,B ),max(x 1,A ,x 1,B ),min(y 0,A ,y 0,B ),max(y 1,A ,y 1,B ))(6);

[0131] Among them, the combined x 0,A,B is the minimum value of x0 of A and x0 of B; the combined x 1,A,B is the maximum value of x1 of A and x1 of B; the combined y 0,A,B is the minimum value of y0 of A and y0 of B; the combined y 1,A,B It is the maximum value of y1 of A and y1 of B.

[0132] Specifically, determine whether the two rectangular ranges A and B are clustered into a block jb A,B Calculated as follows:

[0133]

[0134] When A and B overlap horizontally but not vertically, they belong to the same block; otherwise, they do not belong to the same block. Research reports usually contain two types of layouts: one with only one block; the other with a two-block structure, with the main text block occupying a larger width and the non-main text block occupying a smaller width.

[0135] See also Figure 3 As shown, in step S3, the process of locating the table range is:

[0136] Step S3101. Sort the text lines of the main text block, referring to the logical organization order of the table cells, from top to bottom and from left to right, so y reverse order is preferred and x positive order is adopted.

[0137] Step S3102. Traverse the text lines of the main text block as header cells: the header cell refers to the first cell of the table, generally in the upper left corner (smallest in the x direction and largest in the y direction).

[0138] Step S3103. Check whether the header cell belongs to the table range: if yes, continue the traversal of step S3102 and check the next header cell; if no, continue the next table check;

[0139] Step S3104. Refer to the header cell and perform overlap analysis on all other text rows: text rows that meet the horizontal (x direction) overlap condition form cells in the same column, text rows that meet the vertical (y direction) overlap condition form cells in the same row, and text rows that meet the horizontal non-overlap condition form horizontal non-overlap cells.

[0140] Step S3105. Check the number of cells in the same row and the same column: if both are greater than 1, proceed to the next step of table check; otherwise, the table condition is not met, and the traversal of step S3102 is continued to check the next header cell;

[0141] Step S3106. Start searching for rows in the table: the number of table rows is initialized to 0; traverse the cells in the same column and use the first cell as the first cell in the row. The first cell in the row refers to the first cell in the table row, usually on the far left (smallest in the x direction).

[0142] Step S3107. Refer to the first cell in the row and check the number of cells in the horizontally non-overlapping cells that overlap (in the same row) with the first cell in the y direction, which represents the number of columns in this row. If it is greater than 1 (usually the number of columns in a table row is greater than 1), find a new table row and increase the number of table rows by 1.

[0143] Step S3108. The traversal of step S3106 is completed, and the number of table rows is checked. If it is greater than 1 (usually the number of table rows is greater than 1), the table is found; otherwise, the traversal of step 2 is continued to check the next header cell.

[0144] Step S3109. After the table is found, the table range is calculated and updated.

[0145] Specifically, in step S3109, the table coordinates (x 0,t ,x 1,t ,y 0,t ,y 1,t ) is as follows:

[0146] (x 0,t ,x 1,t ,y 0,t ,y 1,t )=(min(x 0,A ,x 0,B ,x 0,C ,…),max(x 1,A ,x 1,B ,x 1,C ,…),min(y 0,A ,y 0,B ,y 0,C ,…),max(y 1,A ,y 1,B ,y 1,C ,…))(8);

[0147] Among them, the x of the table 0,t is the minimum value of x0 of all cells; 1,t is the maximum value of x1 for all cells; y of the table 0,t is the minimum value of y0 of all cells; the y 1,t is the maximum value of y1 for all cells.

[0148] Specifically, in step S3109, updating the table range is performed as follows: first, all text lines of the main text block are traversed to check whether the text lines belong to the table range, and the text lines that do not belong to the table range constitute non-table text lines; then, the non-table text lines are traversed to aggregate text lines near the table;

[0149] Whether the rectangular range A corresponding to the text line is aggregated with the rectangular range B corresponding to the table A,B , calculated as follows:

[0150]

[0151] Among them, t tRepresents the distance threshold of text near the table; when A and B overlap horizontally and the vertical distance is less than the threshold t t In other cases, no aggregation;

[0152] After the aggregation conditions are met, the table coordinates are calculated according to the formula for the combined rectangle, and the table range is updated.

[0153] See also Figure 4 As shown, in step S3, the process of locating the range of the curve graph is:

[0154] Step S3201. After the table search is completed, for all text lines of the main text block, determine whether they are in any table, filter out the text lines in the table range, and obtain non-table text lines; traverse the text lines;

[0155] Step S3202. Check all the straight line elements to see if they form straight line related texts with the text line, and calculate the combined rectangular range for the straight line related texts that meet the conditions; check all the rectangular elements to see if they form rectangular related texts with the text line, and calculate the combined rectangular range for the rectangular related texts that meet the conditions; after the traversal is completed, obtain the straight line related texts and the rectangle related texts;

[0156] The main features to look for in a curve graph are line-related text and rectangle-related text:

[0157] 1) The legend of the curve graph consists of a rectangle and a text line.

[0158] 2) The description of the curve graph includes a straight line and a text line.

[0159] Check whether the rectangular range A corresponding to the rectangular element and the rectangular range B corresponding to the text line are combined into the legend jtl A,B The formula is as follows:

[0160]

[0161] Among them, t lt Indicates the distance threshold of the legend; the legend is characterized by the rectangular element being on the left side of the text line, and the range of the text line in the y direction includes the range of the rectangle in the y direction; so y1 of A is less than or equal to y1 of B, y0 of A is greater than or equal to y0 of B, x1 of A is less than x0 of B, and the distance between the text line and the rectangular element is x. 0,B -x 1,A , the distance is less than the threshold t lt ;

[0162] When the combined legend conditions are met, the combined coordinates are calculated according to the calculation formula of the combined rectangle.

[0163] Check whether the rectangular range A corresponding to the straight line element and the rectangular range B corresponding to the text line are combined into a curve diagram.A,B The formula is as follows:

[0164]

[0165] Description: sm1 Represents the horizontal distance threshold of the curve diagram, t sm2 Indicates the vertical distance threshold of the curve description; the characteristics of the curve description are that the straight line elements are above or below the text line, and the horizontal coordinates of A and B have constraints; A's x0 is less than B's x1, A's x1 is greater than B's x1, and the absolute value of the difference between A's x0 and B's x0 is less than t sm1 , and the absolute value of the difference between y0 of A and y1 of B is less than t sm2 , the absolute value of the difference between y1 of A and y0 of B is less than t sm2 ,When these conditions are met, A and B are combined into a curve graph; in other cases, A and B are not aggregated;

[0166] After the conditions for the combined curve diagram are met, the combined coordinates are calculated according to the calculation formula of the combined rectangle.

[0167] Step S3203. Traverse the curve graph, which includes two basic elements: curve elements and graph elements;

[0168] Step S3204. First, perform rectangle-related text aggregation: refer to the curve graph, perform aggregation analysis on all rectangle-related texts, and update the range of the curve graph; then perform line-related text aggregation: refer to the curve graph, perform aggregation analysis on all line-related texts, and update the range of the curve graph;

[0169] Whether the rectangular range A corresponding to the line-related text or the rectangle-related text is aggregated with the rectangular range B corresponding to the curve element A,B Calculated as follows:

[0170]

[0171] Among them, t qx Represents the distance threshold between the curve element and the text associated with the nearby straight line or rectangle; A and B meet the horizontal overlap, and the vertical distance is less than the threshold t qx Aggregate in the case of; do not aggregate in other cases;

[0172] When the aggregation conditions of the line-related text or rectangle-related text and the curve are met, the range of the merged curve graph is calculated according to the calculation formula of the combined rectangle coordinates.

[0173] Whether the rectangular range A corresponding to the line-related text or rectangle-related text is aggregated with the rectangular range B corresponding to the graphic element A,B Calculated as follows:

[0174]

[0175] Among them, t tu Represents the distance threshold between the graphic element and the text associated with the line or rectangle nearby; A and B meet the horizontal overlap, and the vertical distance is less than the threshold t tu Aggregate in the case of; do not aggregate in other cases;

[0176] When the aggregation conditions of the line-related text or rectangle-related text and graph are met, the range of the merged curve graph is calculated according to the calculation formula of the combined rectangle coordinates.

[0177] Step S3205. Final graph and text aggregation: refer to the graph, perform aggregation analysis on all non-table text lines, and finally determine the range of the graph;

[0178] Whether the rectangular range A corresponding to the text line is aggregated with the rectangular range B corresponding to the curve graph A,B Calculated as follows:

[0179]

[0180] Among them, t w Represents the distance threshold of the text near the curve graph; A and B meet the horizontal overlap, and the vertical distance is less than the threshold t w Aggregate in the case of; do not aggregate in other cases;

[0181] After the aggregation conditions of the text line and the curve graph are met, the range of the merged curve graph is calculated according to the calculation formula of the combined rectangle coordinates.

[0182] Specifically, in step S4, it is determined whether the rectangular range A corresponding to the text line is within the rectangular range B corresponding to the table or the curve graph. A,B The formula is as follows:

[0183]

[0184] Among them, when A and B overlap horizontally and vertically, A is within the area of ​​B; in other cases, A is not within the area of ​​B.

[0185] The main process of extracting the main text content is:

[0186] (1) For all text lines of the main text block, determine whether they are in any table or any graph, filter out the text lines in the table and graph range, and obtain the main text lines of the main text block.

[0187] (ii) Referring to the logical organization order of the text lines (from top to bottom), sort the main text lines of the main text block (in reverse y order) to obtain the main content of the research report in an orderly organized manner.

[0188] The illustrative examples of research report analysis are as follows:

[0189] 1. Character clustering into lines and text lines clustering into main text blocks.

[0190] As Figure 5 shown, the origin of coordinates is at the lower left corner of the layout.

[0191] (1) Character clustering: As shown in the text line "In the field of life sciences in China... the foundation in [year]", the characters in this line all meet the conditions of vertical overlap and horizontal distance less than the threshold, until the character "生" in the next line ends.

[0192] (2) Text line clustering: First, find the reference text line of the main text block. The text line "In the field of life sciences in China... the foundation in [year]" has the largest width and is determined as the reference text line. The text lines on the left side of the layout, including the short text lines of the table, all meet the text line clustering conditions with the reference text line, and the left side is the main text block.

[0193] 2. Search for the table:

[0194] (1) After sorting the text lines of the main text block, the order of the text lines is from top to bottom first, and then from left to right.

[0195] (2) Refer Figure 5 , when traversing the table header cells, starting from the text line "Novoprotein (******)" until the text line "Continue and centralized procurement policy... ;", no peer cells are found.

[0196] (3) When traversing to the cell "Financial data and valuation", the peer cells include "****" etc., the cells in the same column include "Total operating revenue (in millions of yuan)" etc., and the data part such as "&&&&" etc. and the peer cells all belong to horizontally non-overlapping cells. The number of peer and same-column cells is greater than 1. Determine the cell "Financial data and valuation" as the table header cell.

[0197] (4) Then search for the table rows: Traverse the cells in the same column, starting from the cell "Financial data and valuation" as the first cell in the row, and check the peer cells in the horizontally non-overlapping cells until the end of the last row of the table.

[0198] (5) Finally, update the table range and merge the text line "Data source..." into the table range.

[0199] 3. Search for the curve graph, as shown in "Figure 10" in Figure 6 :[[]]END]]

[0200] (1) Rectangle-related text: The text line of the legend "Market size (in billions of US dollars)" clusters with the rectangle on the left, and the text line "Growth rate (%)" clusters with the rectangle on the left.

[0201] (ii) Line-related text: The line of text “Figure 10: Global…” and the cluster of lines below it, and “Source…” and the cluster of lines above it.

[0202] (III) Rectangle-related text aggregation and line-related text aggregation: The curve graph is first aggregated with the rectangle-related text represented by the legend, and then aggregated with the line-related text represented by the curve graph caption.

[0203] (iv) Text aggregation: In this example, there is no more nearby text, and the range of the curve graph has been determined.

[0204] 4. Filter the range of tables and graphs to refine text content.

[0205] (a) If Figure 7 As shown, after filtering the table range and sorting, the main text content is orderly and accurate.

[0206] (ii) Figure 8 As shown, after filtering the curve chart range and sorting, the main text content is orderly and accurate.

[0207] In the description of the specification, the description with reference to the terms "one embodiment", "preferably", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention, and the schematic expressions of the above terms in this specification do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0208] The present invention is not limited to the above-mentioned embodiments. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention. The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

Claims

1. A method for in-depth analysis of PDF research reports, characterized in that: The following steps are involved: Step S1. Basic parsing, reading PDF to obtain basic elements; the basic elements include character elements, curve elements, straight line elements, graphic elements and rectangular elements; the spatial range of each basic element is described by a rectangular range, a coordinate axis is established, and the coordinate value (x0, x1, y0, y1) of the rectangular range of each basic element is determined, where x0 represents the minimum value of the horizontal coordinate, x1 represents the maximum value of the horizontal coordinate, y0 represents the minimum value of the vertical coordinate, and y1 represents the maximum value of the vertical coordinate; Step S2. Basic layout analysis: cluster all character elements into text lines, and then find the main text block through text line clustering; Step S3. Complex cluster analysis, accurately locating the table range and the curve chart range in the main text block; Step S4. extract the main text content, filter out the text lines in the main text block that are in the table range and the curve chart range, and sort the remaining text lines in the main text block to obtain the main text content; In step S3, the process of locating the table range is: Step S3101. Sort the text lines of the main text block, using y reverse order first and x positive order first; Step S3102. Traverse the text lines of the main text block as table header cells; Step S3103. Check whether the header cell belongs to the table range: if yes, continue the traversal of step S3102 and check the next header cell; if no, continue the next table check; Step S3104. Referring to the header cell, all other text lines are subjected to overlap analysis: text lines that meet the horizontal overlap condition form cells in the same column, text lines that meet the vertical overlap condition form cells in the same row, and text lines that meet the horizontal non-overlap condition form horizontal non-overlap cells; Step S3105. Check the number of cells in the same row and the same column: if both are greater than 1, proceed to the next step of table check; otherwise, the table condition is not met, and the traversal of step S3102 is continued to check the next header cell; Step S3106. Start searching for rows in the table: initialize the number of rows in the table to 0; traverse the cells in the same column and use them as the first cell in the row; Step S3107. Refer to the first cell of the row and check the number of cells that overlap with the first cell of the row in the y direction among the horizontally non-overlapping cells, which represents the number of columns of this row. If it is greater than 1, a new table row is found and the number of table rows is increased by 1. Step S3108. After the traversal of step S3106 is completed, check the number of table rows. If it is greater than 1, the table is found; otherwise, continue the traversal of step S3102 and check the next header cell; Step S3109. After the table is found, the table range is calculated and updated.

2. The method for in-depth analysis of PDF research reports as claimed in claim 1, characterized in that: In step S2, the process of clustering character elements into text lines is as follows: Step S2101. Select two character elements for cluster analysis. After the clustering condition is met, a text line is formed, and the coordinate value of the rectangular range corresponding to the text line is obtained; Step S2102. Continue to perform cluster analysis on the text line and other character elements, and continuously expand the scope of the text line; Step S2103: After character element clustering is completed, the final range of the text line is determined.

3. The method for in-depth analysis of PDF research reports as claimed in claim 2, characterized in that: In step S2, the process of clustering text lines into main text blocks is as follows: Step S2201. Sort all text lines by width, and select the text line with the largest width as the reference text line of the main text block; Step S2202. Perform cluster analysis on the reference text line and other text lines. When the clustering condition is met, a main text block is formed, and the coordinate values ​​of the rectangular range corresponding to the main text block are obtained; Step S2203. The main text block continues to perform cluster analysis with other text lines, and the main text block range is continuously updated; Step S2204: After the text line clustering is completed, the final range of the main text block is determined.

4. The method for deeply analyzing a PDF research report as claimed in claim 2 or 3, characterized in that: The specific process of the cluster analysis is as follows: Step S2301. Calculate the horizontal distance d between the two rectangular ranges A and B x,A,B , then d x,A,B =min (|x 0,A - x 1,B |,|x 1,A - x 0,B |) (1); Among them, |x 0,A -x 1,B | represents the absolute value of the difference between x0 of A and x1 of B. 1,A -x 0,B | represents the absolute value of the difference between x1 of A and x0 of B; min represents the minimum operation, |x 0,A -x 1,B |with|x 1,A -x 0,B The minimum value of | is the horizontal distance between A and B; Step S2302. Calculate the vertical distance d between the two rectangular ranges A and B y,A,B , then d y,A,B =min (|y 0,A - y 1,B |,|y 1,A - y 0,B |) (2); Among them, |y 0,A -y 1,B | represents the absolute value of the difference between y0 of A and y1 of B. 1,A -y 0,B | represents the absolute value of the difference between y1 of A and y0 of B; |y 0,A -y 1,B |with|y 1,A -y 0,B The minimum value of | is the longitudinal distance between A and B; Step S2303: Determine whether the two rectangular areas A and B overlap vertically. y,A,B , then Among them, when y0 of A is greater than y1 of B, or y1 of A is less than y0 of B, A and B do not overlap longitudinally; in other cases, A and B overlap longitudinally; Step S2304: Determine whether the two rectangular areas A and B overlap horizontally. x,A,B , then Among them, when x0 of A is greater than x1 of B, or x1 of A is less than x0 of B, A and B do not overlap horizontally; in other cases, A and B overlap horizontally; Step S2305. Determine whether the two rectangular ranges A and B meet the clustering condition j1 A,B , then Among them, when A and B overlap horizontally and the vertical distance is less than the threshold t l Then cluster and arrange vertically in rows; when A and B overlap vertically and the horizontal distance is less than the threshold t l In other cases, they are clustered and arranged in rows horizontally; in other cases, they are not clustered; Step S2306. If the two rectangular ranges A and B meet the clustering condition, the combined rectangular coordinates (x 0,A,B ,x 1,A,B ,y 0,A,B ,y 1,A,B ) is calculated as follows: (x 0,A,B ,x 1,A,B ,and 0,A,B ,and 1,A,B )=(min(x 0,A ,x 0,B ),max(x 1,A ,x 1,B ),min(and 0,A ,and 0,B ),max(and 1,A ,and 1,B ))(6); Among them, the combined x 0,A,B is the minimum value of x0 of A and x0 of B; the combined x 1,A,B is the maximum value of x1 of A and x1 of B; the combined y 0,A,B is the minimum value of y0 of A and y0 of B; the combined y 1,A,B It is the maximum value of y1 of A and y1 of B.

5. The method for in-depth analysis of PDF research reports as claimed in claim 4, characterized in that: Determine whether the two rectangular ranges A and B are clustered into blocks jb A,B Calculated as follows: Among them, when A and B overlap horizontally but do not overlap vertically, they belong to the same block; in other cases, they do not belong to the same block.

6. The method for in-depth analysis of PDF research reports as claimed in claim 1, characterized in that: In step S3109, the table coordinates (x 0,t ,x 1,t ,y 0,t ,y 1,t ) is as follows: (x 0,t ,x 1,t ,and 0,t ,and 1,t )=(min(x 0,A ,x 0,B ,x 0,C ,…),max(x 1,A ,x 1,B ,x 1,C ,…),min(and 0,A ,and 0,B ,and 0,C ,…),max(and 1,A ,and 1,B ,and 1,C ,…))(8); Among them, the x of the table 0,t is the minimum value of x0 of all cells; x 1,t is the maximum value of x1 for all cells; y of the table 0,t is the minimum value of y0 of all cells; the y 1,t is the maximum value of y1 for all cells.

7. The method for in-depth analysis of PDF research reports as claimed in claim 1, characterized in that: In step S3109, updating the table range is performed as follows: first, all text lines of the main text block are traversed to check whether the text lines belong to the table range, and the text lines that do not belong to the table range constitute non-table text lines; then, the non-table text lines are traversed to aggregate text lines near the table; Whether the rectangular range A corresponding to the text line is aggregated with the rectangular range B corresponding to the table A,B , calculated as follows: Among them, t t Represents the distance threshold of text near the table; when A and B overlap horizontally and the vertical distance is less than the threshold t t In other cases, no aggregation; After the aggregation conditions are met, the table coordinates are calculated and the table range is updated.

8. The method for in-depth analysis of PDF research reports as claimed in claim 1, characterized in that: In step S3, the process of locating the range of the curve graph is as follows: Step S3201. After the table search is completed, for all text lines of the main text block, determine whether they are in any table, filter out the text lines in the table range, and obtain non-table text lines; traverse the text lines; Step S3202. Check all the straight line elements to see if they form straight line related texts with the text line, and calculate the combined rectangular range for the straight line related texts that meet the conditions; check all the rectangular elements to see if they form rectangular related texts with the text line, and calculate the combined rectangular range for the rectangular related texts that meet the conditions; after the traversal is completed, obtain the straight line related texts and the rectangle related texts; Check whether the rectangular range A corresponding to the rectangular element and the rectangular range B corresponding to the text line are combined into the legend jtl A,B The formula is as follows: Among them, t lt Indicates the distance threshold of the legend; the legend is characterized by the rectangular element being on the left side of the text line, and the range of the text line in the y direction includes the range of the rectangle in the y direction; so y1 of A is less than or equal to y1 of B, y0 of A is greater than or equal to y0 of B, x1 of A is less than x0 of B, and the distance between the text line and the rectangular element is x. 0,B -x 1,A , the distance is less than the threshold t lt ; After the combined legend conditions are met, the combined coordinates are calculated; Check whether the rectangular range A corresponding to the straight line element and the rectangular range B corresponding to the text line are combined into a curve diagram. A,B The formula is as follows: Description: sm1 Represents the horizontal distance threshold of the curve diagram, t sm2 Indicates the vertical distance threshold of the curve description; the characteristics of the curve description are that the straight line elements are above or below the text line, and the horizontal coordinates of A and B have constraints; A's x0 is less than B's x1, A's x1 is greater than B's x1, and the absolute value of the difference between A's x0 and B's x0 is less than t sm1 , and the absolute value of the difference between y0 of A and y1 of B is less than t sm2 , the absolute value of the difference between y1 of A and y0 of B is less than t sm2 ,When these conditions are met, A and B are combined into a curve graph; in other cases, A and B are not aggregated; After the conditions for the combined curve graph are met, the combined coordinates are calculated; Step S3203. Traverse the curve graph, which includes two basic elements: curve elements and graph elements; Step S3204. First, perform rectangle-related text aggregation: refer to the curve graph, perform aggregation analysis on all rectangle-related texts, and update the range of the curve graph; then perform line-related text aggregation: refer to the curve graph, perform aggregation analysis on all line-related texts, and update the range of the curve graph; Whether the rectangular range A corresponding to the line-related text or the rectangle-related text is aggregated with the rectangular range B corresponding to the curve element A,B Calculated as follows: Among them, t qx Represents the distance threshold between the curve element and the text associated with the nearby straight line or rectangle; A and B meet the horizontal overlap, and the vertical distance is less than the threshold t qx Aggregate in the case of; do not aggregate in other cases; After the aggregation conditions of the line-related text or rectangle-related text and the curve are met, the range of the merged curve graph is calculated; Whether the rectangular range A corresponding to the line-related text or rectangle-related text is aggregated with the rectangular range B corresponding to the graphic element A,B Calculated as follows: Among them, t tu Represents the distance threshold between the graphic element and the text associated with the line or rectangle nearby; A and B meet the horizontal overlap, and the vertical distance is less than the threshold t tu Aggregate in the case of; do not aggregate in other cases; After the aggregation conditions of the line-related text or rectangle-related text and graph are met, the range of the merged curve graph is calculated; Step S3205. Final graph and text aggregation: refer to the graph, perform aggregation analysis on all non-table text lines, and finally determine the range of the graph; Whether the rectangular range A corresponding to the text line is aggregated with the rectangular range B corresponding to the curve graph A,B Calculated as follows: Among them, t w Represents the distance threshold of the text near the curve graph; A and B meet the horizontal overlap, and the vertical distance is less than the threshold t w Aggregate in the case of; do not aggregate in other cases; After the aggregation conditions of the text line and the curve chart are met, the merged curve chart range is calculated.

9. The method for in-depth analysis of PDF research reports as claimed in claim 1, characterized in that: In step S4, it is determined whether the rectangular range A corresponding to the text line is within the rectangular range B corresponding to the table or the curve graph. A,B The formula is as follows: Among them, when A and B overlap horizontally and vertically, A is within the area of ​​B; in other cases, A is not within the area of ​​B.

Images