A method and device for extracting information from PDF files

Abstract

The present invention relates to the field of information processing, and in particular to a method and device for extracting information from a PDF file. The method comprises the following steps: for a PDF file, generating a corresponding tree structure for the information in each page; counting the information of each node in the tree structure corresponding to each page, identifying and extracting the title, text, chart title and chart ending from each page; summarizing and classifying the titles, extracting charts according to the chart titles and chart endings, mapping the text and charts to the corresponding titles and chart titles, and finally generating structured data of the PDF file. In this way, the title, text, charts and the like in the PDF file can be structuredly extracted, and fine analysis and effective extraction of chart information can be performed, thereby providing data support for realizing search in the vertical field of industry research reports, accurate information positioning and content mining, and greatly simplifying the time for users to analyze the content of the research reports.

Description

Technical Field

[0001] The present invention relates to the field of information processing, and in particular to a method and device for extracting information from a PDF file. Background Art

[0002] In order to facilitate industry analysts to retrieve the desired industry field report content from numerous industry research reports and to dig out the content fragments with the best quality and most representative of the current status of industry analysis from a large number of research reports, it is necessary to finely structure the content in the industry research reports, identify the parent and child titles, content fragments, chart content and other information belonging to each title and organize them organically.

[0003] In the existing technology, information extraction for PDF files of industry research reports is mainly carried out on the text data therein, and there is no better method for parsing images and tables in PDF files. In particular, there is currently no solution for fine parsing and chart data extraction for PDF files of industry research reports. Summary of the invention

[0004] The embodiment of the present invention provides a method and device for extracting information from a PDF file, so as to solve the problem in the prior art that the information content of a PDF file cannot be precisely parsed and chart information cannot be effectively extracted.

[0005] The specific technical solutions provided by the embodiments of the present invention are as follows:

[0006] A method for extracting information from a PDF file, comprising:

[0007] For a PDF file, respectively obtain information in each page of the PDF file, and generate a corresponding tree structure for the information in each page according to the type of information;

[0008] For each page's corresponding tree structure, the information of each node in the tree structure is counted respectively, and the title, text, chart title, and chart ending of each page are identified and extracted from each page according to the position, height, and width of the information of each node, and the icon title and icon ending are paired according to the preset rules;

[0009] Summarize the titles, texts, paired chart titles and chart endings identified and extracted from each page, classify the titles according to their heights and styles, identify and extract charts from each page according to the positions of paired chart titles and chart endings, and map the texts and charts to the corresponding titles and chart titles, respectively, to obtain the summary results of the PDF file;

[0010] According to the summary result of the PDF file, structured data of the PDF file is generated.

[0011] Preferably, further comprising:

[0012] Obtain a PDF file, and decompose the PDF file by page to generate a new PDF file corresponding to each page;

[0013] Get the width and height of the new PDF file corresponding to each page respectively;

[0014] According to the width and height of the new PDF file corresponding to each page, the new PDF file corresponding to each page is enlarged by a preset multiple and converted into a corresponding image.

[0015] Preferably, the information of each node in the tree structure corresponding to each page is counted respectively, and the title, text, chart title, and chart end of each page are identified and extracted from each page according to the position, height, and width of the information of each node, specifically including:

[0016] The position and height of the row text nodes in the tree structure corresponding to each page are counted respectively to determine the position and height of the text in each page;

[0017] Counting the information of the nodes in the preset range of the tree structure corresponding to each page respectively, and comparing the information in the preset range in each page, and determining the position of the head and tail of each page according to the similarity of the information in the preset range in each page;

[0018] According to the positions of the head and the tail of each page and the position of the body text in each page, the main body text including the body text is identified and extracted from each page;

[0019] Identify headings from the extracted main body text containing the main text, and divide the main text into paragraphs;

[0020] According to the preset regular expression, the chart title and icon end are identified and extracted from each page respectively.

[0021] Preferably, the position and height of the line text nodes in the tree structure corresponding to each page are counted respectively to determine the position and height of the text in each page, which specifically includes:

[0022] The X-axis offset and height of the line text nodes in the corresponding tree structure of each page are counted respectively, and the number of line texts with the same X-axis offset and height is the largest, and the corresponding X-axis offset and height are used as the X-axis offset and height of the main text in each page; wherein the position at least includes the X-axis offset relative to the preset coordinate origin.

[0023] Preferably, the title is identified from the extracted main body text including the main text, and the main text is divided into paragraphs, specifically including:

[0024] According to the height, width, and distance of the context, whether there is a title mark, the main body text that meets the preset strategy is used as the recognized title title; wherein the preset strategy is one or any combination of the following: the height is greater than the text height + 1 and less than 30, the distance from the previous text is greater than 9 and the distance from the next text is greater than 9 and there is a title mark, the distance from the previous text is greater than 10 and the height is the text height and the width is less than 60% of the width of the next line of text;

[0025] Divide the text into paragraphs based on line break node information, context distance, and figure titles;

[0026] For the identified and extracted titles and paragraphs, the misidentified titles are filtered out according to the set conditions.

[0027] Preferably, the titles are graded according to their height and style, including:

[0028] According to the style of the title, the titles with the same style are grouped into one category to obtain the titles belonging to each category;

[0029] Based on the height of the titles, and in order of title height from large to small, determine the hierarchical relationship between titles of each category and classify the titles.

[0030] Preferably, the charts are identified and extracted from each page according to the positions of the paired chart titles and the chart ends, including:

[0031] According to the positions of the paired chart title and the end of the chart, determine whether the chart spans pages. If so, calculate the positions of the chart spanning pages in each corresponding page. Otherwise, calculate the positions of the chart in each page.

[0032] According to the position of the chart, the chart is identified and extracted from the corresponding picture on each page, and the charts across the pages are spliced ​​to obtain the final chart.

[0033] Preferably, further comprising:

[0034] According to actual needs, the structured data of the PDF file is saved in a corresponding database.

[0035] A device for extracting information from a PDF file, comprising:

[0036] The page parsing unit is used to count the information of each node in the tree structure corresponding to each page according to the tree structure generated by the overall parsing unit, and identify and extract the title, text, chart title, and chart ending of each page according to the position, height, and width of the information of each node, and pair the icon title and icon ending according to the preset rules;

[0037] The overall parsing unit is used to obtain information from each page of the PDF file respectively, and generate a corresponding tree structure for the information in each page according to the type of information, and summarize the title, text, paired chart title and chart ending identified and extracted from each page, classify the title according to the height and style of the title, and identify and extract charts from each page according to the positions of the paired chart title and chart ending, and map the text and chart to the corresponding title and chart title respectively, to obtain the summary result of the PDF file, and generate structured data of the PDF file according to the summary result of the PDF file.

[0038] Preferably, the method further comprises a pre-processing unit for:

[0039] Obtain a PDF file, and decompose the PDF file by page to generate a new PDF file corresponding to each page;

[0040] Get the width and height of the new PDF file corresponding to each page respectively;

[0041] According to the width and height of the new PDF file corresponding to each page, the new PDF file corresponding to each page is enlarged by a preset multiple and converted into a corresponding image.

[0042] Preferably, the information of each node in the tree structure corresponding to each page is counted respectively, and the title, text, chart title, and chart end of each page are identified and extracted from each page according to the position, height, and width of the information of each node.

[0043] The overall parsing unit is specifically used to count the information of the nodes in the tree structure corresponding to each page within a preset range, and compare the information within the preset range in each page, determine the position of the head and tail of each page according to the similarity of the information within the preset range in each page, and send the position of the head and tail of each page to the page parsing unit;

[0044] The page parsing unit is specifically used for:

[0045] The position and height of the row text nodes in the tree structure corresponding to each page are counted respectively to determine the position and height of the text in each page;

[0046] According to the positions of the header and the footer of each page sent by the overall parsing unit and the position of the body text in each page, the main body text including the body text is identified and extracted from each page respectively;

[0047] Identify headings from the extracted main body text containing the main text, and divide the main text into paragraphs;

[0048] According to the preset regular expression, the chart title and icon end are identified and extracted from each page respectively.

[0049] Preferably, the position and height of the line text nodes in the tree structure corresponding to each page are counted respectively, and when determining the position and height of the text in each page, the page parsing unit is specifically used to:

[0050] The X-axis offset and height of the line text nodes in the corresponding tree structure of each page are counted respectively, and the number of line texts with the same X-axis offset and height is the largest, and the corresponding X-axis offset and height are used as the X-axis offset and height of the main text in each page; wherein the position at least includes the X-axis offset relative to the preset coordinate origin.

[0051] Preferably, when identifying the title from the extracted main body text including the main text and dividing the main text into paragraphs, the page parsing unit is specifically used to:

[0052] According to the height, width, and distance of the context, whether there is a title mark, the main body text that meets the preset strategy is used as the recognized title title; wherein the preset strategy is one or any combination of the following: the height is greater than the text height + 1 and less than 30, the distance from the previous text is greater than 9 and the distance from the next text is greater than 9 and there is a title mark, the distance from the previous text is greater than 10 and the height is the text height and the width is less than 60% of the width of the next line of text;

[0053] Divide the text into paragraphs based on line break node information, context distance, and figure titles;

[0054] For the identified and extracted titles and paragraphs, the misidentified titles are filtered out according to the set conditions.

[0055] Preferably, when the titles are graded according to their height and style, the overall parsing unit is specifically used to:

[0056] According to the style of the title, the titles with the same style are grouped into one category to obtain the titles belonging to each category;

[0057] Based on the height of the titles, and in order of title height from large to small, determine the hierarchical relationship between titles of each category and classify the titles.

[0058] Preferably, when identifying and extracting the charts from each page according to the positions of the paired chart titles and chart ends, the device further comprises a picture cutting unit, wherein:

[0059] The overall parsing unit is specifically used to determine whether the chart spans pages according to the positions of the paired chart title and the end of the chart. If so, the positions of the chart spanning pages in each corresponding page are calculated respectively. Otherwise, the positions of the chart in each page are calculated respectively, and the positions of the charts are sent to the image cutting unit.

[0060] The image cutting unit is used to identify and extract the charts from the corresponding pictures on each page according to the positions of the charts, and to splice the charts across pages to obtain the final charts.

[0061] Preferably, further comprising:

[0062] The output unit is used to save the structured data of the PDF file into a corresponding database according to actual needs.

[0063] The beneficial effects of the embodiments of the present invention are as follows:

[0064] In an embodiment of the present invention, for a PDF file, information in each page of the PDF file is obtained respectively, and a corresponding tree structure is generated for the information in each page according to the type of information; for the tree structure corresponding to each page, information of each node in the tree structure corresponding to each page is counted respectively, and the title, text, chart title, and chart ending in each page are identified and extracted from each page according to the position, height, and width of the information of each node, and the icon title and icon ending are paired according to preset rules; the title, text, paired chart title, and chart ending identified and extracted from each page are summarized, and the title is paired according to the height and style of the title. According to the position of paired chart titles and chart ends, the charts are identified and extracted from each page respectively, and the text and charts are mapped to the corresponding titles and chart titles respectively to obtain the summary results of the PDF file; according to the summary results of the PDF file, the structured data of the PDF file is generated, so that the title, text, charts, etc. in the PDF file of the research report industry can be structured extracted, the information content of the PDF file can be finely analyzed and the chart information can be effectively extracted, which provides data support for the search in the vertical field of industry research reports, accurate information positioning and content mining, and greatly simplifies the time for industry analysts to analyze the content of research reports. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a schematic diagram of the structure of a device for extracting information from a PDF file in an embodiment of the present invention;

[0066] Figure 2 This is a message interaction sequence diagram of a device for extracting information from a PDF file in an embodiment of the present invention;

[0067] Figure 3 This is a flowchart summarizing a method for extracting information from a PDF file in an embodiment of the present invention;

[0068] Figure 4 This is a schematic diagram of the structure of DomTree in an embodiment of the present invention;

[0069] Figure 5 The tree structure corresponding to the structured data of the PDF file in the embodiment of the present invention;

[0070] Figure 6 A detailed flow chart of the execution process of the overall parsing unit in an embodiment of the present invention;

[0071] Figure 7 A detailed flow chart of the execution process of the page parsing unit in an embodiment of the present invention;

[0072] Figure 8 This is a schematic diagram of information parsing results in a PDF file in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] In order to solve the problem that the prior art cannot perform fine analysis of the information content of PDF files and effectively extract chart information, in an embodiment of the present invention, for a PDF file, the title, text, pictures, and tables (wherein, in the embodiment of the present invention, pictures and tables are collectively referred to as "charts") are identified and extracted, and they are organized according to the sequential structure in the PDF file to ultimately generate structured data.

[0075] The scheme of the present invention is described in detail below through specific embodiments. Of course, the present invention is not limited to the following embodiments.

[0076] See also Figure 1 As shown, in an embodiment of the present invention, the information extraction device of a PDF file includes:

[0077] The preprocessing unit 10 is used to obtain a PDF file, decompose the PDF file by page, generate a new PDF file corresponding to each page, obtain the width and height of the new PDF file corresponding to each page, and enlarge the new PDF file corresponding to each page by a preset multiple according to the width and height of the new PDF file corresponding to each page, and convert it into a corresponding image.

[0078] The overall parsing unit 11 is used to obtain the information in each page of the PDF file respectively, and generate a corresponding tree structure for the information in each page according to the type of information, and summarize the title, text, paired chart title and chart end identified and extracted from each page, classify the title according to the height and style of the title, and identify and extract charts from each page according to the positions of the paired chart title and chart end, and map the text and chart to the corresponding title and chart title respectively, to obtain the summary result of the PDF file, and generate the structured data of the PDF file according to the summary result of the PDF file.

[0079] The page parsing unit 12 is used to count the information of each node in the tree structure corresponding to each page according to the tree structure generated by the overall parsing unit 11, and identify and extract the title, text, chart title, and chart ending of each page according to the position, height, and width of the information of each node, and pair the icon title and icon ending according to preset rules.

[0080] The image cutting unit 13 is used to extract the chart according to the position of the chart sent by the overall parsing unit 11, and save the extracted chart locally.

[0081] The output unit 14 is used to save the structured data of the PDF file into a corresponding database according to actual needs.

[0082] Among them, the preprocessing unit 10 is connected to the overall parsing unit 11, and the preprocessing unit 10 will also output PDF files and image files. The overall parsing unit 11 is respectively connected to the page parsing unit 12, the image cutting unit 13, and the output unit 14. The page parsing unit 12 will read the image file and generate the segmented file.

[0083] Preferably, the information of each node in the tree structure corresponding to each page is counted respectively, and the title, text, chart title, and chart end of each page are identified and extracted from each page according to the position, height, and width of the information of each node.

[0084] The overall parsing unit 11 is specifically used to count the information of the nodes in the tree structure corresponding to each page within a preset range, and compare the information within the preset range in each page, determine the position of the head and tail of each page according to the similarity of the information within the preset range in each page, and send the position of the head and tail of each page to the page parsing unit 12;

[0085] The page parsing unit 12 is specifically used for:

[0086] The position and height of the row text nodes in the tree structure corresponding to each page are counted respectively to determine the position and height of the text in each page;

[0087] According to the positions of the header and the footer of each page sent by the overall parsing unit 11 and the position of the body text in each page, the main body text including the body text is identified and extracted from each page respectively;

[0088] Identify headings from the extracted main body text containing the main text, and divide the main text into paragraphs;

[0089] According to the preset regular expression, the chart title and icon end are identified and extracted from each page respectively.

[0090] Preferably, the position and height of the line text nodes in the tree structure corresponding to each page are counted respectively, and when determining the position and height of the text in each page, the page parsing unit 12 is specifically used to:

[0091] The X-axis offset and height of the line text nodes in the corresponding tree structure of each page are counted respectively, and the number of line texts with the same X-axis offset and height is the largest, and the corresponding X-axis offset and height are used as the X-axis offset and height of the main text in each page; wherein the position at least includes the X-axis offset relative to the preset coordinate origin.

[0092] Preferably, when identifying the title from the extracted main body text including the main text and dividing the main text into paragraphs, the page parsing unit 12 is specifically used for:

[0093] According to the height, width, and distance of the context, whether there is a title mark, the main body text that meets the preset strategy is used as the recognized title title; wherein the preset strategy is one or any combination of the following: the height is greater than the text height + 1 and less than 30, the distance from the previous text is greater than 9 and the distance from the next text is greater than 9 and there is a title mark, the distance from the previous text is greater than 10 and the height is the text height and the width is less than 60% of the width of the next line of text;

[0094] Divide the text into paragraphs based on line break node information, context distance, and figure titles;

[0095] For the identified and extracted titles and paragraphs, the misidentified titles are filtered out according to the set conditions.

[0096] Preferably, when the titles are graded according to their height and style, the overall parsing unit 11 is specifically used for:

[0097] According to the style of the title, the titles with the same style are grouped into one category to obtain the titles belonging to each category;

[0098] Based on the height of the titles, and in order of title height from large to small, determine the hierarchical relationship between titles of each category and classify the titles.

[0099] Preferably, when identifying and extracting the charts from each page according to the positions of the paired chart titles and chart ends, the device further comprises a picture cutting unit 13, wherein:

[0100] The overall parsing unit 11 is specifically used to determine whether the chart spans pages according to the positions of the paired chart title and the end of the chart. If so, the positions of the chart spanning pages in each corresponding page are calculated respectively. Otherwise, the positions of the chart in each page are calculated respectively, and the positions of the charts are sent to the image cutting unit 13.

[0101] The image cutting unit 13 is used to identify and extract the charts from the corresponding pictures on each page according to the positions of the charts, and to splice the charts across pages to obtain the final charts.

[0102] See also Figure 2 As shown, a message interaction sequence diagram of a device for extracting information from a PDF file in an embodiment of the present invention.

[0103] Step 200: The pre-processing unit obtains a PDF file.

[0104] Step 201: the preprocessing unit sends a new PDF file corresponding to each page obtained after preprocessing to the overall parsing unit.

[0105] Step 202: The overall parsing unit generates a DomTree for each new PDF file corresponding to each page, and sends the DomTree to the page parsing unit.

[0106] Step 203: The page parsing unit counts the information of each node in each DomTree and sends it to the overall parsing unit.

[0107] Step 204: The overall parsing unit calculates the positions of the head and tail of each page according to the statistical node information sent by the page parsing unit, and sends the calculated positions to the page parsing unit.

[0108] Step 205: The page parsing unit sends the parsing result (ie, structured data) of each page to the overall parsing unit.

[0109] Step 206: The overall parsing unit sends the position of the chart to the image cutting unit.

[0110] Step 207: The overall parsing unit generates structured data of the entire PDF file and sends it to the output unit.

[0111] Step 208: The output unit saves the structured data of the PDF file to a corresponding database.

[0112] That is to say, firstly, the PDF file is preprocessed by the preprocessing unit.

[0113] Then, the overall parsing unit is notified to generate DomTree for each new PDF file corresponding to each page of the PDF file, and the DomTree is handed over to the page parsing unit for statistical analysis of the DomTree.

[0114] Then, the page parsing unit sends the parsing results of each page to the overall parsing unit, and the overall parsing unit summarizes the parsed title, chart, and text of each page to generate the structured data of the title, text, and chart of the PDF file, and sends the location of the chart to the image cutting unit for chart extraction.

[0115] Finally, the overall parsing unit sends the structured data to the output unit, which encapsulates it into different types of structures according to different requirements and saves it to the corresponding database.

[0116] For example, there are three types of databases: mongodb, mysql, and redis.

[0117] In the embodiment of the present invention, the above specific implementation method will be described in detail in the subsequent PDF file information extraction method, and will not be described in detail here.

[0118] See also Figure 3 As shown, in the embodiment of the present invention, the specific process of the method for extracting information from a PDF file is as follows:

[0119] Step 300: for a PDF file, respectively obtain information in each page of the PDF file, and generate a corresponding tree structure for the information in each page according to the type of information.

[0120] Each node in the tree structure (DomTree) is divided according to the type of information in each page of the PDF file.

[0121] See also Figure 4The figure shows the structure diagram of DomTree.

[0122] Among them, LTPage is the node of the entire page;

[0123] LTTextBox is a text container, which can contain one or more LTTextLine nodes;

[0124] LTTextLine is a container for each line of text, which can contain one or more LTChar, LTText, and LTAnno nodes;

[0125] LTChar is the node for each word;

[0126] LTAnno is the node where the letters in the text are actually represented as Unicode strings;

[0127] LTText is a line break node;

[0128] LTFigure is a container for the image area, which can contain one or more LTImage;

[0129] LTImage is an image node, which can contain one or more LTCruve nodes;

[0130] LTCruve is the curve node;

[0131] LTLine is a straight line node;

[0132] LTRect is a rectangular node.

[0133] Among them, the container node will contain attributes such as width, height, x0, y0, x1, y1, etc.; among them, width represents the width of the container, height represents the height of the container, x0 is the X-axis offset of the lower left corner of the container in the entire page, y0 is the Y-axis offset of the lower left corner of the container in the entire page, x1 is the X-axis offset of the upper right corner of the container in the entire page, y1 is the Y-axis offset of the upper right corner of the container in the entire page, and, in an embodiment of the present invention, the page uses the lower left corner as the (0,0) coordinate, the horizontal direction is the X-axis, and the vertical direction is the Y-axis.

[0134] Before executing step 300, the PDF file is further preprocessed, specifically including:

[0135] First, a PDF file is obtained, and the PDF file is decomposed by page to generate a new PDF file corresponding to each page.

[0136] For example, obtain a PDF file of an industry research report, use the pypdf interface to decompose the PDF file by page, generate a new PDF file for each page, and save it.

[0137] Then, get the width and height of the new PDF file corresponding to each page respectively.

[0138] For example, use the pdfminer interface to obtain the width (width) and height (height) of the new PDF corresponding to each page.

[0139] Finally, according to the width and height of the new PDF file corresponding to each page, the new PDF file corresponding to each page is enlarged by a preset multiple and converted into a corresponding image.

[0140] This is because the width and height of the new PDF file corresponding to each page may not be very large, which makes it inconvenient and difficult to find the corresponding position when cutting and extracting the chart later, and the result may be inaccurate.

[0141] For example, using ImageMagic, the new PDF file corresponding to each page is enlarged by a preset multiple according to its width and height, for example, 5 times, and the parameter resolution of wand.image is set to 500, and then converted into a corresponding image, and the generated image is saved.

[0142] Among them, pypdf, pdfminer and ImageMagic are tools in a third-party software package of python. Of course, other methods can also be used to implement the preprocessing process of PDF files, which is not limited in the embodiments of the present invention.

[0143] In this way, after the PDF file is preprocessed, the overall parsing unit can be notified to start parsing the PDF file.

[0144] In this way, after the PDF file is preprocessed, when step 300 is executed, a tree structure is generated for each page of the PDF file through the PDF parsing engine. For example, a corresponding DomTree is generated for each new PDF file corresponding to each page using pdfminer.

[0145] Specifically, how to use pdfminer to generate DomTree can adopt existing technical methods, which will not be described in detail in the embodiments of the present invention.

[0146] Step 310: For each page's corresponding tree structure, count the information of each node in the tree structure of each page, identify and extract the title, text, chart title, and chart ending of each page according to the position, height, and width of the information of each node, and pair the icon title and icon ending according to preset rules.

[0147] When executing step 310, it specifically includes:

[0148] First, the position and height of the row text nodes in the tree structure corresponding to each page are counted respectively to determine the position and height of the text in each page.

[0149] The position of the text to be determined is preferably the X-axis offset of the text.

[0150] Specifically, the LTTextLine containers on the x0 coordinate and the LTTextLine containers with the same height are counted, and the x0 and height corresponding to the maximum number of lines of text are calculated, and used as the X-axis offset of the text and the height of the text.

[0151] Then, the information of the nodes in the preset range of the tree structure corresponding to each page is counted respectively, and the information in the preset range in each page is compared, and the position of the head and tail of each page is determined according to the similarity of the information in the preset range in each page.

[0152] Specifically: 1) counting the information of nodes in the tree structure corresponding to each page within a preset range;

[0153] For example, count the 1 / 5 of LTRect, LTFigure, and LTLine containers on the Y axis of each page, and count the 1 / 5 of LTRect, LTFigure, and LTLine containers below the Y axis of each page.

[0154] 2) Calculate the similarity of information within a preset range in each page and determine the position of the head and tail of each page.

[0155] Specifically: calculate the similarity of information between the upper 1 / 5 container and the lower 1 / 5 container of each page, that is, count the number of LTRect, LTFigure, and LTLine containers with the same x0, y0, x1, and y1. Among the containers whose similarity is greater than 80% of the number of pages in the entire PDF file, calculate the position of the head (that is, the container with the smallest y0) and the tail (that is, the container with the largest y1) of each page.

[0156] Among them, it is worth mentioning that the determination of the position of the header and tail of each page is performed by the overall parsing unit, because it is necessary to summarize the information in each page. After that, the calculated position of the header and tail of each page can be sent to each page parsing unit to enable it to execute subsequent steps.

[0157] Then, according to the positions of the head and the tail of each page and the position of the main text in each page, the main body text including the main text is identified and extracted from each page.

[0158] Specifically, the information between the positions of the head and the tail, and between 100 pixels before and after the X-axis offset, is determined as the recognized main body text containing the body.

[0159] Then, titles are identified from the extracted body text containing the main text, and the main text is divided into paragraphs.

[0160] Specifically: 1) First, identify whether it is a title based on features such as height, width, and distance from the context, and whether there is a title mark (for example, starting with a number or a Chinese number), and use the main text that meets the preset strategy as the identified title, where the preset strategy is specifically, for example: 1. The height is greater than the text height + 1 and less than 30; 2. The distance to the previous text is greater than 9 and the distance to the next text is greater than 9 and there is a title mark; 3. The distance to the previous text is greater than 10 and the height is the text height and the width is less than 60% of the width of the next line of text, etc.

[0161] The setting of each value in the above preset strategy is only a preferred method, and different values ​​can also be set according to actual needs.

[0162] Of course, in the embodiment of the present invention, it is not limited to the above-mentioned preset strategy, and a variety of different strategies can also be set according to the characteristics of the title in the PDF file.

[0163] 2) Then, the text is divided into paragraphs based on features such as line break node information, context distance, and figure title.

[0164] 3) Finally, for the identified and extracted titles and paragraphs, some misidentified titles are filtered out, for example, 4 consecutive lines are all identified as titles, or there are multiple spaces or tabs in the title.

[0165] Finally, according to the preset regular expressions, the chart titles and icon endings are identified and extracted from each page respectively, and the icon titles and icon endings are paired according to the preset rules.

[0166] Specifically, for each line of text node information on each page, a preset regular expression is used to determine whether it is a chart title or a chart end, and the information is recorded in order. According to the coordinates (x0, y0) of the lower left corner of the chart title and the end of the chart image, the chart title and the end of the chart image are paired based on the closest Euclidean distance method, and the chart titles are arranged in the order of appearance.

[0167] It is worth mentioning that for PDF files of industry research reports, the chart titles or chart endings generally have fixed formats or styles. Therefore, the above regular expression can be set based on the characteristics of PDF files of industry research reports.

[0168] That is to say, step 310 mainly extracts the information in each page of the PDF file, and obtains the text, chart title, icon ending, title and other information in each page.

[0169] Step 320: Summarize the titles, texts, paired chart titles and chart endings identified and extracted from each page, classify the titles according to their heights and styles, identify and extract charts from each page according to the positions of the paired chart titles and chart endings, and map the texts and charts to the corresponding titles and chart titles, respectively, to obtain the summary results of the PDF file.

[0170] First, the titles identified and extracted from each page are summarized and divided into different levels according to their height and style.

[0171] Specifically: according to the style of the title, the titles with the same style are grouped into one category to obtain the titles belonging to each category; and according to the height of the title, in descending order of the height of the title, the hierarchical relationship between the titles of each category is determined and the titles are divided into levels.

[0172] For example, the same title styles (naming rules) that start with One, Two…, or (One), (Two)…, or 1, 2…, or 1.1, 1.2… are taken as a category, and the inclusion relationship of the titles is determined from large to small according to the height of the titles. A total of 5 levels of titles are set from large to small, and the titles are then divided into different levels, thus generating a hierarchical relationship of the titles.

[0173] Then, according to the level and position of the title, each paragraph of the text is mapped to the corresponding title.

[0174] Specifically, the various paragraphs of the main text in each page are traversed in order, and according to the different levels and positions of the headings, the various paragraphs of the main text are mapped to the headings that are closest to their own positions.

[0175] Then, the position of the chart is calculated according to the positions of the paired chart title and the end of the chart, and the chart is identified and extracted from the corresponding picture on each page according to the position of the chart.

[0176] Specifically: 1) Determine whether the chart spans pages based on the positions of the paired chart title and chart end. If so, calculate the positions of the chart spanning pages in each corresponding page. Otherwise, calculate the positions of the chart in each page.

[0177] That is, the paired chart titles and icon endings are collected in the order of each page, and the charts with the same X-axis offset are counted, and the coordinates of the upper left corner and the lower right corner of the chart on the page are calculated according to the position of the first position of the chart. Furthermore, for the chart that spans across pages, the coordinates of its upper left corner and the lower right corner are calculated respectively.

[0178] 2) According to the position of the chart, identify and extract the chart from the corresponding picture on each page, and splice the charts across pages to obtain the final chart.

[0179] For example, according to the coordinates of the chart, through the ImageMagic cutting interface, the chart is identified and extracted from the corresponding picture on each page, and the charts across the pages are spliced ​​to generate the final chart.

[0180] Finally, the extracted charts are mapped to the corresponding chart titles.

[0181] Step 330: Generate structured data of the PDF file according to the summary result of the PDF file.

[0182] Specifically, according to the summary results of the PDF file, a tree structure is generated, that is, the structured data of the title, text, and chart of the PDF file is obtained. Figure 5 As shown, the tree structure corresponding to the structured data of the PDF file is a multi-branch tree structure generated according to the hierarchical relationship of the titles, and the mapping relationship between the text, chart titles, chart endings, and charts in each title. Its height is, for example, 7, and the root node is the file name of the PDF file. The titles of each level are located level by level, and the leaf nodes are the contents (i.e., the paragraphs divided into the text, chart titles, and chart endings).

[0183] Furthermore, according to different requirements, it is saved in different databases.

[0184] For example, it can be converted into JSON format and stored in MongoDB, or converted into each row of data and stored in MySQL, or converted into Thrift data type serialization and stored in Redis as key-value type.

[0185] Based on the above embodiments, several specific application scenarios are used below to respectively describe in detail the specific execution processes of the overall parsing unit and the page parsing unit.

[0186] See also Figure 6 As shown, in the embodiment of the present invention, the specific execution process of the overall parsing unit is:

[0187] Step 600: Generate DomTree by using the pdfminer interface for each page of the PDF file.

[0188] Step 601: Send DomTree to the page parsing unit for statistical analysis.

[0189] Step 602: Calculate the similarity of the information (images and lines) in the pages and determine the positions of the head and tail of each page.

[0190] Step 603: Send the positions of the header and the footer to the page parsing unit for information extraction.

[0191] Step 604: Summarize the title extraction results and classify the titles.

[0192] Step 605: Determine the mapping relationship between titles according to the levels of the titles.

[0193] Step 606: Summarize the chart title and chart ending.

[0194] Step 607: Determine whether the chart spans across pages, if so, execute step 609, otherwise, execute step 608.

[0195] Step 608: Calculate the position of the chart in the same page and send it to the picture cutting unit.

[0196] Step 609: Calculate the position of the chart in the multiple pages and send it to the picture cutting unit.

[0197] Step 610: Generate structured data of the PDF file according to the summary results.

[0198] See also Figure 7 As shown, in the embodiment of the present invention, the specific execution process of the page parsing unit is:

[0199] Step 700: traverse each node in DomTree.

[0200] Step 701: Count the X-axis offset and height of each node's information respectively.

[0201] Step 702: Calculate the X-axis offset of the text.

[0202] Step 703: Extract the text from each page according to the X-axis offset.

[0203] Step 704: Identify the chart title and the chart end using a preset regular expression.

[0204] Step 705: Filter and map the chart title and the chart end to obtain a paired chart title and chart end.

[0205] Step 706: Count the height of the extracted text.

[0206] Step 707: Identify the title and divide the text into paragraphs.

[0207] Step 708: Title filtering.

[0208] Based on the above embodiment, taking a page in a PDF file as an example, refer to Figure 8 , which is a schematic diagram of information parsing results in a PDF file in an embodiment of the present invention.

[0209] in, Figure 8 It is the information contained in one page of a PDF file in the field of industry research reports. It can be seen that based on the embodiment of the present invention, the title, text, chart title, chart end and other information in the PDF file can be identified and extracted, that is, the information in the PDF file can be finely analyzed, which is convenient for users to perform search, statistics, information mining and other applications.

[0210] To sum up, in the embodiment of the present invention, for a PDF file, the information in each page of the PDF file is obtained respectively, and according to the type of information, a corresponding tree structure is generated for the information in each page; for the tree structure corresponding to each page, the information of each node in the tree structure corresponding to each page is counted respectively, and according to the position, height and width of the information of each node, the title, text, chart title and chart ending in each page are identified and extracted respectively from each page, and the icon title and icon ending are paired according to preset rules; the title, text, paired chart title and chart ending identified and extracted from each page are summarized, and according to the height and style of the title, the icon title and icon ending are paired The titles are graded, and according to the positions of the paired chart titles and the end of the charts, charts are identified and extracted from each page respectively, and the text and charts are mapped to the corresponding titles and chart titles respectively, to obtain the summary results of the PDF file; according to the summary results of the PDF file, the structured data of the PDF file is generated, so that the title, text, charts, etc. in the PDF files of the research report industry can be structured extracted, the information content of the PDF files can be finely analyzed and the chart information can be effectively extracted, which provides data support for the search in the vertical field of industry research reports, accurate information positioning and content mining, and greatly simplifies the time for industry analysts to analyze the content of research reports.

[0211] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0212] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0213] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0214] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0215] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0216] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for extracting information from a PDF file, characterized in that: include: For a PDF file, respectively obtain information in each page of the PDF file, and generate a corresponding tree structure for the information in each page according to the type of information; For each page of the tree structure, the information of each node in the tree structure is counted respectively, and the title, text, chart title, and chart ending in each page are identified and extracted respectively according to the position, height, and width of the information of each node, and the chart title and chart ending are paired according to the preset rules, wherein the chart ending represents the annotation of the chart at the end of the chart; Summarize the titles, texts, paired chart titles and chart endings identified and extracted from each page, classify the titles according to their heights and styles, identify and extract charts from each page according to the positions of paired chart titles and chart endings, and map the texts and charts to the corresponding titles and chart titles, respectively, to obtain the summary results of the PDF file; According to the summary result of the PDF file, structured data of the PDF file is generated.

2. The method according to claim 1, characterized in that Further including: Obtain a PDF file, and decompose the PDF file by page to generate a new PDF file corresponding to each page; Get the width and height of the new PDF file corresponding to each page respectively; According to the width and height of the new PDF file corresponding to each page, the new PDF file corresponding to each page is enlarged by a preset multiple and converted into a corresponding image.

3. The method according to claim 1 or 2, characterized in that The information of each node in the tree structure corresponding to each page is counted respectively. According to the position, height and width of the information of each node, the title, text, chart title and chart end of each page are identified and extracted respectively from each page, including: The position and height of the row text nodes in the tree structure corresponding to each page are counted respectively to determine the position and height of the text in each page; Counting the information of the containers of the nodes in the tree structure corresponding to each page within a preset range respectively, and comparing the information of the containers within the preset range in each page, determining the positions of the head and tail of each page according to the similarity of the information of the containers within the preset range in each page, wherein the similarity represents the number of containers with the same coordinate attribute, the head represents the container whose lower left corner is in the container with the smallest Y-axis offset in the corresponding page among the containers whose similarity is greater than the threshold of the number of pages of the PDF file, and the tail represents the container whose upper right corner is in the container with the largest Y-axis offset in the corresponding page among the containers whose similarity is greater than the threshold of the number of pages of the PDF file, and the lower left corner of each page is the coordinate (0,0), the horizontal direction is the X-axis, and the vertical direction is the Y-axis; According to the positions of the head and the tail of each page and the position of the body text in each page, the main body text including the body text is identified and extracted from each page; Identify headings from the extracted main body text containing the main text, and divide the main text into paragraphs; According to the preset regular expression, the chart title and chart end are identified and extracted from each page respectively.

4. The method according to claim 3, characterized in that The position and height of the row text nodes in the tree structure corresponding to each page are counted respectively to determine the position and height of the text in each page, specifically including: The X-axis offset and height of the line text nodes in the corresponding tree structure of each page are counted respectively, and the number of line texts with the same X-axis offset and height is the largest, and the corresponding X-axis offset and height are used as the X-axis offset and height of the main text in each page; wherein the position at least includes the X-axis offset relative to the preset coordinate origin.

5. The method according to claim 3, characterized in that Identify the titles from the extracted main text containing the main text and divide the main text into paragraphs, including: According to the height, width, and distance of the context, whether there is a title mark, the main body text that meets the preset strategy is regarded as the recognized title; wherein the preset strategy is one or any combination of the following: the height is greater than the text height + 1 and less than 30, the distance from the previous text is greater than 9 and the distance from the next text is greater than 9 and there is a title mark, the distance from the previous text is greater than 10 and the height is the text height and the width is less than 60% of the width of the next line of text; Divide the text into paragraphs based on line break node information, context distance, and figure titles; For the identified and extracted titles and paragraphs, the misidentified titles are filtered out according to the set conditions.

6. The method according to claim 1, 2, 4 or 5, characterized in that The titles are classified into different levels according to their height and style, including: According to the style of the title, the titles with the same style are grouped into one category to obtain the titles belonging to each category; Based on the height of the titles, and in order of title height from large to small, determine the hierarchical relationship between titles of each category and classify the titles.

7. The method according to claim 2, characterized in that Identify and extract the charts from each page based on the positions of the paired chart titles and chart ends, including: According to the positions of the paired chart title and the end of the chart, determine whether the chart spans pages. If so, calculate the positions of the chart spanning pages in each corresponding page. Otherwise, calculate the positions of the chart in each page. According to the position of the chart, the chart is identified and extracted from the corresponding picture on each page, and the charts across the pages are spliced ​​to obtain the final chart.

8. The method according to claim 1, characterized in that Further including: According to actual needs, the structured data of the PDF file is saved in a corresponding database.

9. A device for extracting information from a PDF file, characterized in that: include: A page parsing unit is used to count the information of each node in the tree structure corresponding to each page according to the tree structure generated by the overall parsing unit, and to identify and extract the title, text, chart title, and chart ending in each page according to the position, height, and width of the information of each node, and to pair the chart title and chart ending according to a preset rule, wherein the chart ending represents the annotation of the chart at the end of the chart; The overall parsing unit is used to obtain information from each page of the PDF file respectively, and generate a corresponding tree structure for the information in each page according to the type of information, and summarize the title, text, paired chart title and chart ending identified and extracted from each page, classify the title according to the height and style of the title, and identify and extract charts from each page according to the positions of the paired chart title and chart ending, and map the text and chart to the corresponding title and chart title respectively, to obtain the summary result of the PDF file, and generate structured data of the PDF file according to the summary result of the PDF file.

10. The device according to claim 9, characterized in that Further comprising a pre-processing unit, for: Obtain a PDF file, and decompose the PDF file by page to generate a new PDF file corresponding to each page; Get the width and height of the new PDF file corresponding to each page respectively; According to the width and height of the new PDF file corresponding to each page, the new PDF file corresponding to each page is enlarged by a preset multiple and converted into a corresponding image.

11. The device according to claim 9 or 10, characterized in that The information of each node in the tree structure corresponding to each page is counted separately. According to the position, height and width of the information of each node, the title, text, chart title and chart end of each page are identified and extracted from each page. The overall parsing unit is specifically used to count the information of the containers of the nodes in the tree structure corresponding to each page within a preset range, and compare the information of the containers within the preset range in each page, determine the positions of the head and tail of each page according to the similarity of the information of the containers within the preset range in each page, and send the positions of the head and tail of each page to the page parsing unit, wherein the similarity represents the number of containers with the same coordinate attributes, the head represents the container with the smallest Y-axis offset in the corresponding page in the container whose similarity is greater than the threshold of the number of pages of the PDF file, the tail represents the container with the largest Y-axis offset in the corresponding page in the container whose similarity is greater than the threshold of the number of pages of the PDF file, and the upper right corner of the container is the largest Y-axis offset in the container in the corresponding page, and the page of each page takes the lower left corner as the (0,0) coordinate, the horizontal direction as the X axis, and the vertical direction as the Y axis; The page parsing unit is specifically used for: The position and height of the row text nodes in the tree structure corresponding to each page are counted respectively to determine the position and height of the text in each page; According to the positions of the head and the tail of each page sent by the overall parsing unit and the position of the body text in each page, the main body text including the body text is identified and extracted from each page respectively; Identify headings from the extracted main body text containing the main text, and divide the main text into paragraphs; According to the preset regular expression, the chart title and chart end are identified and extracted from each page respectively.

12. The device according to claim 11, characterized in that When counting the position and height of the line text nodes in the tree structure corresponding to each page and determining the position and height of the text in each page, the page parsing unit is specifically used to: The X-axis offset and height of the line text nodes in the corresponding tree structure of each page are counted respectively, and the number of line texts with the same X-axis offset and height is the largest, and the corresponding X-axis offset and height are used as the X-axis offset and height of the main text in each page; wherein the position at least includes the X-axis offset relative to the preset coordinate origin.

13. The device according to claim 11, characterized in that When identifying titles from the extracted main body text containing the main text and dividing the main text into paragraphs, the page parsing unit is specifically used to: According to the height, width, and distance of the context, whether there is a title mark, the main body text that meets the preset strategy is regarded as the recognized title; wherein the preset strategy is one or any combination of the following: the height is greater than the text height + 1 and less than 30, the distance from the previous text is greater than 9 and the distance from the next text is greater than 9 and there is a title mark, the distance from the previous text is greater than 10 and the height is the text height and the width is less than 60% of the width of the next line of text; Divide the text into paragraphs based on line break node information, context distance, and figure titles; For the identified and extracted titles and paragraphs, the misidentified titles are filtered out according to the set conditions.

14. The device according to claim 9, 10, 12 or 13, characterized in that When the titles are graded according to their height and style, the overall parsing unit is used to: According to the style of the title, the titles with the same style are grouped into one category to obtain the titles belonging to each category; Based on the height of the titles, and in order of title height from large to small, determine the hierarchical relationship between titles of each category and classify the titles.

15. The device according to claim 10, characterized in that When identifying and extracting the charts from each page according to the positions of the paired chart titles and chart ends, the device further includes a picture cutting unit, wherein: The overall parsing unit is specifically used to determine whether the chart spans pages according to the positions of the paired chart title and the end of the chart. If so, the positions of the chart spanning pages in each corresponding page are calculated respectively. Otherwise, the positions of the chart in each page are calculated respectively, and the positions of the charts are sent to the image cutting unit. The image cutting unit is used to identify and extract the charts from the corresponding pictures on each page according to the positions of the charts, and to splice the charts across pages to obtain the final charts.

16. The device according to claim 9, characterized in that Further including: The output unit is used to save the structured data of the PDF file into a corresponding database according to actual needs.

Images