Method and device for extracting image-text pairs of portable document format documents

Abstract

The present invention provides a method, device, equipment and storage medium for extracting text-image pairs of a PDF document, belonging to the field of data processing technology, and the method comprises: parsing a target PDF document, obtaining text block data and text coordinates corresponding to the text block data, and converting the target PDF document page by page into a document image; filtering and deduplicating the text block data, obtaining candidate legend texts, and determining candidate legend text coordinates corresponding to the candidate legend texts, processing the document image based on image processing technology, and extracting candidate illustration regions; matching candidate legend texts and candidate illustration regions based on candidate legend text coordinates, filtering failed candidate legend texts and failed candidate illustration regions, merging candidate illustration regions belonging to the same legend, and obtaining the text-image pair matching results of the target PDF document. Thus, the text-image pairs in the PDF document can be accurately identified and extracted, the operation process is simple, and the hardware resource overhead is low.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for extracting document image-text pairs in a portable document format. Background Art

[0002] At present, with the rise of deep learning technology, the mainstream scientific research paradigm is shifting from the traditional small data + small model paradigm for specific tasks and fields to the current big data + large model paradigm for general tasks and fields. In order to improve the multimodal information processing capabilities of large models, it is necessary to collect a large number of image-text pairs for model learning and training. Portable Document Format (PDF) documents, especially PDF documents of scientific literature, contain a large number of high-quality image-text pairs. How to quickly, efficiently and accurately extract them and save them in a unified standard format is critical to the learning and development of large models.

[0003] At present, although there are many PDF processing tools on the market, such as PyMuPDF, pdf2image, PyPDF4, etc., their functions in image-text extraction are often relatively basic, mainly focusing on simple text extraction or page image export, which is difficult to meet the needs of in-depth analysis of image-text relationships. Most of these tools are based on rule matching or simple image recognition technology. When faced with complex layouts (such as multi-column text, nested tables, mixed charts, etc.), extraction errors or omissions are prone to occur. In addition, existing solutions often ignore contextual relevance, making it difficult to accurately determine which graphic element a certain text description corresponds to, affecting the subsequent intelligent analysis and content reuse efficiency. Although there are also some works that use semantic segmentation or target detection technology in the field of computer vision to extract illustrations, tables, paragraphs and other information from PDF documents, such methods have some obvious disadvantages, such as: model learning relies on manually annotated data, and annotation work is time-consuming and laborious; model training and reasoning have a large overhead on hardware resources; extraction efficiency is low and error-prone, and versatility is poor; it is impossible to match illustrations and text, etc. Summary of the invention

[0004] The present invention provides a method and device for extracting image-text pairs from a portable document format document, which is used to solve the defects in the prior art that it is difficult to accurately match illustrations and text pairs when processing PDF documents and the resource overhead is large, and can accurately identify and extract image-text pairs in PDF documents, and has a simple operation process and low hardware resource overhead.

[0005] The present invention provides a method for extracting image-text pairs from a PDF document, comprising the following steps:

[0006] Parsing the target PDF document to obtain text block data and text coordinates corresponding to the text block data, and converting the target PDF document page by page into a document image;

[0007] Screening and deduplicating the text block data to obtain candidate legend texts, and determining candidate legend text coordinates corresponding to the candidate legend texts, and processing the document image based on image processing technology to extract candidate illustration areas;

[0008] Based on the candidate legend text coordinates, the candidate legend text and the candidate illustration area are matched, the candidate legend text and the candidate illustration area that fail to match are filtered out, and the candidate illustration areas belonging to the same legend are merged to obtain the image-text pair matching result of the target PDF document.

[0009] According to a method for extracting image-text pairs from a PDF document provided by the present invention, after obtaining the image-text pair matching result of the target PDF document, the method further includes:

[0010] Based on the coordinates of the illustration area in the image-text pair matching result, cutting out the corresponding illustration area from the target PDF document and saving it as an independent illustration file; and / or,

[0011] The image-text pair matching result is saved as structured data.

[0012] According to a method for extracting image-text pairs from a PDF document provided by the present invention, the document image is processed based on image processing technology to extract candidate illustration areas, specifically comprising:

[0013] grayscale and binarize the document image to obtain a first binary image;

[0014] After deleting the pixels of the candidate legend text from the first binary image, extracting the first connected component from the first binary image, and filling the coordinates of the bounding box of the first connected component to obtain a second binary image;

[0015] performing connected component extraction on the second binary image to obtain second connected components;

[0016] Based on the second connected component and a preset threshold, obtaining a small connected component set;

[0017] Based on the second connected component and the small connected component set, respectively generate a third binary image and a fourth binary image, and perform morphological transformation on the third binary image and the fourth binary image;

[0018] respectively extracting connected components from the third binary image after the morphological transformation and the fourth binary image after the morphological transformation to obtain a third connected component and a fourth connected component;

[0019] Filter the fourth connected component from the third connected component to obtain a fifth connected component;

[0020] Based on the fifth connected component, generate a fifth binary image, and perform morphological transformation on the fifth binary image;

[0021] Connected components are extracted from the fifth binary image after the morphological transformation to obtain a sixth connected component, and the sixth connected component is used as a candidate illustration region.

[0022] According to a method for extracting image-text pairs from a PDF document provided by the present invention, performing morphological transformation on the third binary image and the fourth binary image specifically includes:

[0023] Based on the first kernel function size, performing a morphological closing operation on the third binary image and the fourth binary image;

[0024] The performing morphological transformation on the fifth binary image specifically includes:

[0025] Based on the second kernel function size, performing a morphological closing operation on the fifth binary image;

[0026] The size of the second kernel function is larger than the size of the first kernel function.

[0027] According to a method for extracting image-text pairs from a PDF document provided by the present invention, matching the candidate legend text with the candidate illustration area based on the candidate legend text coordinates specifically includes:

[0028] Based on the candidate legend text coordinates, determining that there are one or more candidate legend texts on the current page of the target PDF document;

[0029] In the case that there is only one candidate legend text on the current page, matching all candidate illustration areas on the current page with the candidate legend text;

[0030] In the case that there are multiple candidate legend texts on the current page, the candidate legend texts are matched with the candidate illustration areas based on the coordinates of each candidate legend text and the coordinates of each candidate illustration area.

[0031] According to a method for extracting image-text pairs from a PDF document provided by the present invention, the image-text pair matching results are saved as structured data, specifically comprising:

[0032] Based on the image-text pair matching result, generating a structured text file of the target PDF document;

[0033] Among them, the structured text file includes the file name of the independent illustration file of the target PDF document, the legend text in the image-text pair matching result, the page where the illustration area in the image-text pair matching result is located, and the coordinates of the illustration area in the image-text pair matching result in the target PDF document.

[0034] The present invention also provides a device for extracting image-text pairs from a PDF document, comprising the following modules:

[0035] A parsing and conversion module, used to parse the target PDF document, obtain text block data and text coordinates corresponding to the text block data, and convert the target PDF document page by page into a document image;

[0036] A screening and extraction module, for screening and deduplicating the text block data, obtaining candidate legend texts, and determining candidate legend text coordinates corresponding to the candidate legend texts, and for processing the document image based on image processing technology to extract candidate illustration areas;

[0037] A matching and merging module is used to match the candidate legend text and the candidate illustration area based on the candidate legend text coordinates, filter the candidate legend text and the candidate illustration area that fail to match, and merge the candidate illustration areas belonging to the same legend to obtain the image-text pair matching result of the target PDF document.

[0038] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for extracting image-text pairs from a PDF document as described in any one of the above is implemented.

[0039] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for extracting image-text pairs from a PDF document as described in any one of the above is implemented.

[0040] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for extracting image-text pairs from a PDF document as described in any one of the above is implemented.

[0041] The present invention provides a method and device for extracting text-image pairs of a portable document format document. The method and device parse the PDF document to obtain text block data and text coordinates corresponding to the text block data, convert the PDF document page by page into a document image, screen and deduplicate the text block data, obtain candidate legend texts and candidate legend text coordinates corresponding to the candidate legend texts, process the document image based on image processing technology to extract candidate illustration areas, then match the candidate legend texts and candidate illustration areas according to the candidate legend text coordinates, filter out candidate legend texts and candidate illustration areas that fail to match, merge candidate illustration areas belonging to the same legend, and obtain the text-image pair matching results of the PDF document. Therefore, the text-image pairs in the PDF document can be accurately identified and extracted with a simple operation process and low hardware resource overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 The present invention provides a flowchart of a method for extracting image-text pairs from a PDF document.

[0044] Figure 2 The present invention provides a flowchart of an embodiment of extracting image and text pairs from a PDF document.

[0045] Figure 3 A schematic diagram of an example image provided by the present invention.

[0046] Figure 4 A schematic diagram of an example cropped image provided by the present invention.

[0047] Figure 5 The present invention is a schematic diagram of the structure of a device for extracting image-text pairs from a PDF document.

[0048] Figure 6 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Figure 1 The flowchart of the method for extracting image-text pairs from a PDF document provided by the present invention is as follows: Figure 1 As shown, the method comprises the following steps:

[0051] Step 100: Parse the target PDF document to obtain text block data and text coordinates corresponding to the text block data, and convert the target PDF document into document images page by page.

[0052] Specifically, the execution subject of the embodiment of the present invention is a device with certain computing and processing capabilities, such as a computer device. The method provided by the embodiment of the present invention is described below using a computer device as an example. The target PDF document is a PDF document for which image and text pairs need to be extracted.

[0053] When the user needs to process the target PDF document, the target PDF document to be processed can be input into the computer device. The specific input method can be online downloading, local storage hardware uploading, Bluetooth connection transmission, other format document conversion, etc. Only a limited number of methods are listed here, and the above methods are not limited. Other acquisition methods can also be used, which are not listed here one by one.

[0054] After inputting the target PDF document, the computer device can use open source tools to parse the target PDF document, obtain all the text block data and the coordinates of the text block data in the target PDF document, and convert the target PDF document page by page into a document image.

[0055] For example, you can use the open source tool PyMuPDF to parse a PDF document page by page to obtain the text and coordinate information of all text blocks on each page.

[0056] Optionally, the computer device may convert the PDF document page by page into page images of a preset resolution, where the preset resolution may be set to a relatively small value, such as 100 dpi, to reduce the image size, which is conducive to speeding up the algorithm processing speed.

[0057] It should be noted that, in the embodiments of the present invention, only the open source tool PyMuPDF is taken as an example, but it is not limited to this tool, and other tools can also be used for parsing, which are not listed one by one here.

[0058] Step 101: Screen and deduplicate the text block data to obtain candidate legend texts, determine the candidate legend text coordinates corresponding to the candidate legend texts, and process the document image based on image processing technology to extract candidate illustration regions.

[0059] Specifically, the text block data parsed from the target PDF document includes all the texts in the PDF document, and a large amount of these texts are irrelevant to the task of graphic-text pair extraction. Therefore, it is necessary to screen and filter the text block data; in addition, there may be duplicate instances in the text block data parsed from the target PDF document, so deduplication processing is required.

[0060] In some embodiments, candidate legend texts and coordinates can be screened out according to keyword matching in combination with the text content. For example, for all text blocks, if their text content starts with keywords such as "Figure", "Fig", "fig", etc., they are listed as candidate legend texts; otherwise, they are determined to be ordinary texts and deleted. It should be noted that only three keywords, namely "Figure", "Fig", "fig", are listed here, but this method is not limited to the above keywords, and other acquisition methods can also be used, which will not be listed one by one here.

[0061] In some embodiments, duplicate candidate legend texts and coordinates can be deleted in combination with the text content and text coordinates. For example, compare all candidate legend texts pairwise. If the texts (after deleting all spaces and line breaks) and coordinates of two candidate legends are exactly the same, they are determined to be duplicate texts, and only one of them is retained.

[0062] After obtaining the candidate legend texts, the candidate legend text coordinates can be determined according to the text coordinates corresponding to the candidate legend texts.

[0063] After converting the target PDF document into a document image page by page, the document image can be processed through image processing technology to extract candidate illustration regions. For example, one or more of image preprocessing, morphological operations, connected component analysis, feature extraction, region-based segmentation can be used to extract candidate illustration regions.

[0064] Step 102: Based on the candidate legend text coordinates, match the candidate legend texts and candidate illustration regions, filter out the candidate legend texts and candidate illustration regions that fail to match, and merge the candidate illustration regions belonging to the same legend to obtain the graphic-text pair matching result of the target PDF document.

[0065] Specifically, after obtaining the candidate legend texts, candidate legend text coordinates, and candidate illustration regions, the candidate legend texts and candidate illustration regions can be matched according to the candidate legend text coordinates.

[0066] Optionally, matching the candidate legend text and the candidate illustration area based on the candidate legend text coordinates includes:

[0067] Based on the candidate legend text coordinates, determining that one or more candidate legend texts exist on the current page of the target PDF document;

[0068] In the case that there is only one candidate legend text on the current page, all candidate illustration areas on the current page are matched with the candidate legend text;

[0069] When there are multiple candidate legend texts on the current page, the candidate legend texts and the candidate illustration areas are matched based on the coordinates of each candidate legend text and the coordinates of each candidate illustration area.

[0070] Specifically, in the process of matching the candidate legend texts and the candidate illustration regions, the candidate legend texts and the candidate illustration regions corresponding to each page in the target PDF document may be determined first.

[0071] If there is only one candidate legend text on a page, all candidate illustration areas of the page can be matched to the candidate legend text; if there are multiple candidate legend texts on a page, the coordinates of the candidate legend text and the coordinates of the candidate illustration area can be combined to design heuristic rules for matching.

[0072] In some implementations, heuristic rules can be designed based on conditions such as the vertical overlap rate between the candidate illustration area and the candidate legend text area, the relative position between the border of the candidate illustration area and the border of the candidate legend text area, and the distance between the border of the candidate illustration area and the border of the candidate legend text area.

[0073] For example, for each candidate legend, if the vertical overlap rate between a candidate illustration region and the candidate legend text region is greater than a preset threshold, and the lower border of the candidate illustration region is above the upper border of the candidate legend text region, and the distance is less than the preset threshold, then the candidate illustration region is matched to the candidate legend text region. The preset thresholds can be determined according to actual conditions.

[0074] After the matching process, the legend text that is not matched to the illustration area and the illustration area that is not matched to the legend text can be deleted, and then multiple candidate illustration areas that match the same legend text are merged, that is, multiple sub-images belonging to the same illustration are merged, so as to obtain the image-text pair matching result of the target PDF document.

[0075] The method for extracting image-text pairs of a PDF document provided by the present invention parses the PDF document to obtain text block data and text coordinates corresponding to the text block data, converts the PDF document page by page into a document image, screens and removes duplicates on the text block data, obtains candidate legend texts and candidate legend text coordinates corresponding to the candidate legend texts, processes the document image based on image processing technology to extract candidate illustration areas, then matches the candidate legend texts and candidate illustration areas according to the candidate legend text coordinates, filters out candidate legend texts and candidate illustration areas that fail to match, merges candidate illustration areas that belong to the same legend, and obtains image-text pair matching results of the PDF document, thereby being able to accurately identify and extract image-text pairs in a PDF document with a simple operation process and low hardware resource overhead.

[0076] According to a method for extracting image-text pairs from a PDF document provided by the present invention, after obtaining the image-text pair matching result of the target PDF document, the method further comprises:

[0077] Based on the coordinates of the illustration area in the image-text pair matching result, crop the corresponding illustration area from the target PDF document and save it as an independent illustration file; and / or,

[0078] Save the image-text pair matching results as structured data.

[0079] Specifically, in the process of merging multiple candidate illustration areas that match the same legend text, the coordinates of the multiple candidate illustration areas can be merged at the same time. Specifically, the coordinates of the overall bounding box surrounding these candidate illustration areas can be merged to obtain the bounding box coordinates of the overall illustration area, which are the coordinates of the illustration area in the image-text pair matching results.

[0080] In some implementations, after obtaining the image-text pair matching result of the target PDF document, the corresponding illustration area can be cropped from the target PDF document according to the coordinates of the illustration area in the image-text pair matching result and saved as an independent illustration file.

[0081] As mentioned above, when converting the target PDF document into a document image, the resolution of the document image can be set to a relatively small value, which is conducive to speeding up the algorithm processing speed. Therefore, different resolutions can be determined according to actual needs, such as 300dpi or 600dpi, and then according to the coordinates of the illustration area in the image-text matching results, the high-definition image of the illustration area can be cropped from the overall page image and saved in a specified image format, such as JPG, PNG, TIF, etc.

[0082] Optionally, some key information may be included in the name of the saved independent illustration file, such as the original PDF file name, the page number described by the illustration, the number of the illustration in the page, etc.

[0083] In some implementations, after obtaining the image-text pair matching result of the target PDF document, the image-text pair matching result can also be saved as structured data, such as structured text file types such as CSV, JSON, XML, etc. The present invention does not limit the type of structured data.

[0084] Optionally, the image-text pair matching result is saved as structured data, specifically including:

[0085] Generate a structured text file of the target PDF document based on the image-text pair matching results;

[0086] Among them, the structured text file includes the file name of the independent illustration file of the target PDF document, the legend text in the image-text pair matching result, the page where the illustration area in the image-text pair matching result is located, and the coordinates of the illustration area in the image-text pair matching result in the target PDF document.

[0087] Specifically, the image-text pair matching results are saved as structured data, which can be that the file name of the independent illustration file of the target PDF document, the legend text in the image-text pair matching results, the page where the illustration area in the image-text pair matching results is located, the coordinates of the illustration area in the image-text pair matching results in the target PDF document, etc. are all saved in the structured text file of the target PDF document, thereby facilitating the reading and calling of the image-text pair related information of the target PDF document.

[0088] According to a method for extracting image-text pairs from a PDF document provided by the present invention, a document image is processed based on image processing technology to extract a candidate illustration area, specifically comprising:

[0089] Gray-scaling and binarization processing is performed on the document image to obtain a first binary image;

[0090] After deleting the pixels of the candidate legend text from the first binary image, extracting the first connected component from the first binary image, and filling the coordinates of the bounding box of the first connected component to obtain a second binary image;

[0091] Performing connected component extraction on the second binary image to obtain a second connected component;

[0092] Based on the second connected component and a preset threshold, a small connected component set is obtained;

[0093] Based on the second connected component and the small connected component set, a third binary image and a fourth binary image are generated respectively, and a morphological transformation is performed on the third binary image and the fourth binary image;

[0094] respectively extracting connected components from the third binary image after the morphological transformation and the fourth binary image after the morphological transformation to obtain a third connected component and a fourth connected component;

[0095] Filter the fourth connected component from the third connected component to obtain a fifth connected component;

[0096] Based on the fifth connected component, generate a fifth binary image, and perform morphological transformation on the fifth binary image;

[0097] Connected components are extracted from the fifth binary image after the morphological transformation to obtain a sixth connected component, and the sixth connected component is used as a candidate illustration region.

[0098] Specifically, the embodiment of the present invention can process the document image based on the image processing technology of Connected Component Analysis (CCA) to extract the candidate illustration area. This method is not only simple and efficient, but also can effectively utilize the visual features and structural rules of the illustration area in the PDF document. In addition, this method also deeply integrates the visual prior knowledge of the document image, aiming to accurately and efficiently extract the illustration area from the complex PDF document image. This method not only improves the accuracy of illustration area extraction, but also lays a solid foundation for subsequent document content understanding, indexing, retrieval and format conversion applications.

[0099] First, the document image is generally a three-channel color image, so it is necessary to grayscale and binarize the document image to obtain the first binary image. In this process, the open source tool OpenCV can be used to convert the three-channel color page image into a single-channel grayscale page image, and then binarize it using the global threshold binarization method based on the grayscale value of each pixel position. Considering that the background of the PDF document image is generally pure white, the global threshold here can be set to a higher value, such as 240. After binarization, the color can be reversed to make the background black and the foreground white to facilitate subsequent processing.

[0100] After obtaining the first binary image, the pixels of the candidate legend text can be deleted from the first binary image. In this process, the pixel values ​​in the corresponding rectangular box in the first binary image can be set to 0 according to the coordinates of the candidate legend text to eliminate the interference of the legend text on the illustration area extraction step. Because the spatial positions of the legend text and the illustration area are generally close, if the interference is not eliminated, the subsequent morphological transformation operation may mistakenly merge the legend text pixels into the illustration area.

[0101] Then, the first connected component is extracted from the first binary image after deleting the pixels of the candidate legend text. In this process, the open source tool OpenCV can be used to extract all connected components from the first binary image after deleting the pixels of the candidate legend text. The connected components here can adopt 8-directional connected components. The subsequent connected component extraction process can be carried out according to this method, which will not be repeated in the following text.

[0102] After obtaining the first connected component, the coordinates of the bounding box of the first connected component can be filled to obtain a second binary image. In this process, all pixels in the bounding box can be set to 255 according to the extracted connected component coordinates to merge connected components with overlapping coordinates.

[0103] After obtaining the second binary image, connected components can be extracted from the second binary image to obtain second connected components. After filling, the connected component extraction operation is performed again. The number of second connected components is greatly reduced compared to the first connected components, and there are no connected components whose external frames overlap with each other, which can facilitate subsequent algorithm processing.

[0104] After obtaining the second connected component, the second connected component can be divided according to a preset threshold value to obtain a set of small connected components. The preset threshold value can be determined according to the actual situation. It can be understood that if the height or width of the external frame of a connected component is less than the preset threshold value, the connected component can be determined to be a small connected component, otherwise, the connected component can be determined to be a large connected component. Among them, the small connected component has a higher probability of belonging to the text category, and the large connected component has a higher probability of belonging to the content such as graphics and images.

[0105] In some implementations, the height of the page image can be and width , calculate the connected component threshold :

[0106]

[0107] in, Indicates the minimum operation. " means integer divisibility, and the unit can be pixel.

[0108] After obtaining the small connected component set, the second connected component and the small connected component set can be mapped to generate a third binary image and a fourth binary image respectively, where the third binary image contains the second connected component and the fourth binary image only contains the small connected component.

[0109] After the third binary image and the fourth binary image are generated, morphological transformation may be performed on the third binary image and the fourth binary image, thereby fusing connected components that are close in space.

[0110] After the third binary image and the fourth binary image are morphologically transformed, connected components can be extracted from the morphologically transformed third binary image and the morphologically transformed fourth binary image to obtain a third connected component and a fourth connected component, wherein the third connected component is obtained by aggregating large connected components and small connected components, and the fourth connected component is obtained by aggregating only small connected components.

[0111] After obtaining the third connected component and the fourth connected component, the fourth connected component can be filtered from the third connected component to obtain the fifth connected component. As mentioned above, the third connected component is obtained by aggregating large connected components and small connected components, while the fourth connected component is obtained by aggregating only small connected components. It can be understood that the purpose of filtering the fourth connected component from the third connected component is to delete the connected components aggregating only small connected components, which are likely to correspond to text block areas.

[0112] In this process, assume that the set of connected components extracted from the third connected component is S1, and the set of connected components extracted from the fourth connected component is S2. For each connected component belonging to S1, find out whether it exists in S2 at the same time. If it exists, it means that it is obtained by merging only small connected components, corresponding to the area containing only text in the document, which can be deleted from S1. Finally, the remaining connected components in S1 are large connected components, which correspond to illustration areas or sub-areas containing frame lines and bitmaps in the document.

[0113] Therefore, the fifth connected component can be mapped to generate a fifth binary image, which contains large connected components. The fifth binary image can be further morphologically transformed to fuse connected components that are close in space.

[0114] After the fifth binary image is morphologically transformed, connected components may be extracted from the morphologically transformed fifth binary image to obtain a sixth connected component, and the sixth connected component may be used as a candidate illustration region.

[0115] According to a method for extracting image-text pairs from a PDF document provided by the present invention, morphological transformation is performed on the third binary image and the fourth binary image, specifically comprising:

[0116] Based on the size of the first kernel function, performing a morphological closing operation on the third binary image and the fourth binary image;

[0117] The fifth binary image is subjected to morphological transformation, specifically including:

[0118] Based on the second kernel function size, performing a morphological closing operation on the fifth binary image;

[0119] The size of the second kernel function is larger than that of the first kernel function.

[0120] Specifically, in the process of performing morphological transformation on the third binary image and the fourth binary image, a morphological closing operation may be performed on the third binary image and the fourth binary image based on the first kernel function size.

[0121] Among them, the size of the first kernel function can be based on the height of the page image and width To calculate, in some implementations, the first kernel function size It can be calculated according to the following formula:

[0122]

[0123] In the formula, Indicates the minimum operation. " indicates integer divisibility.

[0124] In the process of performing morphological transformation on the fifth binary image, a morphological closing operation may be performed on the fifth binary image based on the second kernel function size.

[0125] The size of the second kernel function can be determined based on the height of the page image. and width To calculate, in some implementations, the second kernel function size It can be calculated according to the following formula:

[0126]

[0127] In the formula, Indicates the minimum operation. " indicates integer divisibility.

[0128] In the embodiment of the present invention, the second kernel function size Than the size of the first kernel function is larger, so that large connected components with farther spatial distance can be merged to generate candidate illustration regions.

[0129] The following further illustrates the PDF document image-text pair extraction method provided by the present invention through embodiments in specific application scenarios.

[0130] Figure 2 The flowchart of the PDF document image-text pair extraction embodiment provided by the present invention is as follows: Figure 2 As shown, the process includes the following steps:

[0131] Step S10, obtaining an input PDF document.

[0132] In this embodiment, the PDF document to be processed is input into the system. The specific input methods can include online download through the network, uploading from local storage hardware, transmission via Bluetooth connection, conversion from other format documents, etc. Only a limited number of such methods are listed here, and the above several methods are not limited. Of course, other acquisition methods can also be adopted, and they will not be listed one by one here.

[0133] Step S20: Use an open-source tool to parse the PDF document to obtain all text block data and their coordinates, and convert the PDF document page by page into a whole-page document image.

[0134] Use the open-source tool PyMuPDF to parse the PDF document page by page to obtain the text and coordinate information of all text blocks within each page, and convert the PDF document page by page into a page image with a preset resolution. Here, the preset resolution is usually set to a relatively small value, such as 100 dpi, to reduce the image size and speed up the algorithm processing speed. Only the open-source tool PyMuPDF is taken as an example here, but it is not limited to this tool. Of course, other tools can also be used for parsing, and they will not be listed one by one here.

[0135] Step S30: Screen and deduplicate the text data to obtain all candidate legend texts.

[0136] Since the text obtained by parsing the PDF using the PyMuPDF tool includes all the text existing in the PDF document, and this text contains a large amount of text irrelevant to the graphic-text pair extraction task, screening and filtering are required; in addition, there may be duplicate instances in the text and coordinates obtained by PyMuPDF parsing, and deduplication is needed.

[0137] Preferably, step S30 further includes the following sub-steps:

[0138] Step S31: Screen out candidate legend texts and coordinates according to keyword matching in combination with the text content;

[0139] Step S32: Delete duplicate candidate legend texts and coordinates in combination with the text content and text coordinates.

[0140] In step S31, the method of "screening out candidate legend texts and coordinates according to keyword matching in combination with the text content" is as follows: For all text blocks, determine whether their text content starts with keywords such as the word "Figure", "Fig", "fig", etc. If so, list them as candidate legend texts; otherwise, determine them as ordinary text and delete them. Only three keywords, namely "Figure", "Fig", "fig", etc., are listed here, but this method is not limited to the above keywords. Of course, other acquisition methods can also be adopted, and they will not be listed one by one here.

[0141] In step S32, "delete duplicate candidate legend texts and coordinates in combination with text content and text coordinates", the method is: compare all candidate legend texts in pairs, if the text (delete all spaces and line breaks) and coordinates of two candidate legends are exactly the same, then they are determined to be duplicate texts, and only one of them is retained.

[0142] Step S40, processing the document image and extracting candidate illustration regions using image processing technology.

[0143] In-depth processing of document images to accurately extract candidate illustration areas is one of the key steps to achieve automated document analysis and enhancement. This embodiment adopts a comprehensive strategy, the core of which is the image processing technology based on CCA. This method is not only simple and efficient, but also can effectively utilize the visual features and structural laws of the illustration area in the PDF document. In addition, this method also deeply integrates the visual prior knowledge of document images, aiming to accurately and efficiently extract illustration areas from complex PDF document images. This comprehensive method not only improves the accuracy of extraction, but also lays a solid foundation for subsequent document content understanding, indexing, retrieval and format conversion applications.

[0144] Preferably, the step S40 further includes the following sub-steps:

[0145] Step S41, graying and binarizing the color document page image;

[0146] Step S42, deleting the candidate legend area pixels from the binary page image;

[0147] Step S43, extracting connected components from the binary page image and filling them according to the coordinates of the bounding box;

[0148] Step S44, extracting connected components from the binary page image again;

[0149] Step S45, splitting the connected components into a large connected component set and a small connected component set according to a preset threshold;

[0150] Step S46, generating a binary graph including all connected components and a binary graph including only small connected components, and performing morphological transformation on each of them;

[0151] Step S47, extracting connected components from the two binary graphs again, and filtering out the connected components existing in the latter from the former, to obtain large connected components;

[0152] Step S48, generating a binary graph containing a large connected component and performing a morphological transformation on it;

[0153] Step S49, re-extracting connected components as illustration candidate regions.

[0154] Figure 3 The schematic diagram of the example image provided by the present invention, the method of "grayscale and binarization of color document page image" in step S41 is: use the open source computer vision tool OpenCV to convert the Figure 3 The three-channel color page image shown in (a) is converted into a single-channel grayscale page image, and then the image is binarized using the global threshold binarization method based on the grayscale value of each pixel position. Considering that the background of PDF document images is generally pure white, the global threshold here is set to a high value, such as 240, to retain as much foreground pixel information as possible. After binarization, the color is reversed to make the background black and the foreground white, as shown in Figure 1. Figure 3 The image after the binary inversion transformation shown in (b) is convenient for subsequent operations such as connected component analysis and morphological transformation.

[0155] The method of "deleting the candidate legend area pixels from the binary page image" in step S42 is as follows: according to the coordinates of the candidate legend text, the pixel values ​​in the corresponding legend text rectangle in the binary page image obtained in step S41 are set as background, that is, pixel 0. Figure 3 (c) in the figure is used to eliminate the interference of the legend text on the illustration area extraction step. Because the spatial positions of the legend text and the illustration are generally close, if the interference is not eliminated, the subsequent morphological transformation operation may mistakenly merge the legend text pixels into the illustration area.

[0156] The method of "extracting connected components from the binary page image and filling them according to the coordinates of the bounding box" in step S43 is as follows: using the open source computer vision tool OpenCV, all connected components are extracted from the binary image obtained in step S42. The connected components here are connected components in 8 directions, and the connected components are obtained. Figure 3 (d) in Figure 3 As shown in (d) in the figure, different connected components are represented by different colors. Then, according to the coordinates of the extracted connected components, all pixels within the bounding box are filled as foreground, that is, pixel 255, and the obtained Figure 3 (e) in Figure 3 As shown in (e) in , this way, connected components with overlapping coordinates can be merged. After filling, this method performs the connected component extraction operation again and obtains Figure 3 (f) in the above equation. Figure 3 As shown in (f) in Figure 3 (d) in Figure 3 The number of connected components in (f) is greatly reduced, and there are no connected components whose outer frames overlap with each other, which facilitates subsequent algorithm processing.

[0157] The method of “splitting the connected components into a large connected component set and a small connected component set according to a preset threshold value” in step S45 is as follows:

[0158] First, according to the height of the page image and width , in pixels, calculate the connected component threshold , the unit is also in pixels:

[0159]

[0160] in, Indicates the minimum operation. ” indicates integer divisibility.

[0161] Then, if the height or width of a connected component's bounding box is less than the threshold, it is determined to be a small connected component, otherwise, it is a large connected component. Among them, small connected components have a higher probability of belonging to text categories, while large connected components have a higher probability of belonging to graphics and images. After that, this method maps all connected components and small connected components into binary images respectively, and obtains Figure 3 (g) and Figure 3 In (h), such as Figure 3 As shown in (g) in , the binary graph contains all connected components, such as Figure 3 As shown in (h) in , the binary graph only contains small connected components.

[0162] The "morphological transformation" in step S46 refers to a morphological closing operation, which includes a dilation operation and an erosion operation, the purpose of which is to merge connected components that are close in space, and the method is:

[0163] First, according to the height of the page image and width , calculate the kernel function size of the morphological closing operation :

[0164]

[0165] Same as above, Indicates the minimum operation. ” indicates integer divisibility.

[0166] Then, based on the calculated kernel function size, the open source tool OpenCV is used to perform mathematical morphological closing operations on the binary image to merge connected components with close spatial distances. Figure 3 (g) and Figure 3 Execute (h) in the above equations to get Figure 3 (i) and Figure 3 After that, this method again Figure 3(i) and Figure 3 The connected component analysis is performed on (j) in the figure to obtain Figure 3 (k) and Figure 3 In (l). Figure 3 The connected components in (k) are simultaneously Figure 3 The large connected components and small connected components in (f) are aggregated, and Figure 3 The connected components in (l) are only composed of Figure 3 It is obtained by aggregating the small connected components in (f).

[0167] The purpose of "filtering out the connected components existing in the latter from the former" in step S47 is to Figure 3 In (k), connected components that are only aggregated from small connected components are deleted. These connected components are likely to correspond to text block areas. The specific method is: Assume that from Figure 3 The connected component set extracted from (k) is S1, Figure 3 The connected component set extracted from (l) is S2. For each connected component in S1, check whether it also exists in S2. If it exists, it means that it is only obtained by merging small connected components. Corresponding to the area containing only text in the document, it is deleted from S1. Finally, the remaining connected components in S1 are large connected components, such as Figure 3 As shown in (m), it is composed of Figure 3 The large connected component (graphics, images, etc.) in (f) and its nearby small connected components (symbols, text, etc.) are merged, corresponding to the illustration area or sub-area containing frame lines and bitmaps in the document.

[0168] The "morphological transformation" in step S48 refers to a morphological closing operation, which includes a dilation operation and an erosion operation, the purpose of which is to merge connected components that are close in space, and the method is:

[0169] First, according to the height of the page image and width , calculate the kernel function size of the morphological closing operation :

[0170]

[0171] Same as above. Indicates the minimum operation. " indicates integer divisibility.

[0172] Then, according to the calculated kernel function size, the open source tool OpenCV is used to perform mathematical morphological closing operations on the binary image to merge connected components with close spatial distances. It should be noted that the kernel function size in step S48 is Generally, it is better than step S46. is larger to merge large connected components farther in spatial distance to generate candidate illustration regions or subregions, see Figure 3 After that, this method performs Figure 3 The connected component analysis is performed in (n), and each connected component obtained corresponds to a candidate illustration region or subregion, see Figure 3 (o) in.

[0173] Step S50, matching and screening the candidate legend texts and candidate illustration areas, and deleting the legend texts and candidate illustrations that fail to match.

[0174] In this embodiment, a page may extract zero, one or more candidate legends and candidate illustration areas or sub-areas. The method needs to match the candidate legends and candidate illustration areas or sub-areas, and delete the candidate legends and candidate illustration areas or sub-areas that fail to match. The specific method is: if there is only one candidate legend on a page, all candidate illustration areas or sub-areas (if any) of the page are matched to the candidate legend; otherwise, it is necessary to design heuristic rules for matching in combination with the coordinates of the candidate legend and the candidate illustration. Specifically, for each candidate legend, if the vertical overlap rate of a candidate illustration area or sub-area and the candidate legend is greater than a preset threshold, such as 0.1, and the lower border of the candidate illustration area or sub-area is above the upper border of the candidate legend area, and the distance is less than a preset threshold, such as hei / 16, then the candidate illustration area is matched to the candidate legend area. After the above operations, the candidate legend areas that are not matched to any illustration areas or sub-areas and the candidate illustration areas or sub-areas that are not matched to any candidate legend areas are deleted to obtain the filtered legend and illustration matching results, see. Figure 3 The blue solid rectangle represents the inset area or sub-area (there is no inset sub-area in this example), and the red dashed rectangle represents the corresponding legend area.

[0175] Step S60 , and merging the candidate illustration regions belonging to the same legend to obtain an image-text pair matching result.

[0176] In this embodiment, after matching and filtering in step S50, a legend area may match multiple illustration sub-areas, which correspond to multiple sub-images belonging to the same illustration. Their coordinates need to be merged to obtain the overall bounding box coordinates surrounding these candidate illustration areas, which correspond to the bounding box coordinates of the overall illustration area.

[0177] Step S70, cropping and saving the illustration according to the coordinates, and saving the image-text matching information as structured data.

[0178] After obtaining the coordinates of the illustration area, it is necessary to crop and save the coordinates of the illustration area from the entire document image. In the above steps, in order to speed up the algorithm, this method converts the PDF document into a page image with a lower resolution, such as 100dpi; in this step, it is necessary to regenerate a high-definition page image of a specified resolution, such as 300dpi or 600dpi, according to actual needs. Figure 4 A schematic diagram of an example cropped image provided by the present invention, Figure 4 The example cropped images in are from Figure 3 The illustration image cropped from the three-channel color page image shown in (a) can be cropped from the overall page image according to the coordinates of each detected illustration area, and the illustration image can be obtained respectively. Figure 4 (a) and Figure 4 (b) in the figure, and save it in a specified image format, such as JPG, PNG, TIF, etc. It should be noted that the saved file name is accompanied by some key information, such as the original PDF file name, the page number of the illustration, and the number of the illustration on the page.

[0179] In addition, in addition to saving each independent illustration file, this method also generates a structured text file for each PDF document, such as a JSON file, which stores all illustration and legend information in the PDF document, including the saved file name of the illustration, the legend text of the illustration, the page to which the illustration belongs, the coordinates of the illustration in the original PDF document, etc., to facilitate subsequent application modules to read and call. Here, only JSON format files are used as examples, but actual applications are not limited to this file format. Of course, other file formats can also be used, which are not listed here.

[0180] The method of this embodiment combines PDF source code parsing and image processing technology, which can quickly and accurately extract illustrations and legends in PDF documents, obtain their matching relationships and save them as structured data. This embodiment does not rely on complex text detection and recognition technology and target detection technology based on deep learning, and does not require any labeled data for model training, which greatly reduces the cost of computing resources, hardware costs and labor costs, so that it can be applied to the processing needs of massive PDF documents. At the same time, this embodiment uses open source tools to parse PDF documents, which can accurately obtain the text and coordinates of the legend, avoiding errors that may be caused by other text detection and recognition algorithms; this embodiment uses image processing technology based on connected component analysis to extract illustration areas, which can ensure that the extracted illustration area coordinates are relatively accurate, avoiding the problem of inaccurate target border extraction by target detection algorithms based on deep learning.

[0181] The device for extracting image-text pairs from a PDF document provided by the present invention is described below. The device for extracting image-text pairs from a PDF document described below and the method for extracting image-text pairs from a PDF document described above can be referred to each other.

[0182] Figure 5 The schematic diagram of the structure of the device for extracting image and text pairs from a PDF document provided by the present invention is as follows: Figure 5 As shown, the device includes the following modules:

[0183] The parsing and conversion module 500 is used to parse the target PDF document, obtain text block data and text coordinates corresponding to the text block data, and convert the target PDF document page by page into a document image;

[0184] The screening and extraction module 510 is used to screen and remove duplicate text block data, obtain candidate legend texts, and determine candidate legend text coordinates corresponding to the candidate legend texts, and process the document image based on image processing technology to extract candidate illustration areas;

[0185] The matching and merging module 520 is used to match the candidate legend text and the candidate illustration area based on the candidate legend text coordinates, filter the candidate legend text and the candidate illustration area that failed to match, and merge the candidate illustration areas belonging to the same legend to obtain the image-text pair matching result of the target PDF document.

[0186] According to a device for extracting image-text pairs from a PDF document provided by the present invention, the device further comprises a storage module for:

[0187] Based on the coordinates of the illustration area in the image-text pair matching result, crop the corresponding illustration area from the target PDF document and save it as an independent illustration file; and / or,

[0188] Save the image-text pair matching results as structured data.

[0189] According to a device for extracting image-text pairs from a PDF document provided by the present invention, a document image is processed based on image processing technology to extract a candidate illustration area, specifically comprising:

[0190] Gray-scaling and binarization processing is performed on the document image to obtain a first binary image;

[0191] After deleting the pixels of the candidate legend text from the first binary image, extracting the first connected component from the first binary image, and filling the coordinates of the bounding box of the first connected component to obtain a second binary image;

[0192] Performing connected component extraction on the second binary image to obtain a second connected component;

[0193] Based on the second connected component and a preset threshold, a small connected component set is obtained;

[0194] Based on the second connected component and the small connected component set, a third binary image and a fourth binary image are generated respectively, and a morphological transformation is performed on the third binary image and the fourth binary image;

[0195] respectively extracting connected components from the third binary image after the morphological transformation and the fourth binary image after the morphological transformation to obtain a third connected component and a fourth connected component;

[0196] Filter the fourth connected component from the third connected component to obtain a fifth connected component;

[0197] Based on the fifth connected component, generate a fifth binary image, and perform morphological transformation on the fifth binary image;

[0198] Connected components are extracted from the fifth binary image after the morphological transformation to obtain a sixth connected component, and the sixth connected component is used as a candidate illustration region.

[0199] According to a device for extracting image-text pairs from a PDF document provided by the present invention, morphological transformation is performed on the third binary image and the fourth binary image, specifically comprising:

[0200] Based on the size of the first kernel function, performing a morphological closing operation on the third binary image and the fourth binary image;

[0201] The fifth binary image is subjected to morphological transformation, specifically including:

[0202] Based on the second kernel function size, performing a morphological closing operation on the fifth binary image;

[0203] The size of the second kernel function is larger than that of the first kernel function.

[0204] According to a device for extracting image-text pairs from a PDF document provided by the present invention, matching candidate legend texts with candidate illustration regions based on candidate legend text coordinates specifically includes:

[0205] Based on the candidate legend text coordinates, determining that one or more candidate legend texts exist on the current page of the target PDF document;

[0206] In the case that there is only one candidate legend text on the current page, all candidate illustration areas on the current page are matched with the candidate legend text;

[0207] When there are multiple candidate legend texts on the current page, the candidate legend texts and the candidate illustration areas are matched based on the coordinates of each candidate legend text and the coordinates of each candidate illustration area.

[0208] According to a PDF document image-text pair extraction device provided by the present invention, the image-text pair matching result is saved as structured data, specifically comprising:

[0209] Generate a structured text file of the target PDF document based on the image-text pair matching results;

[0210] Among them, the structured text file includes the file name of the independent illustration file of the target PDF document, the legend text in the image-text pair matching result, the page where the illustration area in the image-text pair matching result is located, and the coordinates of the illustration area in the image-text pair matching result in the target PDF document.

[0211] Figure 6 A schematic diagram of the structure of an electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the method for extracting the image-text pair of the PDF document, and the method includes:

[0212] Parsing the target PDF document to obtain text block data and text coordinates corresponding to the text block data, and converting the target PDF document page by page into a document image;

[0213] Screening and deduplicating the text block data, obtaining candidate legend texts, and determining candidate legend text coordinates corresponding to the candidate legend texts, and processing the document image based on image processing technology to extract candidate illustration areas;

[0214] Based on the candidate legend text coordinates, the candidate legend texts and candidate illustration areas are matched, the candidate legend texts and candidate illustration areas that fail to match are filtered out, and the candidate illustration areas belonging to the same legend are merged to obtain the image-text pair matching results of the target PDF document.

[0215] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0216] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the PDF document image-text pair extraction method provided by the above methods, the method includes:

[0217] Parsing the target PDF document to obtain text block data and text coordinates corresponding to the text block data, and converting the target PDF document page by page into a document image;

[0218] Screening and deduplicating the text block data, obtaining candidate legend texts, and determining candidate legend text coordinates corresponding to the candidate legend texts, and processing the document image based on image processing technology to extract candidate illustration areas;

[0219] Based on the candidate legend text coordinates, the candidate legend texts and candidate illustration areas are matched, the candidate legend texts and candidate illustration areas that fail to match are filtered out, and the candidate illustration areas belonging to the same legend are merged to obtain the image-text pair matching results of the target PDF document.

[0220] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the PDF document image-text pair extraction method provided by the above methods, the method comprising:

[0221] Parsing the target PDF document to obtain text block data and text coordinates corresponding to the text block data, and converting the target PDF document page by page into a document image;

[0222] Screening and deduplicating the text block data, obtaining candidate legend texts, and determining candidate legend text coordinates corresponding to the candidate legend texts, and processing the document image based on image processing technology to extract candidate illustration areas;

[0223] Based on the candidate legend text coordinates, the candidate legend texts and candidate illustration areas are matched, the candidate legend texts and candidate illustration areas that fail to match are filtered out, and the candidate illustration areas belonging to the same legend are merged to obtain the image-text pair matching results of the target PDF document.

[0224] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0225] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0226] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting image-text pairs of a portable document format, characterized in that: include: Parsing the target PDF document to obtain text block data and text coordinates corresponding to the text block data, and converting the target PDF document page by page into a document image; Screening and deduplicating the text block data to obtain candidate legend texts, and determining candidate legend text coordinates corresponding to the candidate legend texts, and processing the document image based on image processing technology to extract candidate illustration areas; Based on the candidate legend text coordinates, the candidate legend text and the candidate illustration area are matched, the candidate legend text and the candidate illustration area that fail to match are filtered out, and the candidate illustration areas that belong to the same legend are merged to obtain the image-text pair matching result of the target PDF document; The document image is processed based on image processing technology to extract candidate illustration areas, specifically including: grayscale and binarize the document image to obtain a first binary image; After deleting the pixels of the candidate legend text from the first binary image, extracting the first connected component from the first binary image, and filling the coordinates of the bounding box of the first connected component to obtain a second binary image; performing connected component extraction on the second binary image to obtain second connected components; Based on the second connected component and a preset threshold, obtaining a small connected component set; Based on the second connected component and the small connected component set, respectively generate a third binary image and a fourth binary image, and perform morphological transformation on the third binary image and the fourth binary image; respectively extracting connected components from the third binary image after the morphological transformation and the fourth binary image after the morphological transformation to obtain a third connected component and a fourth connected component; Filter the fourth connected component from the third connected component to obtain a fifth connected component; Based on the fifth connected component, generate a fifth binary image, and perform morphological transformation on the fifth binary image; Performing connected component extraction on the fifth binary image after the morphological transformation to obtain a sixth connected component, and using the sixth connected component as a candidate illustration region; Based on the candidate legend text coordinates, matching the candidate legend text with the candidate illustration area specifically includes: Based on the candidate legend text coordinates, determining that there are one or more candidate legend texts on the current page of the target PDF document; In the case that there is only one candidate legend text on the current page, matching all candidate illustration areas on the current page with the candidate legend text; In the case that there are multiple candidate legend texts on the current page, the candidate legend texts and the candidate illustration areas are matched based on the coordinates of each candidate legend text and the coordinates of each candidate illustration area; for each candidate legend text, if the vertical overlap rate between a candidate illustration area and the candidate legend text is greater than a preset threshold, and the lower border of the candidate illustration area is above the upper border of the candidate legend text, and the distance between the lower border of the candidate illustration area and the upper border of the candidate legend text is less than a preset threshold, then the candidate illustration area is matched to the candidate legend text.

2. The method for extracting image-text pairs of a portable document format according to claim 1, characterized in that: After obtaining the image-text pair matching result of the target PDF document, the method further includes: Based on the coordinates of the illustration area in the image-text pair matching result, cutting out the corresponding illustration area from the target PDF document and saving it as an independent illustration file; and / or, The image-text pair matching result is saved as structured data.

3. The method for extracting image-text pairs of a portable document format according to claim 1, characterized in that: Performing morphological transformation on the third binary image and the fourth binary image specifically includes: Based on the first kernel function size, performing a morphological closing operation on the third binary image and the fourth binary image; The performing morphological transformation on the fifth binary image specifically includes: Based on the second kernel function size, performing a morphological closing operation on the fifth binary image; The size of the second kernel function is larger than the size of the first kernel function.

4. The method for extracting image-text pairs of a portable document format according to claim 2, characterized in that: Saving the image-text pair matching result as structured data specifically includes: Based on the image-text pair matching result, generating a structured text file of the target PDF document; Among them, the structured text file includes the file name of the independent illustration file of the target PDF document, the legend text in the image-text pair matching result, the page where the illustration area in the image-text pair matching result is located, and the coordinates of the illustration area in the image-text pair matching result in the target PDF document.

5. A device for extracting document image-text pairs in a portable document format, characterized in that: include: A parsing and conversion module, used to parse the target PDF document, obtain text block data and text coordinates corresponding to the text block data, and convert the target PDF document page by page into a document image; A screening and extraction module, for screening and deduplicating the text block data, obtaining candidate legend texts, and determining candidate legend text coordinates corresponding to the candidate legend texts, and for processing the document image based on image processing technology to extract candidate illustration areas; A matching and merging module, for matching the candidate legend text and the candidate illustration area based on the candidate legend text coordinates, filtering the candidate legend text and the candidate illustration area that failed to match, and merging the candidate illustration areas belonging to the same legend to obtain the image-text pair matching result of the target PDF document; The document image is processed based on image processing technology to extract candidate illustration areas, specifically including: grayscale and binarize the document image to obtain a first binary image; After deleting the pixels of the candidate legend text from the first binary image, extracting the first connected component from the first binary image, and filling the coordinates of the bounding box of the first connected component to obtain a second binary image; performing connected component extraction on the second binary image to obtain second connected components; Based on the second connected component and a preset threshold, obtaining a small connected component set; Based on the second connected component and the small connected component set, respectively generate a third binary image and a fourth binary image, and perform morphological transformation on the third binary image and the fourth binary image; respectively extracting connected components from the third binary image after the morphological transformation and the fourth binary image after the morphological transformation to obtain a third connected component and a fourth connected component; Filter the fourth connected component from the third connected component to obtain a fifth connected component; Based on the fifth connected component, generate a fifth binary image, and perform morphological transformation on the fifth binary image; Performing connected component extraction on the fifth binary image after the morphological transformation to obtain a sixth connected component, and using the sixth connected component as a candidate illustration region; Based on the candidate legend text coordinates, matching the candidate legend text with the candidate illustration area specifically includes: Based on the candidate legend text coordinates, determining that there are one or more candidate legend texts on the current page of the target PDF document; In the case that there is only one candidate legend text on the current page, matching all candidate illustration areas on the current page with the candidate legend text; In the case that there are multiple candidate legend texts on the current page, the candidate legend texts and the candidate illustration areas are matched based on the coordinates of each candidate legend text and the coordinates of each candidate illustration area; for each candidate legend text, if the vertical overlap rate between a candidate illustration area and the candidate legend text is greater than a preset threshold, and the lower border of the candidate illustration area is above the upper border of the candidate legend text, and the distance between the lower border of the candidate illustration area and the upper border of the candidate legend text is less than a preset threshold, then the candidate illustration area is matched to the candidate legend text.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for extracting document image-text pairs in a portable document format as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for extracting document image-text pairs in a portable document format as claimed in any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for extracting document image-text pairs in a portable document format as claimed in any one of claims 1 to 4 is implemented.

Images