Method of structuring PDF files based on OCR and large models

Abstract

The present application provides a method for structuring a PDF file based on OCR and a large model, which includes: traversing a single-page PDF in the PDF file, and performing layout analysis and content recognition on the single-page PDF based on an OCR model to determine the embedded images and all plain text blocks of the single-page PDF; enhancing the embedded images based on a multimodal large language model to generate enhanced text content in a targeted manner; calculating the positional association relationship between the enhanced text content and the plain text block; vectorizing the embedded images and the corresponding enhanced text content, the plain text block and the corresponding text content, respectively, to obtain an embedded image description vector and a plain text block description vector; generating structured data of the single-page PDF based on the embedded image description vector, the positional association relationship, and the plain text block description vector; generating structured data of the PDF file based on the structured data of all page PDFs in the PDF file.

Description

Technical Field

[0001] The present application relates to the field of large model technology, and in particular to a method for structuring PDF files based on OCR and large models. Background Art

[0002] RAG (Retrieval-Augmented Generation) technology is an innovative method that combines information retrieval and generative models to improve the accuracy and relevance of text generation. Specifically, the RAG model first uses a retrieval component to filter out documents that are closely related to the input query from a large collection of documents. These documents are then used as input to a generative model (such as GPT-3) to produce high-quality, targeted answers. This method shows significant advantages when dealing with generation tasks that require deep background knowledge or detailed contextual information.

[0003] 1. Retrieval phase: One of the core functions of the RAG model is to accurately select documents that match a specific query from a pre-built knowledge base through its built-in retrieval component. These documents not only provide rich background materials and specific details for the generative model, but also significantly enhance the model's understanding of the query content.

[0004] 2. Generative phase: After successfully retrieving relevant documents, the RAG model immediately starts its generative components (such as the large language model GPT) to create answers based on the content of these documents. The generative model cleverly combines the retrieved information with its own language generation capabilities, and ultimately outputs text that is both accurate and coherent.

[0005] It is worth noting that the quality of the knowledge base has a crucial impact on the quality of the answers generated by the RAG model. In current technical practices, the knowledge base is usually formed by parsing various types of documents (such as PDF files, etc.) and identifying the content in them to construct structured data. However, due to the complexity of the content of PDF files (including plain text, tabular images, plain images, etc.), it is relatively difficult to structure them, which may lead to the low quality of the final structured data, thereby indirectly affecting the overall quality of the knowledge base. Summary of the invention

[0006] The purpose of this application is to propose a method for structuring PDF files based on OCR and large models, so as to solve or alleviate the technical problems existing in the prior art.

[0007] A method for structuring a PDF file based on OCR and a large model, comprising:

[0008] Traversing a single-page PDF in the PDF file, and performing layout analysis and content recognition on the single-page PDF based on an OCR model to determine embedded images and all plain text blocks of the single-page PDF;

[0009] Performing enhancement processing on the embedded image based on a multimodal large language model to generate enhanced text content in a targeted manner;

[0010] Calculating a position association relationship between the enhanced text content and the plain text block;

[0011] Vectorizing the embedded image and the corresponding enhanced text content, the plain text block and the corresponding text content respectively to obtain an embedded image description vector and a plain text block description vector;

[0012] Generate structured data of the single-page PDF based on the embedded image description vector, the position association relationship, and the plain text block description vector;

[0013] The structured data of the PDF file is generated based on the structured data of all the pages in the PDF file.

[0014] The solution of the embodiment of the present application has the following technical advantages:

[0015] (1) First, the OCR (Optical Character Recognition) model performs a detailed layout analysis on each page in the PDF file to accurately identify embedded images and plain text blocks. The OCR model is able to handle complex layouts, including text in different fonts, sizes, and arrangements, thus ensuring comprehensive and accurate recognition of PDF content.

[0016] (2) Traditional OCR methods may not be able to effectively recognize embedded images in PDFs, especially those containing tables, charts, or complex graphics. This solution introduces a multimodal large language model and uses its deep understanding of image content to generate enhanced text content related to the image content. This greatly improves the extraction rate of image information and enables the effective use of image content that was originally difficult to structure.

[0017] (3) By calculating the positional association between the enhanced text content and the plain text blocks, the scheme ensures the spatial consistency of the structured data. This helps to maintain the logical structure and information integrity of the original PDF file in the subsequent knowledge base construction, thereby improving the overall quality of the knowledge base.

[0018] (4) Vectorization is a key step in converting unstructured data into structured data. This solution vectorizes all the identified content (including images, enhanced text, and plain text) to generate description vectors. These vectors are not only easy to store and retrieve, but also provide a basis for subsequent structured data generation.

[0019] (5) Using the description vectors and position associations generated in the previous steps, the content of a single page of PDF is integrated into structured data. This structured representation makes the information clearer and easier to query and use.

[0020] (6) Finally, the structured data of each page is integrated to form the structured data of the entire PDF file. In this way, the originally complex and difficult to process PDF files are converted into high-quality, easy-to-use structured data, providing a richer and more accurate background knowledge base for the RAG model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Hereinafter, some specific embodiments of the present application will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:

[0022] Figure 1 This is a flow chart of a method for structuring PDF files based on OCR and a large model according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0024] Figure 1 This is a flow chart of a method for structuring PDF files based on OCR and large models according to an embodiment of the present application. Figure 1 As shown, the method for structuring PDF files based on OCR and large models includes:

[0025] Traversing a single-page PDF in the PDF file, and performing layout analysis and content recognition on the single-page PDF based on an OCR model to determine embedded images and all plain text blocks of the single-page PDF;

[0026] Performing enhancement processing on the embedded image based on a multimodal large language model to generate enhanced text content in a targeted manner;

[0027] Calculating a position association relationship between the enhanced text content and the plain text block;

[0028] Vectorizing the embedded image and the corresponding enhanced text content, the plain text block and the corresponding text content respectively to obtain an embedded image description vector and a plain text block description vector;

[0029] Generate structured data of the single-page PDF based on the embedded image description vector, the position association relationship, and the plain text block description vector;

[0030] The structured data of the PDF file is generated based on the structured data of all the pages in the PDF file.

[0031] The solution of the embodiment of the present application realizes efficient and accurate recognition and structuring of PDF file content by combining the OCR model with the multimodal large language model. The OCR model can comprehensively analyze the PDF layout and accurately identify embedded images and plain text blocks, while the multimodal large language model further improves the extraction rate of image information and converts difficult-to-structure image content into usable enhanced text. The solution also ensures the spatial consistency of structured data by calculating position associations, vectorizes all recognized content to generate description vectors that are easy to store and retrieve, and finally integrates this information into high-quality, easy-to-use structured data. This not only maintains the logical structure and information integrity of the original PDF file, but also provides a richer and more accurate background knowledge base for the RAG model, significantly improving the overall quality and practicality of the knowledge base.

[0032] Optionally, before the embedded image is enhanced based on the multimodal large language model to generate enhanced text content in a targeted manner, the method includes: performing global text extraction on the single-page PDF to use all the obtained text contents as the first text content;

[0033] When the embedded image is enhanced based on the multimodal large language model to generate enhanced text content in a targeted manner, if the embedded image is a table-type image, the text content in the table is extracted as the third text content, and the enhanced text content includes the third text content;

[0034] After the embedded image is enhanced based on the multimodal large language model to generate enhanced text content in a targeted manner, the method further includes: calculating the similarity between the first text content and the third text content, correcting the third text content based on the first text content when the similarity is greater than a set similarity threshold, and determining the first text content when the similarity is less than the set similarity threshold as the text content corresponding to all the plain text blocks and using it as the fourth text content.

[0035] To this end, before the embedded image is enhanced based on the multimodal large language model, global text extraction is first performed on the single-page PDF to obtain all text content as the first text content. This step can preliminarily screen out most of the text information on the PDF page and provide a comprehensive text basis for subsequent steps. Through global text extraction, it is possible to avoid missing any important text in subsequent processing and improve the integrity of information extraction. In addition, when the embedded image is a table-like image, the scheme specifically extracts the text content in the table as the third text content and incorporates it into the enhanced text content. This specific processing for table-like images fully considers the importance and particularity of the table in the PDF file. Tables usually contain structured data, which is of great value for subsequent structured data generation and knowledge base construction. By specifically extracting the table content, it can be ensured that these key information will not be missed or misidentified. Furthermore, after generating the enhanced text content, the scheme calculates the similarity between the first text content and the third text content (i.e., the text extracted from the table). The purpose of this step is to verify and correct the accuracy of the table text content. When the similarity is greater than the set similarity threshold, the corresponding part in the first text content is considered to match the table text content and can be used to correct the table text. This correction process helps to eliminate recognition errors or inconsistencies that may exist in the table text and improve the accuracy of the text content. At the same time, for the first text content whose similarity is less than the set similarity threshold, the solution determines it as the text content corresponding to the plain text block (i.e., the fourth text content), which helps to distinguish between table text and other ordinary text content, and provide a clearer classification for subsequent structured data generation.

[0036] Optionally, calculating the positional association relationship between the enhanced text content and the plain text block includes: determining a text block that is closest to the embedded image among all the plain text blocks, and establishing a positional association relationship between the enhanced text content and the four text contents based on the position of the closest text block.

[0037] By determining the text block closest to the embedded image and establishing a position association relationship between the enhanced text content and the text content corresponding to the plain text block based on its position, the following technical benefits are achieved:

[0038] (1) When calculating the positional association between the enhanced text content and the plain text block, the solution specifically selects the text block closest to the embedded image as the reference point. This step is based on the fact that, in general, text blocks that are closely adjacent to an image tend to have a higher correlation with the image content. By determining the nearest text block and establishing a positional association between the enhanced text content and the plain text content (here it should be the "fourth text content" mentioned above, but for consistency, it is still expressed as "the text content corresponding to the plain text block") based on its position, the logical structure and spatial relationship between the content in the original PDF file can be more accurately reflected.

[0039] (2) Accurately establishing positional associations is crucial for the subsequent generation of structured data. By considering the relative positions between text blocks and images, we can ensure that the layout and arrangement of the original content are correctly reflected in the structured data. This helps maintain the integrity and accuracy of the information and avoids the loss or misunderstanding of important information during data conversion.

[0040] (3) Accurate positional associations also help enhance the practicality of the knowledge base. When the RAG model uses the knowledge base to generate text, it can take into account the spatial relationship and logical connection between the content, thereby generating more coherent and consistent answers. This is particularly important for generation tasks that require deep background knowledge or detailed contextual information, and can significantly improve the quality and relevance of the generated text.

[0041] Optionally, when the embedded image is enhanced based on the multimodal large language model to generate enhanced text content in a targeted manner, if the embedded image is a pure image, semantic analysis is performed on the pure image to generate a description text of the pure image as the second text content, and the enhanced text content also includes the second text content.

[0042] By performing semantic analysis on pure images and generating description text, the following technical benefits are achieved:

[0043] (1) When the embedded image is a pure image (i.e., an image that does not contain any text or table content), the solution uses a multimodal large language model to perform semantic analysis on it to generate a description text of the pure image and include it as the second text content in the enhanced text content. This step is based on the consideration that although pure images do not contain directly readable text information, their visual content often carries important information and meaning, which is crucial for understanding the overall content and background of the PDF file.

[0044] (2) By generating pure image description text through semantic analysis, the solution can extract key information and features in the image, such as objects, scenes, colors, etc., thereby improving the comprehensiveness of information extraction. These description texts not only enrich the enhanced text content, but also provide more useful information for subsequent structured data generation and knowledge base construction.

[0045] (3) Incorporating the description text of pure images into the enhanced text content can further enhance the representation ability of the knowledge base. When the RAG model uses the knowledge base to generate text, it can take into account the information and artistic conception conveyed by the pure images, thereby generating more vivid, vivid and appropriate answers. This is particularly important for generation tasks that require describing or explaining the content of the image, and can significantly improve the quality and relevance of the generated text.

[0046] (3) For users, the description text of pure images can provide additional information and explanations, helping users better understand the content and background of PDF files. This improves the user experience to a certain extent, allowing users to more easily obtain and use the information in PDF files.

[0047] Optionally, performing global text extraction on the single-page PDF to use all obtained text contents as first text contents includes:

[0048] Calling the created PDF parsing library to perform structure and format parsing on the single-page PDF to generate a PDF page content instruction sequence;

[0049] The created PDF text instruction set is called to analyze the text index structure of the PDF page content instruction sequence, so as to extract all text contents on the single PDF page as the first text content according to the text index structure.

[0050] By calling a professional PDF parsing library and text instruction set to perform global text extraction, the following technical benefits are achieved:

[0051] (1) By calling a specially created PDF parsing library, the structure and format of a single-page PDF are parsed to generate a PDF page content instruction sequence. Subsequently, the text index structure of these instruction sequences is analyzed using the created PDF text instruction set, so as to accurately extract all the text content on the single-page PDF as the first text content.

[0052] (2) Through professional PDF parsing libraries and text instruction sets, the solution can more accurately parse the structure and format of PDF files, ensuring that the extracted text content is complete and correct. This avoids the problems of format confusion and text omissions that may be encountered in traditional text extraction methods, and improves the accuracy of text extraction.

[0053] (3) PDF files often contain complex structures and formats, such as multiple layers of nesting, different fonts and typesetting methods, etc. The solution can better handle these complex situations through professional parsing libraries and instruction sets, ensuring the comprehensiveness and accuracy of text extraction. This is especially important for processing PDF files containing large amounts of text and complex layouts.

[0054] (4) Accurate and comprehensive text extraction provides a solid foundation for subsequent data processing and text generation. By extracting high-quality first text content, the solution can more efficiently perform subsequent tasks such as structured data generation, knowledge base construction, and text generation, thereby improving overall processing efficiency.

[0055] (5) The created PDF parsing library and text instruction set have good scalability and maintainability. When the format or structure of the PDF file changes, the parsing library and instruction set only need to be updated and adjusted accordingly to adapt to the new requirements. This reduces the maintenance cost of the solution and provides greater flexibility for its application in different fields.

[0056] Optionally, the performing layout analysis and content recognition on the single-page PDF based on the OCR model to determine the embedded images and all plain text blocks of the single-page PDF includes:

[0057] Calling a PDF rendering library to convert the single-page PDF into a page image, and preprocessing the page image to obtain a preprocessed page image;

[0058] Performing grayscale histogram statistics on the preprocessed page image, and performing feature energy analysis based on the histogram, so as to perform layout analysis on the single-page PDF to obtain multiple layout content areas;

[0059] Color space conversion is performed for each layout content area to determine the embedded images and all plain text blocks of the single-page PDF based on the hue, transparency and brightness data of each layout content area in the target color space.

[0060] The technical benefits of performing layout analysis and content recognition steps on a single-page PDF based on the OCR model are as follows:

[0061] (1) Through grayscale histogram statistics and feature energy analysis, different content areas in PDF pages can be identified more accurately. Grayscale histogram can reflect the grayscale distribution of an image, while feature energy analysis can further extract feature information in the image, thereby helping to distinguish different types of content areas such as text, pictures, and tables. This layout analysis method based on image processing has higher flexibility and accuracy than traditional rule-based or template-based methods.

[0062] (2) Color space conversion can more accurately identify the color characteristics of different content areas. By comparing the hue, transparency, and brightness data of different content areas in the target color space, text and image areas can be distinguished more effectively. This method is particularly effective for processing PDF files containing complex backgrounds, multiple colors, or font styles, and can significantly enhance the processing capabilities of complex PDFs.

[0063] (3) Since the text and image areas have been accurately distinguished through layout analysis, the OCR model and image recognition model can be more effectively applied to recognize text and images. This targeted processing can significantly improve the accuracy of text and image recognition and reduce the occurrence of misidentification or missed recognition.

[0064] (4) Accurate layout analysis and content recognition results can optimize subsequent processing flows. For example, when generating enhanced text content, text and image information can be extracted and integrated more accurately to generate content that better meets user needs. In addition, accurate layout information can also be used for tasks such as automatic classification, archiving, and retrieval of PDF documents, improving the efficiency and accuracy of document management.

[0065] Optionally, the method of performing enhancement processing on the embedded image based on the multimodal large language model to generate enhanced text content in a targeted manner includes the following steps:

[0066] Converting the embedded image into a linear embedding input sequence of the multimodal large language model;

[0067] Performing feature extraction based on position embedding on the linear embedding input sequence to obtain spatial features and hierarchical features of the embedded image;

[0068] Calculating the semantic dependency relationship between the spatial features and the hierarchical features to obtain a semantic feature vector;

[0069] Based on the semantic feature vector, the embedded image is enhanced to obtain the enhanced text content.

[0070] Based on the provided technical solution, the following are the technical benefits of the embedded image enhancement processing steps based on the multimodal large language model:

[0071] The technical benefits of the embedded image enhancement processing step based on a multimodal large language model are as follows:

[0072] (1) By converting embedded images into linear embedding input sequences of a multimodal large language model, the model is able to process image data more efficiently. Subsequently, feature extraction using position embedding can capture the spatial and hierarchical features of the image, which helps the model understand the image content more comprehensively. This processing method fully utilizes the advantages of the multimodal large language model in processing complex data and significantly improves the accuracy and efficiency of image enhancement processing. Compared with traditional image recognition methods, it can more accurately determine whether the embedded image is a pure image or a table-like image.

[0073] (2) Calculate the semantic dependency between spatial features and hierarchical features to obtain semantic feature vectors. This process enables the model to deeply explore the semantic information in the image content. This not only enhances the model's ability to understand the image content, but also enables the model to more accurately grasp the key information in the image, thereby making more precise judgments. This is of great significance for improving the accuracy and reliability of image recognition.

[0074] (3) The multimodal large language model itself has strong generalization capabilities and can process multiple types of data. This advantage of the model can be further exploited by converting embedded images into linear embedding input sequences and using position embedding for feature extraction. This processing method enables the model to better adapt to different types of image data and improves the ability to recognize unknown images. This is of great significance for improving the practicality and application scope of the model.

[0075] (4) Different types of images can be processed differently. For example, for pure images, detailed description text can be generated; for table images, structured data in the table can be extracted. This targeted processing method optimizes the subsequent processing flow and improves processing efficiency and accuracy. At the same time, it also provides a richer source of information for subsequent data analysis and application.

[0076] Optionally, the calculating the semantic dependency relationship between the spatial features and the hierarchical features to obtain a semantic feature vector includes:

[0077] Performing phase perception on the spatial features and the hierarchical features to align the spatial features and the hierarchical features;

[0078] Performing neighborhood propagation on the aligned spatial features and hierarchical features to construct a multi-hop attention intensity transmission queue;

[0079] Performing semantic dependency analysis on the multi-hop attention strength transfer queue to create a semantic dependency meta-graph;

[0080] An adjacency matrix and node features of the semantic dependency metagraph are determined to calculate the semantic feature vector based on the adjacency matrix and the node features.

[0081] To this end, the following technical benefits are achieved through the steps of stage-aware alignment, neighborhood propagation, semantic dependency analysis, and semantic feature vector calculation:

[0082] (1) By performing stage-aware alignment on spatial and hierarchical features, the semantic consistency and complementarity of these features can be ensured. Stage-aware alignment takes into account the different stages and levels in the feature extraction process, thereby more accurately capturing the intrinsic connections between features. Accurate alignment is the basis for subsequent processing steps, which helps reduce redundancy and conflict between features and improves the efficiency and accuracy of subsequent processing.

[0083] (2) By performing neighborhood propagation on the aligned spatial features and hierarchical features and constructing a multi-hop attention intensity transmission queue, the semantic association between features can be further enhanced. Neighborhood propagation allows information to flow and interact between features, thereby revealing deeper semantic dependencies. The enhanced semantic association helps the model understand the image content more accurately and improves the accuracy of the enhancement processing. At the same time, it also provides richer information for subsequent semantic analysis and applications.

[0084] (3) Performing semantic dependency analysis on the multi-hop attention intensity transfer queue can create a semantic dependency metagraph. Semantic dependency analysis is a grammatical analysis method that reveals the dependency relationship between words. It helps the model to more accurately understand the semantic structure in the image content. Accurate semantic dependency analysis is one of the key steps in extracting semantic feature vectors. It helps the model to more accurately capture the semantic information in the image content and provide stronger support for subsequent processing and application.

[0085] (4) By determining the adjacency matrix and node features of the semantic dependency metagraph, the semantic feature vector can be calculated based on this information. The adjacency matrix describes the connection relationship between nodes in the metagraph, while the node features represent the semantic information of each node. The optimized semantic feature vector calculation method fully considers the semantic structure in the image content and the intrinsic connection between features, so that more representative semantic feature vectors can be extracted. These feature vectors not only help the model understand the image content more accurately, but also provide stronger support for subsequent processing and application.

[0086] Optionally, the adjacency matrix and node features of the semantic dependency metagraph are determined based on a graph convolutional network using the following formula to calculate the semantic feature vector based on the adjacency matrix and node features, including:

[0087] ,in, is the first The node characteristics of the layer, is an integer greater than 0, is the first The node characteristics of the layer, is the adjacency matrix Plus self-loop , yes The degree matrix of It is The weight matrix of the layer, is the activation function, and the semantic feature vector is calculated using the node features of the last layer in the graph convolutional network and is denoted as .

[0088] To this end, in the framework of graph neural networks (GNNs), especially when applied to determine whether an embedded image is a pure image or a table-like image, using the weight matrix W(k) of the kth layer and the related feature conversion and update mechanism has the following technical benefits:

[0089] (1) Through the weight matrix of the kth layer, GNN can effectively extract and transform the specific features of the embedded image. This targeted processing enables the model to more accurately capture the key information in the image, whether it is the visual elements in the pure image or the structured data in the table image.

[0090] (2) The multi-layer structure of GNN allows the model to capture the local neighborhood information of nodes at each layer and gradually integrate global information by stacking layers. For embedded images, this means that the model can consider both the local details and the overall structure of the image, thereby making more accurate classification decisions.

[0091] (3) The learning process of the weight matrix W (k) enables GNN to capture the complex dependencies between nodes (in this case, pixels or feature points in the image). This ability is crucial for distinguishing between pure images and table images, as the two types of images have significant differences in visual elements and structures.

[0092] (4) Since the weight matrix is ​​learned during the training process through the back-propagation algorithm, GNN can adaptively adjust the feature learning method according to the specific task and data set. This adaptability enables the model to maintain high performance when processing different types of embedded images.

[0093] (5) Through multi-layer feature conversion and updating, GNN can learn richer node representations that contain key information and features in the image. These rich feature representations help improve the accuracy of the model in judging the type of embedded images.

[0094] (6) The multi-layer structure and weight matrix learning mechanism of GNN make the model robust and generalizable when dealing with noise and unseen data. This is very important for processing various complex and diverse embedded images in practical applications.

[0095] Optionally, the enhancing processing of the embedded image based on the semantic feature vector includes: calculating a manifold description feature of the embedded image based on the semantic feature vector, and enhancing the embedded image based on the manifold description feature to obtain the enhanced text content.

[0096] The manifold description features of the embedded image are calculated based on the semantic feature vector, and these features are used for classification prediction to determine whether the embedded image is a pure image or a table-like image. This has the following technical benefits:

[0097] (1) Manifold description features can capture low-dimensional manifold structures in high-dimensional data. In image recognition, they can reveal the intrinsic structure of image data, such as shape, texture, and layout. By calculating manifold description features, the model can better understand the geometric and topological properties of embedded images.

[0098] (2) Manifold features usually have stronger expressive power and can capture nonlinear relationships and complex patterns in images. Compared with traditional linear features, manifold features can more comprehensively describe the content of an image. The enhanced feature expressive power makes the model more sensitive and accurate in distinguishing between pure images and table-like images.

[0099] (3) Manifold description features have advantages in dealing with noise and redundant information in high-dimensional data. By calculating manifold description features, the model can better cope with noise and changes in image data and improve the robustness of classification.

[0100] (4) Manifold description features perform well when processing data with complex patterns. In image recognition, pure images and table-like images often have different visual patterns and layout features. By calculating manifold description features, the model can better recognize these complex patterns, thereby showing higher accuracy and stability when enhancing pure images and table-like images.

[0101] Optionally, the determining the text block closest to the embedded image among all the plain text blocks, and establishing a position association relationship between the enhanced text content and the four text contents based on the position of the closest text block, includes:

[0102] Calculate the Euclidean distance between the center of each plain text block and the center of the embedded image, select the plain text block with the smallest Euclidean distance as the text block closest to the embedded image, and mark the position direction of the closest text block and the embedded image;

[0103] Based on the position direction of the nearest text block and the embedded image and their corresponding Euclidean distances, a position association relationship between the enhanced text content and the four text contents is established.

[0104] By calculating the Euclidean distance between the center of each plain text block and the center of the embedded image to determine the position association relationship, the following technical benefits are achieved:

[0105] (1) By calculating the Euclidean distance, we can accurately find the plain text block closest to the embedded image. This method is based on geometric distance and provides an objective and accurate metric. Accurate location helps subsequent processing steps, such as text-image association analysis and content layout, to ensure accuracy and consistency of processing.

[0106] (2) By marking the position and direction of the nearest text block and embedded image, the spatial relationship between them can be intuitively understood. This intuitive position association helps users or subsequent processing systems quickly understand the structure and content of the document, especially in application scenarios that require analyzing the relationship between text and images.

[0107] (3) By establishing a positional relationship between the enhanced text content and the four text contents, the text information around the image can be integrated to provide more clues for understanding the image content and context. Enhanced content understanding helps improve the intelligent level of document processing, such as performance and accuracy in applications such as automatic summarization, information retrieval, and content recommendation.

[0108] (4) By calculating the Euclidean distance and marking the location and direction, the document processing and analysis process can be simplified, reducing manual intervention and subjective judgment. The simplified processing process helps to improve processing efficiency and accuracy and reduce processing costs and time costs, especially in application scenarios that require processing a large number of documents.

[0109] Optionally, the edit distance between characters in the first text content and the third text content is calculated according to the following formula, so that the edit distance obtained in the last iteration is used as the edit distance between characters in the first text content and the third text content. , is the length of the first text content, is the length of the third text content:

[0110]

[0111] in, represents the minimum edit distance between the first i characters of the first text content S1 and the first j characters of the third text content S2, represents the weight of the kth character in S1, represents the weight of the kth character in S2, represents the context influence factor when i=0 and j=0, represents the sum of the weights of the first i characters in S1, represents the sum of the weights of the first j characters in S2, represents the character at position i in S1, represents the character at position j in S2, represents the contextual influence factor between the i-th character in S1 and the j-th character in S2, It is the penalty value calculated when the i-th character in S1 does not match the j-th character in S2. The penalty value reflects the difference in character weights and contextual influence factors. represents the minimum edit distance between the first i−1 characters of S1 and the first j−1 characters of S2; α is used to adjust the importance of weight difference in calculating penalty value, and β is used to adjust the importance of context influence factor in calculating penalty value. represents the weight of the i-th character in S1 and the weight of the jth character in S2 The absolute difference of .

[0112] To this end, based on and Represents the weights of the characters in strings S1 and S2. By assigning different weights to different characters, the semantic or grammatical importance of the characters can be reflected. For example, nouns or verbs may have higher weights than prepositions or articles because they carry more information in a sentence. The introduction of weights enables the edit distance to more accurately capture the important differences between strings, thereby giving more consideration to key information when comparing.

[0113] The parameter α adjusts the influence of weight differences in the edit distance calculation. By adjusting α, the influence of weight differences on the final edit distance can be controlled, allowing the algorithm to flexibly adapt to different application scenarios.

[0114] The parameter β is used to adjust the influence of the context influence factor in the edit distance calculation. Taking into account the relationship between characters in a specific context, the introduction of β allows the algorithm to adjust the edit distance according to the contextual information, thus improving the accuracy of the comparison.

[0115] The introduction of factors representing contextual influence enables the algorithm to identify and process differences that are significant only in specific contexts, thereby increasing the algorithm's sensitivity to contextual changes.

[0116] when When the two are the same, a penalty value is calculated; when the two are the same, the penalty value is 0. This conditional expression enables the algorithm to distinguish between character matches and mismatches. For mismatches, by introducing weight differences and context influence factors, the "cost" required to convert one character into another can be more accurately calculated.

[0117] Optionally, the similarity between the first text content and the third text content is calculated according to the following formula: : , is the average of all character weights in S1 and S2, The range used to adjust the similarity score.

[0118] To do this, divide the edit distance by , the edit distance is normalized so that the similarity score is proportional to the length of the string and the average weight of the characters. This allows for a fair benchmark when comparing strings of different lengths. The parameter γ allows for further tuning of the range of similarity scores. If desired, the sensitivity of the score can be increased or decreased by adjusting γ to suit specific application needs. Edit Distance The weights of characters and contextual influence factors are taken into account, so that the similarity score can more accurately reflect the actual similarity between strings. In addition, the above algorithm balances the impact of string length and content differences on the similarity score. Longer strings are not necessarily dissimilar to shorter strings, and through normalization, their similarities can be evaluated more fairly. Furthermore, since multiple parameters are included in the formula, it provides a high degree of flexibility and can be adjusted according to different application scenarios and data sets. For example, in some cases, it may be necessary to consider weight differences more, while in other cases, it may be necessary to consider contextual influence more.

[0119] Based on the above method for structuring a PDF file based on OCR and a large model, an embodiment of the present application further provides a device for structuring a PDF file based on OCR and a large model, which includes:

[0120] The first program unit is used to traverse a single-page PDF in the PDF file, and perform layout analysis and content recognition on the single-page PDF based on an OCR model to determine embedded images and all plain text blocks of the single-page PDF;

[0121] A second program unit is used to enhance the embedded image based on a multimodal large language model to generate enhanced text content in a targeted manner;

[0122] A third program unit is used to calculate the position association relationship between the enhanced text content and the plain text block;

[0123] A fourth program unit is used to vectorize the embedded picture and the corresponding enhanced text content, the plain text block and the corresponding text content, respectively, to obtain an embedded picture description vector and a plain text block description vector;

[0124] A fifth program unit, configured to generate structured data of the single-page PDF based on the embedded image description vector, the position association relationship, and the plain text block description vector;

[0125] The sixth program unit is used to generate the structured data of the PDF file based on the structured data of all the pages PDF in the PDF file.

[0126] The exemplary explanation of each program unit in the above-mentioned device based on OCR and large model structured PDF file can be found in the above-mentioned Figure 1 Description.

[0127] Based on the above method of structuring PDF files based on OCR and large models, the embodiment of the present application also provides a method for constructing a knowledge base, which includes:

[0128] Obtaining structured data of the PDF file generated according to the above method;

[0129] A knowledge base is constructed based on the structured data of the PDF file.

[0130] Based on the above-mentioned method for structuring PDF files based on OCR and large models, an embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein a computer executable program is stored in the memory, and the processor is used to run the computer executable program to implement the operating steps in the above-mentioned method for structuring PDF files based on OCR and large models.

[0131] The expressions "first", "second", "the first" or "the second" used in various embodiments of the present disclosure may modify various components regardless of order and / or importance, but these expressions do not limit the corresponding components. The above expressions are only configured for the purpose of distinguishing an element from other elements. For example, a first user device and a second user device represent different user devices, although both are user devices. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the present disclosure.

[0132] When one element (e.g., a first element) is referred to as being "(operably or communicatively) coupled" or "(operably or communicatively) coupled to" or "connected to" another element (e.g., a second element), it is understood that the one element is directly connected to the other element or the one element is indirectly connected to the other element via yet another element (e.g., a third element). Conversely, it is understood that when an element (e.g., a first element) is referred to as being "directly connected" or "directly coupled" to another element (the second element), no element (e.g., a third element) is interposed between the two.

[0133] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for structuring PDF files based on OCR and large models, characterized in that: include: Traversing a single-page PDF in the PDF file, and performing layout analysis and content recognition on the single-page PDF based on an OCR model to determine embedded images and all plain text blocks of the single-page PDF; Performing enhancement processing on the embedded image based on a multimodal large language model to generate enhanced text content in a targeted manner; Calculating a position association relationship between the enhanced text content and the plain text block; Vectorizing the embedded image and the corresponding enhanced text content, the plain text block and the corresponding text content respectively to obtain an embedded image description vector and a plain text block description vector; Generate structured data of the single-page PDF based on the embedded image description vector, the position association relationship, and the plain text block description vector; Generate structured data of the PDF file based on structured data of all pages in the PDF file; Before the embedded image is enhanced based on the multimodal large language model to generate enhanced text content in a targeted manner, the method includes: performing global text extraction on the single-page PDF to use all the obtained text content as the first text content; When the embedded image is enhanced based on the multimodal large language model to generate enhanced text content in a targeted manner, if the embedded image is a table-type image, the text content in the table is extracted as the third text content, and the enhanced text content includes the third text content; After the embedded image is enhanced based on the multimodal large language model to generate enhanced text content in a targeted manner, the method further includes: calculating the similarity between the first text content and the third text content, so as to correct the third text content based on the first text content when the similarity is greater than a set similarity threshold, and determining the first text content with a similarity less than the set similarity threshold as the text content corresponding to all the plain text blocks and using it as the fourth text content; The calculating the positional association relationship between the enhanced text content and the plain text block includes: determining the text block closest to the embedded image among all the plain text blocks, and establishing a positional association relationship between the enhanced text content and the fourth text content based on the position of the closest text block.

2. The method according to claim 1, characterized in that When the embedded image is enhanced based on the multimodal large language model to generate enhanced text content in a targeted manner, if the embedded image is a pure image, semantic analysis is performed on the pure image to generate a description text of the pure image as the second text content, and the enhanced text content also includes the second text content.

3. The method according to claim 2, characterized in that The performing global text extraction on the single-page PDF to use all obtained text contents as first text contents includes: Calling the created PDF parsing library to perform structure and format parsing on the single-page PDF to generate a PDF page content instruction sequence; The created PDF text instruction set is called to analyze the text index structure of the PDF page content instruction sequence, so as to extract all text contents on the single PDF page as the first text content according to the text index structure.

4. The method according to claim 3, characterized in that The OCR model is used to perform layout analysis and content recognition on the single-page PDF to determine the embedded images and all plain text blocks of the single-page PDF, including: Calling a PDF rendering library to convert the single-page PDF into a page image, and preprocessing the page image to obtain a preprocessed page image; Performing grayscale histogram statistics on the preprocessed page image, and performing feature energy analysis based on the histogram, so as to perform layout analysis on the single-page PDF to obtain multiple layout content areas; Color space conversion is performed for each layout content area to determine the embedded images and all plain text blocks of the single-page PDF based on the hue, transparency and brightness data of each layout content area in the target color space.

5. The method according to claim 4, characterized in that When the embedded image is enhanced based on the multimodal large language model to generate enhanced text content in a targeted manner, the following steps are included: Converting the embedded image into a linear embedding input sequence of the multimodal large language model; Performing feature extraction based on position embedding on the linear embedding input sequence to obtain spatial features and hierarchical features of the embedded image; Calculating the semantic dependency relationship between the spatial features and the hierarchical features to obtain a semantic feature vector; Based on the semantic feature vector, the embedded image is enhanced to obtain the enhanced text content.

6. The method according to claim 5, characterized in that The calculating the semantic dependency relationship between the spatial features and the hierarchical features to obtain a semantic feature vector includes: Performing phase perception on the spatial features and the hierarchical features to align the spatial features and the hierarchical features; Performing neighborhood propagation on the aligned spatial features and hierarchical features to construct a multi-hop attention intensity transmission queue; Performing semantic dependency analysis on the multi-hop attention strength transfer queue to create a semantic dependency meta-graph; An adjacency matrix and node features of the semantic dependency metagraph are determined to calculate the semantic feature vector based on the adjacency matrix and the node features.

7. The method according to claim 6, characterized in that Based on the graph convolutional network, the adjacency matrix and node features of the semantic dependency metagraph are determined using the following formula to calculate the semantic feature vector based on the adjacency matrix and node features, including: ,in, is the first The node characteristics of the layer, is an integer greater than 0, is the first The node characteristics of the layer, is the adjacency matrix Plus self-loop , yes The degree matrix of It is The weight matrix of the layer, is the activation function, and the semantic feature vector is calculated using the node features of the last layer in the graph convolutional network and is denoted as .

8. The method according to claim 7, characterized in that The enhancing process of the embedded image based on the semantic feature vector includes: calculating the manifold description feature of the embedded image based on the semantic feature vector, and enhancing the embedded image based on the manifold description feature to obtain the enhanced text content.

9. The method according to claim 8, characterized in that The edit distance between the characters in the first text content and the third text content is calculated according to the following formula, so that the edit distance obtained in the last iteration is used as the edit distance between the characters in the first text content and the third text content. , is the length of the first text content, is the length of the third text content: in, represents the minimum edit distance between the first i characters of the first text content S1 and the first j characters of the third text content S2, represents the weight of the kth character in S1, represents the weight of the kth character in S2, represents the context influence factor when i=0 and j=0, represents the sum of the weights of the first i characters in S1, represents the sum of the weights of the first j characters in S2, represents the character at position i in S1, represents the character at position j in S2, represents the contextual influence factor between the i-th character in S1 and the j-th character in S2, It is the penalty value calculated when the i-th character in S1 does not match the j-th character in S2. The penalty value reflects the difference in character weights and contextual influence factors. represents the minimum edit distance between the first i-1 characters of S1 and the first j-1 characters of S2; α is used to adjust the importance of weight difference in calculating penalty value, and β is used to adjust the importance of context influence factor in calculating penalty value. represents the weight of the i-th character in S1 and the weight of the jth character in S2 The absolute difference of .

Images