PDF (Portable Document Format) identification and preprocessing method for large model application

Abstract

The invention discloses a PDF (Portable Document Format) identification and preprocessing method for large model application. The method comprises the following steps: S1, file reading and primary processing: reading and processing the content of a PDF file; s2, document content replacement: identifying and extracting a text segment by using LLamaIndex, and replacing the text segment with the text content extracted by PDFP luber; s3, overlapping the content between the pages: from the second page, splicing the content tail of the previous page with the content head of the current page; s4, node manufacturing and processing; s5, performing preliminary segmentation; s6, vectorization processing: calculating the cosine similarity of the embedded vector, and converting the cosine similarity into a cosine distance; s7, threshold setting and semantic segmentation: setting a distance threshold, and segmenting the text into a plurality of paragraphs; s8, summarizing and storing text segments; s9, processing the table; and S10, processing the picture. According to the method, the PDF content is efficiently preprocessed by adopting the PDFP blue library, the OCR and the semantic segmentation technology, and the method has the advantages of high data accuracy, high processing efficiency and wide application range.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a PDF recognition and preprocessing method for large model applications. Background Art

[0002] With the development of big data and artificial intelligence technology, especially the rise of large model applications, the importance of PDF recognition and preprocessing technology has become increasingly prominent. As a common unstructured data format, PDF files are widely used in various documents, reports, papers and other information carriers. However, the format of PDF file content is complex and diverse, containing multiple elements such as text, pictures, tables, etc., which brings great challenges to data extraction and processing. In order to improve the performance of large models in processing PDF data, optimizing the preprocessing process is an essential link.

[0003] Traditional PDF processing methods mainly focus on simple text extraction and image recognition, and cannot effectively handle complex document structures and diverse content formats. With the development of retrieval-augmented generation (RAG) technology, by combining retrieval models and generation models, the accuracy and contextual relevance of information can be improved in complex tasks. However, to achieve this goal, high-quality preprocessing of unstructured data in PDF is crucial.

[0004] In the field of PDF recognition and processing, the optimization preprocessing technology for unstructured data mainly focuses on three aspects: improving data quality, improving processing efficiency and expanding the scope of application.

[0005] In terms of specific technical implementation, existing PDF recognition and preprocessing methods have achieved certain results. For example, by combining the PDFPlumber library and OCR technology, the text, images and tables in PDF files can be effectively extracted and recognized. The PDFPlumber library can read the content of PDF files and extract the text and image information; OCR technology can recognize the text in the image and extract the text content in the image.

[0006] However, these methods still have certain limitations when dealing with complex document structures and diverse content formats. For example, table recognition and processing in PDF files has always been a difficult point. Traditional table recognition methods usually rely on simple line detection and coordinate calculation, which is difficult to handle complex table structures and irregular cell arrangements. In addition, semantic segmentation and vectorization of text are also important links in preprocessing. By calculating the similarity between text paragraphs, text segmentation and semantic analysis can be performed more accurately.

[0007] Therefore, how to provide a PDF recognition and preprocessing method for large model applications is an urgent problem that technicians in this field need to solve. Summary of the invention

[0008] One purpose of the present invention is to propose a PDF recognition and preprocessing method for large model applications. The present invention adopts PDFPlumber library, OCR technology and semantic segmentation technology to achieve efficient recognition and preprocessing of PDF document content. Through node production, vectorization processing and multimodal data processing, the accuracy and consistency of data are ensured, the extraction accuracy and processing efficiency of PDF content are significantly improved, and it has the advantages of efficiently processing complex documents, improving the application effect of large models and expanding the application scope of intelligent document processing, and is suitable for a variety of practical application scenarios.

[0009] A PDF recognition and preprocessing method for large model applications according to an embodiment of the present invention includes the following steps:

[0010] S1. File reading and preliminary processing: Use the PDFPlumber library to read the content of the PDF file and determine whether the PDF page contains images, tables, and text;

[0011] S2. Document content replacement: Use LLamaIndex to identify and extract text segments, replace the identified text segments with the text content extracted by PDFPlumber, perform layout recognition and judgment on the content of each page, and cross-process the content between pages;

[0012] S3, processing the overlapping content between pages: traverse all pages, starting from the second page, and splice the tail of the content of the previous page with the head of the content of the current page;

[0013] S4, node creation and processing: create nodes for the document content and generate corresponding node objects;

[0014] S5, preliminary segmentation: segment the extracted text content according to punctuation marks and line breaks to generate smaller context paragraphs, and merge adjacent contexts;

[0015] S6, vectorization processing: perform vectorization processing on each segmented context paragraph, generate a corresponding embedding vector, calculate the cosine similarity between the embedding vectors of the current context and the next context, and convert the similarity into a cosine distance;

[0016] S7, threshold setting and semantic segmentation: set the distance threshold, determine the position of text segmentation according to the cosine distance, and divide the text into multiple paragraphs;

[0017] S8, text segment summary and storage: summarize the segmented text segments and store the processed text segments into a vector library;

[0018] S9. Table processing: By defining the precise position of each cell, cell cutting is performed on various tables, and the table contents are processed;

[0019] S10, image processing: after the image is recognized, the image is extracted and saved, an image path is generated, semantics of the image is extracted, and the extracted text semantics are strongly aligned with the image.

[0020] Furthermore, the preliminary processing specifically includes:

[0021] If the page contains an image, the image is converted into binary data and text recognition is performed using OCR technology;

[0022] Use the extract_text method in PDFPlumber to extract the text content.

[0023] Furthermore, the S8 specifically includes:

[0024] S81, passing the segmented text segments into the large model for summarization;

[0025] S82, storing the summary information of the current text segment into the documents data of the next text segment to form a context association;

[0026] S83. Add the processed text segment, file name, document path, document directory, paragraph summary and page number information to the metadata and store it in the vector library.

[0027] Furthermore, the S9 specifically includes:

[0028] S91. Cell position definition: The position of a cell is controlled by three variables, including the number of lines, line position, and line length, which together constitute the joint probability distribution of the cell position;

[0029] S92, cell coordinate recognition: after successfully identifying the coordinates of each cell in the table, perform OCR model recognition on the text in each small table;

[0030] S93. Table merging: Merge tables with similar cell coordinates and export them to CSV files;

[0031] S94, data storage: concatenate the column name and content of each row to form a text block, add the text block, file name, file path, and document page number to the metadata, and store them in the vector library.

[0032] Furthermore, the processing of the number of lines specifically includes:

[0033] When performing table recognition, the number of lines is obtained by obtaining the text density extreme points of each cell in the table;

[0034] The number of lines is four times the text density extreme point, and is subject to the distribution of four times the text density extreme point.

[0035] Furthermore, the processing of the line position specifically includes:

[0036] Set two bounded variables θ and σ, which represent the distance from the extreme point of cell text density to the upper and left cell boundaries respectively;

[0037] In the text density probability distribution function, except for the section with the highest probability of cell text density, the rest is the probability distribution of θ and σ;

[0038] The line position is determined by symmetrically completing the probability distribution functions of θ and σ and introducing the control variable w to distinguish θ and σ.

[0039] Furthermore, the processing of the line length specifically includes:

[0040] Get the coordinates of the text density poles and the lengths of θ and σ;

[0041] Calculate the coordinates of the line endpoints;

[0042] The length of the line is obtained using the distance formula.

[0043] The beneficial effects of the present invention are:

[0044] (1) Improve data quality and consistency: By using the PDFPlumber library to read the content of PDF files and combining it with OCR technology to recognize text in images, the text, table, and image information in PDF can be effectively extracted. The document content replacement step uses LLamaIndex to identify and extract text segments, ensuring the accuracy and consistency of the text content. In addition, the content overlap between pages is processed to avoid content loss and ensure the continuity and integrity of the document content.

[0045] (2) Enhanced text semantic segmentation and vectorization processing: The present invention uses advanced semantic segmentation and vectorization processing technology to perform accurate text segmentation and semantic analysis by calculating the cosine similarity between context paragraphs. The distance threshold is set and the location of text segmentation is determined based on the cosine distance, making the text segmentation more accurate and retaining the semantic relevance between paragraphs. This processing method improves the understanding and analysis capabilities of the large model when processing PDF data.

[0046] (3) Improve the processing capabilities of tables and images: The present invention provides an efficient table processing method. By defining the position variables of the cells (number of lines, line positions, and line lengths), it achieves accurate cutting and OCR recognition of table cells, ensuring the integrity and accuracy of the table content. At the same time, the processing of images includes semantic extraction and alignment, ensuring the accurate extraction of image information and semantic consistency with the text. This multimodal data processing capability enables the large model to better process complex PDF files containing tables and images.

[0047] (4) Improve processing efficiency and storage management: In terms of text segment summarization and storage, the method of the present invention improves data storage management efficiency by summarizing the segmented text segments and storing the processed text segments in a vector library. The processing results of tables and pictures are also effectively stored in metadata to form a structured database, which is convenient for subsequent data retrieval and analysis. This storage management method significantly improves the processing efficiency of the system and reduces the consumption of computing resources.

[0048] (5) Expanding the application scope of large models: Through optimized preprocessing technology, the method of the present invention enables large models to process more complex PDF files, expanding the application scenarios of large models. Whether in applications such as intelligent document search, automated document processing, or knowledge question and answer, the method of the present invention has demonstrated powerful processing capabilities and practical application value. In particular, in scenarios of real-time processing and large-scale data analysis, it can significantly improve the response speed and accuracy of large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0050] Figure 1 A flowchart of a PDF recognition and preprocessing method for large model applications proposed by the present invention;

[0051] Figure 2 This is a structural diagram of the PDF processing module of a PDF recognition and preprocessing method for large model applications proposed by the present invention. DETAILED DESCRIPTION

[0052] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0053] refer to Figure 1 and Figure 2 , a PDF recognition and preprocessing method for large model applications, comprising the following steps:

[0054] S1. File reading and preliminary processing: Use the PDFPlumber library to read the content of the PDF file and determine whether the PDF page contains images, tables, and text;

[0055] S2. Document content replacement: Use LLamaIndex to identify and extract text segments, replace the identified text segments with the text content extracted by PDFPlumber, perform layout recognition and judgment on the content of each page, and cross-process the content between pages;

[0056] S3, processing the overlapping content between pages: traverse all pages, starting from the second page, and splice the tail of the content of the previous page with the head of the content of the current page;

[0057] S4, node creation and processing: create nodes for the document content and generate corresponding node objects;

[0058] S5, preliminary segmentation: segment the extracted text content according to punctuation marks and line breaks to generate smaller context paragraphs, and merge adjacent contexts;

[0059] S6, vectorization processing: perform vectorization processing on each segmented context paragraph, generate a corresponding embedding vector, calculate the cosine similarity between the embedding vectors of the current context and the next context, and convert the similarity into a cosine distance;

[0060] S7, threshold setting and semantic segmentation: set the distance threshold, determine the position of text segmentation according to the cosine distance, and divide the text into multiple paragraphs;

[0061] S8, text segment summary and storage: summarize the segmented text segments and store the processed text segments into a vector library;

[0062] S9. Table processing: By defining the precise position of each cell, cell cutting is performed on various tables, and the table contents are processed;

[0063] S10, image processing: after the image is recognized, the image is extracted and saved, an image path is generated, semantics of the image is extracted, and the extracted text semantics are strongly aligned with the image.

[0064] In this implementation, the preliminary processing specifically includes:

[0065] If the page contains an image, the image is converted into binary data and text recognition is performed using OCR technology;

[0066] Use the extract_text method in PDFPlumber to extract the text content.

[0067] In this implementation manner, S8 specifically includes:

[0068] S81, passing the segmented text segments into the large model for summarization;

[0069] S82, storing the summary information of the current text segment into the documents data of the next text segment to form a context association;

[0070] S83. Add the processed text segment, file name, document path, document directory, paragraph summary and page number information to the metadata and store it in the vector library.

[0071] In this implementation manner, the S9 specifically includes:

[0072] S91. Cell position definition: The position of a cell is controlled by three variables, including the number of lines, line position, and line length, which together constitute the joint probability distribution of the cell position;

[0073] S92, cell coordinate recognition: after successfully identifying the coordinates of each cell in the table, perform OCR model recognition on the text in each small table;

[0074] S93. Table merging: Merge tables with similar cell coordinates and export them to CSV files;

[0075] S94, data storage: concatenate the column name and content of each row to form a text block, add the text block, file name, file path, and document page number to the metadata, and store them in the vector library.

[0076] In this implementation manner, the processing of the number of lines specifically includes:

[0077] When performing table recognition, the number of lines is obtained by obtaining the text density extreme points of each cell in the table;

[0078] The number of lines is four times the text density extreme point, and is subject to the distribution of four times the text density extreme point.

[0079] In this implementation manner, the processing of the line position specifically includes:

[0080] Set two bounded variables θ and σ, which represent the distance from the extreme point of cell text density to the upper and left cell boundaries respectively;

[0081] In the text density probability distribution function, except for the section with the highest probability of cell text density, the rest is the probability distribution of θ and σ;

[0082] The line position is determined by symmetrically completing the probability distribution functions of θ and σ and introducing the control variable w to distinguish θ and σ.

[0083] In this implementation manner, the processing of the line length specifically includes:

[0084] Get the coordinates of the text density poles and the lengths of θ and σ;

[0085] Calculate the coordinates of the line endpoints;

[0086] The length of the line is obtained using the distance formula.

[0087] Embodiment 1:

[0088] The embodiment of the present invention provides a PDF recognition and preprocessing method for large model applications. Figure 1 The detailed process of PDF recognition and preprocessing method is given in Figure 2 The structure of PDF file preprocessing and the way of data storage are shown in the figure. It mainly includes the following steps:

[0089] (1) Read PDF files: Use llama index and pdfplumber libraries to read the content of PDF files, and perform layout recognition on each page to determine whether there are images, tables, and text content. First, extract the text content and use the extract_text method of the pdfplumber library to extract the text from the PDF. However, since the text recognition provided by llama index is inaccurate, it is necessary to re-read the PDF file content and modify the text content in the document to ensure the accuracy of text extraction.

[0090] (2) Page content cross processing: In order to prevent the loss of content between pages after text cutting, a method is customized here to achieve content overlap between pages: traverse the read document, modify the text value from the second page, and achieve overlap between pages. Get the overlapping content of the previous page and splice it with the text content of the current page to ensure the continuity of the page content. Then make nodes for the document content and generate corresponding node objects.

[0091] (3) Semantic segmentation and vectorization: First, the extracted text segment is segmented into smaller context paragraphs according to punctuation and line breaks. Subsequently, in order to more completely retain the semantic information, adjacent context paragraphs are merged to enhance the relevance between them. Next, each merged context paragraph is vectorized and the cosine similarity of the embedding vectors between these paragraphs is calculated. After that, a value representing the difference between paragraphs is obtained by subtracting the cosine similarity from 1 (this can be regarded as the cosine distance). Finally, a distance threshold is set, and the text is further segmented into more appropriate paragraphs based on this threshold.

[0092] (4) Text segmentation: Use the percentile function of numpy to obtain the distance threshold, which will be regarded as the demarcation line of outliers. If more segmentation is desired, the threshold can be adjusted appropriately. The index with a distance greater than the threshold is obtained to determine the location of the text segmentation, and the text is segmented according to these threshold nodes to form segmented paragraphs.

[0093] (5) Paragraph summary processing: The segmented text segments are passed into the large model for text segment summary processing, and the summary of the current text segment is stored in the document data of the next text segment to ensure the context relevance.

[0094] (6) Metadata storage: The segmented text segments, extracted file names, document paths, document directories, document paragraph summaries, and page numbers are added to the metadata and stored in the vector library.

[0095] (7) Table processing: First, use the PDF processing library to read the PDF file and identify the table in it. Next, by analyzing the text density extreme points in the table, determine the number of lines, which is four times the text density extreme points. Then, set the variables θ and σ to represent the distance from the text density extreme points to the upper and left boundary lines of the cell, respectively. By symmetrically filling the probability distribution functions of θ and σ and introducing the control variable w, the line position of each cell is determined. By calculating the coordinates of the text density extreme points and the lengths of θ and σ, the coordinates of the line endpoints are obtained, and the distance formula is used to calculate the line length, thereby determining the exact position of each cell.

[0096] Perform OCR text recognition on each identified small table to extract the text content. Then merge the coordinates of adjacent small tables to form a complete table structure. Export the table as a CSV file, and concatenate the column name and content of each row to form a text block. Add the concatenated text block, file name, file path, document page number and other information to the metadata, and store the above metadata in the vector library for subsequent query and processing.

[0097] (8) Image processing: When an image is recognized, the image is extracted and saved, and the image path is generated. The image is semantically extracted to extract the text semantics, and the image and text are strongly aligned.

[0098] In order to further verify the practicability of the method of the present invention, a company conducted an on-site test when providing legal consulting services to customers. The test sites included four branches, A, B, C and D, and each branch selected 50 complex legal documents for processing.

[0099] Table 1 Comparison of test data of traditional method and method of the present invention

[0100]

[0101] From the data analysis of Table 1, it can be concluded that the average processing time of the traditional method is relatively long in each city, while the processing time is significantly reduced after the method of the present invention is adopted. Specifically, the average processing time of the traditional method is 21.98 minutes, while the average processing time of the method of the present invention is 4.78 minutes, and the processing efficiency is improved by 359%. This significant time reduction shows that the method of the present invention greatly improves the processing speed. In terms of text extraction accuracy, the method of the present invention shows a significant advantage. The accuracy of the traditional method is 84.58% on average in various regions, while the accuracy of the method of the present invention is increased to 97.95% on average, an increase of 15.8%. This increase in accuracy shows that the method of the present invention has high accuracy and reliability in text recognition and extraction. In terms of table processing accuracy, the accuracy of the traditional method is 74.63% on average, while the accuracy of the method of the present invention is increased to 94.72%, an increase of 26.9%. This significant improvement shows that the method of the present invention has extremely high accuracy in table recognition and processing. By defining the position variables of cells, using the OCR model to identify the table content, and performing precise cell cutting and merging, the method of the present invention can more accurately process the table content in the PDF document. In terms of image processing accuracy, the accuracy of the traditional method is 60.35% on average, while the accuracy of the method of the present invention is increased to 90.28%, an increase of 49.6%. This improvement shows that the method of the present invention has higher accuracy and reliability in image recognition and processing. By using OCR technology to perform image text recognition, semantic extraction and alignment, the method of the present invention can more accurately process the image content in the PDF document.

[0102] In summary, the method of the present invention has significant advantages in processing PDF documents. It is significantly better than traditional methods in terms of processing time, text extraction accuracy, table processing accuracy, and image processing accuracy. Specifically, the processing time was reduced by 359%, the text extraction accuracy was improved by 15.8%, the table processing accuracy was improved by 26.9%, and the image processing accuracy was improved by 49.6%. These data fully demonstrate the excellent performance and broad application prospects of the method of the present invention in practical applications. Through efficient preprocessing technology and precise recognition methods, the method of the present invention can significantly improve the efficiency and accuracy of PDF document processing, and provide a solid technical foundation for the practical application of large models.

[0103] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A PDF recognition and preprocessing method for large model applications, characterized in that: The steps include: S1. File reading and preliminary processing: Use the PDFPlumber library to read the content of the PDF file and determine whether the PDF page contains images, tables, and text; S2. Document content replacement: Use LLamaIndex to identify and extract text segments, replace the identified text segments with the text content extracted by PDFPlumber, perform layout recognition and judgment on the content of each page, and cross-process the content between pages; S3, processing the overlapping content between pages: traverse all pages, starting from the second page, and splice the tail of the content of the previous page with the head of the content of the current page; S4, node creation and processing: create nodes for the document content and generate corresponding node objects; S5, preliminary segmentation: segment the extracted text content according to punctuation marks and line breaks to generate smaller context paragraphs, and merge adjacent contexts; S6, vectorization processing: perform vectorization processing on each segmented context paragraph, generate a corresponding embedding vector, calculate the cosine similarity between the embedding vectors of the current context and the next context, and convert the similarity into a cosine distance; S7, threshold setting and semantic segmentation: set the distance threshold, determine the position of text segmentation according to the cosine distance, and divide the text into multiple paragraphs; S8, text segment summary and storage: summarize the segmented text segments and store the processed text segments into a vector library; S9. Table processing: By defining the precise position of each cell, cell cutting is performed on various tables, and the table contents are processed; S10, image processing: after the image is recognized, the image is extracted and saved, an image path is generated, semantics of the image is extracted, and the extracted text semantics are strongly aligned with the image.

2. A PDF recognition and preprocessing method for large model applications according to claim 1, characterized in that: The preliminary processing specifically includes: If the page contains an image, the image is converted into binary data and text recognition is performed using OCR technology; Use the extract_text method in PDFPlumber to extract the text content.

3. The PDF recognition and preprocessing method for large model applications according to claim 1, characterized in that: The S8 specifically includes: S81, passing the segmented text segments into the large model for summarization; S82, storing the summary information of the current text segment into the documents data of the next text segment to form a context association; S83. Add the processed text segment, file name, document path, document directory, paragraph summary and page number information to the metadata and store it in the vector library.

4. The PDF recognition and preprocessing method for large model applications according to claim 1, characterized in that: The S9 specifically includes: S91. Cell position definition: The position of a cell is controlled by three variables, including the number of lines, line position, and line length, which together constitute the joint probability distribution of the cell position; S92, cell coordinate recognition: after successfully identifying the coordinates of each cell in the table, perform OCR model recognition on the text in each small table; S93. Table merging: Merge tables with similar cell coordinates and export them to CSV files; S94, data storage: concatenate the column name and content of each row to form a text block, add the text block, file name, file path, and document page number to the metadata, and store them in the vector library.

5. A PDF recognition and preprocessing method for large model applications according to claim 4, characterized in that: The processing of the number of lines specifically includes: When performing table recognition, the number of lines is obtained by obtaining the text density extreme points of each cell in the table; The number of lines is four times the text density extreme point, and is subject to the distribution of four times the text density extreme point.

6. A PDF recognition and preprocessing method for large model applications according to claim 4, characterized in that: The processing of the line position specifically includes: Set two bounded variables θ and σ, which represent the distance from the extreme point of cell text density to the upper and left cell boundaries respectively; In the text density probability distribution function, except for the section with the highest probability of cell text density, the rest is the probability distribution of θ and σ; The line position is determined by symmetrically completing the probability distribution functions of θ and σ and introducing the control variable w to distinguish θ and σ.

7. The PDF recognition and preprocessing method for large model applications according to claim 4, characterized in that: The processing of the line length specifically includes: Get the coordinates of the text density poles and the lengths of θ and σ; Calculate the coordinates of the line endpoints; The length of the line is obtained using the distance formula.

Images