PDF document splitting method based on KNN algorithm, medium and system

Abstract

The invention relates to a PDF document splitting method based on a KNN algorithm, a medium and a system. The method comprises the following steps of obtaining a PDF document to be split, performing classification prediction on texts in the PDF document to be split through a KNN classification model, identifying title pages in the PDF document, collecting page numbers where the texts of the title pages are located, and obtaining a title page set; splitting the PDF document segment by segment according to the title page set to obtain a plurality of independent PDF documents; wherein the training steps of the KNN classification model are as follows: obtaining a PDF document, traversing each page of the PDF document, and extracting text information; constructing a feature matrix by taking the word size information as an independent feature; marking a text of the PDF document as a title page and a non-title page according to the word size information, and storing label information of the title page and the non-title page in a label vector; and in combination with the feature matrix and the label vector, constructing a training data set, and training the KNN classification model. Compared with the prior art, the splitting precision is improved, and human intervention is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a PDF document splitting method, medium and system based on a KNN algorithm. Background Art

[0002] With the popularization of digital information, PDF (Portable Document Format) is widely used in academic publishing, business reports and other fields. Many publications will collect multiple articles in the same PDF document for easy dissemination and archiving. However, this format brings certain difficulties to document management and information retrieval: when users search, quote or analyze specific articles, they often face complex structures, resulting in low efficiency.

[0003] Existing splitting methods mostly rely on manual operations or simple rule algorithms. For example, patent application CN106649229A discloses a method for splitting a PDF file, including: obtaining page parameter information; splitting the PDF file to be split according to the page parameter information to generate a split PDF file. This method realizes the function of automatically splitting the PDF document by page number or bookmark by determining the page parameter information. These methods have the disadvantages of insufficient accuracy and low efficiency, and it is difficult to meet the growing demand for document processing. Automated splitting methods based on machine learning have emerged, among which the KNN (K-Nearest Neighbors) algorithm has become a potential solution due to its intuitive and easy-to-implement characteristics. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a PDF document splitting method, medium and system based on KNN algorithm, which can improve the splitting accuracy while reducing human intervention and adapt to various PDF document structures.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A PDF document splitting method based on KNN algorithm includes the following steps:

[0007] Obtain a PDF document to be split, perform classification prediction on text in the PDF document to be split using a KNN classification model, and identify a title page in the PDF document;

[0008] Collect the page numbers of the texts identified as title pages to obtain a title page set;

[0009] Splitting the PDF document segment by segment according to the title page set to obtain multiple independent PDF documents;

[0010] The training steps of the KNN classification model are as follows:

[0011] Obtain a PDF document, traverse each page of the PDF document, and extract text information of the PDF document;

[0012] Using the font size information in the text information as an independent feature to construct a feature matrix;

[0013] Marking the text of the PDF document as a title page and a non-title page according to the font size information, and storing the label information of the title page and the non-title page in a label vector;

[0014] A training data set is constructed by combining the feature matrix and the label vector, and the KNN classification model is trained by the training data set.

[0015] Furthermore, the specific steps of splitting the PDF document into sections are: whenever a new title page is identified, a new PDF document is created, and the text of the title page and its subsequent content are merged into a new document until the next title page is identified.

[0016] Furthermore, the split PDF documents are saved in a designated directory, and the page number of the title page is used as the file name of each split document.

[0017] Furthermore, the text information includes text content, font size information and the page number where the text is located.

[0018] Furthermore, the text information is a dictionary structure including text blocks, text lines and text spans.

[0019] Furthermore, the text content and font size information are extracted from the text span.

[0020] Furthermore, the accuracy of the KNN classification model is verified by standard evaluation indicators.

[0021] Furthermore, the standard evaluation indicators include multiple ones of precision, recall rate and accuracy rate.

[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the PDF document splitting method based on the KNN algorithm can be implemented.

[0023] According to another aspect of the present invention, a PDF document splitting system based on a KNN algorithm is provided, comprising:

[0024] A classification prediction module is used to obtain a PDF document to be split, perform classification prediction on the text in the PDF document to be split by using a KNN classification model, and identify a title page in the PDF document;

[0025] A title page set acquisition module is used to collect the page numbers of the texts identified as title pages and obtain a title page set;

[0026] The document splitting module is used to split the PDF document into sections according to the title page set to obtain multiple independent PDF documents.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. The present invention obtains a PDF document to be split, classifies and predicts the font size of each text segment in the PDF document to be split through a KNN classification model, and uses the classification result and the preset font size to judge the title page and non-title page. The title page in the PDF document can be effectively identified, thereby improving the accuracy and efficiency of PDF document splitting.

[0029] 2. The present invention obtains a title page set by collecting page numbers where title page texts are located, and splits the PDF document segment by segment according to the title page set. Whenever a new title page is identified, a new PDF document is created, and the text of the title page and its subsequent contents are merged into a new document until the next title page is identified, thereby obtaining multiple independent PDF documents, realizing automatic splitting of PDF documents, and reducing the need for manual intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of the process of the PDF document splitting method based on the KNN algorithm proposed in the present invention. DETAILED DESCRIPTION

[0031] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0032] Abbreviations involved:

[0033] K-Nearest Neighbors, KNN

[0034] Example 1

[0035] This embodiment provides a PDF document splitting method based on the KNN algorithm. Figure 1 As shown, the following steps are included:

[0036] S1. Obtain the PDF document to be split, classify and predict the text in the PDF document to be split through the KNN classification model, and identify the title page in the PDF document.

[0037] The KNN classification model is used to predict the font size of each paragraph of text in the PDF document. The output prediction result is used to identify whether the text is a title page and record the page number of each title page. In this embodiment, text with a font size greater than or equal to 19 is marked as a title page; text with a font size less than 19 is marked as a non-title page.

[0038] S2, collecting the page numbers of the texts identified as title pages, and obtaining a title page set;

[0039] Collect the page numbers of the texts identified as title pages to form a title page set for subsequent document splitting.

[0040] S3. Split the PDF document into sections according to the title page set to obtain multiple independent PDF documents.

[0041] Split the PDF document into sections based on the title page set. Find each title page and merge it and all subsequent pages into a new document until the next title page. Whenever a new title page is identified, create a new PDF document and copy the title page and its subsequent content into the new document. The naming convention for the new document is "Document_PageNumber.pdf"

[0042] Save the split PDF documents to the specified directory (for example, C: / Users / XXX / Output).

[0043] Each split PDF document uses the page number of the title page as the file name, such as "Document_1.pdf", to ensure that the file name clearly identifies the chapter to which it belongs.

[0044] The training steps of the KNN classification model include:

[0045] Use PyMuPDF (fitz library) to open the target PDF document, traverse each page of the PDF document, and use page.get_text("dict") to extract the text information of the PDF document. This process returns a dictionary structure containing multiple text blocks, text lines, and text spans. Extract the text content and font size information from each text span to form a triple list (text, size, page_number) containing the text content, font size information, and the page number where the text is located.

[0046] A feature dataset is constructed based on the font size information extracted from the PDF document, where the font size information of each paragraph of text is used as an independent feature of the feature matrix X. The text of the PDF document is marked as a title page and a non-title page based on the font size information, and the label information of the title page and the non-title page is stored in the label vector y; if the text font size is greater than or equal to 19, the text is marked as a title page (label is 1); if the text font size is less than 19, the text is marked as a non-title page (label is 0). The feature matrix X input during training is a dataset containing font size information, and the label vector y is a label set of the title page or non-title page. Combine the feature matrix X and the label vector y to construct a training dataset for learning and optimizing the classification model.

[0047] According to the training data set, the model is trained using the KNN algorithm. Select k = 3 as the number of neighbors (which can be adjusted according to the data), use the KNeighborsClassifier class in the sklearn library for model training, and implement model fitting through knn.fit(X, y).

[0048] After training is completed, the accuracy of the KNN classification model is verified by standard evaluation indicators to ensure the classification effect. Standard evaluation indicators include precision, recall rate, and accuracy. According to the actual splitting effect, if the classification effect is not good, the parameters of the KNN algorithm, feature extraction method, or label marking rules can be adjusted to further optimize the classification model.

[0049] Example 2

[0050] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a PDF document splitting method based on a KNN algorithm can be implemented.

[0051] The rest is the same as in Example 1.

[0052] Example 3

[0053] This embodiment provides a PDF document splitting system based on a KNN algorithm, including:

[0054] The classification prediction module is used to obtain the PDF document to be split, classify and predict the text in the PDF document to be split through the KNN classification model, and identify the title page in the PDF document;

[0055] A title page set acquisition module is used to collect the page numbers of the texts identified as title pages and obtain a title page set;

[0056] The document splitting module is used to split the PDF document into sections according to the title page set to obtain multiple independent PDF documents.

[0057] The rest is the same as in Example 1.

[0058] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A PDF document splitting method based on KNN algorithm, characterized in that: The following steps are involved: Obtain a PDF document to be split, perform classification prediction on text in the PDF document to be split using a KNN classification model, and identify a title page in the PDF document; Collect the page numbers of the texts identified as title pages to obtain a title page set; Splitting the PDF document segment by segment according to the title page set to obtain multiple independent PDF documents; The training steps of the KNN classification model are as follows: Obtain a PDF document, traverse each page of the PDF document, and extract text information of the PDF document; Using the font size information in the text information as an independent feature to construct a feature matrix; Marking the text of the PDF document as a title page and a non-title page according to the font size information, and storing the label information of the title page and the non-title page in a label vector; A training data set is constructed by combining the feature matrix and the label vector, and the KNN classification model is trained by the training data set.

2. The PDF document splitting method based on the KNN algorithm according to claim 1 is characterized in that: The specific steps of splitting the PDF document into sections are: whenever a new title page is identified, a new PDF document is created, and the text of the title page and its subsequent content are merged into a new document until the next title page is identified.

3. The PDF document splitting method based on the KNN algorithm according to claim 1 is characterized in that: The split PDF documents are saved in a designated directory, and the page number of the title page is used as the file name of each split document.

4. The PDF document splitting method based on the KNN algorithm according to claim 1 is characterized in that: The text information includes text content, font size information and the page number where the text is located.

5. The PDF document splitting method based on the KNN algorithm according to claim 4 is characterized in that: The text information is a dictionary structure including text blocks, text lines and text spans.

6. The PDF document splitting method based on the KNN algorithm according to claim 5 is characterized in that: The text content and font size information are extracted from the text span.

7. The PDF document splitting method based on the KNN algorithm according to claim 1 is characterized in that: The accuracy of the KNN classification model was verified by standard evaluation metrics.

8. The PDF document splitting method based on the KNN algorithm according to claim 7 is characterized in that: The standard evaluation metrics include multiple ones of precision, recall and accuracy.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the PDF document splitting method based on the KNN algorithm according to any one of claims 1 to 8 can be implemented.

10. A PDF document splitting system based on KNN algorithm, characterized in that: include: A classification prediction module is used to obtain a PDF document to be split, perform classification prediction on the text in the PDF document to be split by using a KNN classification model, and identify a title page in the PDF document; A title page set acquisition module is used to collect the page numbers of the texts identified as title pages and obtain a title page set; The document splitting module is used to split the PDF document into sections according to the title page set to obtain multiple independent PDF documents.

Images