Text extraction method, device and equipment of PDF file and storage medium
By constructing a row feature matrix and using clustering methods to classify and concatenate text in PDF files, the problem of low text extraction accuracy is solved, and accurate and coherent text extraction is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA CONSTRUCTION BANK
- Filing Date
- 2023-10-18
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies suffer from low accuracy in extracting text from PDF files, particularly in the error of splitting a sentence into two.
By constructing a line feature matrix, each line of text in the PDF file is clustered based on text features to generate a first type of text and a second type of text. It is then determined whether the first type of text needs a line break, and the text is concatenated according to text features and positional relationships to generate the target text.
It improves the accuracy of text extraction, ensures the coherence and integrity of the text, reduces erroneous line breaks, and improves the performance of subsequent natural language processing tasks.
Smart Images
Figure CN117216279B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, device and storage medium for extracting text from PDF files. Background Technology
[0002] Portable Document Format (PDF) is a cross-platform file format that encapsulates text, fonts, graphics, images, colors, layout, and printing equipment-related parameters into a single file. It maintains the integrity of page elements during network transmission, printing, and plate-making output, offering high integration and security. As a result, it has been widely used across various industries.
[0003] Currently, to edit text in a PDF file, you need to first extract the text and save it to another file format before editing. In practice, some text in PDF files can be directly copied and extracted, while text in others cannot and requires Optical Character Recognition (OCR) to recognize the text.
[0004] However, regardless of which of the above methods of extracting text, there is a problem of incorrectly wrapping a sentence to a newline, resulting in poor accuracy of text extraction. Summary of the Invention
[0005] This application provides a method, apparatus, device, and storage medium for extracting text from PDF files, in order to solve the problem of low accuracy in text extraction from PDF files in the prior art.
[0006] Firstly, this application provides a method for extracting text from a PDF file, including:
[0007] Extract text features from each line of text in the PDF file to be processed, and construct a line feature matrix of the PDF file to be processed. The number of rows in the line feature matrix is the same as the number of rows in the PDF file to be processed, and the number of columns in the line feature matrix is the same as the number of text features. The text features include line length.
[0008] Based on the row feature matrix, each row of text in the PDF file to be processed is clustered to generate a first type of text and a second type of text, wherein the row length of the first type of text is greater than the row length of the second type of text.
[0009] For each text in the first type of text, if the last character of the text is a preset segmentation symbol, then a newline character is added after the last character to generate the processed text; otherwise, the text is determined to be the processed text.
[0010] Add the newline character after the last character of each text in the second type of text to generate the processed text;
[0011] Based on the position of each line of text in the PDF file to be processed, all the processed text is concatenated to generate the target text.
[0012] In one possible design, extracting the text features of each line of text in the PDF file to be processed and constructing the line feature matrix of the PDF file to be processed includes:
[0013] Extract text features from each line of text in the PDF file to be processed, and construct an initial row feature matrix of the PDF file to be processed. The number of rows in the initial row feature matrix is the same as the number of rows in the PDF file to be processed, and the number of columns in the initial row feature matrix is the same as the number of text features.
[0014] The data in each column of the initial row feature matrix is centered to generate the row feature matrix of the PDF file to be processed.
[0015] In one possible design, the process of centering each column of data in the initial row feature matrix to generate the row feature matrix of the PDF file to be processed includes:
[0016] Through the formula: Calculate the standardized data corresponding to each data point in the initial row feature matrix; where z is the standardized data corresponding to any data point in any row, x is the data point, u is the mean of all data points in any row, and s is the standard deviation of all data points in any row.
[0017] The initial row feature matrix is updated based on the standardized data corresponding to each data point in the initial row feature matrix to generate the row feature matrix of the PDF file to be processed.
[0018] In one possible design, the text features also include font size and / or number of characters.
[0019] In one possible design, the step of clustering each line of text in the PDF file to be processed based on the row feature matrix to generate a first type of text and a second type of text includes:
[0020] Based on the row feature matrix, each line of text in the PDF file to be processed is clustered to obtain a first center point and a second center point, wherein the row length of the first center point is greater than the row length of the second center point;
[0021] Calculate the first feature distance between each text and the first center point, and the second feature distance between each text and the second center point;
[0022] Based on the first feature distance and the second feature distance of each text, each text is classified to the center point corresponding to the smallest feature distance, and a first type of text and a second type of text are obtained. The center point of the first type of text is the first center point, and the center point of the second type of text is the second center point.
[0023] In one possible design, the step of concatenating all processed text according to the position of each line of text in the PDF file to be processed to generate the target text includes:
[0024] Based on the position of each line of text in the PDF file to be processed, determine the connection relationship between the text and other texts;
[0025] Based on the connection relationship between each line of text and other texts, the corresponding processed text is concatenated with other processed texts to generate the target text.
[0026] Secondly, this application provides a text extraction device for PDF files, comprising:
[0027] A construction module is used to extract text features from each line of text in a portable file (PDF) to be processed, and construct a row feature matrix of the PDF to be processed. The number of rows in the row feature matrix is the same as the number of rows in the PDF to be processed, and the number of columns in the row feature matrix is the same as the number of text features. The text features include line length.
[0028] The clustering module is used to cluster each line of text in the PDF file to be processed based on the line feature matrix, generating a first type of text and a second type of text, wherein the line length of the first type of text is greater than the line length of the second type of text.
[0029] The processing module is used to, for each text in the first type of text, add a newline character after the last character if the last character of the text is a preset segmentation symbol, and generate the processed text; otherwise, the text is determined to be the processed text.
[0030] The processing module is also used to add the newline character after the last character of each text in the second type of text to generate the processed text;
[0031] The splicing module is used to splice all the processed text according to the position of each line of text in the PDF file to be processed, and generate the target text.
[0032] In one possible design, the building module is specifically used for:
[0033] Extract text features from each line of text in the PDF file to be processed, and construct an initial row feature matrix of the PDF file to be processed. The number of rows in the initial row feature matrix is the same as the number of rows in the PDF file to be processed, and the number of columns in the initial row feature matrix is the same as the number of text features.
[0034] The data in each column of the initial row feature matrix is centered to generate the row feature matrix of the PDF file to be processed.
[0035] The text extraction module from PDF files is specifically used for:
[0036] Through the formula: Calculate the standardized data corresponding to each data point in the initial row feature matrix; where z is the standardized data corresponding to any data point in any row, x is the data point, u is the mean of all data points in any row, and s is the standard deviation of all data points in any row.
[0037] The initial row feature matrix is updated based on the standardized data corresponding to each data point in the initial row feature matrix to generate the row feature matrix of the PDF file to be processed.
[0038] In one possible design, the text features also include font size and / or number of characters.
[0039] In one possible design, the clustering module is specifically used for:
[0040] Based on the row feature matrix, each line of text in the PDF file to be processed is clustered to obtain a first center point and a second center point, wherein the row length of the first center point is greater than the row length of the second center point;
[0041] Calculate the first feature distance between each text and the first center point, and the second feature distance between each text and the second center point;
[0042] Based on the first feature distance and the second feature distance of each text, each text is classified to the center point corresponding to the smallest feature distance, and a first type of text and a second type of text are obtained. The center point of the first type of text is the first center point, and the center point of the second type of text is the second center point.
[0043] In one possible design, the splicing module is specifically used for:
[0044] Based on the position of each line of text in the PDF file to be processed, determine the connection relationship between the text and other texts;
[0045] Based on the connection relationship between each line of text and other texts, the corresponding processed text is concatenated with other processed texts to generate the target text.
[0046] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the processor to perform the text extraction method for PDF files as described in the first aspect and various possible designs of the first aspect.
[0047] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the text extraction method for PDF files as described in the first aspect and various possible designs of the first aspect.
[0048] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the text extraction method for PDF files as described in the first aspect and various possible designs of the first aspect.
[0049] The text extraction method, apparatus, device, and storage medium for PDF files provided in this application involve extracting text features from each line of text in the PDF file to be processed, constructing a line feature matrix for the PDF file, clustering each line of text in the PDF file to generate a first type of text and a second type of text based on the line feature matrix, and adding a newline character after the last character of each line of text in the first type of text if the last character of the text is a preset segmentation symbol, thus generating the processed text; otherwise, the text is determined as the processed text. A newline character is added after the last character of each line of text in the second type of text to generate the processed text, and all the processed texts are concatenated according to their position in the PDF file to generate the target text. The number of rows in the line feature matrix is the same as the number of rows in the PDF file to be processed, and the number of columns in the line feature matrix is the same as the number of text features. The text features include line length, and the line length of the first type of text is greater than the line length of the second type of text. In this technical solution, each line of text in the PDF file to be processed is classified according to its text features, and it is further determined whether to add line breaks to the first type of text that may be incomplete sentences, thereby improving the accuracy of text extraction. Attached Figure Description
[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0051] Figure 1 A schematic diagram of the PDF file provided for an embodiment of this application;
[0052] Figure 2 A schematic diagram of the text structure provided for embodiments of this application. Figure 1 ;
[0053] Figure 3 A flowchart illustrating the text extraction method for PDF files provided in this application embodiment. Figure 1 ;
[0054] Figure 4 A schematic diagram of the text structure provided for embodiments of this application. Figure 2 ;
[0055] Figure 5 A flowchart illustrating the text extraction method for PDF files provided in this application embodiment. Figure 2 ;
[0056] Figure 6 A schematic diagram of the structure of the text extraction device for PDF files provided in the embodiments of this application;
[0057] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0058] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0059] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0060] It should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals. The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0061] Next, the application background of this application will be explained.
[0062] Nowadays, many documents are formatted as PDFs for easy transmission and reading in work and study. However, a problem arises: the text in some PDF documents is difficult to recognize and extract. Due to the unique way PDFs store text, most existing PDF text recognition methods only perform text recognition while preserving the existing paragraph and line breaks in the PDF file.
[0063] For example, Figure 1 This is a schematic diagram of a PDF file provided for an embodiment of this application. Figure 1 The PDF file shown was processed for text extraction. The extracted text structure can be referenced. Figure 2 As shown. Figure 2 A schematic diagram of the text structure provided for embodiments of this application. Figure 1 .in, Figure 2 The numbers on the right represent the line numbers of the text, and the text content is on the right.
[0064] Reference Figure 1 and Figure 2 Because of the way PDFs store text, after extracting text from a PDF file, much text that shouldn't have been wrapped is wrapped. For example,... Figure 2 Lines 9 and 10 in the text are actually a complete sentence:
[0065] The "Several Measures on Promoting High-Quality Development of the Cultural and Tourism Industry" has been approved by the leaders and is hereby issued to you. Please implement it conscientiously in light of your actual circumstances.
[0066] However, the extracted text became:
[0067] The "Several Measures on Promoting the High-Quality Development of the Cultural and Tourism Industry" have been reviewed by all leaders.
[0068] With the approval of the leadership, this document is hereby issued to you. Please implement it conscientiously in light of your specific circumstances.
[0069] Here, \n represents a newline character.
[0070] The text extracted in this way is very unfavorable for subsequent analysis and processing of the file. For example, if you need to analyze the longest sentence in the PDF file later, because the longest sentence was forcibly segmented during extraction, other sentences will be incorrectly identified as the longest sentence in the PDF file. Furthermore, if you want to perform natural language processing tasks such as summarizing or classifying topics from PDF files, incomplete sentences will negatively impact the performance of subsequent tasks.
[0071] In summary, existing methods of storing text in PDFs often incorrectly break sentences into two when extracting text, resulting in poor accuracy in text extraction.
[0072] To address the aforementioned technical problems, this application proposes the following technical concept: Based on the characteristics of erroneously segmented text, this application clusters each line of text in the PDF file to be processed, obtaining a first category of text and a second category of text. The first category of text consists of text that needs further determination regarding whether line breaks are unnecessary, while the second category consists of text that requires line breaks. Further, it is determined whether the last character of each line in the first category is a preset segmentation symbol. If so, the text is determined to require a line break; otherwise, it is determined not to. In this way, a line break character can be added after the last character of the text requiring line breaks. By concatenating all the characters, semantically correct plain text can be obtained, improving the accuracy of text extraction.
[0073] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0074] Figure 3 A flowchart illustrating the text extraction method for PDF files provided in this application embodiment. Figure 1 .like Figure 3 As shown, the method for extracting text from this PDF file may include the following steps:
[0075] S31. Extract the text features of each line of text in the PDF file to be processed, and construct the line feature matrix of the PDF file to be processed.
[0076] The execution subject of this application embodiment is an electronic device, which can be a terminal device or a server. In practical applications, whether the electronic device is a terminal device or a server can be determined according to the actual situation, and this application embodiment does not impose specific restrictions on this.
[0077] In this step, the inventors analyzed the erroneous line breaks extracted from the PDF file using existing technology and found that these texts share characteristics such as: occupying the entire line length, having a similar (but not necessarily identical) number of characters, and having similar font sizes. Therefore, text features of each line of text in the PDF file to be processed can be extracted. Based on these text features, a line feature matrix can be constructed to characterize the PDF file, allowing subsequent determination of whether line breaks need to be added to each line of text.
[0078] The text feature includes line length, and may also include font size and / or number of characters.
[0079] In this row feature matrix, the number of rows is the same as the number of rows in the PDF file to be processed, and the number of columns is the same as the number of text features.
[0080] In practical applications, existing open-source applications can be used to parse the text content, font, font size, and text box size of each line of text in the PDF file to be processed. Then, based on the content obtained from the above parsing, the text features of each line of text in the PDF file to be processed can be extracted.
[0081] Furthermore, in one possible approach, text features of each line of text in the PDF file to be processed can be extracted to construct a line feature matrix of the PDF file. Here, each line of data in the line feature matrix represents the text features of the corresponding text in the PDF file.
[0082] In another possible approach, text features can be extracted from each line of text in the PDF file to be processed, constructing an initial row feature matrix for the PDF file. Here, each row of data in the initial row feature matrix represents the text features of the corresponding text in the PDF file.
[0083] Since the initial row feature matrix has multiple dimensions, in order to eliminate the differences between dimensions and focus more on the differences between different data in the subsequent clustering process, it is necessary to center the data in each column of the initial row feature matrix to generate the row feature matrix of the PDF file to be processed, so that the data contained in the row feature matrix are all data that follow a standard distribution with a mean of 0 and a standard deviation of 1.
[0084] This can be achieved through the following formula: Calculate the standardized data corresponding to each data point in the initial row feature matrix; where z is the standardized data corresponding to any data point in any row, x is any of the aforementioned data points, u is the mean of all data points in any row, and s is the standard deviation of all data points in any row.
[0085] After calculating the standardized data corresponding to each data point in the initial row feature matrix according to the above formula, the initial row feature matrix can be updated based on the standardized data corresponding to each data point in the initial row feature matrix to generate the row feature matrix of the PDF file to be processed.
[0086] S32. Based on the row feature matrix, cluster each line of text in the PDF file to be processed to generate the first type of text and the second type of text.
[0087] In this step, each line of text in the PDF file to be processed can be clustered using the k-means algorithm or the k-median algorithm. The distance function used in the clustering process can be Manhattan distance (also known as L1 distance) or Euclidean distance (also known as L2 distance).
[0088] The line length of the first type of text is greater than that of the second type of text.
[0089] In one possible implementation, S32 can be achieved through the following steps (321) to (323):
[0090] Step (321): Based on the row feature matrix, cluster each line of text in the PDF file to be processed to obtain the first center point and the second center point.
[0091] The row length of the first center point is greater than the row length of the second center point.
[0092] For example, the number of cluster centers can be pre-configured to 2 to cluster the text in the PDF file to be processed into two categories. After obtaining two center points through clustering, the text with incorrect line breaks has characteristics such as: occupying the entire line length, having a similar (but not necessarily identical) number of characters, and having similar font sizes. Therefore, the center point with the longest line length is determined as the class to be confirmed (i.e., the first category), and the other center point is determined as the normal class (i.e., the second category).
[0093] Step (322): Calculate the first feature distance between each text and the first center point, and the second feature distance between each text and the second center point.
[0094] Step (323): Based on the first feature distance and the second feature distance of each text, classify each text to the center point corresponding to the smallest feature distance, and obtain the first type of text and the second type of text.
[0095] Among them, the center point of the first type of text is the first center point, and the center point of the second type of text is the second center point.
[0096] S33. For each text in the first type of text, if the last character of the text is a preset segmentation symbol, then add a newline character after the last character to generate the processed text; otherwise, the text is determined as the processed text.
[0097] In this step, because the lines of the first type of text are relatively long, they may not be complete sentences, with some content remaining on the next line. Therefore, it is necessary to further determine whether the first type of text is a complete sentence. Since each complete sentence has the characteristic that its last character is a segmentation mark, we can determine whether the line of text is a complete sentence by checking whether the last character of each text is a preset segmentation mark.
[0098] Furthermore, if it is determined that the text in the line is a complete sentence, a newline character can be added directly after the last character, i.e., "\n" can be added; otherwise, it means that the text in the line is the same sentence as the text in the next line, so no newline character needs to be added.
[0099] For example, the preset segmentation symbols can be "\t", "\n", ";", ";", ".", "。", "?", "?", "!", or other segmentation symbols. They can be preset according to the actual situation, and there are no specific restrictions on them.
[0100] S34. Add a newline character after the last character of each text in the second type of text to generate the processed text.
[0101] In this step, since each line in the second type of text is shorter, indicating that it is a complete sentence, we can simply add a newline character after the last character.
[0102] S35. According to the position of each line of text in the PDF file to be processed, concatenate all the processed text to generate the target text.
[0103] In this step, after determining whether a newline character needs to be added after the last character of each line of text, and adding newline characters for the text that needs to be added, all the processed texts are simply concatenated according to the original order of each line of text in the PDF to be processed.
[0104] In one possible implementation, the connection relationship between each line of text and other text can be determined based on the position of each line of text within the PDF file to be processed. Based on this connection relationship, the corresponding processed text is then concatenated with the other processed text to generate the target text.
[0105] For example, suppose the connection relationship between text 1, text 2, and text 3 is: text 1 is connected to text 2 after it, and text 3 is connected to text 2 before it. Then, the corresponding processed texts are concatenated according to the above connection relationship, and the generated target text is: processed text 1 - processed text 2 - processed text 3.
[0106] In one possible implementation, the first line of text in the PDF file to be processed can be retrieved, and its processed text can be concatenated with the processed text of the next line. This process is repeated for each subsequent line, concatenating the processed text of that line with the processed text of the next line, until the last line is reached, thus generating the target text.
[0107] The text extraction method for PDF files provided in this application involves extracting text features from each line of text in the PDF file to be processed, constructing a line feature matrix for the PDF file, clustering each line of text in the PDF file based on the line feature matrix to generate a first type of text and a second type of text, and for each text in the first type of text, if the last character of the text is a preset segmentation symbol, a newline character is added after the last character to generate the processed text; otherwise, the text is determined as the processed text. A newline character is added after the last character of each text in the second type of text to generate the processed text, and all the processed texts are concatenated according to the position of each line of text in the PDF file to generate the target text. The number of rows in the line feature matrix is the same as the number of rows in the PDF file to be processed, and the number of columns in the line feature matrix is the same as the number of text features. The text features include line length, and the line length of the first type of text is greater than the line length of the second type of text. In this technical solution, each line of text in the PDF file to be processed is classified according to its text features, and it is further determined whether to add line breaks to the first type of text that may be incomplete sentences, thereby improving the accuracy of text extraction.
[0108] Based on any of the above embodiments, a specific example will be used to explain and illustrate them below.
[0109] The open-source application reads the PDF file to be processed and obtains the text features of each line of text.
[0110] The text features of some lines can be achieved using the following format:
[0111]
[0112]
[0113] Among them, line text is the content of the text, line size is the line length of the text, font size is the font size of the text, and char num is the number of characters in the text. These three are the text features.
[0114] Based on the above text features, an m×3 array (i.e., the initial row feature matrix) corresponding to the PDF file to be processed is constructed, where m is the number of rows in the PDF file and 3 represents the three text features. For example, the initial row feature matrix of the PDF file to be processed can be represented by the following matrix:
[0115] array([[245.21469116, 10.56000042, 12],
[0116] [6.73284912, 10.56000042, 2],
[0117] [143.06469727, 10.56000042, 12],
[0118] [7.847229, 10.56000042, 2],
[0119] [403.82881927, 21.95999908, 20]])
[0120] In the initial row feature matrix above, the first column value is the row length, the second column value is the font size, and the third column value is the number of characters.
[0121] Furthermore, each feature in the initial row feature matrix is centered to generate a new row feature matrix. For example, this row feature matrix can be represented by the following matrix:
[0122] array([[0.37814819, -0.49159217, 0.02905271],
[0123] [-1.29791709, -0.49159217, -1.27831922],
[0124] [-0.33976834, -0.49159217, 0.02905271],
[0125] [-1.29008515, -0.49159217, -1.27831922],
[0126] [1.49289819, 3.16328867, 1.07495025]])
[0127] Based on the above row feature matrix, each line of text in the PDF file to be processed is clustered, resulting in two cluster centers:
[0128] array([[-0.53581853,-0.49159217,-0.52406618],
[0129] [1.39312817, 1.27813965, 1.36257207]])
[0130] Since the first feature (row length) of the second center point is 1.39312817, which is higher than the first center point's -0.53581853, the second center point is determined as the first center point, and the first center point is determined as the second center point.
[0131] Based on the first centroid and the first centroid mentioned above, each line of text in the PDF file to be processed is clustered, and the following clustering results are obtained.
[0132] array([0,0,0,0,1])
[0133] In the above clustering results, each number corresponds to a line of text, where 0 represents that the line of text belongs to the first category and 1 represents that the line of text belongs to the second category.
[0134] Finally, based on the clustering results, newline characters are added to the text that needs to be added in each category of text, and all texts are concatenated after processing to generate the target text.
[0135] In practical applications, based on this technical solution... Figure 1 The PDF file shown is used to extract text. The extracted text format can be referenced. Figure 4 As shown.
[0136] Figure 4 A schematic diagram of the text structure provided for embodiments of this application. Figure 2 .like Figure 4 As shown, the text extracted using this technical solution does not contain the incorrect line break in the sentence, "The 'Several Measures on Promoting the High-Quality Development of the Cultural and Tourism Industry' has been approved by the leaders and is hereby issued to you. Please implement it conscientiously in light of your actual circumstances." This effectively improves the accuracy of text extraction.
[0137] Figure 5 A flowchart illustrating the text extraction method for PDF files provided in this application embodiment. Figure 2 .like Figure 5 As shown, the method for extracting text from this PDF file may include the following steps:
[0138] S51. Use the PDF reading module to obtain each line of text in the PDF file to be processed.
[0139] The PDF reading module can be implemented using existing open-source applications.
[0140] S52. Collect the text features of each line of text in the PDF file to be processed, and construct a centralized line feature matrix.
[0141] S53. Cluster each line of text in the PDF file to be processed, and add line breaks to the text that needs to be added based on the clustering results.
[0142] S54. Concatenate the processed text.
[0143] This technical solution can also be used for semantic restoration of parsed plain text. That is, after extracting text using existing techniques, the obtained text is reprocessed to remove incorrect line breaks and retain correct line breaks. Determining whether a line break is incorrect mainly involves clustering the extracted text using existing techniques to generate a first-class text and a second-class text. The line length of the first-class text is greater than that of the second-class text. The second-class text needs to retain correct line breaks, and in the first-class text, any text whose last character is a preset segmentation symbol needs to retain correct line breaks; otherwise, they are deleted.
[0144] Existing technologies can also extract text using recurrent neural networks (RNNs) and word2vec. However, RNNs and word2vec have the following problems:
[0145] Recurrent Neural Networks: Because they are processed sequentially over time, long-term information needs to be traversed sequentially through all units before entering the current processing unit. Each unit has 4 linear layers (Multilayer Perceptron (MLP) layers) running in each sequential time step, which requires a lot of storage bandwidth and computation time, thus limiting the performance of the model.
[0146] word2vec: A neural network that uses context prediction. It requires word segmentation of the text, consumes a lot of memory when loading the model, and requires a large number of samples to avoid the randomness of the results.
[0147] The K-means method used in this application is an unsupervised learning method for cluster analysis. It can automatically find patterns and group samples with similar attributes together without training labels (no need for a large amount of labeled data) or a lot of time to train the model. It can label the samples through simple and easy-to-understand iterative loops. It has the advantages of using a controllable amount of samples, unlabeled samples, and automatic grouping, which effectively increases the efficiency of text extraction.
[0148] In summary, this technical solution has the following technical effects:
[0149] 1. Based on the characteristics of PDF text with incorrect segmentation, a feature array (row feature matrix) was designed, which includes text features such as text line length, font, and number of characters. The three text features of the text line can be used to clearly distinguish the title, body text, and text with segmentation errors.
[0150] 2. Based on the characteristics of erroneous segments in PDF text, a two-centroid clustering method is used to determine the centers of the erroneous segments among the two centroids, transforming the unsupervised learning clustering algorithm into a binary classification algorithm. The algorithm is simple, uses few parameters, and is easy to understand. Because it uses an unsupervised learning clustering algorithm from machine learning, no manual data and model training is required during the entire computation process, saving significant time and resources. Furthermore, it does not consider the semantic information of the text itself, thus saving considerable computational resources.
[0151] 3. This technical solution is based on the judgment results of the binary classification method, and combines suspected erroneous segmentation with semantic habits to splice the processed text to maintain the coherence of the text.
[0152] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.
[0153] Figure 6 This is a schematic diagram of the structure of a text extraction device for PDF files provided in an embodiment of this application. Figure 6 As shown, the text extraction device 60 for the PDF file includes:
[0154] Module 61 is used to extract text features from each line of text in the portable PDF file to be processed, and to construct a row feature matrix of the PDF file to be processed. The number of rows in the row feature matrix is the same as the number of rows in the PDF file to be processed, and the number of columns in the row feature matrix is the same as the number of text features. The text features include the line length.
[0155] Clustering module 62 is used to cluster each line of text in the PDF file to be processed based on the line feature matrix, generating a first type of text and a second type of text, wherein the line length of the first type of text is greater than the line length of the second type of text.
[0156] Processing module 63 is used to, for each text in the first type of text, add a newline character after the last character if the last character of the text is a preset segmentation symbol, and generate the processed text. Otherwise, the text is determined to be the processed text.
[0157] The processing module 63 is also used to add a newline character after the last character of each text in the second type of text to generate the processed text.
[0158] The splicing module 64 is used to splice all the processed text according to the position of each line of text in the PDF file to be processed, and generate the target text.
[0159] In one possible design, module 61 is specifically used for:
[0160] Extract text features from each line of text in the PDF file to be processed, and construct an initial row feature matrix for the PDF file to be processed. The number of rows in the initial row feature matrix is the same as the number of rows in the PDF file to be processed, and the number of columns in the initial row feature matrix is the same as the number of text features.
[0161] The data in each column of the initial row feature matrix is centered to generate the row feature matrix of the PDF file to be processed.
[0162] Text extraction from PDF files, module 61, is specifically used for:
[0163] Through the formula: Calculate the standardized data for each data point in the initial row feature matrix. Here, z represents the standardized data for any data point in any row, x represents any data point, u represents the mean of all data points in any row, and s represents the standard deviation of all data points in any row.
[0164] The initial row feature matrix is updated based on the standardized data corresponding to each data point in the initial row feature matrix to generate the row feature matrix of the PDF file to be processed.
[0165] In one possible design, text features also include font size and / or number of characters.
[0166] In one possible design, clustering module 62 is specifically used for:
[0167] Based on the row feature matrix, each row of text in the PDF file to be processed is clustered to obtain the first center point and the second center point. The row length of the first center point is greater than the row length of the second center point.
[0168] Calculate the first feature distance between each text and the first center point, and the second feature distance between each text and the second center point.
[0169] Based on the first feature distance and the second feature distance of each text, each text is classified to the center point corresponding to the smallest feature distance, and the first type of text and the second type of text are obtained. The center point of the first type of text is the first center point, and the center point of the second type of text is the second center point.
[0170] In one possible design, the splicing module 64 is specifically used for:
[0171] Determine the connection relationships between each line of text and other text based on the position of each line of text in the PDF file to be processed.
[0172] Based on the connection relationship between each line of text and other texts, the corresponding processed text is concatenated with other processed texts to generate the target text.
[0173] The text extraction device for PDF files provided in this application embodiment can be used to execute the text extraction method for PDF files in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be described again here.
[0174] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. These modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. Each module can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored as program code in the device's memory, and its functions can be called and executed by a processing element. Furthermore, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the processor element or through software instructions.
[0175] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device may include a processor 71 and a memory 72.
[0176] Processor 71 executes computer execution instructions stored in memory, causing processor 71 to perform the scheme in the above embodiments. Processor 71 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0177] The memory 72 is connected to the processor 71 via the system bus and completes communication between them. The memory 72 is used to store computer program instructions.
[0178] Alternatively, the electronic device may also include a transceiver.
[0179] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.
[0180] The electronic device provided in this application embodiment can be used to execute the technical solution of the PDF file text extraction method provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.
[0181] This application also provides a chip for executing instructions, which is used to execute the text extraction method for PDF files described in the above embodiments.
[0182] This application also provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the text extraction method for PDF files described in the above embodiments.
[0183] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor can read the computer program from the computer-readable storage medium, and when the processor executes the computer program, it can implement the technical solution of the text extraction method for PDF files in the above embodiments.
[0184] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.
[0185] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.
[0186] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.
[0187] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.
[0188] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.
[0189] The memory may include high-speed RAM, or it may also include non-volatile storage (NVM), such as disk storage, or it may be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.
[0190] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0191] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.
[0192] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.
[0193] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A method for extracting text from a PDF file, characterized in that, include: Extract text features from each line of text in the portable file format PDF file to be processed, and construct a row feature matrix of the portable file format PDF file to be processed. The number of rows in the row feature matrix is the same as the number of rows in the portable file format PDF file to be processed, and the number of columns in the row feature matrix is the same as the number of text features. The text features include the line length. Based on the row feature matrix, each line of text in the PDF file to be processed is clustered to obtain a first center point and a second center point, wherein the line length of the first center point is greater than the line length of the second center point. Calculate the first feature distance between each line of text and the first center point, and the second feature distance between each line of text and the second center point; Based on the first feature distance and the second feature distance of each line of text, each line of text is classified to the center point corresponding to the smallest feature distance, and a first type of text and a second type of text are obtained. The center point of the first type of text is the first center point, and the center point of the second type of text is the second center point. The line length of the first type of text is greater than the line length of the second type of text. For each line of text in the first type of text, if the last character of the text is a preset segmentation symbol, then a newline character is added after the last character to generate the processed text; otherwise, the text is determined to be the processed text. Add the newline character after the last character of each line of text in the second type of text to generate the processed text; Based on the position of each line of text in the PDF file to be processed, all processed text is concatenated to generate the target text.
2. The method according to claim 1, characterized in that, The step of extracting text features from each line of text in the portable PDF file to be processed, and constructing the line feature matrix of the portable PDF file to be processed, includes: Extract text features from each line of text in the portable file format PDF file to be processed, and construct an initial row feature matrix of the portable file format PDF file to be processed. The number of rows in the initial row feature matrix is the same as the number of rows in the portable file format PDF file to be processed, and the number of columns in the initial row feature matrix is the same as the number of text features. The data in each column of the initial row feature matrix is centered to generate the row feature matrix of the PDF file to be processed.
3. The method according to claim 2, characterized in that, The step of centering each column of data in the initial row feature matrix to generate the row feature matrix of the portable PDF file to be processed includes: Through the formula: Calculate the standardized data corresponding to each data point in the initial row feature matrix; where, For any data point in any row of data, the standardized data is the data corresponding to any data point. For any of the data, The mean of all data in any given row is given. The standard deviation of all data in any given row of data; The initial row feature matrix is updated based on the standardized data corresponding to each data point in the initial row feature matrix to generate the row feature matrix of the portable PDF file to be processed.
4. The method according to any one of claims 1 to 3, characterized in that, The text features also include font size and / or number of characters.
5. The method according to any one of claims 1 to 3, characterized in that, The step of concatenating all processed text according to the position of each line of text in the PDF file to be processed, to generate the target text, includes: The connection relationship between each line of text and other texts is determined based on the position of each line of text in the PDF file to be processed; Based on the connection relationship between each line of text and other texts, the corresponding processed text is concatenated with other processed texts to generate the target text.
6. A text extraction device for PDF files, characterized in that, The PDF file text extraction device is used to implement the PDF file text extraction method according to any one of claims 1-5, the device comprising: A construction module is used to extract text features from each line of text in a portable file format PDF file to be processed, and construct a row feature matrix of the portable file format PDF file to be processed. The number of rows in the row feature matrix is the same as the number of rows in the portable file format PDF file to be processed, and the number of columns in the row feature matrix is the same as the number of text features. The text features include line length. The clustering module is used to cluster each line of text in the portable file PDF to be processed based on the line feature matrix, generating a first type of text and a second type of text, wherein the line length of the first type of text is greater than the line length of the second type of text. The processing module is used to, for each text in the first type of text, add a newline character after the last character if the last character of the text is a preset segmentation symbol, and generate the processed text; otherwise, the text is determined to be the processed text. The processing module is also used to add the newline character after the last character of each text in the second type of text to generate the processed text; The concatenation module is used to concatenate all processed text according to the position of each line of text in the PDF file to be processed, and generate the target text.
7. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 5.
9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 5.