Methods, systems, media, and devices for parsing PDF tables based on financial transaction records.

Abstract

This invention provides a method, system, medium, and device for parsing PDF tables based on financial transaction records, belonging to the field of computer software technology. The method includes the following steps: S1, importing the PDF transaction record file to be processed and preprocessing the PDF transaction record file, including decryption, watermark removal, and institution matching; S2, determining the table type based on the content of the preprocessed PDF transaction record file, determining the table processing flow according to the table type, and parsing the table according to the table processing flow to obtain the table text data; standardizing and structuring the table text data to obtain readable structured data; the embodiments of this invention can effectively identify and filter invalid data rows, and solve the problem of table content misalignment caused by text wrapping, significantly improving the accuracy and completeness of data extraction.

Description

Technical Field

[0001] This application relates to the field of computer software technology, and in particular to methods, systems, media, and devices for parsing PDF tables based on financial transaction records. Background Technology

[0002] With the digitalization of financial services, a large amount of transaction data is stored and transmitted in PDF format. To achieve data reuse and analysis, efficiently and accurately extracting tabular data from PDF files has become a key technology. Currently, various tools and technologies exist for extracting tabular data from PDFs, with Tabula being a typical open-source tool. Tabula focuses on extracting tabular data from PDF files and converting it into structured formats (such as CSV and Excel) for easier subsequent data processing and analysis. Its core technology involves using Java-based libraries (such as PDFBox) to perform low-level parsing of PDF files, obtaining text content and its layout information, including the position coordinates of text blocks, font size, and character spacing. Tabula does not rely on OCR (Optical Character Recognition) technology; instead, it identifies table areas by parsing the internal structure of the PDF and automatically determines the rows and columns of the table based on the text's arrangement on the page, thus reconstructing the original table structure. This method is highly efficient and accurate when processing electronic PDFs (non-scanned images).

[0003] However, other PDF parsing tools similar to these still have certain limitations in functionality. For example, while traditional PDF parsing tools such as iText and PDFBox can extract text content from PDFs, they cannot effectively recognize and restore table structures, making the extracted data difficult to use directly for analysis. On the other hand, OCR tools such as Tesseract are mainly designed for scanned or image-based PDFs, extracting text content through image recognition. Although these tools can achieve certain results when the image quality is good, their processing speed is slow and they are easily affected by factors such as image resolution and noise interference, resulting in unstable recognition accuracy.

[0004] In contrast, Tabula's advantage lies in its ability to directly parse the vector data structure of PDFs without relying on OCR technology. This not only improves processing efficiency but also enhances the stability and accuracy of table recognition. Especially when dealing with borderless tables, Tabula intelligently infers the table structure based on text position and layout patterns. Nevertheless, Tabula still carries the risk of errors or omissions when dealing with complex layouts, multiple merged cells, and tables spanning multiple pages, making it difficult to fully meet the high-precision data extraction needs of the financial sector.

[0005] With the rapid development of the financial industry, banks and other financial institutions generally provide customers with PDF export functions for transaction records. Currently, credit review, risk assessment, and other operations still heavily rely on manual reading and analysis of customer-provided transaction records to determine income, expenses, and liabilities, which is inefficient and prone to errors. Although some systems have attempted to automatically parse transaction data, the inconsistent PDF formats output by various financial institutions, especially the prevalence of borderless tables, line breaks in tables, and irregular table styles, make it difficult for existing tools to accurately identify and extract structured information. Therefore, there is an urgent need to develop a universal PDF transaction record parsing method that adapts to multiple formats to improve the automation and accuracy of financial data analysis. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a method, system, medium, and device for parsing PDF tables based on financial transaction records, so as to solve at least one of the above-mentioned technical problems.

[0007] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a method for parsing PDF tables based on financial transaction records, comprising the following steps:

[0008] S1, import the PDF transaction log file to be processed, and preprocess the PDF transaction log file, including decryption processing, watermark removal processing and classification processing;

[0009] S2, determine the table type based on the content of the preprocessed PDF transaction log file, determine the table processing flow according to the table type, parse the table according to the table processing flow to obtain table text data; standardize and structure the table text data to obtain readable structured data; the table type includes bordered tables, borderless tables, and tables that require line drawing.

[0010] Secondly, embodiments of the present invention provide a PDF table parsing system based on financial transaction records, comprising the following modules:

[0011] The import and preprocessing module is used to import the PDF transaction log file to be processed and preprocess the PDF transaction log file. The preprocessing includes decryption processing, watermark removal processing, and classification processing.

[0012] The table processing module is used to determine the table type based on the content of the preprocessed PDF transaction log file, determine the table processing flow according to the table type, parse the table to obtain table text data according to the table processing flow, and standardize and structure the table text data to obtain readable structured data; the table types include bordered tables, borderless tables, and tables that require line drawing.

[0013] Thirdly, embodiments of the present invention provide an electronic device, comprising:

[0014] One or more processors;

[0015] A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement a PDF table parsing method based on financial transaction records as described in the first aspect.

[0016] Fourthly, embodiments of the present invention provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the PDF table parsing method based on financial transaction records described in the first aspect.

[0017] The above technical solution has the following beneficial effects:

[0018] The embodiments of the present invention can accurately identify and extract tabular data from financial transaction records, overcome the limitations of existing technologies in this scenario, and improve data parsing efficiency and accuracy;

[0019] The embodiments of the present invention can effectively identify and filter invalid data rows (such as blank rows, comment rows, etc.), and at the same time solve the problem of table content misalignment caused by text wrapping, which significantly improves the accuracy and completeness of data extraction.

[0020] The embodiments of the present invention are based entirely on PDF text layout information for parsing, without relying on OCR technology or additional image processing procedures. They have high parsing efficiency, low resource consumption, are suitable for large-scale batch processing scenarios, and do not incur additional software and hardware costs.

[0021] The embodiments of the present invention have strong versatility and can be adapted to PDF transaction files of different formats from multiple financial institutions without the need for customized development for specific templates, which greatly improves the system's adaptability and deployment efficiency. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a PDF table parsing method based on financial transaction records provided in an embodiment of the present invention;

[0024] Figure 2 This is a detailed flowchart of step S1 in an embodiment of the present invention;

[0025] Figure 3 This is a detailed flowchart of step S13 in an embodiment of the present invention;

[0026] Figure 4 This is a detailed flowchart of step S12 in an embodiment of the present invention;

[0027] Figure 5 This is a detailed flowchart of step S14 in an embodiment of the present invention;

[0028] Figure 6 This is a flowchart of the frameless table processing according to an embodiment of the present invention;

[0029] Figure 7 This is a flowchart of the special frame table processing in an embodiment of the present invention;

[0030] Figure 8 This is a functional block diagram of a PDF table parsing system based on financial transaction records according to an embodiment of the present invention;

[0031] Figure 9 This is a functional block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0032] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0033] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0034] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" indicates "at least one embodiment"; the term "another embodiment" indicates "at least one additional embodiment"; the term "some embodiments" indicates "at least some embodiments". Definitions of other terms will be given in the description below.

[0035] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0036] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0037] This invention improves and extends the open-source tool Tabula to enhance its ability to parse complex PDF tables. While traditional Tabula performs well with structured tables, it has limitations when dealing with borderless tables, data wrapping, and irregularly styled tables. Therefore, this invention introduces several innovative modules and techniques. For example, it adds file decryption, watermark removal, and authentication processing. File decryption can handle encrypted PDF files, ensuring the ability to parse protected document content; watermark removal can embed digital watermarking technology to identify the source or add copyright information to the parsing results, enhancing data security and traceability; authentication processing can verify the integrity and authenticity of PDF files, preventing data tampering or forgery. For the numerous borderless, data-wrapped, and irregularly styled tables in financial cloud transaction logs, this invention proposes a PDF table parsing method based on financial transactions, specifically a borderless table processing method. This method combines graphics algorithms and processing techniques to automatically identify table regions, infer logical structures, and correctly parse cross-line data. This embodiment improves adaptability to non-standard tables and expands the application scope of Tabula.

[0038] Terminology Explanation:

[0039] Transaction history: refers to all transaction details of a customer's account within a certain period of time, as recorded by a financial institution. This includes information such as income, expenses, and balance, and is an important basis for assessing a customer's financial situation.

[0040] PDF (Portable Document Format): A universal file format that preserves the original document's layout, fonts, and images, and is widely used for storing and transmitting electronic documents.

[0041] Tabula: An open-source PDF table data extraction tool that excels at extracting table content from PDFs without image scanning and supports exporting to structured formats such as CSV and Excel.

[0042] iText PDF: iText PDF is a Java library for creating and manipulating PDF documents. It supports generating, modifying, and encrypting PDFs, and provides rich document processing functions.

[0043] PDF Box: Apache PDF Box is an open-source Java library for processing PDF documents, supporting functions such as content extraction, text recognition, form filling, and low-level object manipulation.

[0044] OCR (Optical Character Recognition): Optical character recognition technology is used to convert text in images or scanned documents into editable text, and is commonly used when processing image-based PDFs.

[0045] COSString (Common Object Structure String): This is a core class in the Apache PDF Box library, belonging to the org.apache.pdfbox.cos package. It is used to represent string-type data in the underlying object (COS, i.e., Common Object Structure) of a PDF file. Example 1

[0046] like Figure 1 As shown in the figure, the flowchart of a PDF table parsing method based on financial transaction records provided by an embodiment of the present invention includes the following steps:

[0047] S1: Import the PDF transaction log file to be processed and preprocess the PDF transaction log file. The preprocessing includes decryption, watermark removal and classification.

[0048] In this embodiment, the transaction log files in PDF format to be processed are first imported into the system. If the transaction log files are encrypted, the password required for decryption is provided by the user through an interface. The iText PDF tool is used to decrypt them first. The specific encryption algorithm is determined by the encryption method used in the original PDF file, and the tool library will automatically identify and process it to ensure that subsequent processing can proceed normally. After decryption, the watermark information in the transaction log files is removed to avoid watermark interference with subsequent data parsing and analysis. The watermark-removed transaction log files are then categorized to provide a basis for subsequent processing. In addition, the authenticity of the transaction log files can be verified to check whether the decrypted transaction log files have been tampered with or forged, to ensure the reliability and integrity of the data.

[0049] S2 determines the table type based on the content of the preprocessed PDF transaction log file, and determines the table processing flow according to the table type. It then parses and processes the table to obtain the table text data according to the table processing flow. The table text data is then standardized and structured to obtain readable structured data. The table types include bordered tables, borderless tables, and tables that require line drawing.

[0050] Specifically, for the preprocessed file, the table type is determined based on the content of the transaction log file: bordered or borderless. If the table has explicit borders, it is a bordered table and is processed using the bordered table processing method, i.e., using Tabula's SPREADSHEEET_EXTRACTION_ALGORITHM method (table extraction method) for parsing. If the table does not have obvious borders, it is a borderless table and is processed using the borderless table processing method, i.e., using a specialized algorithm to identify and parse this type of table. For special tables, i.e., tables that cannot be parsed successfully by the above two methods, a table line drawing method is used for parsing; parsing refers to the process of extracting table data from the PDF format transaction log file and converting it into computer-readable structured information. Regardless of whether the table has borders, has no borders, or is a special table, after processing, the text data in the table is standardized and structured to extract valuable information. For example, due to significant differences in the PDF transaction record formats used by different financial institutions (e.g., date formats may be YYYY-MM-DD, MM / DD / YYYY, or different year-month-day formats, and amounts may include currency symbols such as ¥ and $, or thousands separators), the parsed data will present diverse formats. Standardization processing can convert these heterogeneous data into a pre-defined standardized format (e.g., standardizing dates to YYYY-MM-DD, retaining the numerical portion of amounts and standardizing units), ensuring data consistency at the format level.

[0051] Furthermore, the parsing process may introduce invalid information (such as blank rows, duplicate rows, and garbled characters in the table) or data errors (such as typos and number confusion caused by OCR recognition errors). Standardization processing identifies and removes meaningless data rows through invalid row detection and data cleaning, eliminating redundant content, correcting obvious errors, and improving the accuracy and purity of the data. Further, the parsed raw text data still has unstructured characteristics (such as merging multiple lines of text and disordered field order), while subsequent business systems (such as credit review systems and risk assessment models) typically require strictly structured data (such as two-dimensional tables and JSON objects). This embodiment converts the parsed text data into a two-dimensional array or a structured object in JSON format and stores the structured table data. Standardization processing can map text data to a preset field system (such as fixed fields like transaction date, transaction amount, and transaction type), ensuring that the data can be directly called and analyzed by downstream systems. In financial business, it is often necessary to summarize and analyze the transaction data of different institutions (such as comprehensively assessing a customer's income and expenditure at multiple banks).

[0052] Standardized data has a unified format and field definitions, enabling cross-institutional and cross-format data analysis and enhancing its versatility and application value. Standardized data formats are easier to store in databases (such as relational databases and data warehouses) and support subsequent statistical analysis and data mining operations (e.g., identifying abnormal transactions through transaction amount distribution and analyzing customer financial stability through income and expenditure frequency), thereby fully leveraging the decision-making support role of transaction flow data in financial operations. In this embodiment, standardization processing of parsed text data eliminates data format differences, improves data quality, and provides a unified foundation for subsequent structured analysis, storage, and application.

[0053] Based on the layout characteristics of borderless financial transaction tables, this invention designs an intelligent recognition mechanism to effectively identify and filter invalid data rows, solve the problem of misaligned table content caused by text wrapping, and improve the accuracy and completeness of data extraction.

[0054] The embodiments of the present invention are based entirely on PDF text layout information for parsing, without relying on OCR (Optical Character Recognition) technology or additional image processing procedures. They have high parsing efficiency, low resource consumption, are suitable for large-scale batch processing scenarios, and do not generate additional software and hardware costs.

[0055] The embodiments of the present invention are highly versatile and can be adapted to PDF transaction files of different formats from multiple financial institutions. No customized development is required for specific templates, which greatly improves the system's adaptability and deployment efficiency.

[0056] like Figure 2 As shown, in some embodiments, step S1 may specifically include the following steps:

[0057] S11 decrypts the encrypted PDF transaction log file, automatically identifying and adapting the encryption algorithm used in the PDF transaction log file during the decryption process.

[0058] In this embodiment, encrypted transaction records can be decrypted using the iText PDF tool. By parsing the encrypted metadata of the PDF file (such as encryption algorithm identifiers, permission settings, etc.), the encryption method can be automatically identified and the corresponding decryption logic can be called without manually specifying the algorithm.

[0059] This embodiment demonstrates how iText PDF can efficiently process financial transaction record PDFs with different encryption algorithms, providing a foundation for subsequent table parsing and data extraction.

[0060] S12: Analyze the PDF text objects page by page in the decrypted PDF transaction log file, calculate the slant of the PDF text objects through the text transformation matrix, and identify and delete PDF text objects with a non-zero slant as watermarks.

[0061] In this embodiment, the position, scaling, and rotation of text objects in the PDF are defined using a Text Transformation Matrix (Tm). Normal text (such as table content) is usually unskewed (skewness is 0), while watermark text is often skewed at 45° or 135° to avoid affecting the readability of the main text, and its transformation matrix includes rotation parameters. The text transformation matrix is ​​a 3×3 matrix, simplified to [a,b,c,d,e,f], where the parameters b and c determine the skew and rotation states. If b=0 and c=0, the text is unskewed (normal content); if b≠0 or c≠0, the text is skewed (for watermarks).

[0062] The text transformation matrix of this invention calculates the slant of the text, which can automatically identify and remove slanted watermarks in PDF transaction records, clearing obstacles for subsequent table parsing and data extraction, and ensuring that the extracted text contains only valid transaction information.

[0063] S13, sequentially match the institutions in the preset institution feature list by PDF digital signature, PDF watermark recognition, and the first page text title line or table header line, determine the financial institution to which the PDF transaction flow file belongs and the file type, and store the feature data of the unmatched PDF transaction flow file as new institution features into the preset institution feature list.

[0064] In this embodiment, the PDF transaction formats of different institutions (e.g., header layout, field naming, encryption methods, watermark styles, etc.) vary significantly (e.g., the transaction formats of banks and third-party payment platforms are completely different). By identifying the institution, the corresponding parsing rules (e.g., field mapping relationships, special symbol processing procedures) can be automatically matched, avoiding errors caused by using a uniform parsing logic for all files (e.g., missing fields, format confusion). Transaction records from the same institution may contain different business types (e.g., personal transfer records, corporate bank statements, credit card statements, etc.), and their data dimensions (e.g., whether they include transaction fees, merchant information) and subsequent uses (e.g., financial auditing, credit approval) differ. After identifying the transaction type, the corresponding processing module can be called in a targeted manner (e.g., in auditing scenarios, it is necessary to focus on extracting counterparties, remarks, and fields), reducing invalid data processing steps. This embodiment analyzes the feature information of PDF files layer by layer, attempting to identify the institution and transaction type in turn.

[0065] like Figure 3 As shown, step S12 may specifically include:

[0066] S121, Load the decrypted PDF transaction log file into the file stream in memory, and count the total number of pages in the PDF transaction log file;

[0067] Specifically, the process reads the PDF transaction log file into a file stream. First, the target PDF transaction log file is loaded into a file stream in memory for subsequent page-by-page parsing and processing. The total number of pages in the PDF file is then counted to prepare for subsequent loop processing of each page.

[0068] S122, Loop through each page according to the total number of pages, and process each page in the PDF transaction log file in turn. If the current page has not been processed, continue to execute steps S123 to S126. If the current page has been processed, the process ends.

[0069] S123, extract all text objects on the current page, determine whether the text type is COSString based on the text object, if the text type is COSString, then proceed to step S124, if the text type is not COSString, it indicates that the text object is not watermark text, then proceed to step S122.

[0070] Specifically, the process reads the PDF text object for that page. For the current page, it extracts all text objects (including the main text and any potential watermarks). These text objects are stored as text elements within the PDF. The process checks if the text type is COSString. If it is, it indicates that the text object is either plain text or watermark text within the PDF, requiring further analysis of its operation type. If it is not COSString, the process skips that text object and proceeds to the next item.

[0071] COSString is a core class in the Apache PDFBox library, belonging to the org.apache.pdfbox.cos package. It represents string data in the underlying Common Object Structure (COS) of a PDF file. Internally, all content in a PDF document is described by a series of COS objects, including numbers, booleans, arrays, dictionaries, etc., and COSString is specifically designed for handling string-type COS objects. It can store various text information in a PDF, such as the document's body text, watermark text, and form field content. When processing PDFs, the COSString class can be used to retrieve and manipulate string data for tasks such as text extraction, watermark recognition, and document content analysis. In this example, we first determine if a text object is of type COSString, and then further analyze whether it is watermark text.

[0072] S124, obtain the operation type of the text object, determine whether the operation type is ShowText, if the operation type is ShowText, then execute step S125; if the operation type is not ShowText, it indicates that the text object is not watermark text, then execute S122.

[0073] Specifically, for text objects confirmed to be of type COSString, their operation type is further analyzed. Text operations in PDFs are defined by specific operation instructions; for example, ShowText is used to display text. The operation type is determined to be ShowText. If it is ShowText, it means the text object is actually displayed on the page, and further determination is needed to determine if it is a watermark. If it is not ShowText, the text object is skipped, and the process moves to the next item.

[0074] In the underlying content stream of a PDF document, ShowText is an operation type used to draw text on the page, belonging to the text drawing instructions defined in the PDF specification. Its function is to display a specified string at a specific location on the PDF page, typically used in conjunction with attributes such as text coordinates, font, and size. In the PDF content stream, ShowText operations are usually represented by instructions (Tj, TJ), where Tj displays a single string and TJ displays an array of strings. When parsing PDF text, the ShowText operation is crucial for extracting visible text, as only text drawn through this operation will be displayed on the page. For example, in watermark detection, by identifying the text object corresponding to the ShowText operation, this type of operation directly determines the actual text content presented on the page.

[0075] S125, Obtain the matrix array information of the text object, calculate the slant of the text object based on the matrix array information, and determine whether the slant of the text object is equal to 0. If the slant is equal to 0, it indicates that the text object is not watermark text, and execute S122; if the slant is not equal to 0, it indicates that the text object is watermark text, and execute step S126.

[0076] Specifically, the process involves obtaining the matrix array information of the text, calculating the slant angle, and extracting the corresponding matrix array information for text objects confirmed to be part of the ShowText operation. The text position and style in a PDF are described by a matrix; by parsing the matrix array, the slant angle of the text can be calculated. The process checks if the slant angle is equal to 0. If it is, the text is horizontally aligned, usually part of the main content, and is marked as "This text is not watermark text," and the deletion operation is skipped. If the slant angle is not equal to 0, the text is slanted, which is one of the characteristics of a watermark, and is marked as "This text is watermark text."

[0077] S126, Remove the watermark text from the current page.

[0078] For text objects marked as "watermarked text," they are removed from the PDF page. This process requires rewriting the filtered text objects into the PDF using a Content Stream Writer, updating the PDF's underlying data structure to ensure the watermark completely disappears without affecting other normal text.

[0079] Specifically, the target document is first loaded using PDFBox's PDDocument to obtain the content stream (PDPage Content Stream) for each page. The content stream is a sequence of drawing instructions, including text drawing (such as ShowText) and graphic drawing operations. All operation instructions in the content stream are iterated through, filtering out ShowText operations containing watermarked text (i.e., the Tj / TJ instructions corresponding to the COSString object already marked as watermarked). Other necessary operations such as drawing instructions for normal text, font settings, and coordinate transformations are retained. The content stream is reconstructed, and a Content Stream Writer object is created to rewrite the filtered instructions back to the page. The specific process is as follows: the original page's content stream is opened and read, instructions are processed line by line, all Show Text operations containing watermarked text are skipped, the remaining valid instructions are written to the new content stream using the ContentStream Writer, the original content stream is closed, and the original page content is replaced with the newly generated content stream. The underlying data structure needs to be updated. The page content of the PDF document references the content stream through indirect objects. After the watermark is removed, the object space occupied by the original content stream needs to be released, the references to the content stream in the Page Dictionary need to be updated, and the document's cross-reference table (XRef Table) needs to be recalculated to ensure the consistency of object references.

[0080] like Figure 5 As shown, in some embodiments, step S13 specifically includes:

[0081] S131, Load the PDF transaction log file into a file stream in memory and obtain a preset list of institutional characteristics.

[0082] First, the target PDF file is loaded into a file stream in memory to prepare for subsequent page-by-page parsing and content analysis.

[0083] S132, parse whether the PDF transaction record file has a digital signature certificate. If a digital signature certificate exists, search for a matching institution record in the institution feature list based on the digital signature certificate to determine the corresponding financial institution and transaction type. If no digital signature certificate exists or no match is found, proceed to step S133.

[0084] Use the iText PDF tool to parse whether the PDF file has a digital signature certificate. If a signature certificate exists, extract the signer information and search for a matching record in the organization feature list; if no signature certificate exists or no match is found, proceed to the next step.

[0085] S133, identify whether there is watermark text in the PDF transaction log file through the watermark detection algorithm. If watermark text exists, search for matching institution records in the preset institution feature list according to the watermark text to determine the corresponding institution and transaction type. If there is no watermark or no match is found, proceed to step S134.

[0086] The watermark detection algorithm is applied to the PDF file to identify whether there is watermark text. If watermark text exists, the system searches for a matching organization record in the preset organization feature list to complete the identification. If there is no watermark or the match is unsuccessful, the system proceeds to the next step.

[0087] S134, extract the first page text title line of the PDF transaction record file, search for matching institution records in the preset institution feature list based on the first page text title line, determine the corresponding institution and transaction type, if no match is found, proceed to step S145;

[0088] Use iText PDF to extract the text content of the first page of the PDF, locate the title line content by keywords (such as "transaction", "turnover", etc.), and match the title line content with the list of institutional features. If a completely matching institutional record is found, the recognition is complete; otherwise, proceed to the next step.

[0089] S135: Obtain the header row of the table through keywords, match the header row with the header template in the institution feature table. If the match is successful, determine the corresponding institution and transaction type. If the match is unsuccessful, determine the PDF transaction file as the transaction file of the new institution, use the header row as the transaction type, and store the feature data of the PDF transaction file in the preset institution feature list.

[0090] Based on the text content of the homepage, the header row of the table is located by keywords (such as "date", "amount", "summary" etc.). The content of the header row is matched with the header template in the organization feature table. If the match is successful, the corresponding organization and transaction type are determined. If the match is still unsuccessful, or if all the above steps fail to match the organization and transaction type, the system determines that the current PDF is a transaction file of a new organization and uses the identified header row as the transaction type. The characteristic information of the transaction (such as title, header, watermark, signature certificate, etc.) is stored in the organization feature table for subsequent identification.

[0091] like Figure 6 As shown, in some embodiments, when the table is a borderless table, the processing flow includes:

[0092] S211, Load the PDF transaction log file into a file stream in memory and count the total number of pages in the PDF transaction log file;

[0093] The process begins by reading the PDF file into a file stream. First, the target PDF file is loaded into the file stream in memory for subsequent page-by-page parsing and processing. Then, the number of pages in the PDF is obtained, and the total number of pages in the PDF file is counted, preparing for subsequent loop processing of each page.

[0094] S212, Loop through each page according to the total number of pages, and process each page in the PDF transaction log file in turn. If the current page has not been processed, continue to execute steps S213 to S217. If the current page has been processed, the process ends.

[0095] This embodiment enters a loop structure, processing each page of the PDF file sequentially. If the current page has not been processed, the following steps continue; otherwise, the process ends.

[0096] S213, Read the text objects of the current page, and divide them into different table rows according to the top coordinate value of each text object;

[0097] The PDF text objects for this page are read. For the current page, all text objects (including body text and possible tables) are extracted and stored as text elements in the PDF. Based on the top coordinate value, different rows are created. Each text object has a specific coordinate position in the PDF, where the top coordinate value represents the text's position in the vertical direction. Text objects in the same row have the same top coordinate value, while the top coordinate values ​​between different rows will differ significantly. This step is fundamental to table recognition, as each row of a table corresponds to a text row in the PDF.

[0098] S214, obtain the leftmost text coordinate value of each table row, calculate the frequency of the leftmost text coordinate value of each row, and determine the table row based on the frequency;

[0099] For each row, obtain the left coordinate value of the leftmost text in that row, find the leftmost text object in that row, and record its left coordinate value. This coordinate value reflects the starting position of the row horizontally. Calculate the frequency of the leftmost value for each row, determine the table rows, and count the frequency of the left coordinate value of the leftmost text in each row. In PDF tables, the starting position of table rows is usually fixed, so the left coordinate values ​​of table rows will show a high frequency. Based on the frequency distribution, filter out the most frequent left coordinate values ​​and mark the corresponding rows as table rows. The left coordinate values ​​of non-table rows (such as titles, descriptions, etc.) usually do not appear repeatedly or have a low frequency. For example, in a PDF page number, the area above the table occupies 20%, the table content occupies 70%, and the area below the table occupies 10%. Then, the left coordinate value of the left text is taken from the left coordinate value that accounts for 70%.

[0100] The specific algorithm is implemented as follows: Traverse the table rows, get the left coordinate value of the leftmost character in each row, and get the left_coords array. Input: left_coords=[100,100,100,98,102,50,50,50,50,50,50,50,300,300], statistical frequency: {100:3,98:1,102:1,50:7,300:2}, high frequency coordinate: 50 (frequency 7), total number of rows=14, percentage=7 / 14=50%, which belongs to the table area.

[0101] S215, determine the table row to which the non-table row belongs based on the similarity of the top coordinate values, and determine the leftmost text coordinate value of the non-table row based on the table row.

[0102] As shown in Tables 1 and 2, in borderless tables, cells do not have explicit border lines. In this embodiment, the table row corresponding to a non-table row is found based on the similarity of the top coordinate values. A non-table row is a sub-row created by line breaks in the text of a table row, and the two are close in vertical position (top coordinate). By calculating the difference in the top coordinates between the non-table row and all table rows, the closest table row is found. Similarity means that the difference between the top coordinate value of the data in the non-table row and the top coordinate value of the table row is less than or equal to the top value of the text. The top coordinate value of the non-table row is set to the top coordinate value of the table row: for text rows marked as non-table rows, their top coordinate values ​​are adjusted to the top coordinate value of the table row closest to them, so that the non-table row and the table row are vertically aligned, which facilitates subsequent column matching.

[0103] Table 1

[0104]

[0105] Table 2

[0106]

[0107] Specifically, extract the top coordinates of all table rows and store them in a list of table tops (e.g., [100, 120, 140, 160]); extract the top coordinates and text content of all non-table rows and store them in a list of non-table rows (each element contains top and text, e.g., [{top:105, text:"Zhang San"}, {top:125, text:"Li Si"}]). For each non-table row, iterate through the top coordinates of all table rows, calculate the absolute value of the difference, and find the table row with the smallest difference. The text in a non-table row should be left-aligned to the corresponding column of its table row. Therefore, the leftmost text coordinate of a non-table row should be the same as the left boundary (left coordinate) of the corresponding column of the table row to ensure column alignment. Since the column structure of the table rows is determined, calculate the original left coordinate of the text in the non-table row, find its corresponding column in the column structure of its table row, and set the leftmost text coordinate of the non-table row to the left boundary of its column to achieve alignment with the table row column. In this embodiment, non-table rows that are vertically offset or slightly horizontally skewed due to line breaks are rebound to the correct table rows and columns, ensuring that the structure of the table rows and columns is rigorous and providing an accurate coordinate reference for subsequent text merging (e.g., concatenating non-table row text into their respective cells).

[0108] S216: Based on the coordinates of the leftmost text in the non-table row, obtain the column corresponding to the table row, and reduce the width of the columns in the non-table row and the table row to 1 / N, where N is the number of rows of text in the table.

[0109] For text in non-table rows, the corresponding column in the table row is found based on its left coordinate value. This process requires matching the column's start left coordinate value and end right coordinate value to ensure that the text in non-table rows is correctly assigned to the appropriate column.

[0110] Non-table rows are not independent of the table text. Rather, they are the result of a table cell's content being too long (e.g., too long to fit on one line) and being automatically split into multiple lines by formats such as PDF. For example, a cell in the "Transaction Description" column of a table might contain the text "Transfer of XX on October 15, 2023 - From XX Company (Salary Payment)". Due to column width limitations, this is split into three rows: Row 1 (Table Row): Transfer of XX on October 15, 2023 - From; Row 2 (Non-Table Row): XX Company (Salary Payment). Row 2 is the "Non-Table Row," but it is essentially part of the "Transaction Description" column content, not a separate new row or content from another column. Mismatched columns will lead to data corruption in the table. The core of a table is the row-column correspondence; each row of text must be explicitly assigned to a specific column to form meaningful data (e.g., a date column corresponds to a date, and an amount column corresponds to an amount). If non-table rows are not matched to their corresponding columns, they may be misassigned to other columns (e.g., in the above example, the non-table row might be misassigned to the adjacent amount column), resulting in confusing data; or non-table rows may be treated as independent rows, adding invalid rows and disrupting the original table's row structure (e.g., one row of data becomes two, leading to errors in subsequent statistics and analysis).

[0111] In addition, this embodiment reduces the width of non-table rows and table rows to 1 / N, where N is the number of rows of text in the table. By reducing the text width by 1 / N, N lines of text can be restored to one line, thus solving the problem of classifying multi-line text and improving the accuracy of column matching. For example, suppose a cell's text is split into 3 lines due to excessive length, with each line's width approximately 1 / 3 of the original column width (original column width 300px, reduced to 100px). Although the left coordinates of the 3 split lines of text are slightly offset, they all fall within the 100px reduction range. The program will determine that these 3 lines of text belong to the same column and eventually merge them into complete content (restoring them to one line). After reducing the column width, the boundaries of the coordinate ranges of adjacent columns are clearer, reducing misjudgments of text crossing columns due to slight coordinate offsets. For example, column A's original range is [200, 500], while column B is [500, 800]—the boundaries are blurred (text left=495 might be misinterpreted as column B); after shrinking, column A becomes [200, 300], and column B becomes [500, 600]—the boundary spacing increases, reducing the probability of misinterpretation. This embodiment ensures that the split multi-line text is correctly merged into the original cell, reduces confusion between adjacent columns, and improves the accuracy of parsing borderless tables.

[0112] S217 Sort the leftmost text coordinate values ​​of the table rows from smallest to largest, count the number of columns in each table row, determine the number of columns in the table based on the frequency of each column number, determine the start left coordinate value and end right coordinate value of each column based on the number of columns in the table, determine the row and column to which the text object of the current page belongs based on the top coordinate value, start left coordinate value, and end right coordinate value, and convert the text object into readable structured data.

[0113] Sort the table rows by left coordinate value in ascending order to ensure the column order matches the actual layout. For each row, count the number of columns. Since the number of columns is usually consistent across rows, counting columns across multiple rows verifies the integrity of the table structure. Count the frequency of column numbers in each row and select the most frequent column number as the standard column number. For example, if most rows have 5 columns, consider the table to have 5 columns. Determine the start left and end right coordinate values ​​for each column based on its column number, and calculate the start and end coordinate values ​​for each column based on the table's column count. Specifically, determine the boundary range of each column based on the distribution of left coordinate values ​​in each column of the table row. For each text object in the PDF, determine its row and column based on its top and left coordinate values. Finally, convert the parsed table data (including rows, columns, and their corresponding text content) into a structured data format (e.g., two-dimensional arrays, JSON objects, etc.) for subsequent data processing and storage.

[0114] This embodiment aligns the top coordinates of non-table rows with those of the corresponding table rows, clearly identifying multiple lines of text within the same cell as belonging to the same row. This avoids splitting single-cell content into multiple independent rows and resolves the issue of blurred rows and columns in borderless tables. Furthermore, by reducing the column width range and matching column affiliation using the left coordinate, it accurately pinpoints the column to which the text belongs, even without border references, thus resolving content mismatch issues caused by blurred column boundaries. By unifying the coordinates and column widths of non-table and table rows, this embodiment achieves more standardized vertical alignment (consistent top) and horizontal range (more concentrated after column width reduction), facilitating data parsing by the program according to row and column structure.

[0115] This embodiment determines the standard column count by statistically analyzing the frequency of row and column occurrences in the table. This ignores the interference of individual abnormal rows and more closely matches the actual structure of the table (for example, if most rows have 5 columns, the table is considered to have 5 columns; abnormal column counts in a few rows can be considered a layout issue). Based on the standard column count, the left and right coordinate ranges of the columns are determined, establishing a unified column boundary standard. This ensures that the content of all subsequent rows (including non-table rows) accurately matches the corresponding columns, avoiding data misalignment caused by ambiguous column boundaries. After the above processing, the table's row and column structure is clear, each text block can be clearly assigned to a specific row and column, and the data's correlation is accurately preserved. This provides a reliable structural foundation for subsequent table data digitization, content retrieval, and format conversion operations, reducing manual proofreading costs.

[0116] like Figure 4 As shown, in some embodiments, the method further includes authentication processing of the decrypted PDF transaction log file based on preset rules, wherein the authentication processing includes format authenticity verification and content logic verification.

[0117] Specifically, the authentication process includes the following steps:

[0118] S31, construct a stream-of-file feature library that includes key feature data such as digital signature certificate information and content, title text, table header text, table footer text, watermark text, and the width and height of PDF pages.

[0119] S32, based on the decrypted serial file classification information, match the corresponding serial type from the serial file feature library and extract the feature data corresponding to the serial type;

[0120] Based on the classification information of the decrypted serial files, the corresponding serial file type is matched from the feature library, and the feature data of that type of serial file is extracted as the basis for subsequent verification.

[0121] S33, perform format authenticity verification and content logic verification on the current PDF transaction log file based on feature data and preset verification rules;

[0122] In this embodiment, during the preprocessing of the PDF transaction log file, the extracted feature information is used to perform format-level verification on the current log file to determine whether it conforms to the expected format specifications, thereby identifying the possibility of format forgery. This includes verifying the digital certificate, watermark text, title text, table footer text, table header text, and page width and height.

[0123] After converting the tabular text data into readable structured data, further logical consistency checks are performed on the transaction records. For example, balance continuity is verified by checking if the current record's balance matches the previous record's balance and transaction amount to determine if there has been any tampering. Key transaction record verification checks for interest-related transactions and verifies if their occurrence dates are concentrated on quarterly interest-paying dates (e.g., March 20-21, June 20-21, September 20-21, December 20-21). If no interest-paying records are found or the interest amount is inconsistent with the balance, there is a risk of forgery. Transaction amount verification uses the Benford algorithm to verify transaction amounts. First, the Benford algorithm extracts the integer part of each transaction amount (removing the decimal point), converts it to a string, and identifies the first non-zero digit as the first digit. Then, the actual distribution frequency of the first digits of all transaction amounts is calculated to generate an actual frequency table. According to the theoretical model of Benford's Law, the distribution of the first digit of transaction amounts generated in normal economic activities is not uniform, but rather exhibits a significant power-law characteristic. For example, amounts starting with 1 account for approximately 30.1%, those starting with 2 for approximately 17.6%, and then decrease digit by digit until amounts starting with 9 account for only about 4.5%. If the actual proportion of certain first digits (especially high-ranking digits such as 9) is significantly higher than the theoretical value, it indicates that the data has been manipulated.

[0124] S34. Obtain the scoring results of format authenticity verification and content logic verification according to the scoring criteria of the verification rules, and sum the scoring results by weight to obtain the overall authenticity score of the PDF serial file.

[0125] The table parsing system in this embodiment has multiple built-in verification rules. Each rule outputs a corresponding score (such as risk level or credibility score) based on the verification result. All rule scores are weighted and summarized to obtain the overall authenticity score of the log file, which is used to help determine its credibility.

[0126] For example, suppose the verification rules and weights for a bank's transaction records are as shown in Table 3:

[0127] Table 3

[0128]

[0129] The individual rule scoring results are as follows: digital signature validity 80 points, watermark feature matching degree 90 points, page size compliance 100 points, header structure consistency 85 points; balance continuity 70 points, interest settlement record standardization 85 points, transaction amount intrinsic value distribution 90 points, field integrity 100 points; based on the weighted sum of the scores for each individual rule, the total score for the format authenticity verification part and the total score for the content logic verification part of the transaction document are obtained, and then the overall authenticity score is obtained. The total score for the format authenticity verification section is (80×0.15) + (90×0.08) + (100×0.07) + (85×0.10) = 34.7 points, and the total score for the content logic verification section is (70×0.20) + (85×0.15) + (90×0.15) + (100×0.05) = 45.25 points. The total score is 34.7 + 45.25 = 79.95 points ≈ 80 points. The score can be evaluated according to the equal score level, and reasonable suggestions can be given. For example, 80 points belongs to "high credibility" (70-89 points range), with slight anomalies (balance calculation deviation, slightly lower interest amount), but no obvious traces of forgery are found overall. No manual review is required, and it can directly enter the subsequent data analysis process. The system automatically records the anomalies for audit traceability.

[0130] This embodiment achieves a comprehensive assessment of the authenticity of log files through multi-dimensional quantitative analysis. It objectively evaluates the authenticity of log files to detect whether they have been tampered with or forged, ensuring the reliability and integrity of subsequent data processing. It retains the efficiency of machine processing while simulating the judgment logic of human review through a weighted algorithm, effectively balancing automation efficiency and verification accuracy. The verification mechanism in this embodiment is based on predefined feature rules and content logic verification. Through a dual verification mechanism of feature comparison and logic verification, it comprehensively verifies the authenticity of files from format to content.

[0131] like Figure 7 As shown, in some embodiments, when the table is a table that requires line drawing, the line drawing process includes:

[0132] S221, Read the text object of the current page and determine the table header row based on the keywords identified in the table header;

[0133] In this embodiment, the header row of the table is located based on keywords (such as "date", "amount", "summary", etc.). If the number of matched keywords reaches a certain percentage (e.g., 60%) of the total number of keywords, it is considered a header row. The text before the header row is ignored.

[0134] S222, Obtain the table tail identifier keyword from the preset parsing configuration table, and determine the table tail row based on the table tail identifier keyword;

[0135] Based on the keywords set in the parsing configuration table, locate the text between the row below the table and the header row (inclusive) and the row below the table (exclusive). This text constitutes the table text. If not configured or not found, the bottom of the entire page will be used as the end position of the table.

[0136] S223, calculate the top coordinates of all text between the header row and the footer row, and cluster the top coordinates to obtain a list of top coordinates;

[0137] Collect the top coordinates of all text lines between the header and footer rows. Sort these top coordinates using coordinate clustering. Iterate through the sorted coordinates, and if the difference between two adjacent coordinates is within a threshold (e.g., [5, 200] pixels), they are considered to belong to the same line (which may be a single line of data composed of multiple lines of text), and the minimum value is retained. Use the clustered list of top coordinates as the Y-axis position of all horizontal lines, that is, determine the horizontal line coordinates based on the list of top coordinates.

[0138] S224: Calculate the minimum left coordinate and maximum right coordinate of all text between the header row and the footer row. Use the minimum left coordinate as the first coordinate and increment it by a fixed step size to generate candidate vertical line positions. Filter out the valid vertical lines from the candidate positions, sort the valid vertical line positions, and obtain the list of left coordinates.

[0139] In this embodiment, the minimum left coordinate (minLeft) and maximum right coordinate (maxRight) of all text are found between the header row and the footer row to determine the left and right boundaries of the table. Specifically, the steps include:

[0140] Initialize by creating an empty list of left coordinates and setting minLeft as the first coordinate (leftmost column line).

[0141] Generate candidate lines, starting from minLeft+step and increasing by a small step (e.g., 10 pixels) until maxRight, generating a series of candidate column positions.

[0142] Filter valid column lines. For each candidate separator, iterate through all text between the header and footer rows, checking if the candidate separator falls between the left and right coordinates of any text. If the candidate separator is not within the rectangular area of ​​any text, it is considered a blank area, suitable for drawing a vertical separator. Add all qualified candidate separators to the left coordinate list, sort the left coordinate list, and perform clustering to remove duplicates (if the difference between adjacent coordinates is too small).

[0143] S225, draw table lines based on the top and left coordinate lists, and generate a fixed PDF transaction log file with clear table borders.

[0144] In this embodiment, complete horizontal and vertical lines are drawn at the identified table row (top coordinate) and column (left coordinate) positions to generate a table with borders. The specific process is as follows:

[0145] Create PDF Stamper or PDF Document objects to modify PDFs. In PDF processing, PDF Stamper (iText5) and PDF Document (iText7) are the core objects for modifying existing PDF files. They are used to open existing PDFs and provide editing capabilities (such as adding table lines, watermarks, text, etc.), and are the basic tools for drawing table lines.

[0146] Obtain a PDF Canvas or PDF Content Byte object for drawing graphics. PDF Canvas (iText7) and PDF Content Byte (iText5) are core objects in the PDF processing library used for drawing graphics, text, and paths. They are equivalent to the "pen" or "canvas" for manipulating PDFs and are direct tools for implementing visualization operations such as drawing table lines and text annotations.

[0147] To draw a horizontal line, iterate through the list of top coordinates. For each y-coordinate, use canvas.moveTo(minLeft,y) and canvas.lineTo(maxRight,y) to draw a horizontal line from the leftmost to the rightmost edge of the table.

[0148] To draw a vertical line, iterate through the list of left coordinates. For each x-coordinate, use canvas.moveTo(x, topOfHeader) and canvas.lineTo(x, bottomOfFooter) to draw a vertical line from the top to the bottom of the table.

[0149] Set the line style (e.g., color, thickness), and call canvas.stroke() to draw the lines. Close PdfStamper or PdfDocument, and save the modified PDF to the specified path. Output a repaired PDF file with clear and complete table lines. Then you can process it according to the workflow for bordered tables.

[0150] The embodiments of the present invention can effectively identify and filter invalid data rows (such as blank rows, comment rows, etc.), and at the same time solve the problem of table content misalignment caused by text wrapping, significantly improving the accuracy and completeness of data extraction.

[0151] The embodiments of the present invention are based entirely on PDF text layout information for parsing, without relying on OCR technology or additional image processing procedures. They have high parsing efficiency, low resource consumption, are suitable for large-scale batch processing scenarios, and do not incur additional software and hardware costs.

[0152] The embodiments of the present invention have strong versatility and can be adapted to PDF transaction files of different formats from multiple financial institutions without the need for customized development for specific templates, which greatly improves the system's adaptability and deployment efficiency.

[0153] Example 2

[0154] like Figure 8 As shown in the figure, an embodiment of the present invention provides a PDF table parsing system based on financial transaction records, which includes the following modules:

[0155] The preprocessing module is used to import the PDF transaction log file to be processed and to preprocess the PDF transaction log file. The preprocessing includes decryption, watermark removal and classification.

[0156] The table processing module is used to determine the table type based on the content of the preprocessed PDF transaction log file, determine the table processing flow according to the table type, parse the table to obtain table text data according to the table processing flow, and perform standardization and structuring processing on the table text data to obtain readable structured data. The table types include bordered tables, borderless tables, and tables that require line drawing.

[0157] In some embodiments, the preprocessing module specifically includes:

[0158] The decryption submodule is used to decrypt encrypted PDF transaction log files. During the decryption process, it automatically identifies and adapts to the encryption algorithm used by the PDF transaction log file.

[0159] The watermark removal submodule is used to analyze the PDF text objects page by page in the decrypted PDF transaction log file, calculate the slant of the PDF text objects through the text transformation matrix, and identify and delete PDF text objects with a non-zero slant as watermarks.

[0160] The classification submodule is used to sequentially match institutions in a preset institutional feature list by PDF digital signature, PDF watermark recognition, and the first page text title line or table header line, to determine the financial institution to which the PDF transaction document belongs and the document type, and to store the feature data of unmatched PDF transaction documents as new institutional features into the preset institutional feature list.

[0161] In some embodiments, the watermark removal submodule specifically includes:

[0162] The import and statistics unit is used to load PDF transaction log files into a file stream in memory and count the total number of pages in the PDF transaction log files;

[0163] The loop unit is used to loop through each page according to the total number of pages, and process each page in the PDF transaction log file in turn. If the current page has not been processed, the first judgment unit to the deletion unit will continue to be executed. If the current page has been processed, the process ends.

[0164] The first judgment unit is used to extract all text objects on the current page. It determines whether the text type is COSString based on the text object. If the text type is COSString, it proceeds to the second judgment unit. If the text type is not COSString, it indicates that the text object is not watermark text, and it proceeds to the loop unit.

[0165] The second judgment unit is used to obtain the operation type of the text object and determine whether the operation type is ShowText. If the operation type is ShowText, then proceed to the third judgment unit; if the operation type is not ShowText, it indicates that the text object is not watermark text and proceed to the loop unit.

[0166] The third judgment unit is used to obtain the matrix array information of the text object, calculate the slant of the text object based on the matrix array information, and determine whether the slant of the text object is equal to 0. If the slant is equal to 0, it means that the text object is not watermark text, and then enters the third judgment unit; if the slant is not equal to 0, it means that the text object is watermark text, and then enters the deletion unit.

[0167] The delete unit is used to remove watermark text from the current page.

[0168] In some embodiments, the classification submodule specifically includes:

[0169] The loading and retrieval unit is used to load the PDF transaction log file into a file stream in memory and retrieve a preset list of institutional characteristics;

[0170] The first judgment and matching unit is used to determine whether the PDF transaction record file has a digital signature certificate. If a digital signature certificate exists, the matching institution record is searched in the institution feature list based on the digital signature certificate to determine the corresponding financial institution and transaction type. If no digital signature certificate exists or the match is unsuccessful, the process proceeds to the second judgment and matching unit.

[0171] The second judgment and matching unit is used to determine whether there is watermark text in the PDF transaction file. If there is watermark text, the corresponding financial institution and transaction type are determined by searching for matching institutional records in the preset institutional feature list based on the watermark text. If there is no watermark text or the match is unsuccessful, the process proceeds to the third judgment and matching unit.

[0172] The third judgment and matching unit is used to judge the text title of the PDF transaction file. If a text title exists, the corresponding financial institution and transaction type are determined by searching for a matching institution record in the preset institution feature list based on the text title. If no match is found, the process proceeds to the fourth judgment and matching unit.

[0173] The fourth judgment and matching unit obtains the table header row through keywords, matches the header row with the institutional feature list, and determines whether a matching header template exists. If a matching header template exists, the corresponding financial institution and transaction type are determined based on the header row; if no matching header template exists, the PDF transaction file is determined as a new institution's transaction file and the feature data of the PDF transaction file is stored in the preset institutional feature list.

[0174] In some embodiments, the table processing module includes a borderless table processing submodule, specifically including:

[0175] The import and statistics unit is used to load PDF transaction log files into a file stream in memory and count the total number of pages in the PDF transaction log files;

[0176] The loop unit is used to loop through each page according to the total number of pages, processing each page in the PDF transaction log file in turn. If the current page has not been processed, it will enter the division unit to the statistics and conversion unit. If the current page has been processed, the process ends.

[0177] The cell division function is used to read the text object on the current page and divide it into different table rows based on the top coordinate value of the text object;

[0178] The table row determination unit is used to obtain the leftmost text coordinate value of each table row, calculate the frequency of the leftmost text coordinate value, and determine the table row based on the frequency of the leftmost text coordinate value.

[0179] The non-table row determination unit is used to determine the table row to which the non-table row belongs based on the similarity of the top coordinate values, and to determine the leftmost text coordinate value of the non-table row based on the table row.

[0180] The determining unit is used to determine the column corresponding to the table row based on the leftmost text coordinate value of the non-table row, and to reduce the width of the columns of the non-table row and the table row to 1 / N, where N is the number of rows of text in the table;

[0181] The statistics and transformation unit is used to sort the leftmost text coordinate values ​​of the table rows from smallest to largest, count the number of columns in each table row, determine the number of columns in the table based on the frequency of each column's occurrence, determine the start left coordinate value and end right coordinate value of each column based on the number of columns in the table, determine the row and column to which the text object of the current page belongs based on the top coordinate value, the start left coordinate value, and the end right coordinate value, and transform the text object into readable structured data.

[0182] In some embodiments, the special table processing module specifically includes:

[0183] The header row determination unit is used to determine the header row of the table based on the core keywords of the preset financial transaction record header.

[0184] The table tail row determination unit is used to obtain the table tail identifier keyword from the preset parsing configuration table and determine the table tail row based on the table tail identifier keyword;

[0185] The top coordinate list retrieval unit is used to extract the top coordinates of all text between the header row and the footer row of the table, and to cluster the top coordinates to obtain a top coordinate list.

[0186] The left coordinate list acquisition unit is used to calculate the minimum left coordinate and maximum right coordinate of all text between the header row and the tail row. The minimum left coordinate is used as the first coordinate and incremented by a fixed step size to generate candidate vertical line positions. Valid vertical lines are selected from the candidate vertical line positions, and the valid vertical line positions are sorted to obtain the left coordinate list.

[0187] The border repair unit is used to draw table lines based on the top coordinate list and the left coordinate list, generating a PDF transaction log file with the table borders repaired.

[0188] In some embodiments, the system further includes: an authentication module, used to perform format authenticity verification and content logic verification on the decrypted PDF transaction log file based on preset rules;

[0189] The authentication module specifically includes:

[0190] The construction submodule is used to build a pipeline file feature library that includes key feature information and feature data such as digital signature certificates, title text, table header text, table footer text, watermark text, and the width and height of PDF pages;

[0191] The matching and judgment submodule is used to match the corresponding transaction type from the transaction file feature library based on the classification information of the decrypted transaction file, and extract the feature data corresponding to the transaction type.

[0192] The verification submodule is used to perform format authenticity verification and content logic verification on the current PDF transaction log file based on feature data and preset verification rules.

[0193] The scoring submodule is used to obtain the scoring results of format authenticity verification and content logic verification according to the scoring criteria of the verification rules, and to weight and summarize the scoring results to obtain the overall authenticity score of the PDF transaction log file.

[0194] Please refer to the method implementation examples for specific details.

[0195] Example 3

[0196] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the above-described methods for parsing PDF tables based on financial transaction records.

[0197] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. Of course, there are other types of readable storage media, such as quantum memories, graphene memories, etc. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted from the content as required by the legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium may not include electrical carrier signals and telecommunication signals.

[0198] Example 4

[0199] The present invention also provides an electronic device. The electronic device of this invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a task execution method based on a dynamic class loading task execution platform provided by the present invention. References are provided below. Figure 9 It shows a schematic diagram of the structure of a computer system 1000 suitable for implementing an electronic device according to embodiments of the present invention. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0200] like Figure 9 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage section 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the computer system 1000. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0201] The following components are connected to I / O interface 1005: an input section including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed.

[0202] In particular, according to the embodiments disclosed in this invention, the processes described in the above main step diagrams can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the main step diagrams. In the above embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit 1001, it performs the functions defined in the system of this invention.

[0203] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0204] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0205] The units described in the embodiments of the present invention can be implemented in software or hardware. The described units can also be located in a processor, and the names of these units do not necessarily limit the specific unit itself.

[0206] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A PDF form parsing method based on financial transaction stream, characterized in that, The method comprises the following steps: S1, importing a PDF transaction file to be processed, and preprocessing the PDF transaction file, the preprocessing comprising decryption processing, watermark removal processing and classification processing; S2, judging a table type according to the content of the preprocessed PDF transaction file, and determining a table processing flow according to the table type, and performing analysis processing on the table to obtain table text data according to the table processing flow; performing standardization and structuring processing on the table text data to obtain readable structured data; the table type comprises a border table, a non-border table and a table requiring line drawing processing; The method further comprises the following step S3: performing authenticity processing on the PDF transaction file based on preset rules, the authenticity processing comprising format authenticity verification and content logic verification; the authenticity processing specifically comprises the following steps: S31, constructing a file feature library comprising digital signature certificates, title text, table header text, table footer text, watermark text, and feature data of the width and height of a PDF page; S32, matching a corresponding file type from the file feature library according to the classification information of the PDF transaction file, and extracting feature data corresponding to the file type; S33, performing format authenticity verification and content logic verification on the current PDF transaction file according to the feature data and preset verification rules; S34, obtaining a scoring result of the format authenticity verification and the content logic verification according to a scoring standard of the verification rules, weighting and summarizing the scoring result, and obtaining an overall authenticity score of the PDF transaction file.

2. The method of claim 1, wherein, The step S1 specifically comprises: S11, decrypting an encrypted PDF transaction file, and automatically identifying and adapting the encryption algorithm used by the PDF transaction file during the decryption process; S12, analyzing a PDF text object page by page for the decrypted PDF transaction file, calculating the inclination of the PDF text object through a text transformation matrix, and determining a PDF text object with a non-zero inclination as a watermark and deleting the watermark; S13, sequentially performing institution matching on the PDF digital signature, the PDF watermark identification, the first page text title row or the table header row in a preset institution feature list, determining the financial institution to which the PDF transaction file belongs and the file type, and storing the feature data of the PDF transaction file that is not matched as new institution features in the preset institution feature list.

3. The method of claim 2, wherein, The step S12 specifically comprises: S121, loading the decrypted PDF transaction file into a file stream in the memory, and counting the total number of pages of the PDF transaction file; S122, according to the total number of pages, processing each page of the PDF transaction file in turn, if the current page is not processed, steps S123 to S126 are executed, and if the current page is processed, the process ends; S123, extract all text objects of the current page, determine whether the text type is COSString according to the text objects, if the text type is COSString, execute step S124, if the text type is not COSString, it indicates that the text object is not a watermark word, execute step S122; S124, obtain the operation type of the text object, determine whether the operation type is ShowText, if the operation type is ShowText, execute step S125; if the operation type is not ShowText, it indicates that the text object is not a watermark word, execute step S122; S125, obtain the matrix array information of the text object, calculate the inclination of the text object according to the matrix array information, determine whether the inclination of the text object is equal to 0, if the inclination is equal to 0, it indicates that the text object is not a watermark word, execute step S122; if the inclination is not equal to 0, it indicates that the text object is a watermark word, execute step S126; S126, remove the watermark word from the current page.

4. The method of claim 2, wherein, The step S13 specifically comprises: S131, load the PDF transaction flow file into the file stream in the memory, and obtain a preset institution feature list; S132, determine whether the PDF transaction flow file exists a digital signature certificate, if the digital signature certificate exists, search for a matched institution record in the institution feature list according to the digital signature certificate, and determine the corresponding financial institution and flow type according to the matched institution record; if the digital signature certificate does not exist or the matching is not successful, execute step S133; S133, determine whether the PDF transaction flow file exists a watermark word, if the watermark word exists, search for a matched institution record in the preset institution feature list according to the watermark word, and determine the corresponding financial institution and flow type according to the matched institution record, if the watermark word does not exist or the matching is not successful, execute step S134; S134, extract the first page text title row of the PDF transaction flow file, search for a matched institution record in the preset institution feature list according to the first page text title row, and determine the corresponding financial institution and flow type according to the matched institution record, if the matching is not successful, execute step S135; S135, obtain the table header row of the table through the keyword, determine whether there is a table header template matched with the table header row in the institution feature table, if there is a matched table header template, determine the corresponding institution and flow type according to the table header template; if there is no matched table header template, determine that the PDF transaction flow file is a flow file of a new institution, and store the feature data of the PDF transaction flow file into the preset institution feature list.

5. The method of claim 1, wherein, When the table is a borderless table, the processing procedure comprises: S211, load the PDF transaction flow file into the file stream in the memory and count the total number of pages of the PDF transaction flow file; S212, according to the total number of pages, each page is circulated, each page in the PDF transaction file is processed in turn, if the current page is not processed, continue to execute step S213 to step S217, if the current page is processed, the process ends; S213, read the text object of the current page, and divide different table rows according to the top coordinate value of the text object; S214, get the leftmost text coordinate value of each table row, and calculate the frequency of the leftmost text coordinate value, and determine the table row according to the frequency of the leftmost text coordinate value; S215, determine the table row to which the non-table row belongs according to the similarity of the top coordinate value, and determine the leftmost text coordinate value of the non-table row according to the table row; S216, determine the column corresponding to the table row according to the leftmost text coordinate value of the non-table row, and reduce the width of the non-table row and the column of the table row to 1 / N, N is the number of rows of text in the table; S217, sort the leftmost text coordinate value of the table row from small to large, count the number of columns of each table row, determine the number of columns of the table according to the frequency of the number of columns of each table row, determine the start left coordinate value and the end right coordinate value of each column according to the number of columns of the table, determine the row and column to which the text object of the current page belongs according to the top coordinate value, the start left coordinate value and the end right coordinate value, and convert the text object into readable structured data.

6. The method of claim 1, wherein, When the table is a table that needs to be processed by drawing lines, the processing flow includes: S221, read the text object of the current page, and determine the table header row of the table according to the keyword of the table header identifier; S222, get the table tail identifier keyword from the preset analysis configuration table, and determine the table tail row according to the table tail identifier keyword; S223, get the top coordinates of all texts between the table header row and the table tail row, and cluster the top coordinates to obtain a top coordinate list; S224, get the minimum left coordinate and the maximum right coordinate of all texts between the table header row and the table tail row, take the minimum left coordinate as the first coordinate, and generate candidate vertical line positions by increasing with a fixed step; From the candidate vertical line positions, filter out the effective vertical lines, sort the positions of the effective vertical lines to obtain a left coordinate list; S225, draw table lines according to the top coordinate list and the left coordinate list, and generate a repaired version of the PDF transaction file containing a clear table border.

7. A PDF form parsing system based on financial transaction stream, characterized by, The system is suitable for the PDF table analysis method based on financial transaction file according to any one of claims 1-6, comprising the following modules: Import and pretreatment module, used for importing the PDF transaction file to be processed, and pretreating the PDF transaction file, the pretreatment including decryption processing, watermark removal processing and classification processing; The table processing module is used for judging a table type according to the content of the preprocessed PDF transaction flow file, determining a table processing flow according to the table type, performing analysis processing on the table according to the table processing flow to obtain table text data, performing standardization and structuring processing on the table text data, and obtaining readable structured data; the table type includes a border table, a non-border table and a table requiring line drawing processing.

8. An electronic device, comprising: Comprise: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement a financial transaction flow based PDF table analysis method as claimed in any one of claims 1-6.

9. A computer readable medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement a financial transaction flow based PDF table analysis method as claimed in any one of claims 1-6.

Images