Document processing method and system

Abstract

The application relates to the technical field of bank receipt processing, and provides a receipt processing method and system. The method realizes multi-source automatic acquisition and unified duplicate elimination of a receipt file through a preset configuration mode; a dynamic regular expression library is used to identify a receipt type, and an optimal analysis template is determined based on multi-template priority matching and an automatic degradation mechanism; the extracted structured data is subjected to multiple verifications, and when the verification fails, the template is corrected and then reanalyzed; a regular matching and format feature analysis are combined to identify a split boundary, multiple receipts are split into independent files, and then are associated, archived and established with multidimensional retrieval indexes. The application realizes full-process automatic closed-loop processing of receipts, and improves the analysis success rate and processing efficiency of multi-bank and multi-format receipts.

Description

Technical Field

[0001] This application relates to the field of bank statement processing technology, and in particular to a document processing method and system. Background Technology

[0002] Currently, document processing relies primarily on manual entry and verification, resulting in low efficiency, high costs, and poor accuracy. Document formats vary significantly between different institutions, making it difficult for existing automation tools to parse documents with multiple formats and layouts. Furthermore, the lack of automatic segmentation, data verification, and intelligent archiving capabilities for multi-page documents prevents the processing flow from forming a closed loop, hindering the fulfillment of enterprises' needs for high-volume, efficient automation. Summary of the Invention

[0003] In view of this, this application aims to propose a document processing method to achieve automatic parsing and archiving of multi-format documents within seconds, thereby improving processing efficiency and accuracy.

[0004] To achieve the above objectives, the technical solution of this application is implemented as follows:

[0005] A document processing method, the method comprising:

[0006] The document file to be processed is obtained through a preset configuration method, and the type of the document file is identified;

[0007] Match the corresponding parsing template according to the type, and select the optimal parsing template according to the preset priority;

[0008] Using the optimal parsing template, key fields are extracted from the document file to obtain structured data;

[0009] The structured data is subjected to multiple validations to obtain normal document files, and the normal document files are then split to generate independent document files.

[0010] The independent document files and their corresponding structured data are linked and archived, and a multi-dimensional retrieval index is established.

[0011] Furthermore, the step of obtaining the document file to be processed through a preset configuration method includes:

[0012] Connect to at least one document file source channel through a preset configuration method. The document file source channel includes at least one of the following: a file transfer server, a shared folder, and a local directory.

[0013] By triggering a scheduled task, the document files to be processed are automatically obtained from the configured document file source channels;

[0014] Calculate the MD5 value of the document to be processed and compare it with the MD5 value record of the processed document. If the MD5 value already exists, it is determined to be a duplicate file and filtered out; otherwise, it is added as a new file and enters the subsequent processing flow.

[0015] Furthermore, the step of obtaining the document file to be processed through a preset configuration method also includes:

[0016] Receive the pending document file manually uploaded by the user, calculate the MD5 value of the pending document file, and compare it with the MD5 value record of the processed document file;

[0017] If the MD5 value already exists, the upload is intercepted and a duplicate message is displayed. If the MD5 value does not exist, the upload is accepted and automatically treated as a new file for subsequent processing.

[0018] Furthermore, if the process fails to obtain the document file to be processed through configuration, a retry process will be initiated.

[0019] If the retries fail within all possible attempts, the retries are considered to have all failed, and a preset alarm message is output.

[0020] The retry process employs an exponential backoff strategy, which sequentially increases the retry waiting time according to a preset initial retry interval, a multiplier factor, and a maximum number of retries.

[0021] Furthermore, the automatic identification of the document type includes:

[0022] Construct a dynamically extensible regular expression library, which supports online addition, modification, or deletion of regular expressions. Each regular expression is configured with a priority identifier and a corresponding type label.

[0023] For the document to be identified, regular expressions are used to match it in descending order of priority, and the organization type or layout type corresponding to the first successfully matched regular expression is determined as the type of the document.

[0024] If all regular expressions fail to match, mark the document file as parsing failed and record the reason for the failure.

[0025] The reasons for failure include at least one of the following: file name, file digest checksum, list of regular expressions that failed to match, and timestamp.

[0026] Furthermore, the step of matching the corresponding parsing template according to the type and matching the optimal parsing template according to a preset priority includes:

[0027] Based on the type of the identified document file, the corresponding template library is called, and each parsed template in the template library is associated with a priority identifier;

[0028] The templates are matched and parsed in descending order of priority, and the first successfully matched template is determined as the optimal parsing template.

[0029] Specifically, if the current parsing template fails to extract or validate fields, it will automatically be downgraded to the next lower priority parsing template and the field extraction and validation process will be repeated. If all priority parsing templates fail, the document file will be marked as parsing failed and the reason for the failure will be recorded.

[0030] Furthermore, the extraction of key fields from the document file includes:

[0031] Configure extraction rules for each key field separately;

[0032] Extract the corresponding field values ​​from the document file according to the configured extraction rules;

[0033] The format of the extracted field values ​​is converted.

[0034] The extraction rules are selected from at least one of coordinate positioning method or regular expression matching method; the format conversion includes at least one of data type conversion, format normalization, and unit conversion.

[0035] Furthermore, the multiple verification of the structured data includes:

[0036] The structured data is subjected to format verification, integrity verification, and logical verification respectively;

[0037] Mark the documents that fail the verification as abnormal and record the reason for the failure;

[0038] The format validation is used to check whether the field conforms to the preset format rules, the integrity validation is used to check whether the core field is empty, and the logic validation is used to check the consistency of business logic between fields. The failure reasons include at least one of the following: validation type, failed field name, and comparison information between expected conditions and actual values.

[0039] Furthermore, the splitting of the normal document file includes:

[0040] Automatically identify split boundaries using regular expression matching and layout feature analysis;

[0041] Based on the splitting boundaries, multiple documents in a single document file are split into multiple independent document files;

[0042] The split independent document files are automatically named and stored according to preset naming rules. At the same time, the MD5 value of each independent document file is calculated and recorded for subsequent file deduplication.

[0043] The regular expression matching is used to identify fixed separators or start and end markers between documents; the layout feature analysis is used to identify layout structure changes or blank areas as split boundaries; the naming rules include at least one of the following: original file name, document type, split sequence number, and split timestamp.

[0044] Compared with related technologies, this application has the following advantages:

[0045] (1) The document processing method of this application realizes the full-process automated closed-loop processing of documents from acquisition to archiving and retrieval through configuration acquisition, automatic type identification, multi-template priority matching and downgrading, key field structured extraction, multiple verification, automatic splitting of multiple documents, associated archiving and multi-dimensional index establishment, thereby improving the parsing success rate and retrieval efficiency.

[0046] (2) By configuring and connecting multiple types of source channels and cooperating with scheduled tasks, multi-source automated acquisition is achieved. By uniformly calculating the MD5 value of automatically acquired and manually uploaded files and comparing it with historical records, unified deduplication of cross-source files is achieved to avoid duplicate processing.

[0047] (3) By calculating the MD5 value of manually uploaded files and comparing it with the historical records, the upload is intercepted and a duplicate is prompted when the MD5 value already exists, and the upload is accepted and included in the processing flow when the MD5 value does not exist, so as to achieve consistency between the manual supplementation and automatic acquisition processes and unified deduplication.

[0048] (4) By entering a retry process with an exponential backoff strategy when acquisition fails and retrying at incremental intervals, and sending an alarm after all retry failures, the system can achieve automatic recovery of transient faults and avoid server overload and timely detection of persistent faults.

[0049] (5) By building a dynamic regular expression library that supports online addition, deletion and modification and each regular expression is configured with priority and type tags and matched in order of priority, and recording the reason for failure including file name, MD5 value, list of failed regular expressions and timestamp when all matches fail, the zero-code instant expansion of new bank types and the traceability of failure identification are realized.

[0050] (6) By calling the associated priority template library according to the file type and matching in order of priority, and automatically downgrading to the next priority template to re-execute when field extraction or verification fails, and marking the parsing failure and recording the reason when all priorities fail, the optimal template is quickly selected and the template-level fault tolerance mechanism is implemented to improve the single file parsing success rate.

[0051] (7) By configuring at least one extraction rule in coordinate positioning or regular expression matching for each key field, and by performing data type conversion, format standardization and unit conversion on the extracted values, the differentiated and accurate extraction of table-type and text-type receipts and the standardization of extracted data are achieved.

[0052] (8) By performing format verification, integrity verification and logical verification on structured data respectively, and marking exceptions and recording verification type, failed field name and expected comparison information when verification fails, the system achieves triple automatic verification of field legality, core field non-emptiness and cross-field business logic consistency, as well as the location of verification failures.

[0053] (9) Identify fixed separators or start and end marks by regular expression matching and identify layout structure changes or blank areas by combining layout feature analysis. Also, split multiple documents into multiple independent files according to the boundary and automatically name and store them according to the naming rules including the original file name, document type, split sequence number and split timestamp. At the same time, calculate and record the MD5 value of each independent file to achieve collaborative identification of split boundaries and standardized naming, unified storage and independent tracking of split files.

[0054] Another objective of this application is to provide a document processing system, comprising:

[0055] The configurable acquisition module is used to acquire the document files to be processed through preset configuration methods;

[0056] The type identification module is used to identify the type of the document file;

[0057] The template matching module is used to match the corresponding parsing template according to the type, and to match the optimal parsing template according to a preset priority.

[0058] The field extraction module is used to extract key fields from the document file using the optimal parsing template to obtain structured data;

[0059] The verification module is used to perform multiple verifications on the structured data to obtain a normal document file;

[0060] The splitting module is used to split the normal document file into independent document files;

[0061] The archive retrieval module is used to associate and archive the independent document files and their corresponding structured data, and to establish a multi-dimensional retrieval index.

[0062] The document processing system of this application achieves a complete automated processing link from document acquisition to archive retrieval by sequentially connecting and transmitting data through a configuration acquisition module, a type identification module, a template matching module, a field extraction module, a verification module, a splitting module, and an archive retrieval module. Attached Figure Description

[0063] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0064] Figure 1 This is a schematic diagram of the overall process of the document processing method described in the embodiments of this application;

[0065] Figure 2 This is a schematic diagram of the document processing system described in the embodiments of this application.

[0066] Explanation of reference numerals in the attached figures:

[0067] Document processing system: 610, Configuration acquisition module; 620, Type recognition module; 630, Template matching module; 640, Field extraction module; 650, Validation module; 660, Splitting module; 670, Archive retrieval module. Detailed Implementation

[0068] To make the technical solution and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0069] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0070] Furthermore, it should be noted that in the description of this application, if terms such as "upper," "lower," "inner," or "outer" appear, indicating orientation or positional relationship, these are based on the orientation or positional relationship shown in the accompanying drawings and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this application. In addition, if terms such as "first" or "second" appear, they are also used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0071] Furthermore, in the description of this application, unless otherwise expressly defined, the terms "installation," "connection," "joining," and "connector" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application in light of the specific circumstances.

[0072] In this application, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0073] The present application will now be described in detail through exemplary embodiments. However, it should be understood that, without further description, elements, structures, and features in one embodiment may be advantageously incorporated into other embodiments.

[0074] In this application, the pre-configured approach refers to the following: through a visual configuration interface provided by the system backend, the administrator pre-defines various rules and parameters required for business processing by means of parameter input, option selection, and expression filling. These parameters and rules are stored in a configuration database in the form of structured data. During runtime, the system dynamically reads the parameter values ​​from the configuration database and executes corresponding business operations based on these values. Modifications to the configuration do not require changes, compilation, or deployment of the program code; the configuration takes effect immediately after saving. The pre-configured approach is reflected in various processing stages of the embodiments of this application, and will be described in detail in the corresponding steps below.

[0075] The first aspect of this application provides a document processing method applied to the automated processing of bank documents in an enterprise financial system. The method obtains the document files to be processed through a preset configuration method and automatically identifies the type. Based on the identified type, it matches the optimal parsing template according to priority for field extraction and verification. After the verified document files are split, associated, archived, and a multi-dimensional retrieval index is established, the method realizes the fully automated processing of documents from acquisition to archiving and retrieval, thereby reducing manual operation costs and improving document processing efficiency and accuracy.

[0076] In related technologies, bank documents serve as core vouchers for fund transfers in corporate finance and business management, and are widely used in various stages such as reconciliation, auditing, tax verification, and compliance management. Bank documents are typically in the form of electronic files in PDF format, issued by various banking institutions, and companies need to process and archive the large number of bank documents generated in their daily operations.

[0077] However, current methods for processing bank documents suffer from low efficiency, poor accuracy, and insufficient compatibility. Specifically, most technologies for processing bank documents rely on manual review, data entry, and verification, or use automated tools with fixed templates for field extraction.

[0078] However, in real-world applications, documents issued by different banking institutions vary significantly in layout, field placement, data format, and identifiers. Even documents generated by the same bank at different times or under different business systems may have different formats, resulting in diverse and dynamically changing document formats. Furthermore, businesses typically need to collaborate with multiple banks, generating tens of thousands of documents in a single month, placing high demands on batch processing capabilities.

[0079] When using manual processing, the overall operation is time-consuming, and manual data entry is prone to omissions and errors, making it unsuitable for enterprises' needs for batch document processing. Furthermore, manually searching for documents requires navigating through folders, hindering rapid response to business demands. While using automated tools with fixed templates presents challenges in batch processing scenarios involving multiple banks and document formats, a single template cannot accommodate the format differences between banks, resulting in poor field extraction accuracy. When the template and document format do not match, the lack of effective fault tolerance and adaptive mechanisms can easily lead to parsing failures.

[0080] For example, when using a fixed-template parsing tool to batch parse bank documents, the parsing accuracy can reach the preset standard. However, for documents issued by other banks with which the company has business dealings, the parsing fails because the fixed template cannot adapt to their format, requiring manual processing of each document individually. Furthermore, the existing solution lacks automatic verification of document data and automatic splitting of multi-page merged documents. Abnormal documents cannot be automatically identified and located, requiring manual splitting of merged files, further increasing the workload.

[0081] In view of this, in order to overcome the shortcomings of related technologies, the document processing method in this embodiment combines... Figure 1 In terms of overall design, it includes the following steps S110 to S150.

[0082] Step S110: Obtain the document file to be processed through a preset configuration method and identify the bank type to which the document file belongs.

[0083] Step S110 is used to automate and configurably acquire the documents to be processed, and automatically determine their bank type, laying the foundation for accurate parsing later. Specifically, this step includes the following sub-steps S111 to S115 executed sequentially.

[0084] Step S111: Configure and connect to multiple channel document file sources.

[0085] Specifically, system administrators can predefine the source channels for document files through the visual configuration interface in the system backend. The system supports connection to various types of source channels, including but not limited to File Transfer Protocol (FTP) servers, shared folders based on the Server Message Block (SMB) protocol, and local directories. Administrators configure the corresponding connection parameters according to the selected channel type.

[0086] For the FTP channel, the required configuration parameters include: server address, port number, login account, login password, remote directory path, and file extension filtering rules. File extension filtering rules are used to restrict the fetching of only files with specified extensions, such as ".pdf" or ".zip". For the shared folder channel, the required configuration parameters include: Uniform Naming Convention (UNC) path, domain account, and access password. For the local directory channel, the required configuration parameters include: local absolute path and file extension filtering rules. All configuration parameters are stored in the system configuration database in structured data format. Configuration takes effect immediately after saving, without requiring a system restart or code development and deployment, enabling zero-code extension access to new channels.

[0087] Step S112: The scheduled task is triggered to automatically retrieve the document file.

[0088] After completing the channel configuration in step S111 above, the system, based on a distributed scheduled task framework, automatically triggers document file retrieval tasks according to the scheduled rules set by the administrator. The scheduled rules are configured using Cron expressions, for example: "0 0 1 * * ?" means execution at 1 AM daily; "0 0 / 3 0 8-18 * * ?" means execution every 30 minutes between 8 AM and 6 PM daily.

[0089] Specifically, after the scheduled task is triggered, the system sequentially iterates through all configured and enabled source channels, establishes connections based on the connection parameters of each channel, and scans the list of document files in the specified directory according to the set document file extension filtering rules. For each acquired document file, the system reads its binary data one by one. For ZIP format compressed files, the system calls the decompression engine to automatically decompress them, extracting the internal PDF format document files, which are then used as pending documents in subsequent processes. During task execution, the system records the task start time, current processing progress, number of processed files, and task status in real time, and supports task execution timeout control, automatically interrupting the task and releasing system resources when the preset time threshold is exceeded.

[0090] In addition, if the process of obtaining the document file to be processed fails through configuration, a retry process is initiated. That is, if the file is obtained from a certain channel due to reasons such as network abnormality, connection timeout, authentication failure, or file reading abnormality, the system does not immediately determine that the acquisition is a final failure, but instead initiates a retry process. If the acquisition fails within all retries, it is determined that all retries have failed, and a preset alarm message is output.

[0091] Specifically, the retry process employs an exponential backoff strategy, sequentially increasing the retry waiting time according to a preset initial retry interval, multiplier factor, and maximum number of retries. For example, the initial retry interval is set to 10 seconds, the multiplier factor to 3, and the maximum number of retries to 5. Upon initial failure, the system waits the initial retry interval (10 seconds) before attempting the first retry. If the first retry also fails, the system multiplies the waiting time by the multiplier factor, i.e., waits 30 seconds before attempting the second retry. If the second retry also fails, the system continues to multiply the waiting time by the multiplier factor, i.e., waits 90 seconds before attempting the third retry. Subsequent retries follow this pattern, with each retry's waiting time multiplied by the multiplier factor, until the maximum number of retries is reached or the acquisition is successful.

[0092] It should be noted that during each retry wait, the system keeps the task suspended and does not release the task context, allowing subsequent processing to continue after a successful retry. If the file is successfully retrieved in a retry within the maximum number of retries, the retry process exits and proceeds to step S113 for deduplication. If the file is successfully retrieved in a retry but is subsequently identified as a duplicate file in the deduplication stage, it is considered processed and no further alarm is triggered. If retrieval fails within all preset maximum number of retries, the system considers all retries to have failed. At this point, the system generates an alarm message and sends it to the operations and maintenance personnel via email, SMS, or internal message channel. Simultaneously, the channel information, timestamp, and reason for the failure are recorded in detail in the system error log. Notably, a single channel retrieval failure will not block the execution of the entire scheduled task; the system skips that channel and continues processing other configured channels to ensure the overall robustness of the task.

[0093] Step S113: Perform deduplication processing on document files based on MD5 values.

[0094] After obtaining the document files to be processed in step S112 above, the system initiates a deduplication and verification mechanism. Specifically, the system calls the MD5 (Message Digest Algorithm 5) calculation engine to read all the binary data of each document file to be processed and calculates the unique MD5 value (i.e., a 128-bit hash value, hereinafter referred to as the MD5 value) corresponding to that document file. Subsequently, the system uses this MD5 value as a query condition to search the database of processed document records.

[0095] The database stores the MD5 hashes of all historically processed documents. If the MD5 hash of a current document matches the database (meaning the MD5 hash already exists), the document is considered a duplicate and automatically filtered out, preventing it from proceeding to the next identification and parsing stage. This filtering operation is logged in the system log. If the MD5 hash of a current document does not match the database, the document is considered a newly created file, marked as pending processing, and proceeds to the next stage. This deduplication process uses file digest values ​​to achieve fast and accurate file deduplication, avoiding the waste of computational resources caused by duplicate processing.

[0096] Step S114: Manually upload supplementary files and remove duplicates.

[0097] As a preferred embodiment, based on the configurable automatic acquisition mechanism implemented in steps S111 to S113 above, the system also provides a manual upload entry as a backup solution. Users can access this entry through the system's front-end interface, select document files from local storage devices for upload, and support both single-file upload and batch file upload modes. Uploaded file formats also support PDF and ZIP formats.

[0098] Specifically, when a user uploads a document file, the system's front-end interface displays an attribute input area, requiring the user to fill in the following attribute information: Bank Type (a required field, with a drop-down list of selectable banks); Transaction Date (a required field, defaulting to the current system date); Uploader (an automatically highlighted field, defaulting to the currently logged-in username). In addition, the system provides a Source Tag field, allowing users to select a reason for submission via a drop-down menu. Optional values ​​include "Resubmission due to channel failure" and "Resubmission due to omission," used for subsequent statistics and verification.

[0099] Furthermore, after the user confirms the upload, the system calculates the MD5 value for each uploaded document file and compares it with the MD5 value record of already processed document files. If the MD5 value already exists, the system intercepts the upload operation for that document file, does not store the file, and displays a pop-up window on the front-end interface prompting the user that the document file already exists and should not be uploaded again. If the MD5 value does not exist, the system accepts the upload of the document file, stores it in the system's temporary file directory, and automatically includes it in the subsequent processing flow. After passing the deduplication check, the subsequent processes of type identification, template matching, field extraction, verification, splitting, and archiving of the above-mentioned manual file supplementation operation are completely consistent with the automatically obtained document files determined as new in step S113, ensuring the consistency of the processing logic.

[0100] Step S115: Dynamic type recognition based on regular expression library.

[0101] In this embodiment, after successfully obtaining the newly added document file through the above steps, the system immediately performs automatic document file type identification. Identifying the document file type specifically refers to identifying the type of banking institution to which the document file belongs. The identification process specifically includes:

[0102] The system constructs a dynamically expandable regular expression library, which is stored in the form of database tables or configuration files. Each regular expression record includes at least the following fields: regular expression pattern string, used to store the regular expression for matching bank identifier text in the document; priority identifier field, represented by a positive integer, with a larger value indicating higher priority; bank type label field, used to store the bank name returned when a match is successful; activation status field, used to indicate whether the record is currently participating in the matching; creation time field and update time field, used to record the configuration change history.

[0103] Specifically, the system provides a web-based visual management interface for system administrators to maintain the regular expression library online. The management interface includes add, modify, and delete functions. The add function allows administrators to input a new regular expression pattern string through a form, select from a drop-down list or manually enter the corresponding bank type label, and set a priority value. After the form is submitted, the system validates the syntax of the regular expression; if valid, the new record is written to the database table. The modify function allows administrators to select an existing regular expression record from the list page, enter the editing interface, modify the regular expression pattern string, adjust the priority value, change the associated bank type label, and enable or disable the record. After saving the changes, the corresponding fields in the database are updated synchronously. The delete function allows administrators to select the regular expression record to be deleted from the list page. Before deleting, the system displays a secondary confirmation dialog box; after confirmation, the system deletes the record from the database table or updates its enabled status to disabled. Add, modify, and delete operations take effect immediately after saving, without requiring a system restart or code redeployment.

[0104] In practical implementation, taking actual business scenarios as an example, the regular expression for recognizing receipts from the Industrial and Commercial Bank of China (ICBC) can be configured as ".(ICBC|ICBC|ICBC)". Its priority flag is set to 100, and the bank type label is set to "ICBC". The regular expression for recognizing receipts from the China Construction Bank (CCB) can be configured as ".(CCB|CCB|CCB)". Its priority flag is set to 100, and the bank type label is set to "CCB". The regular expression for recognizing receipts from China Merchants Bank (CMB) can be configured as ".(CMB|CMBCHINA|CMB)". Its priority flag is set to 90, and the bank type label is set to "CMB". Multiple regular expression records can be configured for the same bank type. For example, one can be used to match Chinese names, one to match English abbreviations, and another to match the bank's unique receipt title format. The matching order of multiple regular expressions is controlled by the priority flag.

[0105] When the system starts, it automatically loads all enabled regular expression records from the database table, sorts them by priority from high to low, compiles the regular expression pattern strings into Pattern objects, and stores them along with the corresponding bank type tags in the system memory cache. During type recognition, the system directly reads the compiled list of regular expressions from the memory cache for matching operations, without needing to query the database each time. When an administrator adds, modifies, or deletes regular expressions in the regular expression library through the backend management interface and saves the changes, the system simultaneously updates the database and refreshes the regular expression list in the memory cache, ensuring that configuration changes take effect immediately after saving.

[0106] During type recognition, the system attempts to match regular expressions sequentially from highest to lowest priority according to the sorted list in the cache. The system first uses the highest priority regular expression to match the text to be recognized. If a match is successful, the system immediately stops attempting to match any subsequent regular expressions, identifies the bank type tag corresponding to the successfully matched regular expression as the document's type, and returns the recognition result. If a match fails, the system downgrades to using the next highest priority regular expression, and so on. This strategy of terminating on the first match ensures that the type recognition result is uniquely determined, avoiding type conflicts that can occur when multiple regular expressions match simultaneously. This type tag serves as the type identifier for the document, used by subsequent step S120. If, after traversing all records in the regular expression library, no regular expression matches successfully, the system determines that the document's type recognition has failed and marks the document as parsing failed.

[0107] Furthermore, if all regular expressions fail to match, the system will process the document file as a failure. Specifically, the system will update the processing status field of the document file to "parsing failed" and record the reason for failure in the failure record table of the database. The reason for failure includes, but is not limited to, the following information: the file name of the document file, the MD5 value of the document file, a list of all regular expressions tried in this matching process, and the timestamp of the current operation.

[0108] In addition, after users supplement or update the regular expression library through the preset configuration management interface, the system provides a re-identification trigger interface for failed files. After users operate the "Retry Identification" button through the front-end interface, the system automatically queries the failure record table for all records marked "parsing failed" with the reason "cannot match bank type", filters out the corresponding original document files, and batch re-executes the type identification process according to the updated regular expression library.

[0109] For example, when a company needs to add support for recognizing receipts from new partner banks, such as processing receipts from Industrial Bank, the system administrator does not need programming skills. They only need to enter the regular expression pattern string ".(Industrial Bank|CIB)." through the backend management interface, select the bank type tab "Industrial Bank," set the priority value, and click save. The system immediately supports automatic type recognition for Industrial Bank receipts. The entire process requires no coding, no compilation, no system restart, and no downtime maintenance. This recognition mechanism allows failed files to be reprocessed after the recognition rules are improved, eliminating the need for users to manually re-upload, thus enhancing the system's usability and closed-loop processing capabilities.

[0110] After completing the type identification in step S115, the system enters the template matching stage in step S120.

[0111] Step S120: Match the corresponding parsing template according to the type, and match the optimal parsing template according to the preset priority.

[0112] In step S120, the template most suitable for the current document format is selected from multiple pre-set parsing templates for the bank. Specifically, this includes the following sub-steps S121 to S123 executed sequentially.

[0113] Step S121: Based on the type of the identified document file, call the corresponding template library. Each parsed template in the template library is associated with a priority identifier.

[0114] Specifically, the system uses the type label identified in step S115 as the query key to retrieve the corresponding parsing template library from the template management database. Each bank type corresponds to an independent template library, and each template library contains one or more parsing templates. The purpose of setting up these multiple parsing templates is to ensure compatibility with different receipt formats generated by the same bank at different times, in different branches, or under different business systems. Each parsing template is configured and has a priority identifier stored in its metadata through a preset configuration method when it is created. This priority identifier is represented by a positive integer, with a larger value indicating a higher priority.

[0115] Step S122: Try to match the parsing templates in order of priority from high to low, and determine the first successfully matched template as the optimal parsing template.

[0116] Specifically, after obtaining the template library corresponding to the current document file type, the system sorts all templates in the template library in descending order of priority. The system then retrieves each template in turn and performs matching and verification between the current document file and the template.

[0117] The specific process of matching and verification is as follows:

[0118] Each parsing template has multiple predefined matching features through a preset configuration method. These features include, but are not limited to: the applicable PDF page size range for the template, a list of specific keywords that must appear in the document, and the expected relative position range of key fields on the page. The system verifies one by one whether the current document file meets all the matching features of the current template. When all features pass verification, the current template is considered to have successfully matched the document file. The system records the currently successfully matched template and immediately stops attempting to match subsequent lower-priority templates.

[0119] In the above steps of the system, the first successfully matched parsing template is determined as the optimal parsing template for the current document file, and the unique identifier of the template is passed to the subsequent field extraction stage.

[0120] Step S123: Template downgrade mechanism and handling of all failures during the matching process.

[0121] Specifically, during the matching process in step S122, and during the field extraction process in subsequent step S130 and the verification process in step S140, if the parsing template of the current priority fails at any stage (for example, template matching fails, the number of fields with empty results when extracting key fields using the template exceeds the threshold, or the data extracted using the template fails the verification stage), the system will automatically perform a template downgrade operation.

[0122] The template downgrade operation refers to the system abandoning the currently prioritized parsing template, automatically switching to the next priority parsing template in the template library, and returning to step S122 to re-perform matching and verification. Upon successful matching, the field extraction and verification process is re-executed. The downgrade process occurs within the template library corresponding to the same type of tag, without switching between types.

[0123] More specifically, if the system fails to complete the parsing of any of the priority parsing templates in the template library after it has traversed all the parsing templates in the template library, and fails to complete any of the matching, extraction, or verification steps, the system will determine that the document file has failed to be parsed, mark it as "parsing failed", and record the reason for the failure. The reason for the failure includes the type label of the current document file, the list of templates that have been tried, and a description of the specific failed step.

[0124] After determining the optimal parsing template in step S120, the system uses the template to extract key information, i.e., to execute step S130.

[0125] Step S130: Using the optimal parsing template, extract key fields from the document file to obtain structured data.

[0126] Specifically, in this step, the system uses the extraction rules defined in the optimal parsing template to perform structured extraction of key information from the document file, which includes the following sub-steps S131 to S133 executed in sequence.

[0127] Step S131: Configure extraction rules for each key field.

[0128] Specifically, in the optimal parsing template, the system configures independent extraction rules for each key field to be extracted through a preset configuration method. Key fields include, but are not limited to: payment account, receiving account, payment account name, receiving account name, transaction amount, transaction date, transaction time, bank transaction number, bank name, and transaction purpose. The extraction rule for each key field is selected from at least one of the following two methods: coordinate positioning or regular expression matching.

[0129] The coordinate positioning method is suitable for table-type receipts with a well-structured layout. Its configuration parameters include: page number, horizontal start coordinates, horizontal end coordinates, vertical start coordinates, and vertical end coordinates of the target area on the page, to accurately locate the area range of the field on the PDF page. The regular expression matching method is suitable for text-type receipts. Its configuration parameter is a regular expression pattern string, which locates and extracts the value of the field by matching keywords near the field.

[0130] Step S132: Extract the corresponding field values ​​from the document file according to the configured extraction rules.

[0131] Specifically, the system extracts the original values ​​of each key field from the document file according to the extraction rules configured in step S131. For fields configured with coordinate positioning rules, the system locates a specified area in the PDF page using coordinate parameters and extracts the text content of that area as the field's original value. For fields configured with regular expression matching rules, the system performs regular expression matching operations on the extracted full-text text and uses the content matched by the first capture group as the field's original value. For fields configured with both of the above rules, the system prioritizes the coordinate positioning method for extraction; if coordinate positioning extraction fails, it automatically switches to the regular expression matching method.

[0132] Step S133: Convert the format of the extracted field values.

[0133] Specifically, after extracting the original values ​​of each field in step S132, the system performs format conversion operations on the original values ​​according to the format conversion rules configured for each field in the optimal parsing template through a preset configuration method. Format conversion includes at least one of the following types: data type conversion, that is, converting string data to numeric or date data; format normalization, that is, converting date strings of different formats into the standard "YYYY-MM-DD" format, or converting amount strings of different formats into the standard numeric format; unit conversion, that is, converting amounts in "ten thousand yuan" to amounts in "yuan".

[0134] Thus, after processing steps S131 to S133, unstructured key information in the document file is extracted and converted into standardized structured data records, which are stored in the system memory in the form of a set of key-value pairs.

[0135] After obtaining the structured data, the system performs multiple verifications and splitting processes on the data, i.e., executes step S140.

[0136] Step S140: Perform multiple validations on the structured data to obtain normal document files, and split the normal document files to generate independent document files.

[0137] In step S140, to ensure the accuracy and completeness of the extracted structured data and to resolve the issue of splitting multiple merged documents, specific steps include verification sub-steps S141 to S142 and splitting sub-steps S143 to S145.

[0138] Specifically, multiple validations are performed on structured data records to obtain normal document files, including:

[0139] Step S141: Perform multiple validations on the structured data.

[0140] Multiple checks include three types: format checks, integrity checks, and logical checks. Specifically:

[0141] Format validation is used to verify whether the values ​​of each field conform to preset format rules. For example, for the account field, it verifies whether it is a 16 to 20-digit number; for the amount field, it verifies whether it is a valid positive value; for the date field, it verifies whether it is a valid "YYYY-MM-DD" format date.

[0142] Integrity checks are performed to verify that core fields are empty or missing. Core fields are predefined by the parsing template using a pre-defined configuration method and include at least the payment account, receiving account, and transaction amount. Integrity checks fail if any core field is empty or missing.

[0143] Logical validation is used to verify whether the business logic between different fields is consistent. For example, it verifies whether the number of digits in the payment account matches the account length specifications of the bank to which the payment account belongs; verifies whether the transaction amount is a positive number greater than zero; and verifies whether the transaction date is within a reasonable business time range.

[0144] Step S142: Verify the exception markers and support re-parsing for failed checks.

[0145] Specifically, during the verification process in step S141 above, if any verification type fails, the system determines the document file as an abnormal file, marks its processing status as "abnormal," and records the specific information of the verification failure in the database's abnormal record table. The reasons for failure include: the failed verification type, i.e., one or more of format verification, integrity verification, or logical verification; the name of the failed field, i.e., which specific field failed the verification; and the comparison information between the expected condition and the actual value.

[0146] Furthermore, after the user corrects the corresponding parsing template or verification rules according to the above-mentioned failure reasons through preset configuration, the system provides a "re-parsing" operation entry through the front-end interface, allowing the user to select a single or batch of documents marked as abnormal, triggering the re-execution of the entire process from steps S110 to S140. Therefore, the above mechanism is used to achieve closed-loop processing of abnormal documents, avoiding the tedious operation of manually correcting each document one by one.

[0147] After completing multiple checks, the system further performs a splitting operation on the document files that are determined to be "normal".

[0148] More specifically, the normal document files are split, including:

[0149] Step S143: Automatically identify split boundaries using regular expression matching and layout feature analysis.

[0150] For document files that pass all checks in steps S141 and S142 and are determined to be "normal," the system further determines whether the file contains multiple independent documents. The system uses a strategy combining regular expression matching and layout feature analysis to automatically identify the split boundaries.

[0151] Specifically, regular expression matching is used to identify fixed separators between documents. The system searches the entire text of the document according to a preset list of separator regular expressions. These separator regular expressions include, but are not limited to: pattern strings for matching consecutive delimiter lines such as "---" or "====", pattern strings for matching page number markers such as "Page X of Y", and pattern strings for matching specific start or end keywords such as "transaction log" or "blank space below this page".

[0152] More specifically, layout feature analysis is used to identify significant changes in layout structure. The system performs rendering analysis on each page of the PDF file, extracting layout feature parameters such as text distribution density, table line position, and large blank areas. When the layout feature parameters of adjacent areas or adjacent pages undergo abrupt changes exceeding a preset threshold, the location of the abrupt change is identified as a possible split boundary.

[0153] Step S144: Based on the split boundaries, split multiple documents in a single document file into multiple independent document files.

[0154] Specifically, based on all the splitting boundary positions identified in step S143, the system determines the start page, end page, and start and end coordinates within the page for each independent document in the original file. The system then calls the PDF page cropping and reassembly engine to split the original file into multiple independent PDF files according to the determined boundary parameters, with each independent file corresponding to a complete document.

[0155] During the splitting process, the system keeps the original page content of each independent file unmodified, and only performs page range truncation and reorganization to ensure the integrity and readability of the split files.

[0156] Step S145: Name and store the split files and record their MD5 hashes.

[0157] After the split is completed, the system automatically generates a filename for each independent document file according to the naming rules set through preset configuration. The naming rules include at least one combination of the following elements: original filename, document type identifier, split sequence number, and split timestamp. For example, the generated filename format is "original filename_bank type_split sequence number_year month day hour minute second.pdf".

[0158] Specifically, the system stores the individual files named according to this rule in the file storage system. Simultaneously, it calls the MD5 calculation engine to calculate the MD5 value of each individual file, associates this MD5 value with the corresponding file storage path, and records it in the file management table of the database for subsequent file deduplication verification. At this point, the multiple documents contained in the original file have been separated at both the physical file level and the logical data level.

[0159] After the split is completed, the system will associate and archive the individual document files with the corresponding structured data, i.e., execute step S150.

[0160] Step S150: Link and archive the independent document files and their corresponding structured data, and establish a multi-dimensional retrieval index.

[0161] In step S150, the system associates each independent document file generated in step S145 with the structured data extracted in step S130 and verified in step S140 to form a complete data package based on a single independent document. Each data package includes the original PDF of the document and its corresponding set of structured key-value pairs.

[0162] Specifically, the system performs archiving and storage operations on the aforementioned data packets according to the archiving rules set through preset configuration. The archiving rules are flexibly configured by the administrator through the system backend, supporting categorization and archiving according to at least one of the following dimensions, and the hierarchical order of each dimension can be customized by the administrator: Institutional type dimension, i.e., creating a first-level directory based on bank type, such as "Industrial and Commercial Bank of China" and "China Construction Bank"; Date dimension, i.e., creating a second-level directory based on transaction date under the institutional type directory, such as "January 2026"; Transaction purpose dimension, i.e., creating a third-level directory based on transaction purpose under the date directory, such as "Payment for Goods" and "Service Fee"; Payer or Payee dimension. During archiving, the system automatically creates the corresponding directory structure, stores the PDF file in the lowest-level directory, and writes the corresponding structured data into a data table in the database, forming a foreign key relationship with the storage path of the PDF file.

[0163] More specifically, during the archiving process described above, the system simultaneously builds a multi-dimensional retrieval index. The index dimensions cover all key fields extracted in step S130, including but not limited to: institution type, transaction date, transaction purpose, full name of payer, payer account number, full name of payee, payee account number, transaction amount, and bank transaction number. Preferably, a database B and a tree index structure are used, and the index data is updated synchronously in real time after archiving is completed to ensure strong consistency of query results.

[0164] Furthermore, once the index is built, the system provides a multi-condition combined query interface, allowing users to select any combination of index dimensions as query conditions through the front-end interface. Since all query conditions are indexed during archiving, the system performs an index scan on the user's query request, achieving rapid location and return of query results within seconds. The system also supports online previewing of search results, single or batch downloading, and data linkage with external financial business systems to achieve data sharing and drill-down queries of receipt data.

[0165] Based on the above overview, through steps S110 to S150, this application embodiment achieves fully automated and intelligent processing of bank documents from acquisition, identification, parsing, verification, splitting to archiving and retrieval, improving the efficiency and accuracy of financial processing, and effectively solving the compatibility problem of processing documents from multiple banks and multiple formats.

[0166] An embodiment of the second aspect of this application provides a document processing system, such as... Figure 2 As shown, the document processing system in this embodiment includes a configuration acquisition module 610, a type identification module 620, a template matching module 630, a field extraction module 640, a verification module 650, a splitting module 660, and an archive retrieval module 670.

[0167] The configuration acquisition module 610 is used to acquire the document file to be processed through a preset configuration method. The type identification module 620 is used to identify the type of the document file. The template matching module 630 is used to match the corresponding parsing template according to the type, and to match the optimal parsing template according to a preset priority. The field extraction module 640 is used to extract key fields from the document file using the optimal parsing template to obtain structured data. The verification module 650 is used to perform multiple verifications on the structured data to obtain normal document files. The splitting module 660 is used to split the normal document file to generate independent document files. The archiving and retrieval module 670 is used to associate and archive the independent document files and their corresponding structured data, and to establish a multi-dimensional retrieval index.

[0168] In the implementation of the document automatic processing system of this embodiment, each module can be an existing module product with data transmission, storage, or computation processing functions. The modules communicate with each other through standardized data interfaces, with the output of one module serving as the input of the next, thus forming a complete automated processing chain. In specific applications, the specific implementation process of the functions of each module in the document automatic processing system of this embodiment can be found in the relevant descriptions in the above method embodiments, and will not be repeated here.

[0169] The above descriptions are merely some embodiments of this application and are not intended to limit this application. The technical features or structures in the foregoing different embodiments can be arbitrarily combined to form other specific technical solutions as needed. For those skilled in the art, this application can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of the claims of this application.

Claims

1. A document processing method for automating bank statement processing in an enterprise financial system, characterized in that, The method includes: The document file to be processed is obtained through a preset configuration method, and the type of the document file is identified; Match the corresponding parsing template according to the type, and select the optimal parsing template according to the preset priority; Using the optimal parsing template, key fields are extracted from the document file to obtain structured data; The structured data is subjected to multiple validations to obtain normal document files, and the normal document files are then split to generate independent document files. The independent document files and their corresponding structured data are linked and archived, and a multi-dimensional retrieval index is established.

2. The document processing method according to claim 1, characterized in that, The step of obtaining the document file to be processed through a preset configuration method includes: The document file source channel is connected to at least one document file source channel through a preset configuration method. The document file source channel includes at least one of the following: a file transfer protocol server, a shared folder, and a local directory. By triggering a scheduled task, the document files to be processed are automatically obtained from the configured document file source channels; Calculate the MD5 value of the document to be processed and compare it with the MD5 value record of the processed document. If the MD5 value already exists, it is determined to be a duplicate file and filtered out; otherwise, it is added as a new file and enters the subsequent processing flow.

3. The document processing method according to claim 2, characterized in that, The step of obtaining the document file to be processed through a preset configuration method also includes: Receive the pending document file manually uploaded by the user, calculate the MD5 value of the pending document file, and compare it with the MD5 value record of the processed document file; If the MD5 value already exists, the upload is intercepted and a duplicate message is displayed. If the MD5 value does not exist, the upload is accepted and automatically treated as a new file for subsequent processing.

4. The document processing method according to claim 3, characterized in that, The method also includes: If the process fails to retrieve the document file to be processed via configuration, a retry process will be initiated. If the retries fail within all possible attempts, the retries are considered to have all failed, and a preset alarm message is output. The retry process employs an exponential backoff strategy, which sequentially increases the retry waiting time according to a preset initial retry interval, a multiplier factor, and a maximum number of retries.

5. The document processing method according to claim 1, characterized in that, The automatic identification of the document file type includes: Construct a dynamically extensible regular expression library, which supports online addition, modification, or deletion of regular expressions. Each regular expression is configured with a priority identifier and a corresponding type label. For the document to be identified, regular expressions are used to match it in descending order of priority, and the organization type or layout type corresponding to the first successfully matched regular expression is determined as the type of the document. If all regular expressions fail to match, mark the document file as parsing failed and record the reason for the failure. The reasons for failure include at least one of the following: file name, file MD5 value, list of regular expressions that failed to match, and timestamp.

6. The document processing method according to claim 1, characterized in that, The step of matching the corresponding parsing template according to the type and matching the optimal parsing template according to a preset priority includes: Based on the type of the identified document file, the corresponding template library is called, and each parsed template in the template library is associated with a priority identifier; The templates are matched and parsed in descending order of priority, and the first successfully matched template is determined as the optimal parsing template. Specifically, if the current parsing template fails to extract or validate fields, it will automatically be downgraded to the next lower priority parsing template and the field extraction and validation process will be repeated. If all priority parsing templates fail, the document file will be marked as parsing failed and the reason for the failure will be recorded.

7. The document processing method according to claim 1, characterized in that, The extraction of key fields from the document file includes: Configure extraction rules for each key field separately; Extract the corresponding field values ​​from the document file according to the configured extraction rules; The format of the extracted field values ​​is converted. The extraction rules are selected from at least one of coordinate positioning method or regular expression matching method; the format conversion includes at least one of data type conversion, format normalization, and unit conversion.

8. The document processing method according to claim 1, characterized in that, The multiple validation of the structured data includes: The structured data is subjected to format verification, integrity verification, and logical verification respectively; Mark the documents that fail the verification as abnormal and record the reason for the failure; The format validation is used to check whether the field conforms to the preset format rules, the integrity validation is used to check whether the core field is empty, and the logic validation is used to check the consistency of business logic between fields. The failure reasons include at least one of the following: validation type, failed field name, and comparison information between expected conditions and actual values.

9. The document processing method according to claim 1, characterized in that, The splitting of the normal document file includes: Automatically identify split boundaries using regular expression matching and layout feature analysis; Based on the splitting boundaries, multiple documents in a single document file are split into multiple independent document files; The split independent document files are automatically named and stored according to preset naming rules. At the same time, the MD5 value of each independent document file is calculated and recorded for subsequent file deduplication. The regular expression matching is used to identify fixed separators or start and end markers between documents; the layout feature analysis is used to identify layout structure changes or blank areas as split boundaries; the naming rules include at least one of the following: original file name, document type, split sequence number, and split timestamp.

10. A document processing system for automating the processing of bank receipts in an enterprise financial system, characterized in that, The system includes: The configurable acquisition module is used to acquire the document files to be processed through preset configuration methods; The type identification module is used to identify the type of the document file; The template matching module is used to match the corresponding parsing template according to the type, and to match the optimal parsing template according to a preset priority. The field extraction module is used to extract key fields from the document file using the optimal parsing template to obtain structured data; The verification module is used to perform multiple verifications on the structured data to obtain a normal document file; The splitting module is used to split the normal document file into independent document files; The archive retrieval module is used to associate and archive the independent document files and their corresponding structured data, and to establish a multi-dimensional retrieval index.

Images