A method, system, device and readable storage medium for processing daily data files

Abstract

The application relates to the technical field of data processing, and discloses a daily data file processing method, a daily data file processing system, a daily data file processing device and a readable storage medium. The daily data file processing method defines a daily file template structure in a client, configures fields and field separators in the template structure, and after the template definition is completed, a secure transmission channel is established based on a preset transmission protocol in response to a successful identity authentication instruction, and a template file is sent to a specified daily directory for storage. The daily data file processing method realizes self-service template configuration, file transmission security and interactive process standardization, and improves the maintainability and business cooperation efficiency of the system.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, system, device, and readable storage medium for processing end-of-day data files. Background Technology

[0002] In the existing end-of-day document processing workflow, such as Figure 1 As shown, the platform typically processes data, generates files, and uploads files sequentially based on business data in the database. The generated daily end-of-day files are then provided to the funding party for download via an SFTP channel based on the SSH protocol. The funding party retrieves the files from the SFTP directory at fixed times each day for reconciliation, data aggregation, and report generation. This model, centered on centralized processing by the platform, features unified management, batch generation, and secure transmission, making it suitable for scenarios with relatively simple data structures and stable output formats in the early stages.

[0003] However, with the increase in the number of participating funding parties and the diversification of business scenarios, the existing processes have gradually revealed their limitations. The main problems are insufficient flexibility, poor scalability, high maintenance costs, and a poor user experience: the platform generates files with fixed formats and field orders, requiring the platform to modify its program for any adjustments needed by funding parties, resulting in long modification cycles and high risks; different funding parties have significantly different needs, requiring the platform to perform repetitive custom development, leading to low response efficiency; frequent format changes increase system testing and communication costs; funding parties cannot configure templates independently and rely on the platform for adjustments, reducing the user experience. The resulting technical contradiction lies in the fact that the platform wants to maintain the stability of a unified batch data output mechanism, while funding parties require flexible and configurable data output capabilities, resulting in a lack of coordination between maintainability and personalization. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, system, device, and readable storage medium for processing end-of-day data files, which can effectively solve the problems in the prior art such as fixed end-of-day file format, non-customizable field order, high platform maintenance costs, and insufficient flexibility for funders.

[0005] In a first aspect, embodiments of this application provide a method for processing end-of-day data files, applied on a client side, including:

[0006] Define the end-of-day document template structure, and configure fields and field separators in the end-of-day document template structure to construct the end-of-day document template;

[0007] Upon successful identity authentication, the end-of-day file template is sent to the designated end-of-day directory for storage via a preset transmission protocol.

[0008] In some embodiments, defining the end-of-day document template structure and configuring fields and field separators in the end-of-day document template structure to construct the end-of-day document template includes:

[0009] Select target fields from the collected preset field set and set the field order of the target fields to obtain a sequential field set;

[0010] Insert a preset delimiter into the sequential field set to obtain a field string with delimiter;

[0011] The delimited field string is written into the end-of-day document template structure to obtain the end-of-day document template.

[0012] In some embodiments, the step of responding to a successful authentication command by sending the end-of-day file template to a designated end-of-day directory for storage via a preset transmission protocol includes:

[0013] Upon successful authentication, an SFTP connection is established based on the SSH protocol to obtain a usable transfer channel.

[0014] The end-of-day file template is uploaded to the virtual root directory path through the available transmission channel, and the end-of-day file template is stored in the virtual root directory path.

[0015] Secondly, embodiments of this application provide a method for processing end-of-day data files, applied on a server side, including:

[0016] Obtain the end-of-day file template stored in the specified end-of-day directory, verify the end-of-day file template, and obtain a verified end-of-day file template; wherein, the end-of-day file template is obtained by the method described in the first aspect;

[0017] The validated end-of-day document template is parsed to extract field information, and corresponding data is extracted from the database based on the field information to generate the target end-of-day document.

[0018] Store the target end-of-day file in the specified end-of-day directory.

[0019] In some embodiments, obtaining the end-of-day document template stored in the specified end-of-day directory, validating the end-of-day document template, and obtaining a valid end-of-day document template includes:

[0020] In response to the timed scan command, the end-of-day file template is read from the specified end-of-day directory to obtain the end-of-day file template to be inspected;

[0021] The change status is determined based on the upload time of the end-of-day document template to be inspected, and the end-of-day document template to be verified is obtained.

[0022] The format and field content of the end-of-day document template to be verified are validated to obtain the verification result;

[0023] If the verification result is satisfactory, a valid end-of-day document template is generated.

[0024] In some embodiments, the step of parsing and validating the end-of-day document template to extract field information, extracting corresponding data from the database based on the field information, and generating the target end-of-day document includes:

[0025] Determine the file type of the verified end-of-day document template to obtain template file type information;

[0026] Based on the template file type information, the validated end-of-day file template is parsed, and the set of field names is extracted to obtain the field information;

[0027] The field information is loaded into the cache to obtain the cached field information;

[0028] Retrieve data corresponding to the cached field information from the database and generate a database result set;

[0029] A formatted dataset is obtained by sequentially combining the database result set and the cached field information.

[0030] The formatted dataset is written into the target end-of-day file, resulting in the target end-of-day file stored in the specified end-of-day directory.

[0031] In some embodiments, the method further includes:

[0032] During the process of generating the target day-end file, the file generation status is monitored, and the monitoring results are obtained.

[0033] When the monitoring result indicates an anomaly, an alarm command is triggered and an anomaly log is recorded, resulting in an alarm log record.

[0034] When the monitoring result indicates that there is no corresponding data, an empty file is generated and stored in the specified end-of-day directory, resulting in an empty file stored in the specified end-of-day directory.

[0035] Thirdly, embodiments of this application provide a day-end data file processing system, including:

[0036] The client is used to define the end-of-day document template structure, configure fields and field separators in the end-of-day document template structure, and construct the end-of-day document template; in response to the authentication success command, the client sends the end-of-day document template to the specified end-of-day directory for storage through a preset transmission protocol;

[0037] The server is used to obtain the end-of-day file template stored in the specified end-of-day directory, verify the end-of-day file template to obtain a verified end-of-day file template; parse the verified end-of-day file template to extract field information, extract corresponding data from the database based on the field information and generate a target end-of-day file, and store the target end-of-day file in the specified end-of-day directory to obtain the target end-of-day file stored in the specified end-of-day directory.

[0038] Fourthly, embodiments of this application provide a terminal device, the terminal device including a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the end-of-day data file processing method of the first aspect described above.

[0039] Fifthly, embodiments of this application provide a computer-readable storage medium, wherein when the computer program is executed on a processor, it implements the end-of-day data file processing methods described in the first and second aspects.

[0040] The embodiments of this application have the following beneficial effects: The client defines the end-of-day document template structure and configures fields and field separators within the template structure to form a standardized document template that meets business requirements. After completing the template definition, the client responds to a successful authentication command, establishes a secure transmission channel based on a preset transmission protocol, and sends the generated template file to the designated end-of-day directory for storage. Through this application's solution, the client can independently configure the template structure and field order, achieving flexible template format construction and differentiated adaptation; the server can perform file reception and parsing based on unified template rules, thereby improving the consistency of template management and the standardization of data interaction. In other words, it effectively achieves self-service template configuration, secure file transmission, and standardized interaction processes, significantly improving the system's scalability, maintainability, and business collaboration efficiency. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This illustration shows a schematic diagram of the end-of-day data file processing method according to an embodiment of this application—a prior art solution.

[0043] Figure 2 This illustration shows a framework diagram of a day-end data file processing method according to an embodiment of this application;

[0044] Figure 3 A flowchart of a day-end data file processing method according to an embodiment of this application is shown;

[0045] Figure 4 This paper illustrates a flowchart of a client-side method for processing end-of-day data files according to an embodiment of this application.

[0046] Figure 5 This paper illustrates a flowchart of a day-end data file processing method according to an embodiment of the present application – on the server side. Detailed Implementation

[0047] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0048] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0049] In the following text, the terms "comprising," "having," and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more combinations thereof. Furthermore, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0050] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0051] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0052] Considering the problems of fixed end-of-day file formats, non-customizable field order, high platform maintenance costs, and insufficient flexibility for fund providers in existing technologies, this application proposes an end-of-day data file processing method. By decoupling the file template definition from the file generation process, it allows fund providers to independently define the file template format and field order. The platform then performs verification, parsing, and target file generation based on the received file template, achieving a balance between flexible configuration and unified management. Figure 2 As shown.

[0053] The following describes the method for processing end-of-day data files using specific examples.

[0054] Figure 3 A flowchart of an end-of-day data file processing system according to an embodiment of this application is shown. Exemplarily, the end-of-day data file processing system includes the following steps:

[0055] In this embodiment, the system comprises two parts: a client and a server. The client is also referred to as the funding party, and the server is also referred to as the platform party. The client is used for defining and transmitting end-of-day document templates, while the server is used for receiving, verifying, parsing, and generating target files from the template files. The two parties interact with each other and maintain directory isolation through a protocol-based secure channel, forming a complete end-of-day data file processing mechanism.

[0056] In step S100, the client defines the end-of-day document template structure and configures fields and field separators in the end-of-day document template structure to construct the end-of-day document template; in response to the authentication success command, the end-of-day document template is sent to the specified end-of-day directory for storage through the preset transmission protocol.

[0057] The end-of-day file template structure is a structural definition describing the arrangement and formatting relationships of fields, used to guide subsequent data parsing on the platform. The default transmission protocol is SFTP, based on the SSH protocol, used for file uploads over an encrypted channel. Field separators are one of the platform-supported symbols, including commas ",", vertical bars "|", &, or @, etc.

[0058] As an example, the funding party selects the target field from the platform's preset list of supported fields, sets the field order and separator, and generates a template file containing the field configuration. For instance, when the fields "Customer ID, Transaction Date, Transaction Amount" are selected, the first line of the generated template file is "Customer ID|Transaction Date|Transaction Amount". The funding party saves the template file as "day_end_template.csv" and uploads it to the virtual root directory path "template" assigned by the platform via SFTP after successful authentication.

[0059] Step S200: The server obtains the end-of-day file template stored in the specified end-of-day directory, verifies the end-of-day file template, and obtains the verified end-of-day file template; it parses the verified end-of-day file template to extract field information, extracts corresponding data from the database based on the field information and generates the target end-of-day file, and stores the target end-of-day file in the specified end-of-day directory, thus obtaining the target end-of-day file stored in the specified end-of-day directory.

[0060] The specified end-of-day directory is the logical directory path on the platform used to store the funding templates and result files. The validation operation is used to verify the format integrity and field validity of the template files. The target end-of-day file is a data file generated from the database based on the template fields and is used to output the end-of-day results.

[0061] As an example, the platform periodically scans a specified directory (e.g., "a / template"), reads the latest uploaded template files, and performs format validation. The system parses the first line of the template, "Customer ID | Transaction Date | Transaction Amount," checks if the field names exist in the platform's field whitelist, and confirms that the delimiter conforms to the rules. After successful validation, the system constructs an SQL query statement based on the field information. For example:

[0062]

[0063] Then, extract the corresponding data from the database and generate a formatted dataset:

[0064] “10001|2025-10-09|520.00”;

[0065] "10002|2025-10-09|418.50";

[0066] “10003|2025-10-09|630.75”;

[0067] The system then creates the target file "dayend_result_20251009.csv", writes formatted data rows into it sequentially, and stores it in the specified end-of-day directory "root / dayend / output".

[0068] In other implementations, if no corresponding data is found in the database, an empty file containing only the field header row is generated for the funding party to access, thus preventing downstream task interruption. If missing or duplicate template fields are detected, the system will record a verification error log and send a template anomaly alarm message; if no corresponding data is found in the database, the system will generate a blank CSV file containing only field rows to maintain the integrity of the task chain.

[0069] In other words, by building a configurable end-of-day document template definition module on the funding side, the funding party can autonomously manage the template structure and field order. The funding party can flexibly select output fields, set delimiters, and file formats according to business needs, thereby achieving rapid file format adjustments without relying on platform code modifications. This significantly improves the flexibility of template configuration and business response speed, reducing communication costs and secondary development investment. Simultaneously, after completing the template definition, the funding party can establish an encrypted transmission channel through the identity authentication module and secure transmission module, ensuring the security and integrity of the template file upload process, preventing data leakage and damage during network transmission, and achieving autonomous and secure control over the template generation and upload process.

[0070] The platform integrates modules for template receiving, format validation, field parsing, and file generation to automate and standardize template file management. Based on the received template structure, the platform can automatically match database fields and generate target files, reducing manual intervention and maintenance. This mechanism supports parallel access from multiple funding parties and differentiated template parsing, significantly improving the system's scalability and compatibility. Simultaneously, the platform can perform status monitoring, anomaly alerts, and log auditing during file generation to ensure the stability and traceability of task execution.

[0071] Figure 4 A flowchart illustrating a day-end data file processing method according to an embodiment of this application is shown. Exemplarily, this day-end data file processing method is applied on a client side and includes the following steps:

[0072] In this embodiment, the client is also referred to as the funding party, and the client is mainly responsible for defining and uploading the end-of-day document template.

[0073] Step S210: Define the end-of-day document template structure, configure fields and field separators in the end-of-day document template structure, and construct the end-of-day document template.

[0074] The file template structure describes the field order and separation method required for the end-of-day data file; the fields are derived from the pre-published support field set of the platform; the field separator is one of the symbols allowed by the platform, such as "|", "&", "@", etc.

[0075] During the implementation process, the funding party selects fields and sets formats for the template structure based on the field support list and file format conventions provided by the platform, ultimately forming a standardized template file that can be recognized by the platform. This template file is used to guide the platform in data field matching and format parsing processing when generating end-of-day data.

[0076] In an optional embodiment, step S210 includes the following sub-steps:

[0077] S211, Select the target field from the collected preset field set and set the field order of the target field to obtain a sequential field set.

[0078] The preset field set is a list of fields that the platform predefines and makes available to funding parties, including multiple data items such as customer number, account number, transaction date, and transaction amount.

[0079] As an example, the funding party selects target fields from a preset field set based on its own business needs, and sets the field arrangement order according to the actual business output requirements, generating a sequential field set. This sequential field set is used for subsequent template format generation and file content mapping.

[0080] For example, when the funding party needs to output the fields "customer number, transaction date and transaction amount", its sequential field set can be constructed in the order of "customer number → transaction date → transaction amount".

[0081] S212, insert a preset delimiter into the sequential field set to obtain a field string with delimiter.

[0082] The preset delimiters are standard field separators defined by the platform and used to divide different fields in the generated template file. Common delimiters include symbols such as "|", "&", and "@".

[0083] As an example, the funding party inserts corresponding delimiters into the sequential field set according to the field's position, causing fields and delimiters to alternate, forming a delimited field string. For example, the sequential field set "Customer ID, Transaction Date, Transaction Amount" is converted to "Customer ID | Transaction Date | Transaction Amount".

[0084] In other implementations, if the template file type is CSV, the delimiter can be replaced with an English comma to conform to the common CSV file standard.

[0085] S213, write the field string with delimiters into the end-of-day document template structure to obtain the end-of-day document template.

[0086] The end-of-day document template is a text file structure that conforms to the platform's requirements, used to store field arrangement and delimiter information. The template file can be in TXT or CSV format, and its content consists of field strings separated by preset delimiters.

[0087] As an example, the funding party writes the delimited field string into a template structure file and saves it as a specified file type to obtain the end-of-day document template. The template file is stored locally by the funding party or in a specified directory, serving as input for subsequent uploading and verification.

[0088] For example, when the funding party selects the CSV format, the generated file template is named "day_end_template.csv", and its content is "customer number, transaction date, transaction amount", which can be directly recognized and parsed by the platform's SFTP module.

[0089] Step S220: In response to the successful identity authentication command, the end-of-day document template is sent to the designated end-of-day directory for storage via a preset transmission protocol.

[0090] Among them, the successful identity authentication instruction is an authentication confirmation signal returned by the platform authentication module after the funder logs into the platform's file transfer service, which is used to confirm that the funder has file upload permissions; the preset transmission protocol is the Secure File Transfer Protocol (SFTP) based on the SSH protocol, which is used to complete the upload operation of template files under an encrypted channel; the specified end-of-day directory is the virtual root directory path allocated by the platform to each funder, which is used to distinguish the end-of-day file storage space of different funders.

[0091] During the process, before uploading, the funding party must authenticate their identity using a unique account and strong password uniformly assigned by the platform. Each account is bound to a unique identifier to ensure that the operation is traceable and auditable. After successful authentication, an SFTP session is initiated and a secure transmission channel is established to transfer the locally generated end-of-day document template file to the platform's designated directory, ensuring that the template file can be correctly saved and read by the platform's file management module.

[0092] In an optional embodiment, step S220 includes the following sub-steps:

[0093] S221 establishes an SFTP connection based on the SSH protocol upon successful authentication, thus obtaining a usable transfer channel.

[0094] SFTP connection is a secure file transfer session established on the basis of SSH protocol, used to realize encrypted data file transfer; available transfer channel is an available data transfer path that has been authenticated and handshaked.

[0095] As an example, after receiving a success instruction from the platform's authentication module, the funding party invokes the transport module to perform connection initialization. During the connection process, the system reads the server address, port number, username, and authentication key from the configuration file, completes authentication and key negotiation through the SSH handshake process, and generates an encrypted channel; subsequently, an SFTP subprotocol instance is loaded into this channel to obtain a usable transport channel.

[0096] For example, when the platform's server address is "sftp.bankhost.com" and the port number is 22, the fund provider will automatically execute the command "ssh -p 22 user@sftp.bankhost.com" after successful authentication, and establish a handshake through the key file "id_rsa", ultimately generating a transport session with the number session_001.

[0097] S222, upload the end-of-day file template to the virtual root directory path through the available transmission channel to obtain the end-of-day file template stored in the virtual root directory path.

[0098] The virtual root directory path is a logical file path space defined by the platform to isolate data files uploaded by different funding parties; the end-of-day file template is a template file generated locally by the funding party, containing field definitions and delimiter information.

[0099] As an example, after establishing a transmission channel, the funding party reads the defined template file into a memory buffer and performs an upload operation using the SFTP `put` method. During the upload process, the system first checks if the target path exists; if not, it creates the corresponding subdirectory under the virtual root directory. Then, the file stream is transmitted in chunks as data blocks. After the transmission is complete, the platform generates the corresponding file in the target directory. Finally, the end-of-day file template is obtained and stored in the virtual root directory path. In other words, the platform uses virtual root directory technology to achieve file system-level isolation among multiple funding parties: each funding party account can only access its own dedicated virtual root directory after logging in (e.g., account A corresponds to / a, which appears as / externally), and cannot perceive or access other funding party directories (e.g., / b). This mechanism achieves logical isolation at the operating system level, eliminating the risk of unauthorized access.

[0100] For example, the funding template file path is C:\dayend\template.csv, and the target virtual directory path is root / dayend;

[0101] After uploading the file using the SFTP command `put C:\dayend\template.csv / root / dayend / template.csv`, a file with the same name "template.csv" will exist in the platform's file system.

[0102] It is understood that the method in this embodiment can be executed by the client in the system of the above embodiments. The method in this embodiment can also be applied to the end-of-day data file processing system of the above embodiments, so it will not be described again here.

[0103] Figure 5A flowchart illustrating a day-end data file processing method according to an embodiment of this application is shown. Exemplarily, this day-end data file processing method, applied on a server, includes the following steps:

[0104] In this embodiment, the server corresponds to the platform and is mainly responsible for receiving, verifying, parsing, and generating target files from template files.

[0105] Step S310: Obtain the end-of-day document template stored in the specified end-of-day directory, verify the end-of-day document template, and obtain a verified end-of-day document template; wherein, the end-of-day document template is obtained by the method of the above embodiment.

[0106] The designated end-of-day directory is a virtual root directory path allocated by the platform to the funding party, used to store uploaded template files; the verification operation is used to confirm the integrity and structural validity of the template files to ensure that the files can be correctly parsed by subsequent modules.

[0107] During implementation, the platform periodically scans a designated directory to detect template files uploaded by funders, and performs multi-level verification based on file format rules, field configuration requirements, and naming conventions to generate a list of template files that have passed verification.

[0108] In an optional embodiment, step S310 includes the following sub-steps:

[0109] S311, responding to the timed scan command, reads the end-of-day file template from the specified end-of-day directory to obtain the end-of-day file template to be inspected.

[0110] Among them, the scheduled scan instruction is a periodic trigger instruction generated by the platform's planned task scheduling module, which is used to start directory file checks at preset times; the end-of-day file template to be checked is the newly uploaded or updated file detected during the scan process.

[0111] As an example, the platform triggers a scan every 10 minutes to read all template files under the specified directory path, detect whether there are any newly uploaded or updated template files, obtain their file names, upload times and file sizes, and form a set of template files to be checked.

[0112] For example, if the file "template_20241008.csv" exists in the directory "root / dayend" and the system detects that its modification time is later than the last scan record, then the file will be marked as a template to be inspected.

[0113] S312, determine the change status based on the upload time of the end-of-day document template to be inspected, and obtain the end-of-day document template to be verified.

[0114] The change status is a marker indicating changes to the file since the last scan, used to identify whether the file has been updated or replaced; the template file to be verified is a file whose upload time has changed.

[0115] As an example, the system compares the upload time of the current scan record with the timestamp of the last scan record in the database. If the timestamps do not match, the file is marked as "pending verification" and added to the verification queue. Furthermore, if a template update is detected within the end-of-day processing window agreed upon by the platform and the funding party (e.g., 06:00 to 08:00 daily), even if the format is valid, the system will still generate an alarm to notify the platform for manual confirmation, preventing interference with ongoing end-of-day batch processing tasks. This scan result will be recorded in the database for subsequent troubleshooting and audit analysis.

[0116] In other implementations, the system may also perform a secondary comparison by combining the file's MD5 digest or SHA256 hash value to avoid misidentification caused by manual modification of the file's time.

[0117] S313, validate the format and field content of the end-of-day document template to be validated, and obtain the validation result.

[0118] The format validation includes file type (txt / csv), encoding format, and consistency of field separators. Field content validation includes whether the field name exists in the platform's field whitelist, whether the field can be found in the database query result set for the current day, and whether the field order conforms to the specifications. Field content validation is considered successful only when all template fields can be successfully mapped in the database result set; otherwise, it is considered a validation failure, triggering an alarm and notifying the platform for manual intervention. It is particularly noteworthy that the platform explicitly states that database fields are only added, not deleted, and not modified. Therefore, if field mapping fails during data integration, it is usually due to incorrect template configuration or improper file upload timing, requiring manual verification.

[0119] As an example, the system first checks if the file extension matches the platform configuration (e.g., .csv or .txt), then reads the first line of the file, parses the delimiter symbols, and compares the set of field names with the field support table defined by the platform. If all field names match and the delimiter conforms to the rules, the file structure is deemed valid.

[0120] For example, when the system detects that the first line of a file contains "Customer ID | Transaction Date | Transaction Amount" and the system-defined separator is "|", the system determines that the format and fields are compliant and generates a validation result of "Pass".

[0121] S314 If the verification result is successful, generate a verified end-of-day document template.

[0122] Among them, the templates that pass the verification are template files whose structure and fields meet the platform requirements, and their verification status is registered in the system database.

[0123] As an example, the system updates the status of the verified file to "VALID" and records the file name, upload time, field set and parsing status in the template management table, generating a verified template object for subsequent parsing and data extraction stages.

[0124] In other implementations, if the file fails verification, the system will record the abnormal field and corresponding error information in the log module, and at the same time send a template abnormality notification to the funder through the message queue to prompt them to re-upload the template file.

[0125] Step S320: Parse the verified end-of-day document template to extract field information, extract corresponding data from the database based on the field information, and generate the target end-of-day document.

[0126] The field information is a set of field names defined in the template file, and the target end-of-day file is a file generated after extracting data from the database based on the field information and performing formatting processing.

[0127] During implementation, the platform performs field parsing operations on the verified template files, extracts the field names contained in the template, and loads the field information into the cache module; then, it obtains the data content of the corresponding fields through the database query interface, generates a formatted data file, and outputs the result file to the specified directory for subsequent distribution or archiving.

[0128] In an optional embodiment, step S320 includes the following sub-steps:

[0129] S321, determine the file type of the verified end-of-day document template and obtain the template file type information.

[0130] The file type information indicates the storage format of the template file (such as CSV, TXT, or DAT) and determines the subsequent parsing method.

[0131] As an example, the system determines the file type based on the file extension. When the template file name ends in "csv", the parsing module uses a comma-separated strategy; when it ends in "txt", a vertical bar-separated strategy is used. File type information is recorded in the template parsing log.

[0132] S322: Based on the daily end-of-day document template that has passed the parsing and verification of the template file type information, extract the set of field names to obtain the field information.

[0133] The field name set is a list of fields defined in the first line of the template file or in the configuration section, used to indicate the query target for the corresponding field in the database.

[0134] As an example, the parsing module reads the first line of the template file, splits the fields according to the file type information, and generates a field list structure. If the template content is "Customer Number|Transaction Date|Transaction Amount", the system generates a set of field names [custNo,tradeDate,amount] and stores it in the memory cache.

[0135] For example, this set of field names will be used as the target fields in a subsequent database query as a SELECT statement to extract matching data.

[0136] S323, load the field information into the cache to obtain the cached field information.

[0137] The cache serves as a temporary storage area within the platform, typically implemented through a distributed cache space. It accelerates the retrieval of field information during subsequent database queries and file generation, while also supporting task sharing across multiple server nodes. Cached field information is stored in a structured format, for example, using the funding institution's ID as the key and a JSON object as the value.

[0138] {"orgNo":"abcdefg","fileType":"txt","fields":["a","b","c",...]};

[0139] After the cache is updated, the old version field information of all nodes is automatically invalidated through the consistency mechanism of the distributed cache (such as broadcast deletion of old keys and version number verification), which avoids multi-node tasks from obtaining expired data and ensures that subsequent tasks use the latest template configuration.

[0140] As an example, the system writes field information into the distributed cache space through a distributed cache management module and sets a uniform cache expiration time (e.g., 15 minutes) for each cache key. This cached field information can be shared across multi-threaded tasks, avoiding duplicate file parsing. For instance, the parsed content can be stored in the distributed cache in a specified format, such as using the organization number as the key and the value in JSON format: {"orgNo":"Organization Number","fileType":"txt","fields":"a|b|c|d...."}, significantly reducing redundant calculations and network overhead.

[0141] In other implementations, if the cache space is insufficient, the system can automatically trigger an old cache cleanup mechanism to ensure the execution efficiency of high-frequency template tasks.

[0142] S324: Retrieve data corresponding to the cached field information from the database and generate a database result set.

[0143] The database result set is the raw data set retrieved by the platform from the business database based on the template field information.

[0144] As an example, the system dynamically constructs SQL query statements based on field information, and the order of fields in the SELECT clause remains consistent with the template definition, for example:

[0145]

[0146] The query results are read at the row level and then written to the result set buffer.

[0147] For example, when the query result contains three records, the result set content is as follows:

[0148] [10001,2025-10-09,520.00],[10002,2025-10-09,418.50],[10003,2025-10-09,630.75];

[0149] [{"a":a1,"b":b1...},{"a":a2,"b":b2...},{"a":a3,"b":b4...}].

[0150] S325, based on the database result set and cached field information, sequentially combines them to obtain a formatted dataset.

[0151] The formatted dataset is an ordered data structure obtained by arranging and combining the field values ​​in the database result set according to the template field order, and is used to generate the final file content.

[0152] As an example, the system iterates through the result set records, concatenating field values ​​one by one according to the order of the cached field information definition: "fields": "a|b|c|d....", and inserting delimiters to form formatted data rows. For example, it generates:

[0153] “10001|2025-10-08|520.00”;

[0154] “10002|2025-10-09|418.50”;

[0155] “10003|2025-10-09|630.75”;

[0156] {"a1|b1...","a2|b2...","a3|b3..."};

[0157] In other implementations, if the template field contains an amount field, the system can perform numerical formatting (such as padding to two decimal places) before combining.

[0158] S326, Write the formatted dataset to the target end-of-day file, resulting in the target end-of-day file stored in the specified end-of-day directory.

[0159] The target end-of-day file is the end-of-day output file generated by the platform system based on the template field information, and the file format is consistent with the template file.

[0160] As an example, the system creates the target file "dayend_result_20251009.csv" and writes the contents of each row of the formatted dataset sequentially, finally saving it to the specified directory "root / dayend / output".

[0161] For example, when the template type is CSV, the first line of the generated file contains field names, and subsequent lines contain corresponding data records, which the platform stores in the corresponding funding party's subdirectory.

[0162] Step S330: Store the target end-of-day file in the specified end-of-day directory.

[0163] The designated end-of-day directory is the storage path for the platform system's output files, used to archive end-of-day result files for download or review.

[0164] As an example, after the writing is completed, the system registers the target file path in the file index table and records the task execution results in the log system, including the file name, number of records, generation time and storage path.

[0165] In other implementations, the platform can also be configured with event triggering rules: upon completion of the end-of-day file storage operation, a message push mechanism is automatically triggered to send a "end-of-day file generation complete" notification to the operations and maintenance center. This notification serves two purposes: first, it allows operations and maintenance personnel to review the file generation results; second, it acts as a trigger signal for downstream related tasks.

[0166] In one optional embodiment, the end-of-day data file processing method applied to the server further includes the following steps:

[0167] Step S340: During the process of generating the target end-of-day file, the file generation status is monitored, and the monitoring results are obtained.

[0168] Among them, the file generation status is the running status identifier of the platform when executing the target day's end-of-day file generation task, which is used to reflect the file generation progress, task execution time and abnormal status; the monitoring result is the status judgment result obtained by analyzing the real-time data collected based on the running status.

[0169] As an example, when performing data extraction and file writing operations in the S320 phase, the platform invokes the task monitoring module to monitor the generation process in real time. The system periodically checks the CPU usage, memory consumption, output file size, and line count changes of the file generation thread and writes these indicators into the task monitoring table. If the generated file size remains zero, the line count is not updated, or a thread interruption signal is detected, the system will mark the monitoring result as "abnormal"; if the file size and record count increase normally, the monitoring result will be marked as "normal".

[0170] In other implementations, the system can also combine the generation time threshold to determine the file generation performance, and automatically trigger a slow task warning signal when the generation time exceeds the preset threshold.

[0171] In one optional implementation, the platform is configured with an SFTP connection pool, with the maximum number of connections dynamically set according to 10% of the total number of funding parties. The system monitors the connection utilization rate in real time, and when the utilization rate reaches a preset threshold (such as 90%), an operation and maintenance alarm is triggered to notify relevant personnel to expand capacity or investigate abnormal connections in a timely manner.

[0172] Step S350: When the monitoring result indicates an anomaly, an alarm command is triggered and an anomaly log is recorded, resulting in an alarm log record. When the monitoring result indicates no corresponding data, an empty file is generated and stored in the specified end-of-day directory, resulting in an empty file stored in the specified end-of-day directory.

[0173] Among them, the alarm command is a trigger signal generated by the platform alarm module based on the abnormal status of the monitoring results, which is used to notify the system operation and maintenance or the upper-layer application task of abnormality; the abnormal log is the fault details recorded by the platform log module; the empty file is a placeholder file generated by the system when no valid data is matched in the database on the same day, which is used to maintain the file continuity of the end-of-day task.

[0174] As an example, when the monitoring result is "abnormal", the system immediately pushes alarm information to the message queue and generates a red high-priority task prompt on the operation and maintenance management terminal. The prompt content includes the task number, funding party code, abnormal time and error type. At the same time, the log module automatically generates alarm log records, recording the abnormal stack information, thread number and output directory path.

[0175] When the monitoring result is "No corresponding data", the system does not trigger an alarm. Instead, it creates a new blank CSV file using the file generation module, writes the field names in the first line of the file but has no data rows, and stores it in the specified end-of-day directory. For example, when the task number is "task_20251009_01" and there is no transaction data, the system generates the file "dayend_result_20251009.csv". The file content only contains the field rows "Customer Number | Transaction Date | Transaction Amount" to ensure the integrity of the file chain and avoid interruption of downstream tasks.

[0176] In one optional embodiment, the platform generates structured logs at key nodes such as template scanning, verification, parsing, cache updating, database querying, file generation, and SFTP uploading. The logs contain funder identifiers, task timestamps, operation types, result statuses, and error details. The logs support retrieval by funder number, date, task ID, and other dimensions, enabling full-chain tracking of a specific funder's file generation process for troubleshooting, security auditing, and compliance review.

[0177] During the process of retrieving data from the database or generating target daily end-of-day documents, target fields containing sensitive information are identified, and the content of the target fields is anonymized or encrypted for storage. Sensitive information includes, but is not limited to, mobile phone numbers, ID card numbers, and bank card numbers. The anonymization process uses a masking method to retain the first and last characters, and replaces the middle characters with "*", for example, "1381234" becomes "1*****4" after anonymization. For intermediate files or logs that need to be stored for a long time, the content of sensitive fields is encrypted and saved using an encryption algorithm (such as AES). The anonymization or encryption operations are driven by a unified configuration policy of the platform to ensure consistent protection of sensitive data in different funding scenarios.

[0178] In addition, the platform generates structured logs at key nodes such as template scanning, verification, parsing, cache updating, database querying, file generation, and SFTP uploading. The logs contain funder identifiers, task timestamps, operation types, result statuses, and error details. The logs can be retrieved by funder number, date, task ID, and other dimensions, enabling full-chain tracking of a specific funder's file generation process for troubleshooting, security auditing, and compliance review.

[0179] It is understood that the method in this embodiment can be executed by the server in the system of the above embodiments. The method in this embodiment is also applicable to the end-of-day data file processing system of the above embodiments, so it will not be described again here.

[0180] This application also provides a terminal device, exemplary of which includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to enable the terminal device to perform the functions of the various modules in the above-described method or system.

[0181] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0182] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving execution instructions.

[0183] This application also provides a computer-readable storage medium for storing the computer program used in the aforementioned terminal device. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0184] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can also be implemented in other ways. The system embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, as an alternative implementation, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0185] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0186] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0187] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for processing end-of-day data files, characterized in that, When applied to a client, the method includes: Select target fields from the collected preset field set and set the field order of the target fields to obtain a sequential field set; Insert a preset delimiter into the sequential field set to obtain a field string with delimiter; Write the delimited field string into the end-of-day document template structure to obtain the end-of-day document template; Upon successful authentication, an SFTP connection is established based on the SSH protocol to obtain a usable transfer channel. The end-of-day file template is uploaded to a virtual root directory path for storage via the available transmission channel, wherein the virtual root directory path is used for file isolation.

2. A method for processing end-of-day data files, characterized in that, When applied on the server side, the method includes: Obtain the end-of-day document template stored in the specified end-of-day directory, verify the end-of-day document template, and obtain a verified end-of-day document template; wherein, the end-of-day document template is obtained by the method described in claim 1; The validated end-of-day document template is parsed to extract field information, and corresponding data is extracted from the database based on the field information to generate the target end-of-day document. The target end-of-day file is stored in the specified end-of-day directory, which is a virtual root directory path used for file isolation.

3. The end-of-day data file processing method according to claim 2, characterized in that, The step of obtaining the end-of-day file template stored in the specified end-of-day directory, verifying the end-of-day file template, and obtaining a verified end-of-day file template includes: In response to the timed scan command, the end-of-day file template is read from the specified end-of-day directory to obtain the end-of-day file template to be inspected; The change status is determined based on the upload time of the end-of-day document template to be inspected, and the end-of-day document template to be verified is obtained. The format and field content of the end-of-day document template to be verified are validated to obtain the verification result; If the verification result is satisfactory, a valid end-of-day document template is generated.

4. The end-of-day data file processing method according to claim 2, characterized in that, The parsing and verification of the end-of-day document template is used to extract field information. Based on the field information, corresponding data is extracted from the database and the target end-of-day document is generated, including: Determine the file type of the verified end-of-day document template to obtain template file type information; Based on the template file type information, the validated end-of-day file template is parsed, and the set of field names is extracted to obtain the field information; The field information is loaded into the cache to obtain the cached field information; Retrieve data corresponding to the cached field information from the database and generate a database result set; A formatted dataset is obtained by sequentially combining the database result set and the cached field information. The formatted dataset is written into the target end-of-day file, resulting in the target end-of-day file stored in the specified end-of-day directory.

5. The end-of-day data file processing method according to claim 2, characterized in that, The method further includes: During the process of generating the target day-end file, the file generation status is monitored, and the monitoring results are obtained. When the monitoring result indicates an anomaly, an alarm command is triggered and an anomaly log is recorded, resulting in an alarm log record. When the monitoring result indicates that there is no corresponding data, an empty file is generated and stored in the specified end-of-day directory, resulting in an empty file stored in the specified end-of-day directory.

6. A daily end-of-day data file processing system, characterized in that, include: The client is used to select target fields from a preset set of collected fields and set the field order of the target fields to obtain a sequential field set; Insert a preset delimiter into the sequential field set to obtain a field string with delimiter; Write the delimited field string into the end-of-day document template structure to obtain the end-of-day document template; Upon successful authentication, an SFTP connection is established based on the SSH protocol to obtain a usable transfer channel; The end-of-day file template is uploaded to a virtual root directory path for storage via the available transmission channel, wherein the virtual root directory path is used for file isolation; The server-side is used to obtain the end-of-day file template stored in the specified end-of-day directory, verify the end-of-day file template to obtain a verified end-of-day file template; parse the verified end-of-day file template to extract field information, extract corresponding data from the database based on the field information and generate a target end-of-day file, and store the target end-of-day file in the specified end-of-day directory to obtain the target end-of-day file stored in the specified end-of-day directory. The specified end-of-day directory is a virtual root directory path, which is used for file isolation.

7. A terminal device, characterized in that, The terminal device includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the end-of-day data file processing method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed on a processor, implements the end-of-day data file processing method according to any one of claims 1-5.

Images