Batch file processing method and device

Abstract

The present invention discloses a batch file processing method and device, which can be used in the financial field, wherein the method comprises: obtaining a file processing basic information table and a file processing record table; scanning the file processing record table, if the number of files in the file processing record table that are in the processing in progress state is less than a preset number, then for multiple files in the file processing state that are in the arrival state, the remaining processing time of each file is determined according to the processing completion time corresponding to each file and the file arrival time, the remaining available time of each file is determined according to the remaining processing time and the maximum processing time, and batch file processing is performed according to the remaining available time of each file. The present invention can perform batch file processing and avoid the problems of memory overflow and uneven distribution of batch process work.

Description

Technical Field

[0001] The present invention relates to the financial field, and in particular to a batch file processing method and device. It should be noted that the batch file processing method and device of the present invention can be used in the financial field, and can also be used in any field other than the financial field, and the application field of the batch file processing method and device of the present invention is not limited. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section.

[0003] The SME online financial service platform has at least 20 different types of files, which need to be received from the DCDS system as batch files to be processed. The system uses a daemon process to call up the service of each type of file in the background for batch processing.

[0004] However, the time and size of DCDS file delivery are uncertain. The existing file processing method is to scan the FTP directory files delivered by DCDS files every three minutes, and start processing if there are files. However, this method has the following problems:

[0005] 1. When there are too many files sent at the same time, several threads for file processing will be called up at the same time. There are many threads for file processing in memory, and memory overflow occurs many times.

[0006] 2. The file processing size and processing time are not arranged in a coordinated manner, and the server batch processing time window is not reasonably utilized. There are often situations where the processing peak and processing idle time are too busy or too idle, and the batch work is unbalanced.

[0007] Therefore, there is an urgent need for a batch file processing solution that can overcome the above problems. Summary of the invention

[0008] An embodiment of the present invention provides a batch file processing method for performing batch file processing to avoid problems of memory overflow and unbalanced distribution of batch process work. The method includes:

[0009] Obtain a file processing basic information table and a file processing record table, wherein the file processing basic information table includes: required processing completion time, maximum processing time, and the file processing record table includes: file arrival time, file processing status, and the file processing status includes: processing in progress status and arrival status;

[0010] Scan the file processing record table. If the number of files in the file processing record table whose processing status is in progress is less than a preset number, then for multiple files whose file processing status is arrival status, determine the remaining processing time of each file according to the processing completion time corresponding to each file and the file arrival time, determine the remaining available time of each file according to the remaining processing time and the maximum processing time, and perform batch file processing according to the remaining available time of each file.

[0011] An embodiment of the present invention provides a batch file processing device for performing batch file processing to avoid problems of memory overflow and unbalanced distribution of batch process work. The device includes:

[0012] An information table acquisition module is used to obtain a file processing basic information table and a file processing record table, wherein the file processing basic information table includes: required processing completion time, maximum processing time, and the file processing record table includes: file arrival time, file processing status, and the file processing status includes: processing in progress status and arrival status;

[0013] The batch file processing module is used to scan the file processing record table. If the number of files in the file processing record table whose processing status is in the processing in progress state is less than a preset number, then for multiple files whose file processing status is the arrival state, the remaining processing time of each file is determined according to the processing completion time corresponding to each file and the file arrival time, and the remaining available time of each file is determined according to the remaining processing time and the maximum processing time, and batch file processing is performed according to the remaining available time of each file.

[0014] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned batch file processing method when executing the computer program.

[0015] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned batch file processing method is implemented.

[0016] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the batch file processing method is implemented.

[0017] Compared with the technical solution in the prior art for performing batch file processing by scanning FTP directory files sent by DCDS files every three minutes, the embodiment of the present invention obtains a file processing basic information table and a file processing record table, the file processing basic information table includes: required processing completion time, maximum processing time, the file processing record table includes: file arrival time, file processing status, the file processing status includes: processing in progress status and arrival status; scan the file processing record table, if the number of files in the file processing record table with the processing in progress status is less than a preset number, then for multiple files with the file processing status as the arrival status, the remaining processing time of each file is determined according to the required processing completion time and the file arrival time corresponding to each file, the remaining available time of each file is determined according to the remaining processing time and the maximum processing time, and batch file processing is performed according to the remaining available time of each file. The embodiment of the present invention analyzes the file processing status through a file processing basic information table and a file processing record table. If the number of files in the file processing record table that are in the processing in progress state is less than a preset number, then for multiple files in the file processing record state that are in the arrival state, the remaining processing time of each file is determined according to the processing completion time corresponding to each file and the file arrival time, and then the remaining available time of each file is determined according to the remaining processing time and the maximum processing time. Batch processing is performed according to the remaining available time to avoid processing conflicts caused by the simultaneous arrival of files and insufficient CPU and memory problems. In addition, resources can be evenly allocated to ensure stable system processing and avoid overly busy or overly idle situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1 Schematic diagram of a batch file processing method in an embodiment of the present invention;

[0020] Figure 2-3 It is a flow chart of batch file processing in a specific embodiment of the present invention;

[0021] Figure 4 It is a structural diagram of a batch file processing device in an embodiment of the present invention;

[0022] Figure 5 It is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0024] First, the technical terms in the embodiments of the present invention are introduced:

[0025] DCDS: Data download platform. This system realizes batch collection of business system data (avoiding repeated data collection), downloads data to the download platform, and performs text-level data processing on the data text: simple verification, encryption and decryption, temporary storage, compression, transmission, and storage. It is physically deployed in the head office node and four regional center node servers. It is a unified batch text data transmission channel between the data source system and the data application target system.

[0026] Source system: refers to the application system that generates and downloads data files in batches.

[0027] Data file: refers to a text file directly formed by the data in the data table in the source system.

[0028] Data application system: refers to the application system that receives and uses data files.

[0029] SME: Bank of China’s online financial service platform for small and medium-sized enterprises.

[0030] In order to process batch files and avoid the problems of memory overflow and uneven distribution of batch process work, an embodiment of the present invention provides a batch file processing method, such as Figure 1 As shown, the method may include:

[0031] Step 101, obtaining a file processing basic information table and a file processing record table, wherein the file processing basic information table includes: required processing completion time, maximum processing time, and the file processing record table includes: file arrival time, file processing status, and the file processing status includes: processing in progress status and arrival status;

[0032] Step 102, scanning the file processing record table. If the number of files in the file processing record table that are in the processing in progress state is less than a preset number, then for multiple files in the file processing record table that are in the arrival state, determine the remaining processing time of each file according to the processing completion time corresponding to each file and the file arrival time, determine the remaining available time of each file according to the remaining processing time and the maximum processing time, and perform batch file processing according to the remaining available time of each file.

[0033] Depend on Figure 1As shown, it can be known that compared with the technical solution in the prior art of performing batch file processing by scanning the FTP directory files sent by the DCDS file every three minutes, the embodiment of the present invention obtains a file processing basic information table and a file processing record table, the file processing basic information table includes: the time required for processing to be completed, the maximum processing time, the file processing record table includes: the file arrival time, the file processing status, the file processing status includes: the processing in progress status and the arrival status; scan the file processing record table, if the number of files in the file processing record table with the processing in progress status is less than the preset number, then for multiple files with the file processing status as the arrival status, the remaining processing time of each file is determined according to the time required for processing to be completed and the file arrival time corresponding to each file, the remaining available time of each file is determined according to the remaining processing time and the maximum processing time, and batch file processing is performed according to the remaining available time of each file. The embodiment of the present invention analyzes the file processing status through a file processing basic information table and a file processing record table. If the number of files in the file processing record table that are in the processing in progress state is less than a preset number, then for multiple files in the file processing record state that are in the arrival state, the remaining processing time of each file is determined according to the processing completion time corresponding to each file and the file arrival time, and then the remaining available time of each file is determined according to the remaining processing time and the maximum processing time. Batch processing is performed according to the remaining available time to avoid processing conflicts caused by the simultaneous arrival of files and insufficient CPU and memory problems. In addition, resources can be evenly allocated to ensure stable system processing and avoid overly busy or overly idle situations.

[0034] In step 101, a file processing basic information table and a file processing record table are obtained. The file processing basic information table includes: required processing completion time, maximum processing time, and the file processing record table includes: file arrival time, file processing status, and the file processing status includes: processing in progress status and arrival status.

[0035] In step 102, the file processing record table is scanned. If the number of files in the file processing record table that are in the processing in progress state is less than a preset number, then for multiple files in the file processing state that are in the arrival state, the remaining processing time of each file is determined according to the processing completion time corresponding to each file and the file arrival time, and the remaining available time of each file is determined according to the remaining processing time and the maximum processing time, and batch file processing is performed according to the remaining available time of each file.

[0036] In one embodiment, batch file processing is performed according to the remaining available time of each file, including:

[0037] Sort the remaining available time of multiple files;

[0038] If the remaining available time of multiple files is not equal, the file with the shortest remaining available time will be processed first.

[0039] In one embodiment, the file processing basic information table further includes: a processing priority;

[0040] The batch file processing is performed according to the remaining available time of each file, and further includes: if the remaining available time of multiple files is equal, the batch file processing is performed according to the processing priority of each file, wherein the file with the highest priority is processed first.

[0041] In one embodiment, the file processing basic information table further includes: average processing time;

[0042] Batch file processing is performed according to the remaining available time of each file, and also includes: if the remaining available time and processing priority of multiple files are equal, batch file processing is performed according to the average processing time of each file, wherein the file with the shortest average processing time is processed first.

[0043] A specific embodiment is given below to illustrate the specific application of batch file processing in the embodiment of the present invention. Figure 2-3 As shown, it is necessary to maintain the file processing basic information table, including the fields file type, processing priority, required processing completion time, average processing time, minimum processing time, maximum processing time, and service interface. The file processing basic information table can be used for the judgment input of file priority processing later. Among them, the average processing time and the maximum processing time need to be updated later, and the sequence number, file name, required processing completion time, and service interface are business configuration data. For example: there are 6 files that need to be processed, and the corresponding file name, processing priority, required completion time, average processing time, maximum processing time, and corresponding service scheduling interface are shown in Table 1.

[0044] Table 1

[0045]

[0046]

[0047] Scan the FTP directory (file processing basic information table) every three minutes, update the file processing record table (file arrival time, file processing start time, processing completion time, file processing status (arrival status, processing in progress status, completion status, processing abnormal status, processing data volume), record the arrival time, and the processing status is the arrival status.

[0048] Run a background daemon service process - file processing scheduler: scan the file record table once a minute to find out whether the number of files being processed is greater than or equal to 3. If it is greater than or equal to 3, no processing is performed. If the number of files being processed is less than 3, the number of files that can be processed again = 3 - the number of files being processed. Combine the file processing basic information table and the file processing record table, sort them from small to large according to (remaining available time - longest processing time), priority from high to low and average processing time from small to large, and obtain the files sorted first.

[0049] In Table 2, three files are already being processed, and there are no extra processes to process the processes with sequence numbers 4, 5, and 6.

[0050] Table 2

[0051]

[0052]

[0053] When 3 is processed at 21:00, a resource is released. Then, according to the priority and time requirements, it can be determined which file 4, 5, and 6 should be scheduled for processing first. Sort by (remaining available time - longest processing time) from small to large, priority from high to low, and average processing time from small to large, and obtain the file in the first order.

[0054] Since K01, NJ6, and PCL all need to be processed within 12 hours, and 1 hour has passed since the file was received, the remaining processing time is 11 hours. According to the (remaining available time - longest processing time), priority, and average processing time, PCL files are processed first. That is, the one with the shortest remaining available time is selected for processing first. When the remaining available time is equal, the one with the highest priority is selected for processing first. When they are the same, the one with the shortest average processing time is selected for processing, as shown in Table 3.

[0055] Table 3

[0056] File name Remaining available time - maximum processing time Priority Average processing time PCL 11-2＝9 2 1 NJ6 11-2＝9 2 2 K01 11-1.5＝9.5 3 1

[0057] After the file processing is completed, the file processing record table is updated with the completion time of the currently processed file, the processing status, the time required for file processing is calculated, (the time required for file processing is calculated + the previous average time) / 2, the longest processing time and the minimum processing time are compared, and the average processing time, minimum processing time and maximum processing time of the file processing basic information table are updated.

[0058] The embodiments of the present invention take into account that the existing methods of immediate file processing have the risk of high concurrency, high pressure on memory and CPU processing, multiple memory overflows in production, and uneven distribution of tasks in batch processing processes, with peak and idle situations. Therefore, a comprehensive judgment is made based on the number of files being processed, file processing priority, the time when files must be processed, and file processing duration to obtain the files that need to be processed most for processing. There can be a maximum of 3 file processing queues, which will be scheduled and supplemented when resources are released. The principle of scheduling selection is: sort by (remaining available time - file processing duration) from small to large, file priority from high to low, and file processing duration from small to large, and select the number of files that can be processed.

[0059] In specific implementation, scan files every three minutes. When there is a new file, move it to the directory of files to be processed, and maintain the information in the file processing record table. The file status is file arrival. Scan the file processing record table every minute to check the number of files being processed and whether the number of files being processed is greater than 3. If it is greater than 3, no processing is performed. If the number of files being processed is less than 3, the number of files that can be processed again = 3-the number of files being processed. Combine the file processing basic information table and the file processing record table, sort them according to the remaining available time-the longest processing time from small to large, the priority from high to low and the average processing time from small to large, and obtain the files sorted in the front. According to the service interface of the file in the file processing basic information table, call up the file processing service. Process the file and record the start time of the file processing. When completed, record the completion time of the file. When there is an exception, record the status as processing exception, and record the key exception log. Calculate the file processing time, the number of processed data, the longest processing time, the minimum processing time, and the average processing time, and update the file processing record table and the file processing basic information table. The average duration is calculated through historical data, and then combined with the minimum completion time, maximum completion time, the time point to be processed, priority and other information, the priority scheduling of file services is calculated. Up to three files can be run at the same time to avoid memory overflow due to excessive file processing. The remaining available time - the longest processing time, the minimum average processing time, and the priority are sorted to select the files that need to be processed most and can be processed quickly to reduce queue backlogs and release resources. Avoid processing conflicts when files arrive at the same time, CPU and memory shortage problems, and evenly distribute resources to ensure stable system processing and avoid being too busy or too idle.

[0060] Based on the same inventive concept, the present invention also provides a batch file processing device, as described in the following embodiments. Since the principles of solving these problems are similar to the batch file processing method, the implementation of the batch file processing device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0061] Figure 4FIG. 1 is a structural diagram of a batch file processing device according to an embodiment of the present invention. Figure 4 As shown, the batch file processing device includes:

[0062] The information table acquisition module 401 is used to obtain a file processing basic information table and a file processing record table, wherein the file processing basic information table includes: required processing completion time, maximum processing time, and the file processing record table includes: file arrival time, file processing status, and the file processing status includes: processing in progress status and arrival status;

[0063] The batch file processing module 402 is used to scan the file processing record table. If the number of files in the file processing record table whose processing status is in progress is less than a preset number, then for multiple files whose file processing status is arrival status, the remaining processing time of each file is determined according to the processing completion time corresponding to each file and the file arrival time, and the remaining available time of each file is determined according to the remaining processing time and the maximum processing time, and batch file processing is performed according to the remaining available time of each file.

[0064] In one embodiment, the batch file processing module 402 is further used to:

[0065] Sort the remaining available time of multiple files;

[0066] If the remaining available time of multiple files is not equal, the file with the shortest remaining available time will be processed first.

[0067] In one embodiment, the file processing basic information table further includes: processing priority;

[0068] The batch file processing module 402 is further used to: if the remaining available time of multiple files is equal, perform batch file processing according to the processing priority of each file, wherein the file with the highest priority is processed first.

[0069] In one embodiment, the file processing basic information table further includes: average processing time;

[0070] The batch file processing module 402 is further used to: if the remaining available time and processing priority of multiple files are equal, perform batch file processing according to the average processing time of each file, wherein the file with the shortest average processing time is processed first.

[0071] Based on the above invention concept, Figure 5As shown, an embodiment of the present invention further provides a computer device 500, including a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520, wherein the processor 520 implements the above-mentioned batch file processing method when executing the computer program 530.

[0072] Based on the aforementioned inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the aforementioned batch file processing method is implemented.

[0073] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the batch file processing method is implemented.

[0074] Compared with the technical solution in the prior art for performing batch file processing by scanning FTP directory files sent by DCDS files every three minutes, the embodiment of the present invention obtains a file processing basic information table and a file processing record table, the file processing basic information table includes: required processing completion time, maximum processing time, the file processing record table includes: file arrival time, file processing status, the file processing status includes: processing in progress status and arrival status; scan the file processing record table, if the number of files in the file processing record table with the processing in progress status is less than a preset number, then for multiple files with the file processing status as the arrival status, the remaining processing time of each file is determined according to the required processing completion time and the file arrival time corresponding to each file, the remaining available time of each file is determined according to the remaining processing time and the maximum processing time, and batch file processing is performed according to the remaining available time of each file. The embodiment of the present invention analyzes the file processing status through a file processing basic information table and a file processing record table. If the number of files in the file processing record table that are in the processing in progress state is less than a preset number, then for multiple files in the file processing record state that are in the arrival state, the remaining processing time of each file is determined according to the processing completion time corresponding to each file and the file arrival time, and then the remaining available time of each file is determined according to the remaining processing time and the maximum processing time. Batch processing is performed according to the remaining available time to avoid processing conflicts caused by the simultaneous arrival of files and insufficient CPU and memory problems. In addition, resources can be evenly allocated to ensure stable system processing and avoid overly busy or overly idle situations.

[0075] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0077] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0078] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0079] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A batch file processing method, characterized in that: include: Obtain a file processing basic information table and a file processing record table, wherein the file processing basic information table includes: required processing completion time, maximum processing time, and the file processing record table includes: file arrival time, file processing status, and the file processing status includes: processing in progress status and arrival status; Scan the file processing record table, if the number of files in the file processing record table that are in the processing-in-progress state is less than a preset number, then for multiple files in the file processing record table that are in the arrival state, determine the remaining processing time of each file according to the processing completion time corresponding to each file and the file arrival time, determine the remaining available time of each file according to the remaining processing time and the maximum processing time, and perform batch file processing according to the remaining available time of each file; The file processing basic information table also includes: processing priority; The method further comprises: if the remaining available time of multiple files is equal, batch processing is performed according to the processing priority of each file, wherein the file with the highest priority is processed first; The file processing basic information table also includes: average processing time; Batch file processing is performed according to the remaining available time of each file, and also includes: if the remaining available time and processing priority of multiple files are equal, batch file processing is performed according to the average processing time of each file, wherein the file with the shortest average processing time is processed first.

2. The batch file processing method according to claim 1, characterized in that: Batch file processing based on the remaining available time of each file, including: Sort the remaining available time of multiple files; If the remaining available time of multiple files is not equal, the file with the shortest remaining available time will be processed first.

3. A batch file processing device, characterized in that: include: An information table acquisition module is used to obtain a file processing basic information table and a file processing record table, wherein the file processing basic information table includes: required processing completion time, maximum processing time, and the file processing record table includes: file arrival time, file processing status, and the file processing status includes: processing in progress status and arrival status; A batch file processing module is used to scan the file processing record table, if the number of files in the file processing record table that are in the processing in progress state is less than a preset number, then for multiple files in the file processing record table that are in the arrival state, the remaining processing time of each file is determined according to the processing completion time corresponding to each file and the file arrival time, the remaining available time of each file is determined according to the remaining processing time and the maximum processing time, and batch file processing is performed according to the remaining available time of each file; The file processing basic information table also includes: processing priority; The batch file processing module is further used to: if the remaining available time of multiple files is equal, batch file processing is performed according to the processing priority of each file, wherein the file with the highest priority is processed first; The file processing basic information table also includes: average processing time; The batch file processing module is further used for: if the remaining available time and processing priority of multiple files are equal, batch file processing is performed according to the average processing time of each file, wherein the file with the shortest average processing time is processed first.

4. The batch file processing device according to claim 3, characterized in that: The batch file processing module is further used to: Sort the remaining available time of multiple files; If the remaining available time of multiple files is not equal, the file with the shortest remaining available time will be processed first.

5. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 2 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

7. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images