Distributed file generation method and device

Abstract

The present invention discloses a distributed file generation method and device, which can be used in the financial field or other technical fields. The method includes: obtaining task configuration information of each file generation task; when the current time reaches the preset determination time of the task to be executed, determining all the file generation tasks to be executed on that day according to the planned execution date of each file generation task, and generating the to-be-executed job information corresponding to each file generation task to be executed on that day in the to-be-executed file generation job queue list; determining a to-be-executed job information with an execution status of to-be-executed from the to-be-executed file generation job queue list; and performing file generation according to the file generation rule configuration information corresponding to the to-be-executed job information. The present invention provides a stable and efficient solution for generating batch files.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed data processing, and in particular to a distributed file generation method and device. Background Art

[0002] In financial and banking application systems, there are a large number of file interactions between different systems, such as data synchronization between different systems, submission of regulatory reporting documents to financial regulatory authorities, bank account reconciliation and other business scenarios. These business scenarios have large amounts of data, and the file export logic and generation cycle are relatively fixed.

[0003] The existing technology is generally implemented through batch files. A server is designated as a batch server, and a batch scheduler performs serial scheduling to execute different file generation jobs on a daily basis to generate batch data files that meet different needs. However, serial scheduling will result in low execution efficiency of file generation. Therefore, the existing technology lacks a more efficient batch file generation solution. Summary of the invention

[0004] In order to solve the technical problems in the above-mentioned background technology, the present invention proposes a distributed file generation method and device.

[0005] In order to achieve the above object, according to one aspect of the present invention, a distributed file generation method is provided, the method comprising:

[0006] Obtaining task configuration information of each file generation task, wherein the task configuration information includes: a planned execution date and a file generation rule configuration information identifier;

[0007] When the current time reaches the preset determination time of the pending tasks, all the file generation tasks to be executed on that day are determined according to the respective planned execution dates of each file generation task, and the pending job information corresponding to each file generation task to be executed on that day is generated in the pending file generation job queue list, wherein the pending job information includes: execution status, and when the pending job information is generated, the execution status of the pending job information is pending;

[0008] Determine a piece of pending job information with an execution status of pending from the pending file generation job queue list, and set the execution status of the pending job information to being executed;

[0009] Obtain the file generation rule configuration information identifier of the file generation task corresponding to the job information to be executed, then determine the file generation rule configuration information corresponding to the obtained file generation rule configuration information identifier, and then perform file generation according to the file generation rule configuration information, and set the execution status of the job information to be executed as successful when the file generation is completed.

[0010] Optionally, the to-be-executed job information further includes: a server identification field and a timestamp field;

[0011] While setting the execution status of the to-be-executed job information to being executed, the method further includes:

[0012] Update the server identification field of the job information to be executed to the local server identification;

[0013] Update the timestamp field of the pending job information to the current time.

[0014] Optionally, the distributed file generation method further includes:

[0015] When the file generation is completed, the timestamp field of the to-be-executed job information is updated to the file generation completion time.

[0016] Optionally, the information of the job to be executed also includes: execution time and priority;

[0017] The step of generating a job queue list from the to-be-executed file to determine a piece of to-be-executed job information with an execution status of to-be-executed specifically includes:

[0018] Searching for all pending job information whose execution time is the current time and whose execution status is pending from the pending file generation job queue list;

[0019] A piece of to-be-executed job information with the highest priority is determined from all the found to-be-executed job information.

[0020] Optionally, the task information corresponding to each file generation task to be executed on the day is generated in the to-be-executed file generation task queue list, specifically including:

[0021] For each file generation task to be executed on the day, first determine whether the task information corresponding to the file generation task already exists in the to-be-executed file generation task queue list;

[0022] If not, then task information corresponding to the file generation task is generated in the to-be-executed file generation job queue list.

[0023] Optionally, the method is applied to any batch file generation server in a distributed batch file generation server cluster.

[0024] Optionally, task configuration information of each file generation task is stored in the same job definition table; the job definition table is stored in a distributed database in the distributed batch file generation server cluster.

[0025] Optionally, the to-be-executed file generation job queue list is stored in a distributed database in the distributed batch file generation server cluster.

[0026] Optionally, the file generation rule configuration information includes: file attributes, file generation logic, and file content organization;

[0027] The file attributes include: file name, output path, encoding format, field separation method, separator and line break attribute information;

[0028] The file generation logic includes: specific SQL statements for file generation;

[0029] The file content organization includes: the field sequence and arrangement rules of the output file.

[0030] In order to achieve the above object, according to another aspect of the present invention, a distributed file generation device is provided, the device comprising:

[0031] The task configuration information acquisition module is used to acquire the task configuration information of each file generation task, wherein the task configuration information includes: a planned execution date and a file generation rule configuration information identifier;

[0032] The to-be-executed task determination module is used to determine all file generation tasks that need to be executed on that day according to the respective planned execution dates of each file generation task when the current time reaches the preset to-be-executed task determination time, and generate to-be-executed job information corresponding to each file generation task that needs to be executed on that day in the to-be-executed file generation job queue list, wherein the to-be-executed job information includes: execution status, and when the to-be-executed job information is generated, the execution status of the to-be-executed job information is to-be-executed;

[0033] A task selection module, used to determine a piece of to-be-executed job information with an execution status of to-be-executed from the to-be-executed file generation job queue list, and set the execution status of the to-be-executed job information to be in execution;

[0034] The task execution module is used to obtain the file generation rule configuration information identifier of the file generation task corresponding to the job information to be executed, then determine the file generation rule configuration information corresponding to the obtained file generation rule configuration information identifier, and then perform file generation according to the file generation rule configuration information. When the file generation is completed, the execution status of the job information to be executed is set to successful execution.

[0035] Optionally, the to-be-executed job information further includes: a server identification field and a timestamp field;

[0036] The task selection module is further configured to update the server identification field of the job information to be executed to the local server identification field and update the timestamp field of the job information to be executed to the current time while setting the execution state of the job information to be executed to being executed.

[0037] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned distributed file generation method when executing the computer program.

[0038] In order to achieve the above object, according to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed in a computer processor, the steps in the above distributed file generation method are implemented.

[0039] The beneficial effects of the present invention are as follows: the embodiment of the present invention configures corresponding file generation rule configuration information for each file generation task, and directly generates the corresponding file according to the file generation rule configuration information when a file needs to be generated, thereby improving the efficiency of file generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0041] Figure 1 is a first flow chart of a distributed file generation method according to an embodiment of the present invention;

[0042] Figure 2 is a second flow chart of the distributed file generation method according to an embodiment of the present invention;

[0043] Figure 3is a third flow chart of the distributed file generation method according to an embodiment of the present invention;

[0044] Figure 4 is a fourth flow chart of the distributed file generation method according to an embodiment of the present invention;

[0045] Figure 5 It is an application scenario diagram of the present invention;

[0046] Figure 6 It is a schematic diagram of each batch file generation server executing a file generation operation of the present invention;

[0047] Figure 7 is a structural block diagram of a distributed file generation device according to an embodiment of the present invention;

[0048] Figure 8 It is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0050] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0051] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatus.

[0052] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0053] The present invention aims at the defects and shortcomings of the existing batch file generation scheme, such as being unsuitable for distributed environment, low system availability, low file generation efficiency, and high development and maintenance cost, and provides an efficient batch file generation method suitable for distributed environment. The present invention comprises two parts. The first part is to design a scheduling method supporting distributed environment to solve the problems of low availability and low efficiency of traditional scheduling. The second part is to standardize the file generation process and realize a configurable file generation method, thereby reducing the development and maintenance cost of batch file generation.

[0054] Figure 5 It is an application scenario diagram of the present invention, such as Figure 5 As shown, the present invention utilizes a distributed batch file generation server cluster to perform batch file generation. Multiple batch file generation servers in the distributed batch file generation server cluster can perform batch file generation operations simultaneously. Figure 6 shown.

[0055] Figure 1 is a flow chart of a distributed file generation method according to an embodiment of the present invention, wherein the implementation subject is any batch file generation server in a distributed batch file generation server cluster, such as Figure 1 As shown, the distributed file generation method of the present invention includes steps S101 to S104.

[0056] Step S101, obtaining task configuration information of each file generation task, wherein the task configuration information includes: a planned execution date and a file generation rule configuration information identifier.

[0057] In one embodiment of the present invention, the task configuration information of each file generation task is stored in the same job definition table. The job definition table is stored in a distributed database in the distributed batch file generation server cluster, so each distributed batch file generation server faces the same job definition table.

[0058] In one embodiment of the present invention, the specific table structure of the job definition table can refer to Table 1. The table structure of the job definition table includes: job identification, job name, priority, planned execution date, planned execution time and file generation rule configuration information identification.

[0059] The job identification field is a unique identifier that distinguishes it from other file generation jobs. The job name field is used to describe the specific function of the job, which batch file is generated. The priority field can be divided into three levels: high, medium, and low by defining different priorities. When there are multiple jobs to be executed in the system, the priority field can be dynamically adjusted to give priority to jobs with high priorities. The planned execution date is defined by a regular expression, which can flexibly support the execution of batch file generation jobs by day, month, or specific day. The planned execution time field defines the specific time of job execution. The file generation rule configuration information identification field is used to associate the file generation rule configuration information, so that the corresponding batch files can be generated according to the specific file generation rule configuration information when executing the file generation job.

[0060]

[0061] Table 1

[0062] Step S102, when the current time reaches the preset time for determining the tasks to be executed, all the file generation tasks that need to be executed on that day are determined according to the planned execution date of each file generation task, and the corresponding to-be-executed job information for each file generation task that needs to be executed on that day is generated in the to-be-executed file generation job queue list, wherein the to-be-executed job information includes: execution status, and when the to-be-executed job information is generated, the execution status of the to-be-executed job information is to-be-executed.

[0063] In one embodiment of the present invention, the preset time for determining the task to be executed may be a fixed time every day, for example, 0:00 every day.

[0064] In one embodiment of the present invention, the to-be-executed file generation job queue list is stored in a distributed database in the distributed batch file generation server cluster, so each distributed batch file generation server faces the same to-be-executed file generation job queue list. In the present invention, the to-be-executed job information is a record in the to-be-executed file generation job queue list.

[0065] In one embodiment of the present invention, the specific table structure of the to-be-executed file generation job queue list can refer to the following Table 2. The to-be-executed file generation job queue list includes fields such as sequence number, job ID, priority, execution date, execution time, server ID, execution status, and timestamp.

[0066] The sequence number field is used to identify the initialized job sequence number, which is unique; the job identification field is used to associate the specific job definition in the job definition table; the priority field is used to specify the order of job execution; the execution date field is used to set the specific date of job execution, which is determined by the pending execution date field in the job definition table; the execution time field is used to set the specific time of job execution, which is determined by the pending execution time field in the job definition table; the server identification field is used to identify the specific distributed batch file generation server that executes the job; the execution status field is used to identify the current status of job execution; and the timestamp field is used to record the time when the job status is updated.

[0067]

[0068] Table 2

[0069] Step S103: determining a piece of to-be-executed job information with an execution status of to-be-executed from the to-be-executed file generation job queue list, and setting the execution status of the to-be-executed job information to being executed.

[0070] Step S104, obtain the file generation rule configuration information identifier of the file generation task corresponding to the job information to be executed, then determine the file generation rule configuration information corresponding to the obtained file generation rule configuration information identifier, and then perform file generation according to the file generation rule configuration information, and set the execution status of the job information to be executed as successful when the file generation is completed.

[0071] It can be seen that the present invention designs a scheduling method supporting a distributed environment to solve the low availability and low efficiency of traditional scheduling. In addition, the present invention standardizes the file generation process to achieve a configurable file generation method, thereby reducing the development and maintenance costs of batch file generation.

[0072] In one embodiment of the present invention, the to-be-executed job information further includes: a server identification field and a timestamp field.

[0073] like Figure 2 As shown, in one embodiment of the present invention, while executing the above step S103 to set the execution status of the to-be-executed job information to be in execution, the method of the present invention further includes steps S201 and S202.

[0074] Step S201: update the server identification field of the to-be-executed job information to the local server identification.

[0075] Step S202: Update the timestamp field of the to-be-executed job information to the current time.

[0076] In an embodiment of the present invention, a distributed batch file generation server uses a preemptive method to query pending job information and perform file generation business processing. The query conditions are that the execution date is equal to the current date, the execution time is less than or equal to the current time, and the execution status is pending, and the priority is sorted from high to low. After querying the pending job information that meets the requirements, the server identification, execution status, and timestamp fields in the pending job information are updated, and the server identification is updated to the identification of the distributed batch file generation server, the execution status is updated to executing, and the timestamp is updated to the current time.

[0077] In one embodiment of the present invention, when the file generation is completed, the above step S104 also updates the timestamp field of the to-be-executed job information to the file generation completion time.

[0078] In one embodiment of the present invention, after the file generation is completed in step S104, the distributed batch file generation server updates the execution status and timestamp fields of the pending job information, and the execution status is updated to execution success and the timestamp is updated to the current time. The distributed batch file generation server continues to query the pending job information in a preemptive manner to perform file generation business processing.

[0079] In one embodiment of the present invention, the information of the job to be executed further includes: execution time and priority.

[0080] like Figure 3 As shown, in one embodiment of the present invention, the above step S103 of generating a job queue list from the to-be-executed file to determine a to-be-executed job information with an execution status of to-be-executed specifically includes step S301 and step S302.

[0081] Step S301, searching for all pending job information whose execution time is the current time and whose execution status is pending from the pending file to generate the job queue list.

[0082] Step S302: determine a piece of to-be-executed job information with the highest priority from among all the to-be-executed job information found.

[0083] like Figure 4 As shown, in one embodiment of the present invention, the above step S102 generates task information corresponding to each file generation task to be executed on the day in the to-be-executed file generation task queue list, specifically including step S401 and step S402.

[0084] Step S401 , for each file generation task to be executed on the day, first determine whether task information corresponding to the file generation task already exists in the to-be-executed file generation job queue list.

[0085] Step S402: If not, then generating task information corresponding to the file generation task in the to-be-executed file generation job queue list.

[0086] In the embodiment of the present invention, each batch file generation server in the distributed batch file generation server cluster can perform the above step S102. Therefore, when generating the task information corresponding to each file generation task to be executed on the day in the pending file generation operation queue list, each batch file generation server first needs to determine whether the target file generation task to be executed on the day has been recorded in the pending file generation operation queue list, that is, other batch file generation servers in the cluster have generated the task information corresponding to the target file in the pending file generation operation queue list. That is, when the distributed batch file generation server of the present invention generates the task information corresponding to each file generation task to be executed on the day in the pending file generation operation queue list, it is also necessary to perform duplicate detection. If it already exists, the task information generation work is no longer performed.

[0087] In one embodiment of the present invention, the file generation rule configuration information of the present invention includes: file attributes, file generation logic and file content organization. The file attributes include: file name, output path, encoding format, field separation method, separator and line break attribute information; the file generation logic includes: specific SQL statements for file generation; the file content organization includes: field order and arrangement rules of the output file.

[0088] The present invention standardizes the batch file generation process, summarizes the rule elements required to generate a batch file through abstraction and induction of batch file generation elements, and customizes the rule configuration information for file generation. Referring to the batch file generation rule configuration information in Table 3 below, the rule configuration information is defined in XML format and specifically includes three parts: file attributes, file generation logic, and file content organization. All elements for generating batch files are defined in the file generation rule configuration information, and generating different batch files only requires configuring the file generation rule configuration information according to specific requirements.

[0089]

[0090] Table 3

[0091] In the batch file generation job, the file generation rule configuration information is read according to the file rule configuration information identifier rule_id, and the file generation rule configuration information is parsed according to the XML syntax. The file attributes, file generation rules, and file content organization definitions in the file generation rule configuration information are loaded into the distributed batch file generation server. The distributed batch file generation server queries the original data that meets the requirements according to the file generation rules, rearranges them according to the field order of the file content organization, and generates specific files according to the file name, encoding format, and other requirements in the file attributes. In this way, each new file generation job does not need to be repeated. Development work only needs to be configured according to the needs. Operation and maintenance does not need to maintain a large number of file generation programs, and only needs to focus on the corresponding file generation rule configuration information.

[0092] When adding a new file generation task, you only need to configure the file generation rule configuration information according to the file format and generation logic, define the file generation job according to the file generation cycle and priority, and store it in the database table. The daily file generation job reads the file generation job definition from the definition table, instantiates the corresponding execution job according to the configured parameters, and stores it in the job execution table. The file generation job reads the file generation job to be executed that meets the conditions from the job execution table, and completes the file generation according to the configured file generation rule engine. During the file generation job process, the distributed batch file generation server can be dynamically adjusted to multiple servers as needed. On the one hand, the concurrent execution of multiple servers improves the efficiency of file generation. On the other hand, the availability of the file generation job is very high, which effectively avoids the problem of file generation failure caused by a single server failure.

[0093] It can be seen from the above embodiments that the present invention provides an efficient batch file generation method applicable to a distributed environment to address the defects and shortcomings of the existing batch file generation solutions, such as being not applicable to a distributed environment, having low system availability, having low file generation efficiency, and having high development and maintenance costs, and achieves at least the following beneficial effects:

[0094] 1. Use preemptive mode to schedule the execution of file generation jobs, which is suitable for distributed environments and improves system availability.

[0095] 2. Generating batch files in a distributed manner can fully improve the concurrency of job scheduling and effectively improve the efficiency of batch file generation.

[0096] 3. The file generation process is abstracted and standardized. To add batch file generation, you only need to add file rule configuration information and define new batch file generation jobs. There is no need to develop new job implementation code for this purpose. The standardized process effectively reduces development costs.

[0097] 4. All file generation jobs are implemented through standardized configuration. Maintenance personnel only need to focus on the job definition table and file rule configuration information, and the system maintenance cost is low.

[0098] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0099] Based on the same inventive concept, an embodiment of the present invention also provides a distributed file generation device, which can be used to implement the distributed file generation method described in the above embodiment, as described in the following embodiment. Since the principle of solving the problem by the distributed file generation device is similar to that of the distributed file generation method, the embodiment of the distributed file generation device can refer to the embodiment of the distributed file generation method, and the repeated parts are not repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiment is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0100] Figure 7 is a structural block diagram of a distributed file generation device according to an embodiment of the present invention. Figure 7 As shown, the distributed file generation device according to the embodiment of the present invention includes:

[0101] The task configuration information acquisition module 1 is used to acquire the task configuration information of each file generation task, wherein the task configuration information includes: a planned execution date and a file generation rule configuration information identifier;

[0102] The pending task determination module 2 is used to determine all the file generation tasks that need to be executed on the day according to the respective planned execution dates of each file generation task when the current time reaches the preset pending task determination time, and generate the pending job information corresponding to each file generation task that needs to be executed on the day in the pending file generation job queue list, wherein the pending job information includes: execution status, when the pending job information is generated, the execution status of the pending job information is pending;

[0103] The task selection module 3 is used to determine a piece of to-be-executed job information with an execution status of to-be-executed from the to-be-executed file generation job queue list, and set the execution status of the to-be-executed job information to be in execution;

[0104] The task execution module 4 is used to obtain the file generation rule configuration information identifier of the file generation task corresponding to the job information to be executed, and then determine the file generation rule configuration information corresponding to the obtained file generation rule configuration information identifier, and then perform file generation according to the file generation rule configuration information, and when the file generation is completed, set the execution status of the job information to be executed as successful.

[0105] In one embodiment of the present invention, the job information to be executed further includes: a server identification field and a timestamp field. The task selection module 3 is further configured to update the server identification field of the job information to be executed to a local server identification field, and update the timestamp field of the job information to be executed to the current time while setting the execution state of the job information to be executed to being executed.

[0106] In one embodiment of the present invention, the distributed file generation device of the present invention is applied to any batch file generation server in a distributed batch file generation server cluster.

[0107] In one embodiment of the present invention, the task execution module 4 is further configured to update the timestamp field of the to-be-executed job information to the file generation completion time when the file generation is completed.

[0108] In one embodiment of the present invention, the information of the job to be executed further includes: execution time and priority.

[0109] The task selection module 3 is specifically used to search for all pending job information with the current execution time and the pending execution status from the pending file generation job queue list; and determine the highest priority pending job information from all the found pending job information.

[0110] In one embodiment of the present invention, the to-be-executed task determination module 2 generates task information corresponding to each file generation task to be executed on the day in the to-be-executed file generation task queue list, specifically including:

[0111] The to-be-executed task determination module 2 first determines, for each file generation task that needs to be executed on the day, whether task information corresponding to the file generation task already exists in the to-be-executed file generation job queue list; if not, then generates task information corresponding to the file generation task in the to-be-executed file generation job queue list.

[0112] In order to achieve the above object, according to another aspect of the present application, a computer device is also provided. Figure 8As shown, the computer device includes a memory, a processor, a communication interface and a communication bus. A computer program that can be executed on the processor is stored in the memory. When the processor executes the computer program, the steps in the above-mentioned embodiment method are implemented.

[0113] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0114] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and units, such as the corresponding program units in the above method embodiments of the present invention. The processor executes various functional applications of the processor and works data processing by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiments.

[0115] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0116] The one or more units are stored in the memory, and when executed by the processor, the method in the above embodiment is performed.

[0117] The specific details of the above-mentioned computer device can be understood by referring to the corresponding related descriptions and effects in the above-mentioned embodiments, and will not be repeated here.

[0118] In order to achieve the above purpose, according to another aspect of the present application, a computer-readable storage medium is also provided, wherein the computer-readable storage medium stores a computer program, and the computer program implements the steps in the above-mentioned distributed file generation method when executed in a computer processor. Those skilled in the art can understand that the implementation of all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program can include the processes of the embodiments of the above-mentioned methods when executed. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0119] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A distributed file generation method, characterized in that: The method is applied to any batch file generation server in a distributed batch file generation server cluster, and the method comprises: Obtaining task configuration information of each file generation task, wherein the task configuration information includes: a planned execution date and a file generation rule configuration information identifier; the task configuration information of each file generation task is stored in the same job definition table; the job definition table is stored in a distributed database in the distributed batch file generation server cluster; When the current time reaches the preset determination time of the pending tasks, all the file generation tasks to be executed on that day are determined according to the respective planned execution dates of each file generation task, and the pending job information corresponding to each file generation task to be executed on that day is generated in the pending file generation job queue list, wherein the pending job information includes: an execution status, and when the pending job information is generated, the execution status of the pending job information is pending; the pending file generation job queue list is stored in a distributed database in the distributed batch file generation server cluster; Determine a piece of pending job information with an execution status of pending from the pending file generation job queue list, and set the execution status of the pending job information to being executed; Obtain the file generation rule configuration information identifier of the file generation task corresponding to the job information to be executed, then determine the file generation rule configuration information corresponding to the obtained file generation rule configuration information identifier, and then perform file generation according to the file generation rule configuration information, and set the execution status of the job information to be executed as successful when the file generation is completed.

2. The distributed file generation method according to claim 1, characterized in that: The to-be-executed job information also includes: a server identification field and a timestamp field; While setting the execution status of the to-be-executed job information to being executed, the method further includes: Update the server identification field of the job information to be executed to the local server identification; Update the timestamp field of the pending job information to the current time.

3. The distributed file generation method according to claim 2, characterized in that: Also includes: When the file generation is completed, the timestamp field of the to-be-executed job information is updated to the file generation completion time.

4. The distributed file generation method according to claim 1, characterized in that: The information of the job to be executed also includes: execution time and priority; The step of generating a job queue list from the to-be-executed file to determine a piece of to-be-executed job information with an execution status of to-be-executed specifically includes: Searching for all pending job information whose execution time is the current time and whose execution status is pending from the pending file generation job queue list; A piece of to-be-executed job information with the highest priority is determined from all the found to-be-executed job information.

5. The distributed file generation method according to claim 1, characterized in that: The task information corresponding to each file generation task to be executed on the day is generated in the to-be-executed file generation task queue list, specifically including: For each file generation task to be executed on the day, first determine whether the task information corresponding to the file generation task already exists in the to-be-executed file generation task queue list; If not, then task information corresponding to the file generation task is generated in the to-be-executed file generation job queue list.

6. The distributed file generation method according to claim 1, characterized in that: The file generation rule configuration information includes: file attributes, file generation logic and file content organization; The file attributes include: file name, output path, encoding format, field separation method, separator and line break attribute information; The file generation logic includes: specific SQL statements for file generation; The file content organization includes: the field sequence and arrangement rules of the output file.

7. A distributed file generation device, characterized in that: The device is applied to any batch file generation server in a distributed batch file generation server cluster, and the device includes: The task configuration information acquisition module is used to acquire the task configuration information of each file generation task, wherein the task configuration information includes: a planned execution date and a file generation rule configuration information identifier; the task configuration information of each file generation task is stored in the same job definition table; the job definition table is stored in a distributed database in the distributed batch file generation server cluster; The to-be-executed task determination module is used to determine all file generation tasks that need to be executed on that day according to the respective planned execution dates of each file generation task when the current time reaches the preset to-be-executed task determination time, and generate to-be-executed job information corresponding to each file generation task that needs to be executed on that day in the to-be-executed file generation job queue list, wherein the to-be-executed job information includes: execution status, when the to-be-executed job information is generated, the execution status of the to-be-executed job information is to-be-executed; the to-be-executed file generation job queue list is stored in the distributed database in the distributed batch file generation server cluster; A task selection module, used to determine a piece of to-be-executed job information with an execution status of to-be-executed from the to-be-executed file generation job queue list, and set the execution status of the to-be-executed job information to be in execution; The task execution module is used to obtain the file generation rule configuration information identifier of the file generation task corresponding to the job information to be executed, then determine the file generation rule configuration information corresponding to the obtained file generation rule configuration information identifier, and then perform file generation according to the file generation rule configuration information. When the file generation is completed, the execution status of the job information to be executed is set to successful execution.

8. The distributed file generation device according to claim 7, characterized in that: The to-be-executed job information also includes: a server identification field and a timestamp field; The task selection module is further configured to update the server identification field of the job information to be executed to the local server identification field and update the timestamp field of the job information to be executed to the current time while setting the execution state of the job information to be executed to being executed.

9. The distributed file generation device according to claim 7, characterized in that: The information of the job to be executed also includes: execution time and priority; The task selection module determines a piece of information of a pending job with an execution status of pending from the pending file generation job queue list, specifically including: The task selection module searches for all pending job information whose execution time is the current time and whose execution status is pending from the pending file generation job queue list; The task selection module determines a piece of to-be-executed job information with the highest priority from all the to-be-executed job information found.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed in a computer processor, the computer program implements the method according to any one of claims 1 to 6.

Images