Batch processing scheduling method and system based on scheduling file extension conversion

Abstract

The present application relates to a kind of batch processing scheduling method and system based on scheduling file extension conversion, respectively configure scene label and execution label after obtaining identification node from source scheduling file and corresponding execution extension and conversion, the automatic analysis and extension of job scheduling are realized by the naming rule and procedure of scheduling, and generate control-M xml scheduling file or batch processing scheduling excel file, and can be according to naming rule, dynamically extend some same type jobs to reduce, so as to reach the extension of sub-database table or other jobs needing extension, can be configured first, then dynamically extended according to actual needs, save time, reduce error.

Description

Technical Field

[0001] This invention relates to the field of automation programs and data processing technology, and in particular to a batch processing scheduling method and system based on scheduling file expansion conversion. Background Technology

[0002] Batch jobs are job scripts that process a large number of objects or functions at once. By applying batch jobs, specific large-scale repetitive tasks can be executed efficiently, thereby significantly improving job execution efficiency and saving manpower and computing resources.

[0003] Ideally, batch processing jobs should execute without manual intervention, requiring only data reading and processing according to preset configurations. However, in real-world scenarios, a large number of different batch processing jobs need to work together in a specific order, making batch processing job scheduling a necessary guarantee for their execution.

[0004] To facilitate more convenient control over batch job scheduling, existing technologies often employ batch scheduling tools to provide job management, monitoring, and scheduling functions. Common batch scheduling tools include visual scheduling tools like Control-M and scheduling tools configured via Excel.

[0005] However, configuring and generating scheduling XML files using the visual scheduling tool Control-M requires repetitive configuration multiple times for the same type of scheduling job, which is time-consuming, error-prone, and cannot dynamically extend to scheduling jobs with sharded databases and tables. Scheduling configured via Excel also suffers from cumbersome configuration, is time-consuming, error-prone, and cannot dynamically extend to scheduling jobs with sharded databases and tables. Furthermore, different batch processing scheduling tools lack universality, and the scheduling file formats they use are incompatible, making it difficult to meet the needs of more flexible batch processing scheduling tasks.

[0006] It is evident that existing batch processing scheduling methods mostly rely on scheduling tools to configure scheduling jobs. For multiple jobs with the same function, it is necessary to add or delete job scheduling and configuration one by one, which is time-consuming, labor-intensive, and prone to errors. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a batch processing scheduling method and system based on scheduling file expansion and conversion. Through scheduling naming rules and programs, it achieves automatic analysis and expansion of job scheduling, generating control-M XML scheduling files or batch processing scheduling Excel files. Furthermore, it can dynamically expand and reduce similar jobs according to naming rules to achieve expanded scheduling for database sharding or other jobs requiring expansion. Basic configuration can be performed first, followed by dynamic expansion based on actual needs, saving time and reducing errors.

[0008] To achieve the above objectives, the technical solution adopted by the present invention includes:

[0009] A batch processing scheduling method based on scheduling file expansion conversion, characterized in that it includes:

[0010] S1. Obtain the source scheduling file, analyze the source scheduling file and extract the corresponding identification nodes. The identification nodes include root nodes, child nodes and job nodes divided according to hierarchy.

[0011] S2. Configure scene tags and execution tags for each identification node. The scene tags include one or more application scenarios matched by the corresponding identification node. The execution tags are set with different default values ​​according to the corresponding identification node level, and the execution tags are in a revisable state by default.

[0012] S3. Obtain the batch processing application scenario and select the corresponding recognition node as the processing node according to the scenario label.

[0013] S4. Determine whether the default value of the execution label of the processing node matches the batch processing application scenario attribute. When the default value of the execution label of the processing node matches the batch processing application scenario attribute, use the default value as the scheduling value.

[0014] S5. When it is determined that the default value of the execution label of the processing node does not match, revise the default value and generate the scheduling value.

[0015] S6. Based on the batch processing application scenario's job requirements, use the scheduling value to expand each processing node accordingly.

[0016] S7. Use the expanded processing node combination to generate the transformed target scheduling file.

[0017] S8. Execute batch jobs using the target scheduling file.

[0018] Furthermore, the execution tag includes job naming rules, associated node relationships, and job dependencies.

[0019] Furthermore, the revised default value includes any one or a combination of the following operations:

[0020] Modify the job naming rules;

[0021] Modify the default associated node relationships;

[0022] Delete the default associated node relationship;

[0023] Add new associated node relationships;

[0024] Modify the default job dependencies;

[0025] Delete the default job dependency;

[0026] Add new job dependencies.

[0027] Furthermore, the batch processing application scenario attributes include any one or more combinations of input / output conditions, required resources for execution, execution frequency, execution time, and execution user;

[0028] The batch processing application scenario job requirements include the number of parallel jobs, database sharding configuration, and table sharding configuration.

[0029] Furthermore, the expanded processing nodes include:

[0030] Identify and process the extended items of the node scheduling value;

[0031] Match extension items based on the job requirements of batch processing application scenarios;

[0032] Add sequentially numbered features to successfully matched expansion items;

[0033] Combine extended terms with sequentially numbered features to generate extended scheduling values;

[0034] Use the extended scheduling value and the processing node to create the corresponding extended node.

[0035] Furthermore, step S8 also includes:

[0036] Extract scheduling information from the target scheduling file;

[0037] Input the scheduling information into the Luban scheduling template to generate the Luban scheduling file;

[0038] Batch processing jobs are executed using Luban scheduling files.

[0039] This invention also relates to a batch processing scheduling system based on scheduling file expansion conversion, characterized in that it includes:

[0040] The identification module is used to analyze the source scheduling file and extract the corresponding identification nodes;

[0041] The tag management module is used to configure scene tags and execution tags for each recognition node.

[0042] The batch processing application judgment module is used to select the corresponding recognition node as the processing node based on the scene label, and to judge whether the default value of the execution label of the processing node matches based on the batch processing application scene attributes.

[0043] The extension module is used to extend each processing node according to the job requirements of the batch processing application scenario using the corresponding scheduling value, and to combine the extended processing nodes to generate the transformed target scheduling file.

[0044] The scheduling and execution module is used to execute batch jobs using the target scheduling file.

[0045] The present invention also relates to a computer-readable storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, implements the above-described method.

[0046] The present invention also relates to an electronic device, characterized in that it includes a processor and a memory;

[0047] The memory is used to store the identification node and target scheduling file;

[0048] The processor is used to execute the above method by invoking the identification node and the target scheduling file.

[0049] The present invention also relates to a computer program product, including a computer program and / or instructions, characterized in that the computer program and / or instructions, when executed by a processor, implement the steps of the above-described method.

[0050] The beneficial effects of this invention are as follows:

[0051] The batch scheduling method and system based on scheduling file expansion and conversion described in this invention automatically analyzes and expands job scheduling through scheduling naming rules and programs, generating control-M XML scheduling files or batch scheduling Excel files. It can dynamically expand and reduce similar jobs according to naming rules, and can configure various batch scheduling settings for database sharding and table partitioning, and dynamically adjust according to the configuration of various environmental resources to achieve expanded database sharding or other job scheduling that requires expansion. Basic configuration can be performed first, and then dynamic expansion can be carried out according to actual needs, resulting in high adaptability. By expanding batch scheduling, the maintenance time of batch scheduling is reduced, and the probability of errors in repetitive work is also reduced, thereby saving time and reducing errors. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the batch processing scheduling method based on scheduling file expansion conversion according to the present invention.

[0053] Figure 2 This is a schematic diagram of the batch processing scheduling system based on the extended conversion of scheduling files according to the present invention. Detailed Implementation

[0054] To better understand the content of this invention, a detailed description will be provided in conjunction with the accompanying drawings and embodiments.

[0055] The first aspect of this invention relates to a process flow as follows: Figure 1 The batch scheduling method based on scheduling file expansion conversion shown includes:

[0056] S1. Obtain the source scheduling file, analyze the source scheduling file, and extract the corresponding identification nodes. The identification nodes include root nodes, child nodes, and job nodes divided according to hierarchy. The source scheduling file can be a batch scheduling XML file corresponding to control-M, or an Excel scheduling file.

[0057] Preferably, based on the hierarchical structure of root nodes, child nodes, and job nodes (jobs), multi-level nesting of child nodes can be configured to achieve classification of job flow scheduling in the functional view. Adding a batch scheduling task typically requires adding a job name, job node, job parent node, job dependencies, etc. Based on the existing production big data batch processing mode, database sharding and table partitioning are usually adopted, and a unified parallel scheduling method for jobs is used to accelerate data processing efficiency. The number of server clusters managed by this tool is usually the number of jobs that can be processed in parallel for the same function, so the same function job is often configured hundreds of times, with only the job name being different.

[0058] For example, the entire credit card batch job is the root node, the billing function module is a child node, and this child node contains multiple billing sub-modules, such as bill detail nodes, bill summary nodes, and bill sending nodes. Each node typically has 1-6 databases and 48 or 96 jobs, with job names following the format 0001-0048. The scheduling tool will poll and schedule the 48 jobs, distributing them to different machines for execution.

[0059] S2. Configure scene tags and execution tags for each identification node. The scene tags include one or more application scenarios matched by the corresponding identification node. The execution tags include job naming rules, associated node relationships and job dependencies. Different default values ​​are set according to the corresponding identification node level. The execution tags are in a revisable state by default.

[0060] Scene labels correspond to the inherent properties of recognition nodes, that is, the specific batch processing scenarios in which a particular recognition node can be applied. Scene labels allow for easy confirmation of whether a recognition node should be selected for a specific scenario.

[0061] The execution tag is completely independent of the scenario tag, and there is no direct correlation or influence between the two. The execution tag is mainly used in scenarios where multiple jobs are executed in parallel under the batch scheduling configuration of database sharding and table partitioning. For example, the same function (job) can be expanded into 1-6 jobs according to the database sharding configuration, and the job needs to be expanded into 48 or 96 according to the table partitioning configuration.

[0062] A typical naming convention for execution tags includes:

[0063] 1) Name the SHELL job XX_DB001_XX.sh according to the database sub-number;

[0064] 2) Name the SHELL job XX_XX_0001.sh according to the sub-table job number;

[0065] 3) Other jobs that need to be supported for expansion or reduction can be added according to the rules.

[0066] S3. Obtain the batch processing application scenario and select the corresponding recognition node as the processing node based on the scenario label.

[0067] It is important to note that when identifying scene labels, a comprehensive filtering method should be selected. Without additional settings for mutual exclusion between different nodes, all nodes that meet the criteria should be selected for identification.

[0068] S4. Determine whether the default value of the execution label of the processing node matches the batch processing application scenario attributes. Specifically, this includes matching based on different attribute settings such as input / output conditions, required resources, execution frequency, execution time, and execution user. When the default value of the execution label of the processing node matches the batch processing application scenario attributes, the default value is used as the scheduling value.

[0069] Preferably, by parsing the processing nodes corresponding to the batch scheduling configuration file through code, each job table, job, and related attributes (such as input / output conditions, required resources, execution frequency, execution time, execution user, etc.) of the batch scheduling are loaded into memory in a hierarchical structure.

[0070] S5. When it is determined that the default value of the execution label of the processing node does not match, revise the default value to generate a scheduling value. Specifically, revising the default value includes any one or more of the following operations: modifying the job naming rule; modifying the default associated node relationship; deleting the default associated node relationship; adding an associated node relationship; modifying the default job dependency relationship; deleting the default job dependency relationship; adding a job dependency relationship.

[0071] S6. Based on the job requirements of the batch processing application scenario, the scheduling values ​​are used to expand each processing node accordingly. The job requirements of the batch processing application scenario include the number of parallel jobs, database sharding configuration, and table sharding configuration.

[0072] Expanding each processing node includes: identifying expansion items for the processing node scheduling value; matching expansion items according to the job requirements of the batch processing application scenario; adding sequential numbering features to successfully matched expansion items; combining expansion items with added sequential numbering features to generate an expanded scheduling value; and using the expanded scheduling value and processing node matching to establish the corresponding expanded node.

[0073] In actual execution, the number of database shards and jobs to be expanded in the job scheduling is configured via a configuration file, and then the expanded database shards and job numbers in the configuration file are loaded into memory. The batch scheduling data in memory is read in a loop, and the naming rules are used to determine whether expansion is needed. If expansion is needed, the jobs are expanded according to the expanded database shards and job numbers, and the jobs are placed in the same table with all related attributes remaining consistent, except for the job table name and the shell script name being expanded. All expansion is then written to new memory. If expansion is not needed, all information is directly written to new memory.

[0074] S7. Use the expanded processing node combination to generate the transformed target schedule file, in particular, rewrite the new in-memory data into the batch schedule file.

[0075] S8. Using the target scheduling file, extract the scheduling information from the target scheduling file and input it into the Luban scheduling template to generate the Luban scheduling file, and execute the batch processing job. Specifically, load the expanded scheduling file to be converted into memory; load the template of the Luban scheduling EXCEL file and parse the relevant structure; read the scheduling file in a loop, convert the relevant information and write it into the Luban job flow scheduling definition, job flow scheduling dependency, job scheduling definition, job flow scheduling dependency, etc.; write the job flow scheduling definition, job flow scheduling dependency, job scheduling definition, and job flow scheduling dependency information into Excel.

[0076] Another aspect of the present invention relates to a batch processing scheduling system based on scheduling file expansion conversion, the structure of which is as follows: Figure 2 As shown, it includes:

[0077] The identification module is used to analyze the source scheduling file and extract the corresponding identification nodes;

[0078] The tag management module is used to configure scene tags and execution tags for each recognition node.

[0079] The batch processing application judgment module is used to select the corresponding recognition node as the processing node based on the scene label, and to judge whether the default value of the execution label of the processing node matches based on the batch processing application scene attributes.

[0080] The extension module is used to extend each processing node according to the job requirements of the batch processing application scenario using the corresponding scheduling value, and to combine the extended processing nodes to generate the transformed target scheduling file.

[0081] The scheduling and execution module is used to execute batch jobs using the target scheduling file.

[0082] By using this system, the aforementioned computational processing methods can be executed and the corresponding technical effects can be achieved.

[0083] Embodiments of the present invention also provide a computer-readable storage medium capable of implementing all the steps of the methods in the above embodiments, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements all the steps of the methods in the above embodiments.

[0084] Embodiments of the present invention also provide an electronic device for performing the above-described method. As an implementation device for the method, the electronic device has at least a processor and a memory. In particular, the memory stores data and related computer programs required for performing the method, such as identification nodes and target scheduling files. The processor calls the data and programs in the memory to execute all the steps of the method and obtain the corresponding technical effects.

[0085] Preferably, the electronic device may include a bus architecture, which may include any number of interconnected buses and bridges. The bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same element, i.e., a transceiver, providing a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used to store data used by the processor during operation.

[0086] Additionally, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operating control) used may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device; the memory may be one or more of a buffer, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices, which can store the aforementioned data information, and may also store programs for executing the information, and the processor can execute the program stored in the memory to achieve information storage or processing, etc.; the input unit is used to provide input to the processor, for example, it can be a button or touch input device; the power supply is used to provide power to the electronic device; the display is used to display images and text, for example, it can be an LCD display. The communication module is a transmitter / receiver that transmits and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, which can be the same as in conventional mobile communication terminals. Based on different communication technologies, multiple communication modules can be incorporated into the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) is also coupled to a speaker and microphone via an audio processor to provide audio output through the speaker and receive audio input from the microphone, thereby enabling typical telecommunications functions. The audio processor can include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor is coupled to a central processing unit, enabling on-device recording via the microphone and on-device playback of stored sound via the speaker.

[0087] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0089] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0090] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0091] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A batch processing scheduling method based on scheduling file expansion and conversion, characterized in that, include: S1. Obtain the source scheduling file, analyze the source scheduling file and extract the corresponding identification nodes. The identification nodes include root nodes, child nodes and job nodes divided according to hierarchy. S2. Configure scene tags and execution tags for each identification node. The scene tags include one or more application scenarios matched by the corresponding identification node. The execution tags are set with different default values ​​according to the corresponding identification node level. The execution tags are in a revisable state by default. The execution tags include job naming rules, associated node relationships and job dependencies. S3. Obtain the batch processing application scenario and select the corresponding recognition node as the processing node according to the scenario label. S4. Determine whether the default value of the execution label of the processing node matches the batch processing application scenario attribute. When the default value of the execution label of the processing node matches the batch processing application scenario attribute, the default value is used as the scheduling value. The batch processing application scenario attribute includes any one or more combinations of input / output conditions, resources required for execution, execution frequency, execution time, and execution user. S5. When it is determined that the default value of the execution label of the processing node does not match, revise the default value and generate the scheduling value. S6. Based on the batch processing application scenario job requirements, the scheduling value is used to expand each processing node accordingly, wherein the batch processing application scenario job requirements include the number of parallel jobs, database sharding configuration, and table sharding configuration. S7. Use the expanded processing node combination to generate the transformed target scheduling file. S8. Execute batch jobs using the target scheduling file.

2. The method as described in claim 1, characterized in that, The revised default value includes any one or a combination of the following operations: Modify the job naming rules; Modify the default associated node relationships; Delete the default associated node relationship; Add new associated node relationships; Modify the default job dependencies; Delete the default job dependency; Add new job dependencies.

3. The method as described in claim 1, characterized in that, The extended processing nodes include: Identify and process the extended items of the node scheduling value; Match extension items based on the job requirements of batch processing application scenarios; Add sequentially numbered features to successfully matched expansion items; Combine extended terms with sequentially numbered features to generate extended scheduling values; Use the extended scheduling value and the processing node to create the corresponding extended node.

4. The method as described in claim 1, characterized in that, Step S8 further includes: Extract scheduling information from the target scheduling file; Input the scheduling information into the Luban scheduling template to generate the Luban scheduling file; Batch processing jobs are executed using Luban scheduling files.

5. A batch processing scheduling system based on scheduling file expansion and conversion, characterized in that, include: The identification module is used to analyze the source scheduling file and extract the corresponding identification nodes; The tag management module is used to configure scene tags and execution tags for each identification node. The execution tags include job naming rules, associated node relationships, and job dependency relationships. The batch processing application judgment module is used to select the corresponding recognition node as the processing node according to the scene label, and to judge whether the default value of the execution label of the processing node matches the batch processing application scene attribute. The batch processing application scene attribute includes any one or more combinations of input and output conditions, execution resources required, execution frequency, execution time and execution user. The extension module is used to extend each processing node according to the job requirements of the batch processing application scenario using the corresponding scheduling value, and to generate the transformed target scheduling file by combining the extended processing nodes. The job requirements of the batch processing application scenario include the number of parallel jobs, database sharding configuration, and table sharding configuration. The scheduling and execution module is used to execute batch jobs using the target scheduling file.

6. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 4.

7. An electronic device, characterized in that, Including processor and memory; The memory is used to store the identification node and target scheduling file; The processor is configured to execute the method of any one of claims 1 to 4 by invoking the identification node and the target scheduling file.

8. A computer program product, comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 4.

Images