Batch job processing method, device, equipment, system and medium

Abstract

The invention provides a batch job processing method, device, equipment, system and medium, relates to the technical field of finance, is applied to a distributed architecture system, and comprises a batch job management platform, a scheduler and an actuator. The batch job management platform is used for configuring at least one job set; at least one type of operation strategy is configured in each operation set; comprising the steps of selecting a target job and a target job strategy from at least one job set in a batch job management platform based on user requirements; issuing the target job to at least one target scheduler in the target application environment for deployment based on the target job strategy, and determining deployment information of the target job; the target operation strategy and the target application environment have a mapping relationship; and in response to the scheduling request, creating the job instance based on the deployment information, issuing the job instance to the target executor for processing, and by introducing a job strategy to configure a scheduling mechanism and a specific execution process, the processing flexibility and reliability are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of finance, and in particular to a batch job processing method, device, equipment, system and medium. BACKGROUND

[0002] With the acceleration of global digital transformation, banks and other financial institutions are facing changing and diversified business needs. Due to the frequent and complex changes in business needs, bank enterprise systems are gradually transforming to micro application architecture, especially distributed architecture applications. By using a distributed framework, the system can perform large-scale parallel batch processing jobs. By distributing tasks to different servers, the computing and storage resources of multiple servers can be fully utilized to shorten the task execution time, and also solve the problem of upper limit of single machine computing capacity.

[0003] In the prior art, the batch processing method based on distributed architecture generally includes task configuration, task scheduling and batch processing process. The task configuration refers to configuring business batch processing tasks according to task information. The task scheduling refers to registering the configured tasks to the plan scheduler container to form an assembled job. The thread pool in the plan scheduler container schedules and executes each job in parallel. The batch processing refers to using the remote sharding technology to perform batch processing on business data according to different business needs.

[0004] However, the above method supports data sharding, but does not support setting independent parameters for each sharding task, lacks a certain flexibility, and directly schedules and executes each job through the scheduler container, which may cause the job to be executed in different environments, increasing the risk of faults and job processing errors. SUMMARY

[0005] The present application provides a batch job processing method, device, equipment, system and medium, which is used to solve the problem that the existing batch processing method based on distributed architecture supports data sharding, but does not support setting independent parameters for each sharding task, lacks a certain flexibility, and directly schedules and executes each job through the scheduler container, increasing the risk of faults and job processing errors.

[0006] In a first aspect, the present application provides a batch job processing method applied to a distributed architecture system. The distributed architecture system includes a batch job governance platform, a scheduler and an executor. The batch job governance platform is used to configure at least one job set. Each job set is configured with at least one type of job policy. The job policy is used to indicate the parameter information required for job processing, the scheduler information for sharding scheduling and the executor information for executing the job. The method comprises:

[0007] select a target job and a target job policy from at least one job set in the batch job governance platform based on user demand;

[0008] issue the target job to at least one target scheduler in a target application environment for deployment based on the target job policy, and determine deployment information of the target job; the target job policy has a mapping relationship with the target application environment;

[0009] In response to a scheduling request, create a job instance based on the deployment information, and issue the job instance to a target executor for processing.

[0010] Optionally, the job policy includes a sharding policy, a unit execution policy, a database sharding policy, a scheduler scheduling policy, and a load balancing policy; the sharding policy is used to indicate sharding parameters of a scheduled job; the unit execution policy is used to indicate executor information of an executed job; the database sharding policy is used to indicate database information of a sharded parameter information; the scheduler scheduling policy is used to indicate scheduler information of a scheduled job; and the load balancing policy is used to indicate calling information of a target scheduler.

[0011] Optionally, determining the deployment information of the target job includes:

[0012] processing configuration information corresponding to the target job policy to determine the deployment information of the target job; the configuration information includes at least one of the parameter information, the sharding parameter, the executor information, the database information, the scheduler information, and the calling information.

[0013] Optionally, the batch job governance platform is also used to approve at least one type of job policy configured for each job set; the method further includes:

[0014] In response to a modification operation of the user on the job set and / or the job policy, generating approval information; the approval information carries modification content;

[0015] based on the batch job governance platform, auditing the approval information;

[0016] After the audit passes, based on first identification information of the modified job set and / or the job policy, determining whether to synchronize the modification content to the at least one target scheduler.

[0017] Optionally, the method further includes:

[0018] If the first identification information and second identification information of the target job policy and / or the target job set to which the target job belongs are the same, the modification content is synchronized to the at least one target scheduler.

[0019] If the first identification information and the second identification information of the target job set and / or the target job policy are different, the modified job set and / or the job policy are stored in the batch job governance platform.

[0020] Optionally, the method further comprises:

[0021] When the at least one job set is configured based on the batch job governance platform, the pre-configured at least one job set is audited, and after the audit is passed, the configured at least one job set is generated.

[0022] In a second aspect, the present application provides a batch job processing device applied to a distributed architecture system; the distributed architecture system comprises a batch job governance platform, a scheduler and an executor; the batch job governance platform is used to configure at least one job set; at least one type of job policy is configured in each job set; the job policy is used to indicate parameter information required for job processing, scheduler information for sharding scheduling jobs and executor information for executing jobs; the device comprises:

[0023] A selection module is configured to select a target job and a target job policy from at least one job set in the batch job governance platform based on user demand;

[0024] A determination module is configured to distribute the target job to at least one target scheduler in a target application environment for deployment based on the target job policy, and determine deployment information of the target job; the target job policy has a mapping relationship with the target application environment;

[0025] A creation module is configured to create a job instance based on the deployment information in response to a scheduling request, and distribute the job instance to a target executor for processing.

[0026] In a third aspect, the present application provides an electronic device comprising a processor and a memory in communication connection with the processor;

[0027] The memory stores computer execution instructions;

[0028] The processor executes the computer execution instructions stored in the memory to realize the method according to any one of the first aspect.

[0029] In a fourth aspect, the present application provides a distributed architecture system; the distributed architecture system comprises a batch job governance platform, a scheduler and an executor; the batch job governance platform is configured to configure at least one job set; each job set is configured with at least one type of job policy; the job policy is used to indicate parameter information required for job processing, scheduler information for sharding scheduling jobs, and executor information for executing jobs; and the distributed architecture system is used to execute the method according to any one of the first aspect.

[0030] In a fifth aspect, the present application provides a computer readable storage medium, which stores computer execution instructions; when the computer execution instructions are executed by a processor, the computer execution instructions are used to implement the method according to any one of the first aspect.

[0031] In summary, the present application provides a batch job processing method, device, equipment, system and medium; by introducing the combination of a batch job governance platform, a scheduler and an executor in a distributed architecture system, a user can filter target jobs that need to be processed from a plurality of job sets configured in the batch job governance platform according to actual business needs, and match corresponding target job policies; the job set and the job policy are uniformly maintained through the governance platform, which reduces redundancy and errors caused by scattered configuration, reduces manual operation costs, and facilitates tracing the association between jobs and policies; further, since the target job policy and the target application environment have a preset mapping relationship, the distributed architecture system can distribute the target job to the target scheduler in the corresponding environment according to the mapping relationship; the target scheduler receives the target job and completes deployment, and generates deployment information as a basis for subsequent execution; this sharding scheduling means that the job can be split into multiple sub-tasks and processed by different scheduler nodes in collaboration; through the mapping of the target job policy and the target application environment, it can be ensured that the job runs in the correct environment and reduces the risk of cross-environment interference; sharding scheduling improves the processing efficiency of large-scale jobs and avoids single-node overload; further, when the system receives a scheduling request, it can create a specific job instance for the target job based on the generated deployment information, and distribute the job instance to the corresponding target executor according to the executor information specified in the target job policy; the target executor performs data processing, calculation and other operations according to the job instance configuration and feeds back the execution result; since the target executor focuses on the processing of the job instance and has clear division of labor with the scheduler, it meets the isolation requirement; at the same time, creating a job instance based on deployment information can ensure that the execution parameters and the policy are consistent, reduce execution deviation, and improve the accuracy of job processing; in this way, through the distributed collaboration of the target scheduler and the target executor, parallel processing of jobs can be realized, the throughput of the overall system can be improved, and it is especially suitable for massive data processing scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0033] Figure 1 An application scenario schematic diagram provided for an embodiment of the application;

[0034] Figure 2 A flowchart schematic diagram of a batch job processing method provided for an embodiment of the application;

[0035] Figure 3 A model structure schematic diagram of a distributed architecture system provided for an embodiment of the application;

[0036] Figure 4 A flowchart of a batch job governance method provided for an embodiment of the application;

[0037] Figure 5 A structure schematic diagram of a batch job processing apparatus provided for an embodiment of the application;

[0038] Figure 6 A structure schematic diagram of an electronic device provided for an embodiment of the application.

[0039] The specific embodiments of the application have been shown through the above-described drawings, and will be described in more detail hereinafter. These drawings and the written description are not intended to restrict the scope of the concept of the application by any means, but to illustrate the concept of the application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0040] In order to clearly describe the technical solutions of the embodiments of the application, in the embodiments of the application, the same items or similar items with basically the same functions and effects are distinguished by using “first”, “second”, etc. For example, the first identification information and the second identification information are only used to distinguish different identification information, and the order is not limited. Those skilled in the art can understand that “first”, “second”, etc. do not limit the quantity and execution order, and “first”, “second”, etc. also do not limit the difference.

[0041] It should be noted that in the present application, the words “exemplary” or “for example” are used to mean serving as an example, instance, or illustration. Any embodiment or design solution described as “exemplary” or “for example” in the present application should not be interpreted as being more preferred or having more advantages than other embodiments or design solutions. Rather, the words “exemplary” or “for example” are used to present related concepts in a specific manner.

[0042] In this application, "at least one" means one or more, "multiple" means two or more. "And / or" describes the association between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the following cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including single or multiple items in any combination. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0043] First, the professional terms involved in this application are explained.

[0044] Micro application: can refer to an independent service unit that encapsulates specific business functions in a distributed architecture, which can provide one or more interfaces; because the size of microservices varies, there are many different understandings of the concept of microservices in the industry, in order to avoid ambiguity caused by the industry's different understanding of the granularity of "microservices", this application refers to microservices in a distributed architecture as micro applications.

[0045] Micro instance: can refer to a single program individual after the micro application is started, and the micro instance is the materialized representation of the micro application runtime. When deployed in a cloud container environment, the micro instance corresponds to the container itself; if running in a Linux virtual machine, it refers to an independent process that carries the micro application function in the environment.

[0046] Batch job governance platform: can refer to a specific micro application dedicated to batch job management, which provides a visual console to enable users to centrally configure, operate and monitor jobs, and achieve efficient and unified visual management and control of batch jobs.

[0047] Scheduler: can refer to a specific micro application that stores job strategy deployment information and job scheduling, and uniformly processes job scheduling execution and running state monitoring.

[0048] Executor: can refer to a specific micro application, which is built based on the SpringBatch framework and is used to define the actual business logic of batch jobs; by receiving scheduling requests from the scheduler, it parses the job definition information contained therein and executes specific batch tasks accordingly.

[0049] Job: can refer to the specific implementation of a complete business process composed of single or multiple steps split by business logic. When the job is started, the executor will follow the preset order to execute these steps one by one. Therefore, the job is essentially a list of steps that can run independently and have a clear execution order.

[0050] Job instance: can refer to a logical entity generated at runtime, each job instance represents an independent execution process of the job, and is distinguished by the job name and job identification parameters.

[0051] Job parameters: can refer to parameter values applied to all job instances created.

[0052] Sharding: can refer to a parallelization strategy for splitting jobs. By setting the number of shards, the job can be divided into multiple independent instances (each shard corresponds to a job instance) and executed concurrently, significantly improving the processing efficiency of data-intensive or time-consuming jobs. Each job instance runs the same code logic, so it can be differentiated by shard index for task allocation and control.

[0053] Sharding parameters: can refer to the parameters required by a specific instance of a job shard during runtime, used to provide differentiated input or configuration for each shard instance in a parallel execution environment.

[0054] Cron expression: can refer to a time-based string configuration format used to define the periodic scheduling rules of jobs or tasks. By combining specific fields, it can accurately set the trigger conditions in time dimensions such as minutes, hours, and dates.

[0055] Routing strategy: can refer to a rule or algorithm used to determine the specific computing node on which a task is executed in a distributed multi-node environment.

[0056] Task sharding: can refer to a distributed processing technology that splits large computing tasks into multiple independent sub-tasks (shards) and distributes them to different computing nodes for parallel execution through a scheduling system.

[0057] Task parameters: can refer to input data transmitted from the outside during task execution, used to dynamically configure task behavior or provide necessary context information and variable values for its execution.

[0058] JobHandler: can refer to the core execution unit in the task scheduling framework that carries out specific business logic. Developers define the actual operation behavior of the task when it is triggered by implementing a specific interface or base class, which is a key component for decoupling job scheduling and business functions.

[0059] In the traditional batch processing mode, bank enterprises generally rely on single machine centralized architecture, usually using IBM (International Business Machines) mainframe or deploying batch scripts on servers directly connected to databases to process tasks. However, with the exponential growth of data, single machine performance gradually becomes a bottleneck of batch processing, and distributed batch processing architecture can achieve flexible expansion through fine-grained service division, and each service can independently manage data storage and dynamically adjust node size to improve processing efficiency. Therefore, the distributed batch processing architecture can solve the problems existing in the single machine centralized architecture, which is no longer limited by a single server.

[0060] By means of distributed batch processing architecture, tasks can be distributed to multiple different servers, making full use of the resources of multiple servers, relying on the high-concurrency batch processing job capability provided by the architecture to significantly shorten the task execution time and break through the single machine computing power limit.

[0061] However, in a distributed environment, there are many services under the business system, and each service contains a large number of independent batch jobs, which makes the job size grow rapidly, thus putting forward higher requirements for job governance. Therefore, it is urgent to realize efficient and safe batch job management through standardized job definition and deployment process.

[0062] In a possible implementation manner, SpringBatch can be used as a batch job development framework or a purchased commercial batch framework as a basic framework, and open source scheduling frameworks such as Quartz and XXL-JOB are generally used at the scheduling level, and are often customized and extended according to actual business needs. For example, Quartz supports configuring jobs through XML or Java code, defining task name, execution time and parameters, etc.; and XXL-JOB can provide visual WorldWide Web (Web) console and Application Programming Interface (API) two task configuration modes, support flexible setting and management of Cron expression, executor, routing strategy, task sharding, task parameters, failure retry mechanism and JobHandler components, etc.

[0063] Among them, the failure retry mechanism refers to automatically retrying to execute the task when the task execution fails.

[0064] However, the Quartz framework lacks a unified task management center and cannot provide visual operation and maintenance capabilities, usually embedded in business applications in the form of Jar package and deployed with the application, which is difficult to realize centralized scheduling management and does not support data sharding function.

[0065] The XXL-JOB framework can provide a visual console and support data sharding, and the scheduler can distribute the shard number and the total number of shards to different executors to process tasks in parallel, but still has many limitations, such as the inability to set independent parameters for sharded tasks, the high coupling between the scheduler and the console, which leads to rigid deployment architecture, and the need for separate deployment of the entire scheduling center to isolate business systems, making it difficult to balance centralized control and system isolation needs.

[0066] In addition, the framework lacks process control for job configuration changes, and any modification takes effect immediately, which poses a stability risk and cannot effectively support differentiated governance across multiple environments.

[0067] It should be noted that the immediate change mechanism may disrupt the environmental isolation between multiple business systems, causing untested changes to directly affect the test environment. In the context of multi-environment collaborative development, changes usually need to follow strict management and approval processes to ensure compliance and stability, and the lack of control over the immediate effect mechanism easily bypasses key audit steps, leading to unauthorized changes entering the production environment, significantly increasing the risk of system errors and operation.

[0068] In another possible implementation, a batch processing method based on a distributed architecture generally includes task configuration, task scheduling, and batch processing; wherein task configuration refers to configuring business batch processing tasks according to task information; task scheduling refers to registering the configured tasks to a plan scheduler container to form an assembled job, and the thread pool in the plan scheduler container schedules and executes each job in parallel; batch processing refers to using remote sharding technology to process business data in batches according to different business needs.

[0069] However, the above method supports data sharding, but does not support setting independent parameters for each sharded task, lacks a certain degree of flexibility, and directly schedules and executes each job through the scheduler container, which may cause jobs to be executed in different environments, increasing the risk of faults and job processing errors.

[0070] To address the aforementioned issues, this application proposes a batch job processing method. By introducing a combination of a batch job governance platform, scheduler, and executor into a distributed architecture system, users can filter target jobs from multiple pre-configured job sets within the batch job governance platform based on actual business needs, and match corresponding target job strategies. This job set and strategy are uniformly maintained through the governance platform, reducing redundancy and errors caused by decentralized configuration, lowering manual operation costs, and facilitating the traceability of job-strategy relationships. Furthermore, since there is a pre-defined mapping relationship between target job strategies and target application environments, the distributed architecture system can distribute target jobs to the target scheduler in the corresponding environment based on this mapping relationship. The target scheduler receives the target job, completes deployment, and generates deployment information as the basis for subsequent execution. This sharded scheduling means that jobs can be split into multiple subtasks, processed collaboratively by different scheduler nodes, through target job... Mapping job strategies to target application environments ensures jobs run in the correct environment, reducing the risk of cross-environment interference. Sharded scheduling improves the processing efficiency of large-scale jobs, avoiding single-node overload. Furthermore, when the system receives a scheduling request, it can create specific job instances for the target job based on the generated deployment information. According to the executor information specified in the target job strategy, the job instance is distributed to the corresponding target executor. This target executor completes data processing, computation, and other operations according to the job instance configuration and returns the execution results. Because this target executor focuses on job instance processing, its division of labor with the scheduler is clear, meeting isolation requirements. Simultaneously, creating job instances based on deployment information ensures that execution parameters are consistent with the strategy, reducing execution deviations and improving job processing accuracy. Thus, through distributed collaboration between the target scheduler and the target executor, parallel processing of jobs can be achieved, improving the overall system throughput, especially suitable for massive data processing scenarios.

[0071] For example, Figure 1 This is a schematic diagram of an application scenario provided in an embodiment of this application, such as... Figure 1 As shown, this application scenario can be applied to batch job processing in a banking system. The application scenario includes a user terminal device 101 and a distributed architecture system 102, which includes a batch job governance platform, a scheduler, and an executor.

[0072] The user logs in to the web management interface of the batch job governance platform based on the terminal device 101. On the batch job governance platform, jobs are represented in the form of job sets. After browsing the job sets, the user finds the target job to be executed. For each job, the platform provides a plurality of pre-configured job strategies, and each job strategy defines the running mode of the job in a specific scenario. Therefore, the user can select a suitable target job strategy according to the current specific demand, for example, whether to run in a test environment or in a production environment. Alternatively, the user inputs the user demand into the web management interface of the batch job governance platform based on the terminal device 101. The batch job governance platform automatically identifies the target job and the target job strategy corresponding to the user demand. The embodiments of the present application do not make specific limitations on the manner of determining the target job and the target job strategy.

[0073] Further, after determining the target job and the target job strategy, the batch job governance platform starts to analyze the target job strategy. The target job strategy includes scheduler information and executor information, as well as a mapping relationship with the target application environment configuration. In this way, the batch job governance platform can deploy the target job to the target scheduler of the corresponding target application environment according to the mapping relationship. Correspondingly, the batch job governance platform also records the deployment information of this deployment.

[0074] In response to the scheduling request, the target scheduler is triggered, and then the target scheduler creates a specific job instance according to the deployment information received and stored from the batch job governance platform. In this way, the target scheduler can distribute the job instance to the correct target executor according to the executor information specified in the deployment information. After receiving the job instance, the target executor loads the business processing code and executes the specific business logic together with the parameter information carried in the job instance.

[0075] Optionally, if the job is configured with sharding, the target scheduler can perform sharding calculation and assign different sharding parameters to different target executors.

[0076] The terminal device can also be referred to as a user equipment (UE), a mobile station (MS), a mobile terminal, a terminal, etc. In actual applications, the terminal device is, for example, a desktop computer, a notebook computer, a personal digital assistant (PDA), a smart phone, a tablet computer, a vehicle-mounted device, a wearable device (such as a smart watch or a smart bracelet), a smart home device (such as a smart display device), etc.

[0077] It should be noted that the application embodiments do not specifically limit the application scenarios of the application, and can be applied to any scenario requiring batch job processing.

[0078] Optionally, the batch job processing method provided by the application is applied to a distributed architecture system, the distributed architecture system comprising a batch job governance platform, a scheduler and an executor; the batch job governance platform is configured to configure at least one job set; each job set is configured to configure at least one type of job strategy; the job strategy is used to indicate parameter information required for job processing, scheduler information for sharding scheduling of a job and executor information for execution of the job.

[0079] It should be noted that, in the batch job processing process, the job set as a core management unit is used to logically group jobs, and one job set can contain multiple job strategies, and the job strategy is used to indicate parameter information required for job processing, scheduler information for sharding scheduling of a job and executor information for execution of the job.

[0080] Among them, the job set can refer to a basic logical unit for job organization and management in the batch job governance platform, which is a container used for aggregation and classification of jobs with common attributes or belonging to the same business category. The job set can be represented by top and bottom job set names, Chinese names of job sets, job set versions and the like.

[0081] The job strategy can refer to a predefined and reusable configuration template, which encapsulates a complete instruction set and rule set for driving a job to run under specific conditions.

[0082] The parameter information can refer to a component of the job strategy, which refers to a collection of external input variables or configuration data relied on by the job in the process of executing its business logic.

[0083] The scheduler information can refer to an identifier of a scheduler instance or cluster specified in the job strategy and responsible for task scheduling, triggering and sharding calculation of the job.

[0084] The executor information can refer to an identifier of an executor instance or cluster specified in the job strategy and responsible for carrying and executing the job task logic.

[0085] It can be understood that the job can be represented by defining basic information of the job such as job name, SpringBatchJobId, execution micro application and the like, and the application embodiments do not specifically limit the forms of the job and the job set.

[0086] In this way, for batch job processing under a distributed architecture system, a unified batch job management platform is introduced to process batch jobs with a job set as the center, centralized management of jobs and job strategies is performed, unified configuration, control and monitoring are realized, decoupling of job definition, task scheduling and task execution is realized through the respective functions of the batch job management platform, the scheduler and the executor, and the maintainability and flexibility of the system are improved.

[0087] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0088] Figure 2 A flowchart of a batch job processing method provided in an embodiment of the present application is shown in the figure. The batch job processing method can be applied to the distributed architecture system described above, as shown in the figure, the batch job processing method comprises the following steps: Figure 2

[0089] S201, selecting a target job and a target job strategy from at least one job set in the batch job management platform based on user demand.

[0090] In an embodiment of the present application, user demand can refer to the type of job to be executed, the processing target and the core requirements such as the data range to be processed, the execution time limit, the parameter rules, etc. specified by the user according to actual business scenarios such as daily early morning transaction data aggregation, monthly financial report generation, etc. The specific content corresponding to user demand is not limited in the embodiment of the present application, which can be set based on specific business scenarios.

[0091] For example, the system receives and parses the job execution demand generated by the user based on the terminal device, and then searches and matches in the multiple job sets configured in the batch job management platform according to the demand content, to locate the target job to be executed from the matched job set. At the same time, according to the optional strategy of the target job and the user demand, the target job strategy to be adopted for running the target job is determined, which contains the parameter information required for running the target job, the specified scheduler information and the executor information.

[0092] S202, deploying the target job in at least one target scheduler in a target application environment based on the target job strategy, and determining the deployment information of the target job; the target job strategy has a mapping relationship with the target application environment.

[0093] ​In the embodiments of the present application, the target application environment can refer to an environment that is pre-established for running a specific business application, has independent resource configuration, and is physically or logically isolated from each other, such as different development environments, test environments, pre-release environments, production environments, and the like.

[0094] The deployment information can refer to data records generated by the governance platform after the job deployment action is completed. Since the scheduler is generally deployed in different environments, and the load can be mounted when the cluster is deployed, it is necessary to issue the job to the scheduler in the corresponding environment. The scheduler saves the job in the database of the scheduler after receiving the job, and becomes a job to be run after the job is deployed. Correspondingly, the batch job governance platform can record the deployment information.

[0095] It should be noted that the target job strategy has a mapping relationship with the target application environment. The environment mapping rule is pre-set to ensure that the application environment of the job issue matches the business demand.

[0096] For example, the distributed architecture system can read the pre-set environment mapping rule in the target job strategy, confirm the target application environment corresponding to the target job strategy, and in the determined target application environment, the system filters out the target scheduler that meets the condition from a plurality of schedulers in the target application environment according to the scheduler information of the target job strategy for sharding scheduling, and issues the determined target job to the filtered target scheduler through an internal communication mechanism. After the target scheduler receives the target job, the target scheduler completes the deployment preparation according to the parameter information required for job processing in the target job strategy.

[0097] Further, after the deployment is completed, the system can automatically summarize and generate the deployment information of the target job. The content can include: job unique identifier, deployed scheduler identifier, sharding parameter, deployment time, job parameter, associated target job strategy identifier, and the like. The embodiments of the present application do not make specific limitations on the content of the deployment information.

[0098] S203, in response to the scheduling request, creating a job instance based on the deployment information, and issuing the job instance to a target executor for processing.

[0099] In the embodiments of the present application, the source of the scheduling request can be automatically triggered or manually triggered. For example, the Quartz component and the like in the target scheduler is automatically triggered according to the Cron expression to generate a scheduling request, or the user clicks the option operation on the batch job governance platform to send a manual scheduling instruction to the target scheduler. The embodiments of the present application do not make specific limitations on the generation mode of the scheduling request.

[0100] Exemplarily, after receiving the scheduling request, the target scheduler queries deployment information of the to-be-run job, creates a job instance according to the deployment information, and issues the job instance to a corresponding target executor for running.

[0101] In this way, compared with the traditional batch processing method which does not support setting independent parameters for each shard task and limits the flexibility of task configuration, the application configures a job set and a job strategy in the batch job management platform, so that each job set can contain multiple types of job strategies, and these job strategies can indicate parameter information required for job processing, such as shard parameters, job parameters, and the like, thereby supporting setting independent parameters for each shard task and improving the flexibility of configuration. In the traditional batch processing method, the scheduler directly schedules and executes each job, which may cause the job to be executed in different environments, increasing the risk of failure and error. The application establishes a mapping relationship between the target job strategy and the target application environment, ensures that the job is executed in a suitable application environment, deploys the target job in the target scheduler in the target application environment, ensures the accuracy and adaptability of scheduling, avoids job processing failure, and further, by responding to the scheduling request, creating a job instance based on the deployment information, and issuing the job instance to a target executor for processing, the reliability of job processing is ensured. The target executor is responsible for specific job execution, reducing the risk of errors caused by inconsistent environments.

[0102] In addition, the method of the application is designed through a distributed architecture, including the separation design of the batch job management platform, the scheduler and the executor, so that each part of the distributed architecture system can be independently expanded and maintained, improving the maintainability of the system.

[0103] In summary, the application introduces a job strategy configuration scheduling mechanism and a specific execution process, solving the problems of insufficient flexibility, inaccurate scheduling and low processing reliability in the traditional distributed architecture batch processing method.

[0104] Optionally, the job strategy includes a shard strategy, a unit execution strategy, a database shard strategy, a scheduler scheduling strategy and a load balancing strategy; the shard strategy is used to indicate shard parameters of the scheduled job; the unit execution strategy is used to indicate executor information of the executed job; the database shard strategy is used to indicate database information of the shard storage parameter information; the scheduler scheduling strategy is used to indicate scheduler information of the scheduled job; and the load balancing strategy is used to indicate calling information of the target scheduler.

[0105] Exemplarily, Figure 3 A model structure schematic diagram of a distributed architecture system provided by an embodiment of the application is as follows, Figure 3As shown, the distributed architecture system is centered on a job set, multiple job sets can be established under one system, each job set has multiple versions, and multiple different job strategies can be established for each specific version of the job set to bind, wherein multiple batch job job strategies can also be defined in the job set, that is, the corresponding execution strategy can be configured for the job in the job definition, such as sharding strategy, unit execution strategy, database sharding strategy, scheduler scheduling strategy, and load balancing strategy.

[0106] Among them, the sharding strategy can refer to a rule configuration for guiding the logical splitting of batch jobs into multiple independent and parallel processing task units, such as defining sharding strategy name, first-level sharding number, second-level sharding number, and other information to represent the corresponding sharding strategy; the unit execution strategy can refer to a directional rule that explicitly specifies the allocation of job tasks to the corresponding target executor to carry and run, such as defining unit execution name, unit ID list, and other information to represent the corresponding unit execution strategy.

[0107] The database sharding strategy can refer to a rule configuration for guiding the system to store the parameters, states, or result data required for job running in different database shards in a distributed manner, such as defining database sharding strategy name, database shard list, and other information to represent the corresponding database sharding strategy.

[0108] The scheduler scheduling strategy can refer to a scheduling rule that explicitly specifies the corresponding scheduler to be responsible for triggering and managing a specific job, such as defining scheduler name, scheduler network address, scheduler micro application name, and other information to represent the corresponding scheduler scheduling strategy.

[0109] The load balancing strategy can refer to an algorithm rule for guiding the selection of a suitable target scheduler from multiple target schedulers to receive job deployment or processing scheduling requests when there are multiple target schedulers to choose from.

[0110] In this way, by defining the above job strategy, different dimensions of job scheduling and execution, such as sharding, execution unit, data storage, scheduler selection, load balancing, are decomposed into independent strategies, which realizes strategy decoupling, facilitates independent optimization and management of each strategy, for example, load balancing strategy and scheduler scheduling strategy enable the system to dynamically allocate tasks and select schedulers according to the load, thereby enhancing the horizontal expansion capability and flexibility of the system in processing high-concurrency load, and through the sharding strategy, database sharding strategy and unit execution strategy, precise control and allocation of computing resources, data resources and executor resources are realized, thereby improving resource utilization and job execution efficiency. Therefore, for batch job governance under a distributed architecture, the design based on the job strategy distinguishes job governance under different business systems, so that job governance under different environments does not interfere with each other, so that the scheduler and the governance platform are independently deployed and do not interfere with each other, and the scheduler is separated from the job configuration and is only responsible for a single scheduling responsibility.

[0111] In addition, by selecting and combining different job strategies for the same job, various runtime scenarios can be easily adapted without modifying the business code of the job itself, thereby improving the flexibility of the application.

[0112] Optionally, the deployment information of the target job is determined, including:

[0113] The configuration information corresponding to the target job strategy is processed to determine the deployment information of the target job; the configuration information includes at least one of the parameter information, the sharding parameter, the executor information, the database information, the scheduler information and the calling information.

[0114] In an embodiment of the application, the parameter information can refer to a set of variables or configuration values that drive the business logic of the job to run, which is used to be injected when the job is instantiated, and is used to control the processing range, behavior mode and running characteristics of the job, etc. The sharding parameter can refer to a specific parameter set used to guide the task scheduling of the job, which provides a specific execution context for the sharding strategy, such as the total number of sharding, the current sharding name, the scheduler name corresponding to the sharding, etc.

[0115] The executor information can refer to the address or identification information of one or more executors that can carry and execute the task logic of the job, which is used to indicate the location of the target executor that executes the job.

[0116] The scheduler information can refer to the address or identification information of one or more schedulers responsible for scheduling and managing the job, which is used to indicate the location of the target scheduler that receives the job scheduling.

[0117] The database information can refer to address or identification information of a database used to store data related to the current job run, and the calling information is used to indicate information corresponding to a target scheduler that meets load balancing.

[0118] For example, the distributed architecture system can extract various configuration information contained in the target job strategy, such as parameter information, sharding parameters, executor information, database information, scheduler information, and calling information. Further, the configuration information is converted into structured data that can be directly used for job deployment, and the converted structured data is integrated into complete deployment information, which is stored in the database of the batch job management platform to form a traceable and queryable deployment archive.

[0119] It should be noted that in this application, the job strategy can be configured based on the job set and deployed to the corresponding scheduler. In this way, the job information used by the scheduler when scheduling the job is the deployed job information.

[0120] In this way, by processing the configuration information corresponding to the target job strategy, the consistency of the information required for job deployment and the target job strategy is ensured. For example, when the job instance is executed, the information in the deployment information can be directly read, and there is no need to repeatedly parse the target job strategy, which reduces the deviation in the information transmission process and improves the collaboration efficiency in the distributed environment. In addition, the scattered configuration parameters are integrated into unified deployment information, which realizes the standardization and automation of the deployment process and reduces errors and differences caused by manual intervention.

[0121] Optionally, the batch job management platform is further configured to approve at least one type of job strategy configured in each job set; and the method further includes:

[0122] In response to a modification operation of the user on the job set and / or the job strategy, generating approval information; the approval information carries the modification content;

[0123] Based on the batch job management platform, the approval information is audited;

[0124] After the audit is passed, based on the first identification information of the modified job set and / or the job strategy, it is determined whether to synchronize the modification content to the at least one target scheduler.

[0125] In the embodiments of the present application, the change content of the job set, such as the creation of the job set, the definition of the job, and the modification, needs to go through a corresponding audit and release process to ensure the management and control of the update, release, and use of the job set.

[0126] Optionally, the modification operation can refer to an adding, deleting, or modifying operation.

[0127] In the present application, the approval can refer to an evaluation and authorization process initiated by the batch job governance platform for a job set or a job strategy change proposal; the approval information is a standardized request object encapsulating the specific content of the change generated automatically by the system after the modification operation is triggered, and correspondingly, the modification content can refer to the specific data part submitted by the user based on the terminal device, which expects to change the original configuration of the job set or the job strategy.

[0128] The first identification information is used to uniquely identify the identification information of a job set or a job strategy, and the present application does not limit the form of the first identification information. For example, the first identification information can be a job set name, a job strategy name, or a job set ID, a job strategy ID, etc.

[0129] For example, when the user performs a modification operation on the job set or the job strategy in the batch job governance platform, the system generates the approval information and automatically triggers the approval process, that is, the approval information is automatically submitted to the audit module of the batch job governance platform and enters the preset audit process. During the audit process, the auditor can evaluate based on the approval information displayed by the platform. If the evaluation is passed, the system extracts the first identification information of the modified job set or job strategy and determines whether to synchronize to at least one target scheduler based on the first identification information.

[0130] In this way, the approval process ensures that all modifications to the job set and the job strategy are authorized and audited, preventing unauthorized changes and improving system security. The above approval mechanism can also avoid configuration errors caused by accidental or intentional modifications, reducing the risk of production environment failures and improving system reliability. Therefore, the platform audit link can intercept non-compliant operations to eliminate the risk of running caused by configuration errors. After the audit is passed, the first identification information is used to intelligently determine whether to synchronize the changes, ensuring the consistency of configuration management and runtime environment, avoiding invalid or conflicting deployments, and improving the stability and security of the system.

[0131] Optionally, the method further comprises:

[0132] If the first identification information and the second identification information of the target job strategy and / or the target job set to which the target job belongs are the same, the modification content is synchronized to the at least one target scheduler;

[0133] If the first identification information and the second identification information of the target job strategy and / or the target job set to which the target job belongs are different, the modified job set and / or the job strategy are stored in the batch job governance platform.

[0134] In the embodiments of the present application, the second identification information can refer to the identification information corresponding to the job strategy or the job set currently bound and synchronized with the at least one target scheduler. Since the job strategy bound to the same version of the same job set under the same system has and only has one job strategy that can be deployed for the same scheduler under the same environment, whether to synchronize the modification content to the at least one target scheduler is determined by comparing whether the first identification information and the second identification information are the same.

[0135] It should be noted that the related configuration information of the jobs in the modified job set and / or job strategy will be synchronized and take effect in the strategy deployment process, that is, the content of the jobs under the job strategy in the deployed state is modified, and the change will be automatically synchronized to the target scheduler that has received the deployment, ensuring that the job configuration of the target scheduler is consistent with the batch job governance platform in real time.

[0136] It can be understood that the above synchronization decision mechanism compares whether the identification information of the modified job set and / or job strategy is consistent with the current target job strategy and / or the target job set to which the target job belongs. When the identification information is consistent, the change is synchronized to the scheduler, avoiding invalid or erroneous deployment operations, ensuring the consistency of the online environment and configuration definition, and realizing instant online update. When the identification information is inconsistent, the modified job set or job strategy is stored as an independent version instead of being directly overwritten, preventing accidental modification from interfering with the running jobs, and realizing version management and safe isolation of the configuration. This processing logic by distinguishing the same and different identification prevents the running jobs from appearing runtime errors or inconsistent states due to sudden changes in the strategy or definition, improves system stability, and also reduces the risk of manual misoperation, making the configuration change process more controllable and reliable.

[0137] Optionally, the method further comprises:

[0138] When the at least one job set is configured based on the batch job governance platform, the pre-configured at least one job set is audited, and after the audit is passed, the configured at least one job set is generated.

[0139] In this step, the user submits the configuration content of the job set based on the terminal device. The batch job governance platform does not immediately make it effective, but treats it as a to-be-audited item, and the corresponding auditor or authorized role evaluates its compliance, correctness and rationality. Only after the audit is passed, the configuration can become a formal and usable configured job set.

[0140] Therefore, the audit mechanism can effectively intercept the job set containing errors, conflicts or non-compliant configurations into the system, thereby avoiding the failure of batch job running caused by configuration errors, greatly reducing the subsequent operation and maintenance risks and repair costs. At the same time, through the above audit mechanism, configuration conflicts, parameter errors or unreasonable resource allocation problems can be detected and avoided, improving the accuracy and reliability of job definition. In addition, through manual or automated audit intervention, the risk of misoperation is reduced, and the change process is more controllable.

[0141] Exemplary, Figure 4 A flowchart of a batch job governance method provided by an embodiment of the present application is shown in Figure 4 The batch job governance method mainly includes three parts: job configuration, job deployment and job running.

[0142] The batch job governance method includes the following steps:

[0143] Step A: new job in the job set, that is, define the job and configure the job, that is, add the job in the job strategy. Defining the job includes defining the basic information, sharding strategy, unit execution strategy, job parameters, sharding parameters and the like related to the job.

[0144] If there is no target job set required in the job set, the target job set can be newly created. If there is no target job strategy defined in the job strategy, the target job strategy can be newly added. It can be understood that if the target job set is newly created or the target job strategy is newly added, the newly created target job set or the newly added target job strategy needs to be sent to the corresponding audit platform for approval. After the approval, it can be synchronized to the target scheduler.

[0145] Step B: select the target job and the target job strategy from the defined job set in the batch job governance platform based on the user demand, and deploy the target job to the scheduler in the specified environment based on the target job strategy, such as the scheduler in the system integration testing (SIT) environment or the scheduler in the user acceptance testing (UAT) environment. The target job becomes a job to be run after deployment.

[0146] If there is no target scheduler defined in the system, the scheduler definition can also be newly added. The scheduler in different application environments corresponds to a scheduler database for storing the information of the scheduler created and defined in the batch job governance platform, and the deployment and the like.

[0147] Step C: The scheduler queries the deployment information of the to-be-run job after receiving the scheduling request, creates a job instance according to the deployment information, and issues the job instance to the target executor for running.

[0148] It should be noted that the information such as the job set, the job, the sharding strategy, the unit execution strategy, the database sharding strategy, the job parameter, the sharding parameter, and the scheduler in the foregoing steps is created and defined in the batch job governance platform, and relevant data are stored in the database of the batch job governance platform.

[0149] It can be understood that, under the distributed application architecture, the batch job governance platform, the scheduler, and the executor are all micro applications, and the databases are independent.

[0150] It should be noted that one job strategy can establish a binding relationship with one or more jobs in a job set. When the deployment operation of the job strategy is performed, the system issues all jobs associated with the job strategy and the configured strategy information to the scheduler in the specified target environment, and after the deployment is successfully completed, the state of the job strategy is updated to "deployed". The scheduler generates a corresponding to-be-run job instance according to the deployment information, stores the to-be-run job instance in the database, and enters a state of waiting to be scheduled.

[0151] In addition, for all job strategies that bind the same job set and the same version, if their target application environments and target schedulers are also the same, the system only allows one of the job strategies to be in the deployed state under this condition.

[0152] In this way, the batch jobs are processed based on the job set, the change of the job set needs to go through the corresponding audit and release process, and the deployment of the job is flexibly deployed by configuring the job strategy, thereby realizing independent governance and isolation across business systems and ensuring that the components do not affect each other. Through unified job definition and job strategy management, the definition and deployment of all jobs can be centrally managed and monitored in the batch job governance platform, and the complexity of use is simplified.

[0153] In the job definition, each shard can configure its own job parameter, and the job parameter and various execution strategy parameters are configured in a parameterized manner, so that the configuration of the job is more flexible and easy to manage, and the job strategy can also be flexibly deployed.

[0154] In the foregoing embodiments, the batch job processing method provided by the embodiments of the present application is introduced, and in order to realize each function in the method provided by the embodiments of the present application, an electronic device as an execution subject can include a hardware structure and / or a software module, and each function is realized in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether a certain function in the foregoing functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application of the technical solution and the design constraint conditions.

[0155] For example, Figure 5 A structure schematic diagram of a batch job processing device provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the batch job processing device 500 is applied to a distributed architecture system; the distributed architecture system comprises a batch job governance platform, a scheduler and an executor; the batch job governance platform is configured to configure at least one job set; each job set is configured to configure at least one type of job policy; the job policy is used to indicate parameter information required by job processing, scheduler information for sharding scheduling jobs and executor information for executing jobs; the batch job processing device 500 comprises: Figure 5 A selection module 501 configured to select a target job and a target job policy from at least one job set in the batch job governance platform based on user demand;

[0156] A determination module 502 configured to distribute the target job to at least one target scheduler in a target application environment for deployment based on the target job policy, and determine deployment information of the target job; the target job policy has a mapping relationship with the target application environment;

[0157] A creation module 503 configured to create a job instance based on the deployment information in response to a scheduling request, and distribute the job instance to a target executor for processing.

[0158] Optionally, the job policy comprises a sharding policy, a unit execution policy, a database sharding policy, a scheduler scheduling policy and a load balancing policy; the sharding policy is used to indicate sharding parameters of scheduled jobs; the unit execution policy is used to indicate executor information of executed jobs; the database sharding policy is used to indicate database information for sharding and storing the parameter information; the scheduler scheduling policy is used to indicate scheduler information of scheduled jobs; and the load balancing policy is used to indicate calling information of a target scheduler.

[0159] Optionally, the determination module 502 is specifically configured to:

[0160] process configuration information corresponding to the target job policy to determine the deployment information of the target job; the configuration information comprises at least one of the parameter information, the sharding parameters, the executor information, the database information, the scheduler information and the calling information.

[0161] Optionally, the batch job governance platform is further configured to approve at least one type of job policy configured in each job set; the batch job processing device 500 further comprises a modification module configured to:

[0162] Optionally, the batch job governance platform is further configured to approve at least one type of job policy configured in each job set; the batch job processing device 500 further comprises a modification module configured to:

[0163] In response to a modification operation of the job set and / or the job strategy by the user, approval information is generated, wherein the approval information carries the modification content.

[0164] The approval information is audited based on the batch job management platform.

[0165] After the audit is passed, it is determined whether to synchronize the modification content to the at least one target scheduler based on the first identification information of the modified job set and / or the job strategy.

[0166] Optionally, the batch job processing apparatus 500 further comprises a synchronization module, configured to:

[0167] If the first identification information and the second identification information of the target job strategy and / or the target job set to which the target job belongs are the same, the modification content is synchronized to the at least one target scheduler.

[0168] If the first identification information and the second identification information of the target job strategy and / or the target job set to which the target job belongs are different, the modified job set and / or the job strategy are stored in the batch job management platform.

[0169] Optionally, the batch job processing apparatus 500 further comprises an audit module, configured to:

[0170] When at least one job set is configured based on the batch job management platform, the pre-configured at least one job set is audited, and after the audit is passed, the configured at least one job set is generated.

[0171] It should be noted that the specific implementation principles and effects of the batch job processing apparatus 500 described above can be referred to the related description and effects of the above-mentioned embodiments, which will not be described in detail here.

[0172] The embodiments of the present application also provide an electronic device, Figure 6 A structural schematic diagram of an electronic device provided by the embodiments of the present application is shown in FIG. 6, which can include a processor 601 and a memory 602 in communication connection with the processor; the memory 602 stores a computer program; the processor 601 executes the computer program stored in the memory 602, so that the processor 601 executes the method described in any of the above-mentioned embodiments. Figure 6

[0173] The memory 602 and the processor 601 can be connected through a bus 603.

[0174] ​The embodiment of the present application further provides a computer readable storage medium, which stores computer program execution instructions, and the computer program execution instructions are used for realizing the method in any one of the preceding embodiments of the present application when executed by a processor.

[0175] The embodiment of the present application further provides a chip running an instruction, which is used for executing the method in any one of the preceding embodiments of the present application executed by an electronic device.

[0176] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program can realize the method in any one of the preceding embodiments of the present application executed by an electronic device when executed by a processor.

[0177] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiment described above is only schematic, for example, the division of the module is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be indirect coupling or communication connection through some interfaces, apparatuses or modules, and can be electrical, mechanical or other forms.

[0178] The module described as a separate component can be or can not be physically separated, and the component displayed as a module can be or can not be a physical unit, that is, can be located in one place, or can be distributed to a plurality of network units. Part or all of the modules can be selected according to actual needs to realize the embodiment scheme.

[0179] In addition, the function modules in each embodiment of the present application can be integrated in one processing unit, or each module can be physically present alone, or two or more modules can be integrated in one unit. The unit composed of the above modules can be realized in the form of hardware, or in the form of hardware plus software function unit.

[0180] The integrated module realized in the form of software function module can be stored in a computer readable storage medium. The software function module stored in a storage medium comprises a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor execute part of the steps of the method described in each embodiment of the present application.

[0181] It should be appreciated that the above processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the application can be directly embodied by hardware processor execution, or by hardware and software module combination in the processor.

[0182] The memory can include a high-speed random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0183] The bus can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0184] The aforementioned storage medium can be realized by any type of volatile or nonvolatile storage devices or a combination thereof, such as a Static Random-Access Memory (SRAM), an Electrically Erasable Programmable Read Only Memory (EEPROM), an Erasable Programmable Read-Only Memory (EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic storage, a flash memory, a magnetic disk, or an optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0185] An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can be a part of the processor. Consistent with the teachings provided herein, the processor and the storage medium can be located in an ASIC. Alternatively, the processor and the storage medium can be located in a circuit.

[0186] It is noted that, for the aforementioned method embodiments, the acts performed as part of the methods are illustrated in a particular order. However, this is not necessarily the order in which the acts are performed. For example, one act can be performed before another act in some embodiments, while in other embodiments, the same act can be performed after the other act. Moreover, it is possible that some acts can not be performed in some embodiments. Further, it is possible that some acts can be performed simultaneously. In addition, the acts can be performed by one or more entities.

[0187] Further, it is noted that, in the flowcharts, each step is represented by a block. However, this representation is merely one of many possible representations. For example, some steps can be represented by more than one block. Moreover, not all of the blocks are necessarily executed in the order shown in the flowcharts. Further, some blocks can be skipped or performed in different orders. Further, some blocks can be performed simultaneously. Further, some blocks can be performed at least in part by one or more entities.

[0188] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments. Each technical feature of the above embodiments can be combined arbitrarily, and in order to make the description concise, each technical feature in the above embodiments is not described in all possible combinations, however, as long as the combination of these technical features does not exist contradictory, it should be considered as the scope disclosed in the specification.

[0189] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0190] The above is merely specific implementation of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited to this, any change or replacement within the technical scope disclosed by the embodiments of the present application should be covered in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

1. A method of processing batch jobs, characterized by, The method is applied to a distributed architecture system, the distributed architecture system comprises a batch job governance platform, a scheduler and an executor, the batch job governance platform is configured to configure at least one job set, and each job set is configured with at least one type of job policy; The job policy is used to indicate parameter information required by job processing, scheduler information for scheduling a sharded job and executor information for executing a job; the method comprises: selecting a target job and a target job policy from at least one job set in the batch job governance platform based on user demand; deploying the target job in at least one target scheduler in a target application environment based on the target job policy, and determining deployment information of the target job; the target job policy has a mapping relationship with the target application environment; in response to a scheduling request, creating a job instance based on the deployment information, and delivering the job instance to a target executor for processing.

2. The method of claim 1, wherein, The job policy comprises a sharding policy, a unit execution policy, a database sharding policy, a scheduler scheduling policy and a load balancing policy; the sharding policy is used to indicate sharding parameters of a scheduled job; the unit execution policy is used to indicate executor information of an executed job; the database sharding policy is used to indicate database information for sharding and storing the parameter information; the scheduler scheduling policy is used to indicate scheduler information of a scheduled job; and the load balancing policy is used to indicate calling information of a target scheduler.

3. The method of claim 2, wherein, The method further comprises: processing configuration information corresponding to the target job policy to determine the deployment information of the target job; the configuration information comprises at least one of the parameter information, the sharding parameters, the executor information, the database information, the scheduler information and the calling information.

4. The method of claim 1, wherein, The batch job governance platform is further configured to approve at least one type of job policy configured in each job set; the method further comprises: in response to a modification operation of the user on the job set and / or the job policy, generating approval information; the approval information carries the modification content; auditing the approval information based on the batch job governance platform; after the audit is passed, determining whether to synchronize the modification content to the at least one target scheduler based on first identification information of the modified job set and / or the job policy.

5. The method of claim 4, wherein, The method further comprises: if the first identification information and second identification information of the target job policy and / or a target job set to which the target job belongs are the same, the modification content is synchronized to the at least one target scheduler; if the first identification information and second identification information of the target job policy and / or the target job set to which the target job belongs are different, the modified job set and / or the job policy are stored in the batch job governance platform.

6. The method according to any one of claims 1-5, characterized in that, The method further comprises: when at least one job set is configured based on the batch job governance platform, preconfigured at least one job set is audited, and after the audit is passed, the configured at least one job set is generated.

7. A batch job processing apparatus characterized by comprising: The application is applied to a distributed architecture system, the distributed architecture system comprises a batch job governance platform, a scheduler and an executor, the batch job governance platform is used for configuring at least one job set, at least one type of job strategy is configured in each job set; The job strategy is used for indicating parameter information required by job processing, scheduler information of sharding scheduling job and executor information of executing job; the device comprises: A selection module is configured to select a target job and a target job strategy from at least one job set in the batch job governance platform based on user demand; A determination module is configured to distribute the target job to at least one target scheduler in a target application environment for deployment based on the target job strategy, and determine deployment information of the target job; the target job strategy has a mapping relationship with the target application environment; A creation module is configured to create a job instance based on the deployment information in response to a scheduling request, and distribute the job instance to a target executor for processing.

8. An electronic device, comprising: Comprise: A processor, and a memory connected with the processor in communication; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to implement the method in any one of claims 1-6.

9. A distributed architecture system; characterized by, The distributed architecture system comprises a batch job governance platform, a scheduler and an executor, the batch job governance platform is used for configuring at least one job set, at least one type of job strategy is configured in each job set; the job strategy is used for indicating parameter information required by job processing, scheduler information of sharding scheduling job and executor information of executing job; the distributed architecture system is used for executing the method in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, the computer execution instructions are executed by the processor to implement the method in any one of claims 1-6.

Images