A method and system for batch task execution

Abstract

The application relates to a method and system for batch task execution, relating to the technical field of data processing, the field of distribution, the field of big data and the field of finance. The method comprises the following steps: defining parameters corresponding to a batch task; receiving and verifying parameters transmitted by a scheduling system of the batch task, and when the parameters transmitted by the scheduling system are defined, the verification is passed, and the batch task is continuously processed; otherwise, the verification fails, and the batch task execution fails; when the verification is passed, a logical table name is extracted according to the parameters transmitted by the scheduling system, and the logical table name is replaced by a physical data table name to generate an SQL containing the physical data table name; a Step Chain is formed according to a logical table and physical data table mapping rule, a logical database and physical database rule and the SQL containing the physical data table name; and the Step Chain is executed to complete the batch task. The method reduces required hardware resources, and does not exist multi-node scheduling, so that the problems of multi-node scheduling and processing result consistency do not need to be considered.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a batch processing task execution method and system. BACKGROUND

[0002] With the continuous popularization and generalization of two-dimensional codes, the scenarios of data query and data processing through two-dimensional codes gradually increase. When two-dimensional code data is downloaded for batch processing, the processing system reads two-dimensional code data for processing, and the processing result is written into a file and downloaded to a downstream system. With the development of distributed technology, the use of separate databases and tables is becoming more common, which means that the design of data download needs to consider the impact of separate databases and tables.

[0003] The processing system uses a relational database MySQL, and in view of the large amount of data in some tables, separate database and table design is performed on two-dimensional code and related tables.

[0004] At present, when two-dimensional code data is downloaded for batch processing, a set of batch processing system needs to be deployed for each two-dimensional code physical database, and a processing job needs to be configured for each two-dimensional code data table. Each set of batch processing system is regarded as a node. The design of the processing system has a large demand for hardware resources, and needs to consider the scheduling between multiple nodes and the consistency of processing results. SUMMARY

[0005] In order to solve the above technical problems existing in the prior art, the present application provides a batch processing task execution method and system, which reduces the required hardware resources and does not exist multiple node scheduling, so there is no need to consider the problem of multiple node scheduling and processing result consistency.

[0006] In a first aspect, the present application provides a batch processing task execution method, which comprises: defining each parameter corresponding to a batch processing task, wherein the each parameter corresponding to the batch processing task includes a request entity class, a structured query language SQL containing a logical table, a logical table and physical data table mapping rule, a logical database and physical database rule, and a SQL parameter type; receiving and verifying the parameters transmitted by a scheduling system of the batch processing task, and when the parameters transmitted by the scheduling system are defined, the verification is passed, and the batch processing task is continued to be processed; otherwise, the verification fails, and the batch processing task execution fails; when the verification is passed, a logical table name is extracted according to the each parameter transmitted by the scheduling system, and the logical table name is replaced by a physical data table name to generate a SQL containing the physical data table name; a Step Chain is formed according to the logical table and physical data table mapping rule, the logical database and physical database rule, and the SQL containing the physical data table name; and the Step Chain is executed to complete the batch processing task.

[0007] The method stores data in a database and a data table respectively, and can define a processing system including a plurality of Steps as a StepChain, and replace a logical table with an actual physical data table. By using the method, hardware resources are reduced, cost is lowered, and there is no multi-node scheduling. There is no need for multi-node scheduling and processing result consistency, and there is also no need for multi-node configuration content consistency. In addition, as the business grows, the system performance is improved by adding units, and the batch processing system only modifies the configuration, adds new database and table mapping rules, and does not need to add a new subsystem, which has high practicability.

[0008] In a possible implementation, the parameters corresponding to the batch processing task further include:

[0009] task name, step name prefix, data block size, text encoding format, text output field sorting, text output format, text name prefix, output path, and file transfer protocol.

[0010] In a possible implementation, the definition of the parameters corresponding to the batch processing task specifically includes:

[0011] The parameters corresponding to the batch processing task are defined in the form of key-value pairs.

[0012] In a possible implementation, the batch processing task corresponds to n physical databases, each of which includes m physical data tables, and n and m are integers greater than 1. The StepChain is formed according to the logical table and physical data table mapping rule, the logical database and physical database rule, and the SQL including the physical data table name, and specifically includes:

[0013] Step 1: defining i = 1, where i represents the i-th physical database, and defining the content of the StepChain as empty;

[0014] Step 2: determining whether i is less than n. If i > n, a file transfer protocol is defined to send a Step to form the StepChain. If i < n, j = 1 is defined, where j represents the j-th physical data table on the i-th physical database.

[0015] Step 3: determining whether j is less than m. If j > m, i is increased by 1, and the process jumps to Step 1. If j < m, a step is defined.

[0016] Step 4: adding the defined step to the StepChain, increasing j by 1, and jumping to Step 3.

[0017] In a possible implementation, the step is defined if j < m.

[0018] loading the SQL containing the physical data table name, and loading the parameter of the SQL containing the physical data table name to complete the definition of the read process of the step;

[0019] defining the processor Processor and the write process of the step.

[0020] In a possible implementation, the executing the step chain to complete the batch processing task specifically includes:

[0021] executing the Step Chain to obtain a task execution result;

[0022] sending the task execution result to a message middleware;

[0023] when the scheduling system listens to that the task execution result is received in the message middleware, obtaining the task execution result from the message middleware to complete the batch processing task.

[0024] In a possible implementation, the method further includes:

[0025] when the first k steps in the Step Chain are executed successfully and the (k+1)th step is executed unsuccessfully, the batch processing task is executed unsuccessfully, the k is a positive integer and the k is less than the product of the n and the m;

[0026] when the batch processing task is executed again, the execution starts from the (k+1)th step.

[0027] In a second aspect, the application further provides a system for batch task execution, comprising a loading verification unit, an input parameter verification unit, an SQL parsing and reconstruction unit, a step splitting unit and an execution unit. The loading verification unit is configured to define parameters corresponding to a batch task, wherein the parameters comprise a request entity class, a structured query language (SQL) containing a logical table, a logical table and physical data table mapping rule, a logical database and physical database rule, and an SQL parameter type. The input parameter verification unit is configured to receive and verify parameters transmitted by a scheduling system of the batch task. When the parameters transmitted by the scheduling system are defined, the verification passes, and the batch task is processed. Otherwise, the verification fails, and the batch task execution fails. The SQL parsing and reconstruction unit is configured to extract a logical table name according to the parameters transmitted by the scheduling system when the verification passes, and replace the logical table name with a physical data table name to generate an SQL containing the physical data table name. The step splitting unit is configured to form a step chain (Step Chain) according to the logical table and physical data table mapping rule, the logical database and physical database rule, and the SQL containing the physical data table name. The execution unit is configured to execute the Step Chain to complete the batch task.

[0028] The system stores data in a database and a data table, and can be defined as a processing system including a Step Chain including multiple Steps, and replaces a logical table with an actual physical data table. By using the system, hardware resources are reduced, costs are lowered, and multi-node scheduling does not exist. There is no need to consider multi-node scheduling and processing result consistency, and there is also no need to consider multi-node configuration content consistency. In addition, as a business grows, a unit is added to improve system performance, and a batch processing system only needs to modify a configuration, add a new database and table mapping rule, and does not need to add a new subsystem, thereby having high practicability.

[0029] In a possible implementation, the loading verification unit is specifically configured to define the parameters corresponding to the batch task in the form of key-value pairs.

[0030] In a possible implementation, the parameters corresponding to the batch task further comprise:

[0031] a task name, a step name prefix, a data block size, a text encoding format, a text output field sorting, a text output format, a text name prefix, an output path and a file transfer protocol.

[0032] In a possible implementation, the loading verification unit is specifically configured to define the parameters corresponding to the batch task in the form of key-value pairs.

[0033] In a possible implementation, the batch processing task corresponds to n physical databases, each of the physical databases includes m physical data tables, n and m are integers greater than 1, and the step splitting unit is specifically configured to implement the following steps:

[0034] Step one: defining i = 1, where i represents an i-th physical database, and defining content of the Step Chain as empty;

[0035] Step two: determining whether i is less than n, if i > n, defining a file transfer protocol to send a Step to form the Step Chain, if i < n, defining j = 1, where j represents a j-th physical data table on the i-th physical database;

[0036] Step three: determining whether j is less than m, if j > m, increasing i by 1 and jumping to the step one, if j < m, defining a step;

[0037] Step four: adding the defined step to the Step Chain, increasing j by 1, and jumping to the step three.

[0038] In a possible implementation, the execution unit is specifically configured to: execute the Step Chain to obtain a task execution result; send the task execution result to a message middleware; and when the scheduling system listens to that the task execution result is received in the message middleware, obtain the task execution result from the message middleware to complete the batch processing task.

[0039] In a possible implementation, the execution unit is further configured to: when the first k Steps in the Step Chain are executed successfully and the k+1 Step is executed unsuccessfully, the batch processing task is executed unsuccessfully, where k is a positive integer and k is less than a product of n and m; and when the batch processing task is executed again, the execution starts from the k+1 Step. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 FIG. 1 is a schematic diagram of a batch processing framework;

[0041] Figure 2 FIG. 2 is a schematic diagram of a system for executing a batch processing task;

[0042] Figure 3 FIG. 3 is a flowchart of a method for executing a batch processing task provided by an embodiment of the present application;

[0043] Figure 4 FIG. 4 is a schematic diagram of a system for executing a batch processing task provided by an embodiment of the present application;

[0044] Figure 5 Another flow chart of a method for batch task execution provided by an embodiment of the present application is provided.

[0045] Figure 6 A schematic diagram of a system for batch task execution provided by an embodiment of the present application is provided. DETAILED DESCRIPTION

[0046] In order to make the person skilled in the art more clearly understand the scheme of the present application, the application scenario of the present application technology is first described below.

[0047] Batch processing in the present application refers to, when facing complex business and mass data processing, without human intervention, only reading batch data periodically, then completing corresponding business processing and performing archiving operation.

[0048] The whole flow of batch processing can be divided into three stages:

[0049] 1. Reading data, which can come from files, databases or message queues;

[0050] 2. Processing data, processing the read data and forming output results, such as fund reconciliation processing of bank reconciliation system;

[0051] 3. Writing data, writing the output results into files, databases or message queues, etc.

[0052] Referring to Figure 1 , the figure is a schematic diagram of a batch processing framework.

[0053] The batch processing framework is Spring Batch, which is a lightweight and complete batch processing framework, and can be used to build robust and efficient batch processing applications.

[0054] Job Launcher is a task launcher, which is responsible for starting the task Job.

[0055] Job is a unit encapsulating the whole batch processing process, and running a batch processing task is to run the content defined by Job.

[0056] Step is the encapsulation of a certain process of Job, and a Job can contain one or more Steps. The Step is executed step by step according to a certain logic, which represents the completion of Job execution.

[0057] Item Reader is the read processing of resources in Step.

[0058] Item Processor often needs to read data, perform a series of business logic operations, and then write data to a specified persistence process in the development process. That is, the processing process in Step.

[0059] Item Writer is an important component for implementing writing, that is, writing resources in Step.

[0060] The Job Repository is specifically responsible for interacting with the database and recording the addition, update, and execution of the entire batch.

[0061] With the popularity of QR codes, the scenarios in which data query and data processing are performed through QR codes are gradually increasing, for example, IC cards (Integrated Circuit Cards) can be applied in the financial field.

[0062] See also Figure 2 ,This figure is a schematic diagram of a current batch task execution system.

[0063] The batch processing system uses the relational database MySQL. In view of the large amount of data in some tables, the QR code and other related tables are designed to be sharded.

[0064] The system needs to collect QR code transaction data daily and transmit it to the data download platform for other systems to subscribe. These tasks involve business processing of large amounts of data and require a batch processing application to implement. The existing implementation solution is to target each database ( Figure 2 The databases in the system (represented by c2bscan1 and c2bscan2) require a batch processing system 12 to be deployed separately. Each batch processing system 12 corresponds to the same scheduling system 11. A text generation job is required for each table in each database (icqtrnpf_0 and icqtrnpf_1). This design requires a large amount of hardware resources. Complex issues such as scheduling and result consistency across multiple batch processing systems must be considered. All batch system configurations must be highly consistent (except for databases and data tables), which presents the risk of missing configuration changes in a particular system. Horizontal expansion (increasing modularization) of the QR code business system requires the batch processing system to scale horizontally as well.

[0065] In order to solve the above problems, the embodiments of the present application provide a method and system for batch task execution. By dynamically configuring the QR code business data source, SQL parsing reconstruction, and parameter decoupling, a text generation step chain (Step Chain) is realized, which solves the problem of needing to deploy multiple batch processing systems and configure multiple JOBs, reduces resource consumption and repeated configuration, and at the same time reduces maintenance costs. In addition, there is no multi-node scheduling, so there is no need to consider the problems of multi-node scheduling and consistency of processing results.

[0066] In order to enable people skilled in the art to understand the present application more clearly, the technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0067] The words "first", "second", etc. in the description of the present application are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated

[0068] The embodiment of the present application provides a method for batch task execution, which is specifically described below with reference to the accompanying drawings.

[0069] Referring to Figure 3 The figure is a flow chart of a method for batch task execution provided by the embodiment of the present application.

[0070] The method comprises the following steps:

[0071] S301: defining each parameter corresponding to the batch task, wherein the each parameter corresponding to the batch task comprises a request entity class, a SQL containing a logical table, a logical table and physical data table mapping rule, a logical library and physical database rule, and a SQL parameter type.

[0072] SQL (Structured Query Language), that is, structured query language.

[0073] The logical library and physical database rule and the logical table and physical table mapping rule are data sources (datasource).

[0074] The each parameter corresponding to the batch task further comprises:

[0075] Defining a task (Job), including Job name, Step name prefix and (Chunk) size.

[0076] Defining a read process (Reader), including an entity class, a SQL containing a logical table, and a SQL parameter type.

[0077] Defining a write process (writer), including an encoding format, an output field sorting, and an output format.

[0078] Defining a file (File), including a text name prefix and an output path.

[0079] Defining a file transfer protocol (File Transfer Protocol, FTP), including an address (Address), a user password, a target path, etc.

[0080] In a possible implementation manner, the each parameter corresponding to the batch task is defined in the form of key-value.

[0081] The parameter definition needs to be checked for necessity, such as whether the entity class exists in the Reader configuration, whether the SQL is written normally, and whether it can be parsed normally.

[0082] S302: Receive and check the parameters passed by the scheduling system of the batch processing task. When the parameters passed by the scheduling system are defined, the check passes, and the batch processing task continues to be processed. Otherwise, the check fails, and the batch processing task execution fails.

[0083] The scheduling system initiates the Job scheduling to pass parameters, and the parameter values correspond to the key values of the configuration items.

[0084] The input parameters are checked to check whether the passed parameters exist in the configuration items in S301. If not, the check fails, and the scheduling task execution fails.

[0085] S303: When the check passes, the logical table name is extracted according to the parameters passed by the scheduling system, and the logical table name is replaced with the physical data table name to generate a SQL containing the physical data table name.

[0086] That is, the Reader SQL is parsed to extract the logical table name.

[0087] S304: According to the logical table and physical data table mapping rule, as well as the logical library and physical database rule and the SQL containing the physical data table name, a Step Chain is formed.

[0088] According to the data source (data source) parameter, the specific database and table splitting rule is matched, and then the Step is split according to the database and table splitting rule to form a Step Chain. Each Step acts on a specific physical database and physical table. This process uses a SQL parsing reconfigurator to generate corresponding physical SQL for a specific physical database and physical table.

[0089] S305: Execute the Step Chain to complete the batch processing task.

[0090] In practical applications, the batch processing task is a two-dimensional code data download task, that is, the data source is a plurality of data sources of two-dimensional code business data. The configuration in S301 is the mapping rule of two-dimensional code business data about database and table splitting. When the two-dimensional code data download task is executed, the parameters specify the above configuration, and the SQL parsing reconfiguration is used to form a separate Step for each physical library and each physical table, and to combine a Step Chain. After the text is generated, the text transmission Step is executed.

[0091] In summary, the method provided in the embodiments of the present application stores data in a database and a data table respectively, can define a processing system including a Step Chain of multiple Steps, and replaces a logical table with an actual physical data table. By using the method, hardware resources are reduced, cost is lowered, and there is no multi-node scheduling. There is no need for multi-node scheduling and consistency of processing results, and there is also no need for consistency of multi-node configuration content. In addition, with the growth of business, adding units improves system performance, and the batch processing system only modifies the configuration, adds new database and table mapping rules, and does not need to add a new subsystem, which has high practicability.

[0092] In order to make the skilled in the art more clearly understand the technical solutions of the present application, first, the implementation of the system for executing the batch processing task is introduced.

[0093] Referring to Figure 4 The figure is a schematic diagram of a system for executing a batch processing task provided in the embodiments of the present application.

[0094] The system configures multiple data sources of the two-dimensional code business data in a dynamic data source manner, configures the two-dimensional code business data in a manner of combination of mapping rules, SQL parsing reconstruction, and parameterization decoupling, and realizes a single system batch processing mechanism for the two-dimensional code data download.

[0095] The configuration file defines multiple physical data sources (dataSource), defines the dataSource (including logical library and physical database rules, logical table and physical data table mapping rules), defines the Spring Batch Job (including Job name, Step name prefix, Chunk size), defines the Spring Batch Reader (including request entity class, SQL containing logical table, SQL parameter type), defines the Spring Batch Writer (including text encoding format, text output field sorting, text output format-fixed length or separator), defines the File (including text name prefix, output path), defines the Ftp (including address, user password, target path), and the like, and all parameter configurations are completely decoupled.

[0096] When the two-dimensional code data download task is executed, the parameters specify the above configurations, SQL parsing reconstruction is used, a separate Step is formed for each physical table of each physical database, and the Steps are combined into a Step Chain; after the text generation is completed, the text transmission Step is executed. The entire task supports breakpoint retry.

[0097] The working principle and implementation of each module in the system 40 for executing the batch processing task are described below.

[0098] The system 40 in which the batch processing task is executed includes a batch processing system 42, a database and table splitting unit 43, and a SQL parsing and reconfigurator 44.

[0099] The batch processing system 42 includes a loading verifier 421 and an input parameter verifier 422.

[0100] The loading verifier 421 defines a plurality of physical data sources, defines a data source (including logical library and physical database rules, logical table and physical data table mapping rules), defines a job (including job name, step name prefix, chunk size), defines a reader (including entity class, SQL containing logical table, SQL parameter type), defines a writer (including encoding format, output field order, output format), defines a file (including text name prefix, output path), defines an Ftp (including address, user password, target path), and the like. Each of the above definitions is defined in the form of key-value (key-value pair), that is, according to the key value, the key is the key, and the value is the value, to ensure high concurrent read-write performance.

[0101] The loading verifier 421 loads the configuration and performs necessary verification, such as whether the entity class in the reader configuration exists, whether the SQL writing is normal, and whether it can be normally parsed.

[0102] The input parameter verifier 422 is used to verify the input parameter (Input Parameter), and the input parameter is sent by the scheduling system 41.

[0103] When the scheduling system 41 initiates job scheduling, it will pass parameters (the parameter value corresponds to the key value of the configuration item), for example:

[0104] "jobParameterStr":

[0105] "{ "job":"c2bscan_icqtrnpf", "reader":"c2bscan_icqtrnpf", "datasource":"c2bscan", "writer":"trn", "file":"icqtrnpf", "ftp":"icqtrnpf"}"

[0106] "job(against the key value of the configuration Job)": "c2bscan_icqtrnpf",

[0107] "reader(against the key value of the configuration Reader)": "c2bscan_icqtrnpf",

[0108] "datasource(against the key value of the configuration DataSource)": "c2bscan",

[0109] "writer(against the key value of the configuration Writer)": "trn",

[0110] "file(against the key value of the configuration File)": "trn",

[0111] "batchdate":"20210101",

[0112] "ftp(against the key value of the configuration Ftp)": "trn"

[0113] },

[0114] The input parameter verifier 422 is used to check the input parameters. When the input parameters are checked, it is checked whether the passed parameters exist in the configuration item. If not, the verification fails, and the task execution fails.

[0115] The SQL parsing reconfigurator 44 parses the Reader SQL and extracts the logical table name. The logical table name is replaced with the physical data table name to generate a new SQL containing the physical data table name.

[0116] The STEP splitter 423 matches the specific database and table splitting rule according to the datasource of the input parameters, and then splits the Step according to the database and table splitting rule to combine into a Step Chain. Each Step acts on a specific physical database and physical data table. This process uses the SQL parsing reconfigurator 44 to generate the corresponding physical SQL for the specific physical database and physical data table.

[0117] The following describes the specific implementation steps of the method for executing the batch processing task.

[0118] Referring to Figure 5 , the figure is a flowchart of another method for executing a batch processing task provided by an embodiment of the present application.

[0119] The method includes the following steps:

[0120] S501: The Job task is scheduled and parameter verification is performed.

[0121] The scheduling system initiates the Job task scheduling and passes parameters. The parameter value corresponds to the key value of the configuration item. The input parameter verifier is used to check the input parameters.

[0122] S502: Match the preset database and table splitting rule. There are n physical databases, and each physical database has m sub-tables.

[0123] Wherein, n and m are integers greater than 0.

[0124] The Step splitter matches the custom database and table splitting mapping rule. The two-dimensional code data corresponds to n physical databases (c2bscan1, c2bscan2,..., c2bscan_n).

[0125] Each database has m sub-tables (icqtrnpf_1, icqtrnpf_2,..., icqtrnpf_m).

[0126] Therefore, it can be disassembled into n*m text generation Steps, and added to the Step Chain.

[0127] S503: Define i=1, and determine whether i is greater than n.

[0128] Define i=1, i represents the i-th physical database c2bscan_i. Define Step Chain, which is empty.

[0129] Determine whether i is greater than n. If i>n, it means that the Step Chain is successfully generated. Then, define the Ftp sending Step, and execute S517. Otherwise, execute S504.

[0130] S504: Define j=1.

[0131] Define j=1, j represents the j-th physical sub-table icqtrnpf_j on the i-th physical database c2bscan_i.

[0132] S505: Determine whether j is greater than m.

[0133] Determine whether j is greater than m. If j>m, execute S516. If j<m, execute S505 and define the specific Step.

[0134] S506: Define step start.

[0135] S507: Define step reader start.

[0136] S508: Load the i-th physical database.

[0137] S509: Parse SQL and obtain the logical table name.

[0138] S510: Reconstruct SQL, replace the logical table name with the j-th physical table name.

[0139] S511: Load SQL and load SQL parameters.

[0140] S512: Define step reader end.

[0141] S513: Define step processor and step writer.

[0142] S514: Define step end, and add the step to the step chain.

[0143] S515: j = j + 1.

[0144] The text generation Step configures the physical database c2bscan_i, parses and reconstructs the SQL containing the physical sub-table icqtrnpf_j, and configures the SQL execution parameter; after the text generation Step is added to the Step Chain, j is incremented by 1.

[0145] S516: i = i + 1.

[0146] S517: define the Ftp sending Step.

[0147] File Transfer Protocol (FTP) is a standard protocol for file transfer over a network.

[0148] S518: execute the Job.

[0149] S519: end.

[0150] In the above method, the Step Chain execution supports breakpoint retry. The Step Chain has n*m text generation Steps, in a Job task scheduling process, the first k Steps are executed successfully, the k+1 Step fails, the subsequent Steps will not be executed, and the entire Job fails. The Job task is triggered again, and the first k Steps are directly skipped, and the execution starts from the k+1 Step.

[0151] The Job execution result can realize asynchronous notification, the scheduling system executes the two-dimensional code data download task, and the message middleware waits for processing results. After the Job is executed, whether it is failed or successful, the processing result is written into the message middleware, and the scheduling system processes. The message middleware is an important component in a distributed system, mainly solving application coupling, asynchronous message, traffic clipping, etc. It can realize high performance, high availability, scalability and eventual consistency architecture. The specific implementation method of the message middleware is a relatively mature technology, and the embodiments of the present application will not be described here.

[0152] In summary, the method provided in the embodiments of the present application stores two-dimensional code data in n databases respectively, each database corresponds to m subtables, and the processing system can be defined as a Step Chain including n*m Steps. The SQL parsing reconfigurator replaces the logical table with an actual physical data table, and the processing result is notified asynchronously to avoid timeout. Therefore, the method provided in the embodiments of the present application reduces hardware resources and costs, and does not require multi-node scheduling. The problems of multi-node scheduling and consistency of processing results do not exist, and the problem of consistency of multi-node configuration content does not exist. In addition, the two-dimensional code system responds to business growth, increases units to improve system performance, and the batch processing system only needs to modify the configuration, add new sublibrary and subtable mapping rules, and does not need to add a new subsystem.

[0153] Based on the method for executing a batch processing task provided in the above embodiments, the embodiments of the present application further provide a device for executing a batch processing task, which will be specifically described below with reference to the accompanying drawings.

[0154] Referring to Figure 6 , the figure is a schematic diagram of a device for executing a batch processing task provided in the embodiments of the present application.

[0155] The device provided in the embodiments of the present application includes a loading verification unit 601, a parameter verification unit 602, a SQL parsing reconfiguration unit 603, a step splitting unit 604, and an execution unit 605.

[0156] The loading verification unit 601 corresponds to the loading verifier 421 in Figure 4 , and is configured to define parameters corresponding to a batch processing task. The parameters corresponding to the batch processing task include a request entity class, a structured query language (SQL) containing a logical table, a logical table and physical data table mapping rule, a logical database and physical database rule, and a SQL parameter type.

[0157] The parameter verification unit 602 corresponds to the parameter verifier 422 in Figure 4 , and is configured to receive and verify parameters transmitted by a scheduling system of a batch processing task. When the parameters transmitted by the scheduling system are defined, the verification passes, and the batch processing task continues to be processed. Otherwise, the verification fails, and the batch processing task fails to be executed.

[0158] The SQL parsing reconfiguration unit 603 corresponds to the SQL parsing reconfigurator 44 in Figure 4 , and is configured to extract a logical table name according to the parameters transmitted by the scheduling system when the verification passes, and replace the logical table name with a physical data table name to generate a SQL containing the physical data table name.

[0159] The step splitting unit 604 corresponds to the step splitting unit 44 in Figure 4a STEP splitter 423 in the logic unit 401, configured to form a step chain according to the logical table-physical data table mapping rule, the logical database-physical database rule, and the SQL containing the physical data table name;

[0160] the execution unit 605, configured to execute the step chain to complete the batch processing task.

[0161] In a possible implementation, the loading and verifying unit 601 is specifically configured to define parameters corresponding to the batch processing task in the form of key-value pairs.

[0162] In a possible implementation, the step splitting unit 604 is specifically configured to implement the following steps:

[0163] Step one: defining i=1, where i represents the i th physical database, and defining the content of the step chain as empty;

[0164] Step two: determining whether i is less than n. If i>n, a file transfer protocol is defined to send a step, so as to form the step chain. If i<n, defining j=1, where j represents the j th physical data table on the i th physical database;

[0165] Step three: determining whether j is less than m. If j>m, i is increased by 1, and the step one is jumped to. If j<m, a step is defined;

[0166] Step four: adding the defined step to the step chain, increasing j by 1, and jumping to the step three.

[0167] In a possible implementation, the execution unit 605 is specifically configured to: execute the step chain to obtain a task execution result; send the task execution result to a message middleware; and when the scheduling system listens to the message middleware and receives the task execution result, obtain the task execution result from the message middleware, so as to complete the batch processing task.

[0168] In a possible implementation, the execution unit 605 is further configured to: when the first k steps in the step chain are executed successfully and the k+1 th step fails, the batch processing task fails, where k is a positive integer and k is less than the product of n and m; and when the batch processing task is executed again, the execution starts from the k+1 th step.

[0169] The system provided by the embodiment of the application stores data in a database and a data table respectively, and can be defined as a processing system including a Step Chain of multiple Steps, and replaces a logical table with an actual physical data table. By using the system, hardware resources are reduced, cost is lowered, and there is no multi-node scheduling. There is no need to consider multi-node scheduling and processing result consistency, and there is also no need to consider multi-node configuration content consistency. In addition, with the growth of business, the system performance is improved by adding units, and the batch processing system only needs to modify the configuration, add new database and table mapping rules, and does not need to add a new subsystem, and has high practicability.

[0170] The system can be run by an electronic device including a processor and a memory. The loading verification unit 601, the input parameter verification unit 602, the SQL analysis and reconstruction unit 603, the step splitting unit 604, and the execution unit 605 are all stored in the memory as program units, and the corresponding functions are implemented by the processor executing the program units stored in the memory.

[0171] The processor includes a core, and the core retrieves the corresponding program unit from the memory. The core can be one or more, and the batch processing task execution is implemented by adjusting the core parameters.

[0172] The memory can include a non-permanent memory in a computer readable medium, a random access memory (RAM), and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory includes at least one memory chip.

[0173] The embodiment of the application provides a storage medium having a program stored thereon, and the program is executed by a processor to implement the batch processing task execution method.

[0174] The embodiment of the application provides a processor for running a program, wherein the program is executed to implement the batch processing task execution method.

[0175] The embodiment of the application provides a device including a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements the batch processing task execution method when executing the program. The device in the application can be a server, a PC, a PAD, a mobile phone, and the like.

[0176] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0177] It should be noted that the method and system for batch task execution provided by the present application can be used in the fields of data processing technology, distribution, big data, finance, etc. The above are only examples and do not limit the application of the method and system for batch task execution provided by the present application.

[0178] It should be understood that in the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0179] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. The device embodiment described above is only illustrative, and the units and modules described as separate components can be or can not be physically separated. In addition, part or all of the units and modules can be selected according to actual needs to achieve the purpose of the embodiment. Those skilled in the art can understand and implement without creative labor.

[0180] The above is only a specific embodiment of the present application. It should be noted that those skilled in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A method for executing batch tasks, characterized in that: The method includes: Defining each parameter corresponding to the batch processing task, where each parameter corresponding to the batch processing task includes a request entity class, a Structured Query Language (SQL) containing logical tables, a mapping rule between the logical table and the physical data table, a rule between the logical database and the physical database, and an SQL parameter type; Receiving and verifying the parameters passed by the scheduling system of the batch processing task. When the parameters passed by the scheduling system are defined, the verification passes, and the batch processing task continues to be processed; otherwise, the verification fails and the batch processing task fails to execute; When the verification passes, extracting the logical table name according to each parameter passed by the scheduling system, and replacing the logical table name with the physical data table name to generate an SQL containing the physical data table name; Forming a Step Chain according to the mapping rule between the logical table and the physical data table, the rule between the logical database and the physical database, and the SQL containing the physical data table name; Executing the Step Chain to complete the batch processing task; The defining of each parameter corresponding to the batch processing task specifically includes: defining each parameter corresponding to the batch processing task in the form of key-value pairs; The batch processing task corresponds to n physical databases, and each physical database includes m physical data tables. The n and m are integers greater than 1. The forming of the Step Chain according to the mapping rule between the logical table and the physical data table, the rule between the logical database and the physical database, and the SQL containing the physical data table name specifically includes: Step 1: Defining i = 1, where i represents the i-th physical database, and defining the content of the Step Chain as empty; Step 2: Judging whether i is less than n. If i > n, defining a file transfer protocol sending step to form the Step Chain; if i < n, defining j = 1, where j represents the j-th physical data table on the i-th physical database; Step 3: Judging whether j is less than m. If j > m, increasing i by 1 and jumping to Step 1; if j < m, defining a step; Step 4: Adding the defined step to the Step Chain, increasing j by 1, and jumping to Step 3.

2. The method for executing batch tasks according to claim 1, wherein: Each parameter corresponding to the batch processing task further includes: Task name, step name prefix, data block size, text encoding format, text output field sorting, text output format, text name prefix, output path, and file transfer protocol.

3. The method for executing batch tasks according to claim 1, wherein: The if j < m, then defining a step; Loading the SQL containing the physical data table name, and loading the parameters of the SQL containing the physical data table name to complete the definition of the read processing of the step; Defining the processor and write processing of the step.

4. The method for executing batch tasks according to claim 3, wherein: The executing of the step chain to complete the batch processing task specifically includes: Executing the Step Chain to obtain the task execution result; Sending the task execution result to the message middleware; When the scheduling system monitors that the task execution result is received in the message middleware, it obtains the task execution result from the message middleware to complete the batch task.

5. The method for executing batch tasks according to claim 4, wherein: The method further includes: When the first k Steps in the Step Chain are successfully executed and the (k + 1)-th Step fails, the batch task fails. Here, k is a positive integer and k is less than the product of n and m. When the batch task is executed again, it starts from the (k + 1)-th Step.

6. A system for executing batch tasks, characterized in that: The system includes: a loading verification unit, an input parameter verification unit, an SQL parsing and reconstruction unit, a step splitting unit, and an execution unit; The loading verification unit is used to define each parameter corresponding to the batch task. Each parameter corresponding to the batch task includes a request entity class, a structured query language SQL containing logical tables, a mapping rule between the logical table and the physical data table, a rule between the logical database and the physical database, and an SQL parameter type. The input parameter verification unit is used to receive and verify the parameters passed by the scheduling system of the batch task. When the parameters passed by the scheduling system are defined, the verification passes and the batch task is continued to be processed; otherwise, the verification fails and the batch task fails. The SQL parsing and reconstruction unit is used to, when the verification passes, extract the logical table names according to the parameters passed by the scheduling system, and replace the logical table names with the physical data table names to generate an SQL containing the physical data table names. The step splitting unit is used to form a step chain Step Chain according to the mapping rule between the logical table and the physical data table, the rule between the logical database and the physical database, and the SQL containing the physical data table names. The execution unit is used to execute the Step Chain to complete the batch task; The loading verification unit is specifically used to define each parameter corresponding to the batch task in the form of key-value pairs. The batch task corresponds to n physical databases, and each physical database includes m physical data tables. n and m are integers greater than 1. The step splitting unit is specifically used to implement the following steps: Step 1: Define i = 1. Here, i represents the i-th physical database, and define the content of the Step Chain as empty. Step 2: Determine whether i is less than n. If i > n, define a file transfer protocol sending Step to form the StepChain; if i < n, define j = 1. Here, j represents the j-th physical data table on the i-th physical database. Step 3: Determine whether j is less than m. If j > m, increase i by 1 and jump to Step 1; if j < m, define step. Step 4: Add the defined step to the Step Chain, increase j by 1, and jump to Step 3.

Images