A method and apparatus for batch processing of services

Abstract

The present invention provides a method and apparatus for batch processing of services, which can be used in the financial field or other technical fields. The method includes: generating a batch processing flow chart according to the batch configuration information of each service of a service unit; generating a batch processing program of the service unit according to the batch processing flow chart, the batch program templates of each service of the service unit, and the batch configuration information; executing the batch processing program of the service unit; wherein, the batch configuration information and batch program templates of each service of the service unit are preset. The apparatus is used to execute the above method. The method and apparatus for batch processing of services provided by the embodiments of the present invention improve the efficiency of batch processing.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a method and device for batch processing of services. Background Art

[0002] Currently, the application scenarios of the database of application systems in financial institutions such as banks can generally be divided into two categories: online transactions and batch processing.

[0003] The existing batch processing mode is usually organized according to the dimension of applications, and each application completes its own batch processing. The current application obtains data files from the upstream application, imports them into the database inside the application, and then performs batch processing. The processing results are exported as data files and then transmitted to the downstream application, finally completing all batch processing of the overall business. During the batch processing, the batch processing results are transmitted between the upstream and downstream applications in the form of data files. The transmission of a large number of data files brings a large resource overhead to storage and network, and a large number of intermediate processing processes and cross-application file delivery processes prolong the time required for batch processing and reduce the efficiency of batch processing. Summary of the Invention

[0004] In view of the problems in the prior art, embodiments of the present invention provide a method and device for batch processing of services, which can at least partially solve the problems existing in the prior art.

[0005] In a first aspect, the present invention proposes a method for batch processing of services, including:

[0006] Generating a batch processing flow chart according to the batch configuration information of each service of the service unit;

[0007] Generating a batch processing program of the service unit according to the batch processing flow chart, the batch program templates of each service of the service unit, and the batch configuration information;

[0008] Executing the batch processing program of the service unit;

[0009] Wherein, the batch configuration information and batch program templates of each service of the service unit are preset.

[0010] In a second aspect, the present invention provides a device for batch processing of services, including:

[0011] A first generation module, configured to generate a batch processing flow chart according to the batch configuration information of each service of the service unit;

[0012] A second generation module, configured to generate a batch processing program of the service unit according to the batch processing flow chart, the batch program templates of each service of the service unit, and the batch configuration information;

[0013] An execution module for executing the batch processing program of the service unit;

[0014] Among them, the batch configuration information and batch program templates of each service of the service unit are preset.

[0015] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the batch processing method of the service described in any of the above embodiments.

[0016] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the batch processing method of the service described in any of the above embodiments.

[0017] In a fifth aspect, the present invention provides a computer program product. The computer program product includes a computer program. When the computer program is executed by a processor, it implements the batch processing method of the service described in any of the above embodiments.

[0018] The batch processing method and device for services provided by the embodiments of the present invention can generate a batch processing flow chart according to the batch configuration information of each service of the service unit, generate a batch processing program for the service unit according to the batch processing flow chart, the batch program templates and batch configuration information of each service of the service unit, and execute the batch processing program of the service unit. Regarding the services related in the service processing flow as an independent and complete service unit, overall batch scheduling is performed, optimizing the batch processing flow and improving the efficiency of batch processing. Description of the Drawings

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. In the drawings:

[0020] Figure 1 It is a schematic diagram of the principle of the batch processing method for services provided by the first embodiment of the present invention.

[0021] Figure 2 It is a schematic flowchart of the batch processing method for services provided by the second embodiment of the present invention.

[0022] Figure 3 It is a schematic flowchart of the batch processing method for services provided by the third embodiment of the present invention.

[0023] Figure 4 It is a schematic diagram of the batch processing flow chart provided by the fourth embodiment of the present invention.

[0024] Figure 5 It is a schematic flow diagram of the batch processing method of the service provided by the fifth embodiment of the present invention.

[0025] Figure 6 It is a schematic diagram of the architecture of the data transmission platform provided by the sixth embodiment of the present invention.

[0026] Figure 7 It is a schematic diagram of the batch processing flow chart provided by the seventh embodiment of the present invention.

[0027] Figure 8 It is a schematic structural diagram of the batch processing device of the service provided by the eighth embodiment of the present invention.

[0028] Figure 9 It is a schematic structural diagram of the batch processing device of the service provided by the ninth embodiment of the present invention.

[0029] Figure 10 It is a schematic structural diagram of the batch processing device of the service provided by the tenth embodiment of the present invention.

[0030] Figure 11 It is a schematic structural diagram of the batch processing device of the service provided by the eleventh embodiment of the present invention.

[0031] Figure 12 It is a schematic diagram of the physical structure of the electronic device provided by the twelfth embodiment of the present invention. Detailed implementation manners

[0032] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer and more understandable, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily.

[0033] To facilitate understanding of the technical solutions provided by the present application, the relevant content of the technical solutions of the present application will be described below first.

[0034] Business service unit: A business service unit is a self - contained set of services involved in a business processing flow. This set includes the services required for a business processing flow and the data to be processed. Viewed vertically according to functional modules, the processing of a business process involves multiple applications, and one application can provide multiple services. Viewed horizontally according to business processes, one business can include multiple business processing flows, and one business processing flow can span multiple services in multiple upstream and downstream applications.

[0035] Batch program template: A batch program template is a parameterized batch program that is pre - set. The processing processes of a batch of batch programs executed in parallel are the same, and only the data objects to be processed are divided by batches. Therefore, the batch numbers and the data files or data tables accessed in the batch program can be parameterized. When releasing the batch version, only the batch program template needs to be released. According to the batch configuration information, the template is instantiated, and a batch of parallel executable batch programs are instantiated according to the batch number and the data corresponding to that batch.

[0036] Batch configuration information: Various information required for batch program scheduling, including which service the batch belongs to, which business service unit it belongs to, how many batches are executed in parallel, what conditions need to be met before execution, what operations need to be performed after execution, what data needs to be provided by upstream batches, and what data needs to be opened to downstream batches.

[0037] Figure 1 It is a schematic diagram of the principle of the batch processing method of the business provided by the first embodiment of the present invention. As Figure 1 shown, in combination with enterprise application architecture design, from the business dimension, according to the principle of "high cohesion and low coupling", the business processing flow is corresponded to the business service unit, such as the reconciliation service unit, the interest settlement service unit, the salary disbursement service unit, the accounting service unit, etc. A business service unit is an independent self - contained set of upstream and downstream processing links related to a certain type of business, which includes the services and related data required to complete the business processing flow. For example, the reconciliation service unit involves deposit services, transfer services, reconciliation services, etc. The data required to process the deposit service, the data required to process the transfer service, the data required to process the reconciliation service, etc. can be stored in the batch - specific database cluster of the reconciliation service unit, so that when the batch processing platform (BPMP) executes the batch processing program of the reconciliation service unit, the relevant data can be called; the reconciliation and accounting service unit involves XX services, information query, accounting services, etc. The data required to process the XX service, the data required to process the information query, the data required to process the accounting service, etc. can be stored in the batch - specific database cluster of the reconciliation and accounting service unit, so that when BPMP executes the batch processing program of the reconciliation service unit, the relevant data can be called.

[0038] Each business service unit is equipped with a dedicated batch database cluster. For each service in a business service unit, the corresponding data can be replicated to the dedicated batch database cluster in near real-time through the Data Transmission Platform (DTP).

[0039] The batch program template is a parameterized batch program. In the batch program, the batch number and the data shard number accessed can be represented by parameter placeholders. Each service in a business service unit corresponds to a batch program template, and the batch program template is set according to the service. The embodiments of the present invention do not make any limitations. The batch configuration information may include the business service unit to which it belongs, the service to which it belongs, the batch number, the pre-dependency conditions, the post-processing rules, the required upstream data shards, the data shards opened to the downstream, etc., and is set according to actual needs. The embodiments of the present invention do not make any limitations. Both the batch program template and the batch configuration information are pre-set before the batch processing of the business is carried out.

[0040] The batch processing management platform can be a server or a server cluster, which is set according to actual needs. The embodiments of the present invention do not make any limitations. The data transmission platform can be a server or a server cluster, which is set according to actual needs. The embodiments of the present invention do not make any limitations.

[0041] Taking the batch processing management platform as the execution entity as an example, the specific implementation process of the batch processing method for the business provided by the embodiments of the present invention will be described below. It can be understood that the execution entity of the batch processing method for the business provided by the embodiments of the present invention is not limited to the batch processing management platform.

[0042] Figure 2 It is a schematic flowchart of the batch processing method for the business provided by the second embodiment of the present invention. As Figure 2 shown, the batch processing method for the business provided by the embodiments of the present invention includes:

[0043] S201. Generate a batch processing flowchart according to the batch configuration information of each service in the business service unit;

[0044] Specifically, the batch processing management platform obtains the batch configuration information of each service in the business service unit, and then performs batch scheduling and orchestration according to the batch configuration information of each service in the business service unit to determine the logical order of processing of each service, and generates a batch processing flowchart. Among them, the batch configuration information of each service in the business service unit is pre-set and is set according to actual needs. The embodiments of the present invention do not make any limitations.

[0045] S202. Generate a batch processing program for the business service unit according to the batch processing flowchart, the batch program templates of each service in the business service unit, and the batch configuration information;

[0046] Specifically, the batch processing management platform obtains the batch program templates of each service of the business service unit, obtains relevant data from the batch configuration information of each service to replace the relevant parameters in the batch program templates of each service, and instantiates the batch program templates into batch programs to obtain the batch processing programs of each service. Then, the batch processing programs of each service are concatenated according to the batch processing flow chart to generate the batch processing program of the business service unit. Among them, the batch program templates of each service of the business service unit are preset.

[0047] S203. Execute the batch processing program of the business service unit;

[0048] Specifically, after obtaining the batch processing program of the business service unit, the batch processing management platform will execute the batch processing program of the business service unit, call the applications of each service of the business service unit to process data, so as to complete the processing of the business processing flow corresponding to the business service unit.

[0049] The batch processing method of the service provided by the embodiment of the present invention can generate a batch processing flow chart according to the batch configuration information of each service of the business service unit, generate a batch processing program of the business service unit according to the batch processing flow chart, the batch program templates and the batch configuration information of each service of the business service unit, and execute the batch processing program of the business service unit, regard the relevant services in the business processing flow as an independent and complete business service unit, and perform batch scheduling as a whole, optimize the batch processing flow, and improve the efficiency of batch processing.

[0050] Figure 3 is a schematic flowchart of the batch processing method of the service provided by the third embodiment of the present invention. As Figure 3 shown, on the basis of the above embodiments, further, the generating a batch processing flow chart according to the batch configuration information of each service of the business service unit includes:

[0051] S301. Obtain the pre-dependency conditions and post-processing rules corresponding to each service from the batch configuration information of each service; among them, the batch configuration information includes pre-dependency conditions and post-processing rules;

[0052] Specifically, the batch processing management platform obtains the pre-dependency conditions and post-processing rules corresponding to each service from the batch configuration information of each service. Among them, the batch configuration information includes pre-dependency conditions and post-processing rules. It can be understood that if a certain service does not require pre-dependency conditions, the pre-dependency conditions can be set to be empty. If a certain service does not require post-processing rules, the post-processing rules can be set to be empty.

[0053] Among them, the precondition dependencies refer to the conditions that need to be satisfied before the batch processing program of the service is executed. For example, it is required that the batch processing program of the upstream service of the current service is executed, and the upstream data is open to the batch execution program of the current service. The post-processing rules refer to the processing that needs to be performed after the batch processing program of the service is executed. For example, which service's batch processing program will be executed sequentially next, and several service's batch processing programs will be selected and executed according to the branch selection conditions. Among them, the precondition dependencies and the post-processing rules are set according to actual needs, and are not limited in the embodiments of the present invention. The branch selection conditions are set according to actual needs, and are not limited in the embodiments of the present invention.

[0054] S302. Determine the processing order of each service according to the precondition dependencies and the post-processing rules corresponding to each service, and generate a batch processing flow chart.

[0055] Specifically, the batch processing management platform can determine the processing order of each service according to the precondition dependencies and the post-processing rules corresponding to each service, and connect each service in series according to the processing order to generate a batch processing flow chart.

[0056] For example, the business service unit 1 includes four services: service A, service B, service C, and service D. According to the precondition dependencies and the post-processing rules of service A, service B, service C, and service D, it is determined that service A, service B, service C, and service D are processed in sequence, and the generated batch processing flow chart is as Figure 4 shown.

[0057] Figure 5 is a schematic flow chart of the batch processing method of the business provided by the fifth embodiment of the present invention. As Figure 5 shown, on the basis of the above embodiments, further, the generating the batch processing program of the business service unit according to the batch processing flow chart and the batch program templates and batch configuration information of each service of the business service unit includes:

[0058] S501. Obtain the batch information and batch processing data information corresponding to each service from the batch configuration information of each service; among them, the batch configuration information includes batch information and batch processing data information;

[0059] Specifically, the batch processing management platform obtains the batch information and batch processing data information corresponding to each service from the batch configuration information of each service. The batch information determines how many batches of data a single service needs to process, and the batch processing data information includes the sources of the data processed in each batch. Among them, the batch configuration information includes batch information and batch processing data information.

[0060] For example, the batch information corresponding to Service A includes 100 batch numbers, from 1 to 100, so 100 batches of data need to be processed. The batch processing data information corresponding to Service A includes the data sources corresponding to each of the 100 batch numbers. For example, the first batch number corresponds to the first column of Data Table A, the second batch number corresponds to the second column of Data Table A, the third batch number corresponds to the third column of Data Table A, and so on. The second batch number corresponds to the 100th column of Data Table A.

[0061] S502. Generate a batch processing program for each service according to the batch information, batch processing data information corresponding to each service, and the batch program template of each service;

[0062] Specifically, the batch processing management platform replaces the corresponding parameters in the batch program template of each service with the batch information corresponding to each service, and replaces the corresponding parameters in the batch program template of each service with the batch processing data information corresponding to each service, generates a processing program for each batch of processing, instantiates the batch program template into a batch processing program, and obtains the batch processing program of each service.

[0063] For example, the batch information corresponding to Service A includes 100 batch numbers, and the batch processing data information corresponding to Service A includes the data sources corresponding to each of the 100 batch numbers. Replace a batch number with the parameter corresponding to the batch number in the batch program template of Service A, and replace the parameter corresponding to the batch processing data information in the batch program template of Service A with the data source corresponding to the batch number, generate a processing program, and repeat the above process to generate a processing program for each batch number. The 100 obtained processing programs are the batch processing programs of Service A, and the above 100 processing programs can be executed in parallel.

[0064] S503. Generate the batch processing program of the business service unit according to the batch processing flow chart and the batch processing programs of each service.

[0065] Specifically, the batch processing management platform determines the execution order of the batch processing programs of each service according to the processing order of each service in the batch processing flow chart, and obtains the batch processing program of the business service unit.

[0066] Based on the above embodiments, further, the generating the batch processing program of the business service unit according to the batch processing flow chart and the batch processing programs of each service includes:

[0067] If there is a sequential execution relationship between the batch processing programs of two services, a virtual batch task is set between the batch processing programs of the two services.

[0068] Specifically, if it is confirmed from the batch processing flow chart that two services are processed sequentially, then the batch processing programs of these two services have a sequential execution relationship, that is, the batch processing program of one service is executed first, and after the execution is completed, the batch processing program of the other service is executed. At this time, a virtual batch task can be set between the batch processing programs of these two services to ensure the sequential execution of the batch processing programs of these two services.

[0069] For example, Service A and Service B are two sequentially processed services. A virtual batch task X is set between the batch processing program of Service A and the batch processing program of Service B. When the batch processing program of the business service unit is executed, the batch task X will query whether the batch processing program of Service A has been executed. After the batch processing program of Service A is executed, the virtual batch task X will trigger the execution of the batch processing program of Service B.

[0070] Based on the above embodiments, further, the execution of the batch processing program of the business service unit includes:

[0071] Access the batch dedicated database of the business service unit to call relevant data to execute the batch processing program of the business service unit; wherein, the batch dedicated database of the business service unit is established in advance and stores the data required for the execution of the batch processing program of the business service unit.

[0072] Specifically, when the batch processing management platform executes the batch processing program of the business service unit, it will access the batch processing program of the business service unit to query relevant data for the execution of the batch processing programs of each service. Wherein, the batch dedicated database of the business service unit is established in advance and stores the data required for the execution of the batch processing program of the business service unit.

[0073] After the batch configuration information of each service of the business service unit is set, the data required for the execution of the batch processing program of the business service unit is determined. The batch processing data information included in the batch configuration information of each service corresponds to the data required for the execution of the batch processing program of the service unit. Through the data transmission platform, the data corresponding to the batch processing data information included in each service of the business service unit is synchronized from the database corresponding to each service to the batch dedicated database of the business service unit respectively, so as to facilitate the call when the batch processing program of the business service unit is executed.

[0074] All the data related to batch processing of the business service unit is aggregated into the batch-specific database of the business service unit. The corresponding relationship between the batch processing program and the data in the batch-specific database is set through the batch configuration information of each service in the business service unit. When the batch processing program of the business service unit is executed, the required data can be obtained from the batch-specific database of the business service unit, avoiding the large-scale batch file transfer between upstream and downstream applications, reducing the operations of data transformation, export, transmission, decoding, and import for a large volume of data, and reducing the CPU, network, and storage resources consumed by these intermediate operation links.

[0075] In the batch processing mode of the prior art, if a piece of data needs to be processed sequentially by N upstream and downstream applications, each application on the processing chain needs to export a copy of the data and pass it to the downstream, and temporary storage space needs to be reserved for each application, and the data storage space is enlarged by N times. For this scenario, due to the batch-specific database of the business service unit established in advance in this application, only one copy of the data needs to be stored in the batch-specific database for the batch processing program of the business service unit to call, avoiding the problem of data storage magnification in the traditional batch processing mode and saving storage space.

[0076] In the batch processing mode of the prior art, batch processing is directly executed on the databases corresponding to each service. To avoid that large-scale batch processing consumes too many system resources and affects the stable operation of real-time customer online transactions, batch processing usually selects the time period with the least business pressure in the early morning and needs to be completed within the specified batch processing time window. Since this application aggregates the batch-related data of each service into the batch-specific database, realizing the separation of online transactions and batch processing in data access, the load of large-scale batch processing is borne by the batch-specific database cluster, without affecting real-time customer online transactions, and improving the system stability. The restriction of the batch processing time window is relaxed, supporting more timely and flexible batch processing.

[0077] In the batch processing mode of the prior art, because the support capabilities of various databases for batch processing are different, batch processing on different databases is separated. For example, if a service has both batch processing based on the MySQL database and batch processing based on the Oracle database, it needs to be split into a batch processing process on the MySQL database and a batch processing process on the Oracle database. This application unifies the aggregation of batch processing on multi-source databases, so a database product with strong HTAP (Hybrid Transaction and Analytical Processing) query capabilities can be uniformly selected to build the batch-specific database, better supporting complex analytical queries with a large amount of data, providing strong batch processing capabilities at the system level, unifying the batch processing method, and simplifying the development of batch programs at the application layer.

[0078] The unified orchestration of batch processing programs for services of multiple upstream and downstream applications of a business service unit, and all batch-related data in a business service unit is aggregated into a dedicated batch database. Therefore, there is no need for the processes of encoding, exporting, transmitting, decoding, and importing large-volume data files between the upstream and downstream applications involved in a business service unit. It only needs to agree on which data the upstream application opens to the downstream application, and the downstream application can access the corresponding data in the dedicated batch database cluster according to the batch configuration information of the upstream application. The batch processing method for business provided by the embodiments of the present invention optimizes the batch processing process, reduces intermediate links such as large-volume file transmission between upstream and downstream, reduces the time required for batch processing, and improves the efficiency of batch processing.

[0079] Based on the above embodiments, further, the batch processing method for business provided by the embodiments of the present invention further includes:

[0080] Checking whether the batch program templates of each service in the business service unit conform to the preset specifications.

[0081] Specifically, before using the batch program templates of each service in the business service unit, the batch processing management platform can check the batch program templates of each service in the business service unit to check whether each batch program template conforms to the preset specifications. Among them, the preset specifications are set in advance and are set according to actual needs, which are not limited in the embodiments of the present invention.

[0082] For example, the preset specification is the enterprise batch program specification, which is used to check whether the format of the batch program template is correct, whether the file size meets the requirements, whether to submit data after processing a preset number of data, etc.

[0083] Based on the above embodiments, further, the business service unit includes services involved in completing a business processing process.

[0084] Specifically, each service involved in a business processing process is included in the business service unit, and batch configuration information and batch program templates are set for each service.

[0085] Figure 6 It is a schematic diagram of the architecture of the data transmission platform provided by the sixth embodiment of the present invention. As Figure 6 shown, the data transmission platform may include a data collection unit, a data parsing unit, a data storage unit, a query registration center, a data sending unit, etc.

[0086] Each application's batch processing-related service registers information such as the IP address of its database server, database type, log file location, the service it belongs to, the business service unit it belongs to, and the data required by the service of the business service unit with the data transmission platform.

[0087] The data collection unit performs data docking and adaptation with databases such as MySQL, PostgreSQL, Oracle, and DB2 of each application according to the registration information. The data collection unit includes agents for various databases to access database data. For open-source databases such as MySQL and PostgreSQL, it accesses the data in the database binlog or xlog; for commercial databases such as Oracle and DB2, it can obtain the data synchronization interface through commercial channels and call the data synchronization interface to obtain near-real-time database change information.

[0088] The data parsing component parses the database change information obtained by the data parsing unit (the log is also a type of database change information, recording the changes in the data in the database), and parses the database logs such as MySQL binlog and PostgreSQL xlog of each application or the data changes obtained by calling through the data synchronization interface into SQL statements.

[0089] The data storage unit temporarily stores the parsed data for consumption by the downstream database (batch dedicated database).

[0090] The query registration center queries which service in the business service unit the current data belongs to, and the IP address of the batch dedicated database in the corresponding business service unit.

[0091] The data sending unit writes the parsed data into the batch dedicated database of the corresponding business service unit.

[0092] In the batch processing mode of the prior art, it is necessary to export the data file from the database and then perform file delivery between different application R & D teams. To meet the data security requirements, it is necessary to perform deformation operations and decoding operations on sensitive data. The present invention realizes the near-real-time synchronization of the database data within each service to the batch dedicated database through database connection, and can realize transmission encryption through the SSL encryption connection function of the database; according to the batch configuration information published by each service, the system can realize the access right to expose only specific table data to the downstream batch through the access right control function of the database. The entire data transmission process provides security guarantee through the system functions at the database level, without exporting data files for manual transmission, and without implementing additional data deformation and data decoding at the application level, improving the security and convenience of data transmission.

[0093] The following takes a specific embodiment to illustrate the specific implementation process of the batch processing method of the business provided by the embodiment of the present invention.

[0094] Suppose a business service unit Y involves two applications, with the upstream application being app_A and the downstream application being app_B. The upstream application app_A provides a batch processing-related service service_A, and the downstream application app_B provides a batch processing-related service service_B. The business service unit Y includes service service_A and service service_B.

[0095] service_A corresponds to a data table db_A.table_A which includes 320 table partitions (part). During the version release, the batch program template prog_A of service_A is released, and it is declared in the batch configuration information released together with service_A that:

[0096] service_A is processed in 16 batches in parallel, with each batch corresponding to a table partition;

[0097] The pre-dependency condition of service_A is Start, and Start means the pre-dependency condition is empty;

[0098] The subsequent processing rule of service_A specifies service_B;

[0099] The batch processing data information corresponding to service_A is part17 - part32 of db_A.table_A;

[0100] The data in part1 - part16 of db_A.table_A is made available for use by service_B of the downstream application app_B.

[0101] service_B corresponds to a data table db_B.table_B which includes 16 table partitions. During the version release, the batch program template prog_B of service_B is released, and it is declared in the batch configuration information released together with service_B that:

[0102] service_B is processed in 16 batches in parallel, with each batch corresponding to a table partition;

[0103] The pre-dependency conditions of service_B include: all batches of service_A have been executed;

[0104] The post-processing rule of service_B is End, and End means no post-processing is required;

[0105] The batch processing data information corresponding to service_B is part1 - part16 of db_A.table_A and part1 - part16 of db_B.table_B;

[0106] The pre - dependency conditions of service_B include: access permissions for parts 1 to 16 of the data table db_A.table_A corresponding to service_A of the upstream application app_A.

[0107] After BPMP receives the batch program templates and batch configuration information of service_A and service_B of the business service unit Y, the specific processing flow is as follows:

[0108] First step: Traverse the batch configuration information of all services in the business service unit Y. Generate a batch processing flow chart according to the pre - dependency conditions and post - processing rules corresponding to service_A, and the pre - dependency conditions and post - processing rules corresponding to service_B. The previous node of service_A is Start, and the subsequent node is service_B. The previous node of service_B is service_A, and the subsequent node is End. Therefore, a batch processing flow chart can be generated as Figure 7 shown.

[0109] Second step: Check whether the pre - dependency conditions of service_A are met. Here, the pre - dependency conditions of service_A are empty, so the subsequent steps can be continued.

[0110] Third step: According to the batch configuration information of service_A, prog_A authorizes the connecting user of prog_B, and BPMP controls that prog_B can only access parts 1 to 16 of db_a.table_A when instantiating the service_B parameters.

[0111] Fourth step: Determine that service_A is processed in 16 batches in parallel according to the batch configuration information of service_A. Replace the corresponding parameters in the batch program template of service_A with the batch numbers and the data corresponding to the batch numbers in the batch configuration information. The batch program prog_A_1 of the first batch accesses part 17 of db_A.table_A, the batch program prog_A_2 of the second batch accesses part 18 of db_A.table_A, and so on. The batch program prog_A_16 of the 16th batch accesses part 32 of db_A.table_A, and finally generates the batch processing program of service_A.

[0112] Fifth step: Check whether the post - processing rules of service_A are met. Here, service_A needs to process service_B subsequently. If it is met, continue the subsequent steps.

[0113] Step 6: Check whether the pre - dependency conditions of service_B are met. If so, proceed to the subsequent steps.

[0114] The previous batch process of service_B is service_A, which is met.

[0115] Service_B needs to access parts 1 to 16 of the data table db_A.table_A corresponding to service_A of app_A, which is met.

[0116] Step 7: Determine that service_B is divided into 16 batches according to the batch configuration information of service_B. Replace the corresponding parameters in the batch program template of service_B with the batch numbers and the data corresponding to the batch numbers. The batch program prog_B_1 of the first batch accesses part1 of db_A.table_A and part1 of db_B.table_B, the batch program prog_B_2 of the second batch accesses part2 of db_A.table_A and part2 of db_B.table_B, and so on. The batch program prog_B_16 of the 16th batch accesses part16 of db_A.table_A and part16 of db_B.table_B, and finally generates the batch processing program of service_B.

[0117] Step 8: Generate the batch processing program of business service unit Y. Service_B needs to start execution after all batches of service_A are completed. Therefore, add a virtual batch task prog_v as a closure between the batch processing programs of service_A and service_B. Prog_v ensures that after prog_A_1 to prog_A_16 are all executed, prog_B_1 to prog_B_16 start to execute. The batch processing program of business service unit Y includes the batch processing program of service_A, prog_v, and the batch processing program of service_B.

[0118] Step 9: Execute the batch processing program of business service unit Y. When executing the batch processing program of service_A, read the data in parts 17 to 32 of db_A.table_A from the batch - specific database of business service unit Y; when executing the batch processing program of service_B, read the data in parts 1 to 16 of db_A.table_A and parts 1 to 16 of db_B.table_B from the batch - specific database of business service unit Y. The batch - specific database of business service unit Y is established in advance through the data transmission platform.

[0119] Figure 8 It is a schematic structural diagram of a batch processing device for services provided by the eighth embodiment of the present invention. As Figure 8 shown, the batch processing device for services provided by the embodiments of the present invention includes a first generation module 801, a second generation module 802, and an execution module 803, where:

[0120] The first generation module 801 is used to generate a batch processing flow chart according to the batch configuration information of each service of the service unit; the second generation module 802 is used to generate a batch processing program of the service unit according to the batch processing flow chart, the batch program templates of each service of the service unit, and the batch configuration information; the execution module 803 is used to execute the batch processing program of the service unit; wherein, the batch configuration information and batch program templates of each service of the service unit are preset.

[0121] Specifically, the first generation module 801 obtains the batch configuration information of each service of the service unit, and then performs batch scheduling and orchestration according to the batch configuration information of each service of the service unit to determine the logical order of processing of each service, and generates a batch processing flow chart. Among them, the batch configuration information of each service of the service unit is preset and set according to actual needs, which is not limited in the embodiments of the present invention.

[0122] The second generation module 802 obtains the batch program templates of each service of the service unit, obtains relevant data from the batch configuration information of each service to replace the relevant parameters in the batch program templates of each service, and instantiates the batch program templates into batch programs to obtain the batch processing programs of each service. Then, the batch processing programs of each service are concatenated according to the batch processing flow chart to generate the batch processing program of the service unit. Among them, the batch program templates of each service of the service unit are preset.

[0123] After obtaining the batch processing program of the service unit, the batch processing program of the service unit will be executed, and the execution module 803 calls the applications of each service of the service unit to perform data processing, so as to complete the processing of the business processing process corresponding to the service unit.

[0124] The batch processing device provided by the embodiment of the present invention can generate a batch processing flow chart according to the batch configuration information of each service in the service unit, generate a batch processing program for the service unit according to the batch processing flow chart, the batch program templates of each service in the service unit, and the batch configuration information, and execute the batch processing program of the service unit, scheduling the relevant services in the business processing process as an independent and complete service unit as a whole, optimizing the batch processing process, and improving the efficiency of batch processing.

[0125] Figure 9 FIG. 4 is a schematic structural diagram of the batch processing device for services provided by the ninth embodiment of the present invention, as Figure 9 shown. On the basis of the above embodiments, further, the first generation module 801 includes an acquisition sub-module 8011 and a first generation sub-module 8012, where:

[0126] The first acquisition sub-module 8011 is used to acquire the pre-dependency conditions and post-processing rules corresponding to each service from the batch configuration information of each service; wherein, the batch configuration information includes pre-dependency conditions and post-processing rules; the first generation sub-module 8012 is used to determine the processing order of each service according to the pre-dependency conditions and post-processing rules corresponding to each service, and generate a batch processing flow chart.

[0127] Figure 10 FIG. 5 is a schematic structural diagram of the batch processing device for services provided by the tenth embodiment of the present invention, as Figure 10 shown. On the basis of the above embodiments, further, the second generation module 802 includes a second acquisition sub-module 8021, a second generation sub-module 8022, and a third generation sub-module 8023, where:

[0128] The second acquisition sub-module 8021 is used to acquire the batch information and batch processing data information corresponding to each service from the batch configuration information of each service; the second generation sub-module 8022 is used to generate a batch processing program for each service according to the batch information and batch processing data information corresponding to each service and the batch program template of each service; the third generation sub-module 8023 is used to generate a batch processing program for the service unit according to the batch processing flow chart and the batch processing programs of each service.

[0129] On the basis of the above embodiments, further, the execution module 803 is specifically used for

[0130] If there is a sequential execution relationship between the batch processing programs of two services, a virtual batch task is set between the batch processing programs of the two services.

[0131] On the basis of the above embodiments, further, the execution module 803 is specifically used for

[0132] Access the batch dedicated database of the service unit to call relevant data and execute the batch processing program of the service unit; wherein, the batch dedicated database of the service unit is pre-established and stores data required for the execution of the batch processing program of the service unit.

[0133] Figure 11 It is a schematic structural diagram of a batch processing device for services provided in the eleventh embodiment of the present invention. As Figure 11 shown, on the basis of the above embodiments, further, the batch processing device for services provided in the embodiments of the present invention further includes a detection module 804, wherein:

[0134] The detection module 804 is specifically configured to check whether the batch program templates of each service of the service unit conform to preset specifications.

[0135] On the basis of the above embodiments, further, the service unit includes services involved in completing a business processing flow.

[0136] The embodiments of the device provided in the embodiments of the present invention can specifically be used to execute the processing flows of the above method embodiments, and its functions will not be elaborated here. Reference can be made to the detailed descriptions of the above method embodiments.

[0137] It should be noted that the batch processing method and device for services provided in the embodiments of the present invention can be used in the financial field and can also be used in any technical field other than the financial field. The embodiments of the present invention do not limit the application fields of the batch processing method and device for services.

[0138] Figure 12 It is a schematic physical structure diagram of an electronic device provided in the twelfth embodiment of the present invention. As Figure 12 shown, the electronic device may include: a processor 1201, a communication interface 1202, a memory 1203, and a communication bus 1204. Among them, the processor 1201, the communication interface 1202, and the memory 1203 communicate with each other through the communication bus 1204. The processor 1201 can call logical instructions in the memory 1203 to execute the following methods: generate a batch processing flow chart according to the batch configuration information of each service of the service unit; generate the batch processing program of the service unit according to the batch processing flow chart, the batch program templates of each service of the service unit, and the batch configuration information; execute the batch processing program of the service unit; wherein, the batch configuration information and batch program templates of each service of the service unit are pre-set.

[0139] In addition, when the logical instructions in the above-mentioned memory 1203 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0140] This embodiment discloses a computer program product. The computer program product includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided in the above-mentioned method embodiments. For example, it includes: generating a batch processing flow chart according to the batch configuration information of each service of the service unit; generating a batch processing program of the service unit according to the batch processing flow chart, the batch program templates of each service of the service unit, and the batch configuration information; executing the batch processing program of the service unit; wherein, the batch configuration information and batch program templates of each service of the service unit are pre-set.

[0141] This embodiment provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. The computer program causes the computer to execute the methods provided in the above-mentioned method embodiments. For example, it includes: generating a batch processing flow chart according to the batch configuration information of each service of the service unit; generating a batch processing program of the service unit according to the batch processing flow chart, the batch program templates of each service of the service unit, and the batch configuration information; executing the batch processing program of the service unit; wherein, the batch configuration information and batch program templates of each service of the service unit are pre-set.

[0142] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0144] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0146] In the description of this specification, the descriptions referring to terms such as "one embodiment", "a specific embodiment", "some embodiments", "for example", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0147] The above-mentioned specific embodiments have further elaborated on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the above is only the specific embodiments of the present invention and is not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for batch processing of services, characterized in that, Including: Generate a batch processing flow chart according to the batch configuration information of each service in the business service unit; Generate the batch processing program of the business service unit according to the batch processing flow chart, the batch program templates of each service in the business service unit, and the batch configuration information; Execute the batch processing program of the business service unit; Wherein, the batch configuration information and batch program templates of each service in the business service unit are preset; Wherein, the execution of the batch processing program of the business service unit includes: Access the batch dedicated database of the business service unit to call relevant data and execute the batch processing program of the business service unit; wherein, the batch dedicated database of the business service unit is pre-established and stores the data required for the execution of the batch processing program of the business service unit, and each business service unit is equipped with a batch dedicated database cluster; Wherein, the generation of the batch processing program of the business service unit according to the batch processing flow chart, the batch program templates of each service in the business service unit, and the batch configuration information includes: Obtain the batch information and batch processing data information corresponding to each service from the batch configuration information of each service; wherein, the batch processing data information includes the sources of the data processed in each batch; Generate the batch processing program of each service according to the batch information and batch processing data information corresponding to each service and the batch program template of each service; Generate the batch processing program of the business service unit according to the batch processing flow chart and the batch processing programs of each service; Wherein, the generation of the batch processing program of the business service unit according to the batch processing flow chart and the batch processing programs of each service includes: If there is a sequential execution relationship between the batch processing programs of two services, set a virtual batch task between the batch processing programs of the two services.

2. The method according to claim 1, characterized in that, The generation of the batch processing flow chart according to the batch configuration information of each service in the business service unit includes: Obtain the pre-dependency conditions and post-processing rules corresponding to each service from the batch configuration information of each service; wherein, the batch configuration information includes pre-dependency conditions and post-processing rules; Determine the processing order of each service according to the pre-dependency conditions and post-processing rules corresponding to each service, and generate a batch processing flow chart.

3. The method according to claim 1, characterized in that, Also including: Check whether the batch program templates of each service in the business service unit conform to the preset specifications.

4. The method according to any one of claims 1 to 3, characterized in that, The business service unit includes the services involved in completing a business processing flow.

5. A device for batch processing of services, characterized in that, Including: A first generation module for generating a batch processing flow chart according to the batch configuration information of each service in the business service unit; A second generation module for generating the batch processing program of the business service unit according to the batch processing flow chart, the batch program templates of each service in the business service unit, and the batch configuration information; An execution module for executing the batch processing program of the business service unit; Wherein, the batch configuration information and batch program templates of each service in the business service unit are preset; Among them, the execution module is specifically configured to access the batch dedicated database of the service unit to call relevant data and execute the batch processing program of the service unit; among them, the batch dedicated database of the service unit is pre-established and stores data required for the execution of the batch processing program of the service unit, and each service unit is equipped with a batch dedicated database cluster; Among them, the second generation module includes a second acquisition sub-module, a second generation sub-module, and a third generation sub-module, where, The second acquisition sub-module is used to obtain the batch information and batch processing data information corresponding to each service from the batch configuration information of each service; among them, the batch processing data information includes the sources of the data processed in each batch; The second generation sub-module is used to generate the batch processing program of each service according to the batch information and batch processing data information corresponding to each service and the batch program template of each service; The third generation sub-module is used to generate the batch processing program of the service unit according to the batch processing flow chart and the batch processing programs of each service; Among them, the execution module is specifically further configured to, if there is a sequential execution relationship between the batch processing programs of two services, set a virtual batch task between the batch processing programs of the two services.

6. A computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 4 is implemented.

8. A computer program product, characterized in that, The computer program product includes a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 4 is implemented.

Images