Batch data processing method and device, electronic equipment and computer program product

Abstract

The invention discloses a batch data processing method and device, electronic equipment and a computer program product, and the method comprises the steps: intercepting a business operation request which is a request for carrying out change operation on data in a business library; when the business library is in the batch running state, a business operation cache table is updated according to the business operation request, and the business operation cache table is used for caching data related to business operation; determining a batch processing job chain to be rerun according to the business operation cache table and the batch processing job chain information; and rerunning the batch processing operation chain to be rerun to obtain a batch processing result. According to the method, the service operation related to the changed data is intercepted, and the cache table is adopted to store the service operation data, so that the data redundancy and the occupation of the storage space are reduced, and the complexity of realizing uninterrupted service by the system is reduced. In addition, the method does not invade an associated system and is independent of a main system process, so that the flexibility of the system is improved, the influence on the associated system is reduced, and the operation and maintenance cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a batch data processing method, device, electronic equipment, and computer program product. Background Art

[0002] With the rapid development and popularization of computer and network technology, information systems play an increasingly important role in banking business. As banks' digital transformation is further promoted, customer service capabilities are continuously improved, and the time and space scope for customers to handle business are also expanding. Therefore, how to ensure that information systems can provide services to customers around the clock and without interruption is a common challenge faced by all banks.

[0003] At present, the industry generally adopts the "double date, double balance" technical solution for accounting systems. This solution realizes cross-day transaction processing and data supply by setting two dates (the last trading day and the last trading day) and two sets of balances (account balance, last account balance and related balance direction). This solution currently has some disadvantages. First, the table structure design is relatively complex, and multiple fields and indexes need to be added. Especially for systems with large transaction volumes, it will greatly increase the storage space and maintenance costs of the database; second, the end-of-day or data supply logic judgment is relatively cumbersome, and multiple situations need to be considered. When there are many batch data interactions with related systems, it may cause data inconsistency or performance degradation, affecting the accuracy and timeliness of the data; in addition, for management systems, due to the low transaction concurrency, the use of this solution will lead to overly complex system design and high operation and maintenance costs. Summary of the invention

[0004] The embodiments of the present application provide a batch data processing method, device, electronic device, and computer program product to reduce data redundancy in the batch processing process and reduce the complexity of the system in achieving uninterrupted service.

[0005] The present application embodiment adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a batch data processing method, the batch data processing method comprising:

[0007] Intercepting a business operation request, wherein the business operation request is a request to change data in a business database;

[0008] When the business database is in a batch running state, updating a business operation cache table according to the business operation request, wherein the business operation cache table is used to cache data related to the business operation;

[0009] Determine the batch processing job chain to be re-run according to the business operation cache table and the batch processing job chain information;

[0010] Re-run the batch processing job chain to be re-run to obtain a batch processing result.

[0011] Optionally, when the business database is in a batch running state, updating the business operation cache table according to the business operation request includes:

[0012] When the business database is in a batch run state, determining whether the business database is in a forced batch run state;

[0013] When the business database is not in a forced batch running state, determining whether a data table of the business database has a write lock;

[0014] If the write lock does not exist or the write lock is released, the business operation request is parsed to obtain a parsing result of the business operation request, wherein the parsing result includes business operation content, business operation type, and data affected by the business operation;

[0015] Updating the business operation cache table according to the parsing result of the business operation request;

[0016] The business operation cache table stores the table name of the business data table affected by the business operation, the business operation type, the interception time and the data processing status.

[0017] Optionally, updating the business operation cache table according to the parsing result of the business operation request includes:

[0018] If the business operation type is modification, then search for data of other fields according to the field data contained in the modification statement, and splice the field data contained in the modification statement and the other found field data into complete field data and save them in the business operation cache table;

[0019] If the business operation type is insert, the field data contained in the insert statement is directly saved in the business operation cache table;

[0020] If the business operation type is deletion, the data inserted into the business operation cache table is deleted according to the field data contained in the deletion statement.

[0021] Optionally, determining the batch job chain to be rerun according to the business operation cache table and the batch job chain information includes:

[0022] When the business database is in a batch running state, read the business operation cache table to obtain the table name of the business data table in an unprocessed state;

[0023] The batch processing job chain to be re-run is determined according to the table name of the unprocessed business data table and the batch processing job chain information.

[0024] Optionally, determining the batch job chain to be rerun according to the table name of the unprocessed business data table and the batch job chain information includes:

[0025] Acquire the batch processing job chain information, the batch processing job chain information includes the execution status of the job chain and the input job information of the job chain, the execution status includes the job chain that has been completed, the job chain that is in operation, and the job chain in queue;

[0026] The table name of the unprocessed business data table is matched with the input job information of the job chain to determine the batch job chain to be re-run.

[0027] Optionally, re-running the batch processing job chain to be re-run to obtain a batch processing result includes:

[0028] For the running job chains and queued job chains in the batch job chain to be rerun, call the job chain stop interface to stop running and queuing;

[0029] Starting the input job of the batch processing job chain to be rerun to rerun it, and obtaining the rerun result of the batch processing job chain to be rerun;

[0030] Writing the rerun results of the batch processing job chain to be rerun into the batch processing result temporary table;

[0031] When all batch processing job chains to be re-run have been re-run, the business data tables associated with all batch processing job chains to be re-run are locked and the business operations of the associated systems are suspended;

[0032] The rerun results in the batch result temporary table are written into the business data table, and the lock of the business data table is released after the writing is completed.

[0033] Optionally, the starting of the input job of the batch processing job chain to be rerun to rerun the batch processing job chain to obtain the rerun result of the batch processing job chain to be rerun includes:

[0034] Determine a data extraction method for the batch job chain to be rerun;

[0035] When the data extraction mode of the batch job chain to be rerun is an incremental mode, extracting incremental data according to the business cache table;

[0036] Re-running is performed based on the incremental data and the extracted data of the batch processing job chain to be re-run, so as to obtain the re-run result of the batch processing job chain to be re-run.

[0037] In a second aspect, an embodiment of the present application further provides a batch data processing device, the batch data processing device comprising:

[0038] An interception component, used to intercept a business operation request, wherein the business operation request is a request for changing data in a business database, and when the business database is in a batch running state, update a business operation cache table according to the business operation request, wherein the business operation cache table is used to cache data related to the business operation;

[0039] The processing component is used to determine the batch processing job chain to be re-run according to the business operation cache table and the batch processing job chain information, and re-run the batch processing job chain to be re-run to obtain a batch processing result.

[0040] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0041] Processor; and

[0042] A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform any of the aforementioned batch data processing methods.

[0043] In a fourth aspect, an embodiment of the present application further provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements any of the aforementioned batch data processing methods.

[0044] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the batch data processing method of the embodiment of the present application first intercepts the business operation request, which is a request for changing the data in the business database; when the business database is in a batch running state, the business operation cache table is updated according to the business operation request, and the business operation cache table is used to cache data related to the business operation; the batch job chain to be rerun is determined according to the business operation cache table and the batch job chain information; the batch job chain to be rerun is rerun to obtain the batch processing result. The batch data processing method of the embodiment of the present application intercepts the business operation request related to the database data change, and stores the business operation data in the form of a cache table. Compared with the existing "dual database and dual table" solution, it reduces data redundancy and storage space occupation, and reduces the complexity of the system to achieve uninterrupted service. In addition, the batch processing solution of the embodiment of the present application can be implemented through a componentized universal design, does not invade the associated system, is independent of the main system process, improves the flexibility of the system, reduces the impact on the associated system, and reduces the operation and maintenance cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0046] Figure 1 A flow chart of a batch data processing method in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of an interception process of a business operation in an embodiment of the present application;

[0048] Figure 3 This is a schematic diagram of a re-running process of a batch processing job chain in an embodiment of the present application;

[0049] Figure 4 This is a structural schematic diagram of a batch data processing device in an embodiment of the present application;

[0050] Figure 5 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0052] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0053] First, we introduce several data processing solutions used in existing technologies:

[0054] 1) Patent document CN1553393A: discloses a data processing method for realizing uninterrupted banking services, including the following steps: executing pre-day-end processing; performing 24-hour mode processing; and restoring normal working conditions. The steps of performing 24-hour mode processing include: the core business system performs the bank's day-end batch processing, while the 24-hour business processing device processes transaction requests from self-service devices; and the 24-hour business processing device performs account replenishment processing. The present invention realizes a method and system for realizing uninterrupted banking services by using two databases, and can provide core banking services for external services around the clock, thereby improving the overall service quality of the bank and bringing great convenience to customers.

[0055] 2) Patent document CN107563887A: discloses a zero downtime daily cutting method based on AS400 bank core accounting system, including the following steps: AB table design: when the core batch runs, the date data files of the journal entries generated by the online transaction are stored separately in the same library, and the file name is suffixed with the AB mark to distinguish, the previous day's online use of A table, the next day's online use of B table, and the next day's use of A table in a cycle; online job pre-start; audit. The present invention solves the drawbacks of multiple online shutdowns at the end of the day and long offline time in the dual database switching account collection method and the dual balance method, and realizes true 7*24 hours uninterrupted service on AS400, avoiding the situation of large-scale transaction failures in various electronic channels during the daily cutting period, ensuring the stability of transaction channels of other banks such as Rural Credit Bank, Second Generation Payment, and UnionPay, and improving customer experience and bank competitiveness.

[0056] 3) Patent document CN115689714A: Provides a data processing method and system that supports uninterrupted service of business systems, including: establishing a data architecture with front-end and back-end layers; performing corresponding data processing through the data architecture; distinguishing the current processing module according to the processing status of the data processing and the flag of each module in the business system, and judging whether the current module processing is completed, if so, the system automatically flips the card; if not, the system will loop until the processing is completed; the data processing includes intelligent identification and processing of dynamic data and static data and daily data cutting processing. The present invention realizes the uninterrupted complex batch data processing of the banking business system while maintaining continuous service in the system front desk through the data architecture with front-end and back-end layers, intelligent identification of dynamic data and static data, accurate and flexible full-process operation status monitoring, modular batch flipping solution, and millisecond-level data daily cutting mechanism.

[0057] 4) Patent document CN115617804A: discloses a data processing method, device, equipment and storage medium for uninterrupted service, which relates to the field of database technology. The method includes: setting the direction of the database view to the first type of physical table in the database, for instructing the external service to read the data in the first type of physical table through the database view; the database stores the first type of physical table and the second type of physical table with the same structure; based on the source data of the current data processing cycle, the second type of physical table is processed; in response to the completion of the data processing operation of the second type of physical table, the database view is redirected to the second type of physical table, for instructing the external service to read the processed data in the second type of physical table through the database view. The technical solution of the embodiment of the present invention can perform batch data processing on the basis of providing external services uninterruptedly.

[0058] In summary, the first patent document adopts a two-database processing method, using a 24-hour business processing device to process requests from self-service devices and perform account replenishment operations, which is costly. The second patent document adopts a dual-table AB marking method, using table A online the previous day and table B online the next day, which is complex to implement. The third patent document achieves uninterrupted service of the business system by establishing a layered data architecture for the front and back ends, and distinguishing between static data and dynamic data. The fourth patent document achieves uninterrupted service by setting up a database view to point to a cyclic operation of the first and second physical tables.

[0059] The above existing solutions have at least the following deficiencies:

[0060] 1) Most of the existing technical solutions are aimed at accounting systems with high concurrency, and have high implementation requirements for management systems. Therefore, management systems are more complex to implement and have high implementation costs. In particular, for systems that need to be modified, the modification is difficult and costly;

[0061] 2) Most current technologies use a redundant data storage solution in the form of dual databases or dual tables to implement system daily switching to avoid conflicts between batch jobs and real-time transactions. For systems with large business scales, this greatly increases the consumption of system database resources and maintenance costs;

[0062] 3) When the existing technical solutions and related systems interact with each other for end-of-day batch data, there are certain defects in the timeliness and completeness of data acquisition. When there are many downstream related systems involved, it may cause data inconsistency or performance degradation, affecting the accuracy and timeliness of the data.

[0063] In order to solve at least one of the above technical problems, the present application embodiment provides a batch data processing method, such as Figure 1 As shown, a flow chart of a batch data processing method in an embodiment of the present application is provided, and the batch data processing method at least includes the following steps S110 to S140:

[0064] Step S110 , intercepting a business operation request, wherein the business operation request is a request for changing data in a business database.

[0065] When performing batch data processing, the embodiment of the present application needs to first intercept the business operation requests initiated by the associated system. The business operation requests can be intercepted by a custom interception component. The business operation requests here are mainly for change operations that affect data in the business database, such as insert, update, and delete operations.

[0066] Step S120, when the business database is in a batch running state, updating a business operation cache table according to the business operation request, wherein the business operation cache table is used to cache data related to the business operation;

[0067] In order to resolve the conflict problem between real-time business and batch data processing and coordinate the processing order between real-time business and batch data processing, the overall processing procedure of the batch data processing method of the embodiment of the present application needs to be executed when the business database is in the batch running state. The batch running state means that only the business database is currently processing batch data. If the business database is not currently in the batch running state, it means that no batch processing operation is being performed. Then the business operation request initiated by the current associated system will naturally not have a conflict problem, and the business operation can be directly processed according to the normal business processing logic.

[0068] If the business database is currently in a batch running state, it means that the batch program is operating the business database, and the real-time business operation request also needs to operate the business database, which may cause data processing conflicts. Therefore, the embodiment of the present application defines a business operation cache table, which caches the current business operation records and the affected data into the business operation cache table first, so as to facilitate the subsequent processing of the data affected by these business operations. The business operation cache table is independent of the original business database and can be regarded as a temporary data table for caching change data and operation records.

[0069] Step S130, determining the batch processing job chain to be re-run according to the business operation cache table and the batch processing job chain information;

[0070] Batch job chain is an orderly execution sequence formed by connecting multiple independent batch jobs through specific scheduling and execution strategies during data batch processing. Batch job chain information mainly includes data tables associated with the batch job chain and execution status.

[0071] As mentioned earlier, the business operation cache table stores data related to real-time business operations. During normal business processing, the data stored in the business database will change as business operations are generated in real time. When batch processing operations on the business database conflict with real-time business operations, the batch processing results may be inaccurate.

[0072] Based on this, the embodiment of the present application combines the data in the business operation cache table and the batch job chain information for matching analysis, and can determine whether the real-time business operations have an impact on the current batch job chain and which batch job chains are specifically affected. These affected batch job chains are the job chains to be re-run, that is, the batch processing logic of the data needs to be re-executed based on the data affected by the real-time business operations.

[0073] Step S140, re-running the batch processing job chain to be re-run to obtain a batch processing result.

[0074] After the batch processing job chains to be re-run are determined, the job chains to be re-run are re-run in sequence according to the specific execution order in the job chains to obtain the batch processing results after the re-run.

[0075] The batch data processing method of the embodiment of the present application intercepts business operation requests related to database data changes and uses a cache table to store business operation data. Compared with the existing "dual database and dual table" solution, it reduces data redundancy and storage space occupation, and reduces the complexity of the system to achieve uninterrupted service. In addition, the batch processing solution of the embodiment of the present application can be implemented through a componentized universal design, does not invade the associated system, is independent of the main system process, improves the flexibility of the system, reduces the impact on the associated system, and reduces the operation and maintenance cost.

[0076] In some embodiments of the present application, when the business database is in a batch-running state, updating the business operation cache table according to the business operation request includes: when the business database is in a batch-running state, determining whether the business database is in a forced batch-running state; when the business database is not in a forced batch-running state, determining whether a write lock exists for a data table of the business database; when there is no write lock or the write lock is released, parsing the business operation request to obtain a parsing result of the business operation request, the parsing result including the business operation content, the business operation type, and the data affected by the business operation; updating the business operation cache table according to the parsing result of the business operation request; wherein, the business operation cache table stores the table name of the business data table affected by the business operation, the business operation type, the interception time, and the data processing status.

[0077] like Figure 2 As shown, a schematic diagram of the interception process of a business operation in an embodiment of the present application is provided. When the business database is in a batch running state, the interception component can further determine whether it is in a forced batch running state. The forced batch running mode is generally triggered by a downgrade processing mechanism, and its function is to prioritize the completion of batch jobs. Under normal circumstances, forced batch running will not be enabled, mainly considering some special high-concurrency scenarios. A large number of frequent real-time business operations will cause the batch processing tasks currently in operation to be unable to complete the operation. The forced batch running mode can be enabled to allow the related batch processing job chain to complete the operation first. In the forced batch running mode, business operations cannot be performed. At this time, the interception component can throw a custom exception and return it to the calling end to remind that it is forced to run the batch at this time.

[0078] If the forced batch run state is not in progress or the forced batch run state has ended, you can further determine whether a write lock exists on the data table affected by the business operation. If a write lock exists, it means that a batch job is writing to the table. In this case, the system will return directly and wait for the write lock to be released before re-initiating the business operation.

[0079] If there is no write lock or the write lock is released, the SQL statement passed in the business operation request is parsed to obtain the type of business operation such as update, delete, insert, etc., the name of the data table affected by the operation, the field name and field value in the where condition, the field name and field value affected by insert or update, and other information.

[0080] After parsing the above information, each business operation record and the affected data are recorded in the business operation cache table, mainly recording the data table name, business operation type, interception time, business hours and corresponding processing status affected by the operation. The processing status is initialized to pending, indicating the conflict between the data table that has not yet processed the business operation and the batch job chain it affects.

[0081] In some embodiments of the present application, updating the business operation cache table according to the parsing result of the business operation request includes: if the business operation type is modification, searching for data of other fields according to the field data contained in the modification statement, and splicing the field data contained in the modification statement and the other found field data into complete field data and saving them in the business operation cache table; if the business operation type is insertion, saving the field data contained in the insert statement directly into the business operation cache table; if the business operation type is deletion, deleting the data inserted into the business operation cache table according to the field data contained in the delete statement.

[0082] When recording the parsing results of the business operation request into the business operation cache table, the corresponding data cache processing is different according to the type of business operation, which is mainly divided into the following situations:

[0083] 1) Update statement: In order to ensure the integrity of the cache table data, all field values ​​of the table need to be saved. A complete full-field data consists of two parts: the fields included in the update statement + other fields of the table. The fields and field values ​​included in the update statement are obtained by parsing the SQL statement of the update, and the other fields of the table are reverse-queried by generating a select statement based on the where condition of the parsed update statement. The reverse-query result and the data included in the update are spliced ​​to form complete data and saved in the cache data table.

[0084] 2) Insert statement: The insert statement contains complete data and saves the parsed data fields into the cache data table.

[0085] 3) Delete statement: The delete statement will delete the inserted data in the cache data table according to the fields in the where condition. Because the delete operation is a full load, the delete operation will not leave any records in the cache data table.

[0086] In some embodiments of the present application, determining the batch job chain to be re-run based on the business operation cache table and the batch job chain information includes: when the business database is in a batch running state, reading the business operation cache table to obtain the table name of the business data table in the unprocessed state; determining the batch job chain to be re-run based on the table name of the business data table in the unprocessed state and the batch job chain information.

[0087] When determining the batch job chain to be rerun, first read the operation records and other data in the business operation cache table to obtain the name of the business data table in the unprocessed state. There are three processing states corresponding to the business data table in the operation record table: unprocessed, in process, and processed. The embodiment of the present application only processes the business data table in the unprocessed state. Matching analysis is performed based on the table name of the business data table in the unprocessed state and the information of the batch job chain, so as to determine the batch job chain to be rerun.

[0088] In some embodiments of the present application, determining the batch job chain to be re-run based on the table name of the unprocessed business data table and the batch job chain information includes: obtaining the batch job chain information, the batch job chain information including the execution status of the job chain and the input job information of the job chain, the execution status including the job chain that has been completed, the job chain that is running, and the job chain in the queue; matching the table name of the unprocessed business data table with the input job information of the job chain to determine the batch job chain to be re-run.

[0089] The embodiment of the present application can obtain the execution status of the batch job chain by calling the job chain query interface, including the job chain that has been completed, the job chain that is running, and the job chain in the queue. A complete batch job chain is composed of multiple types of batch components, mainly including input components, upsert components, commit pre-components and commit components. Among them, the input component is the source of the batch job, which generally extracts data from the business database to the Hive table for subsequent batch components to perform logical processing.

[0090] Match the table name of the unprocessed business data table with the input job name of the job chain, so as to determine which job chains need to be rerun, and determine the rerun order between the job chains to be rerun according to the job priority order. The embodiment of the present application will record which input jobs of each job chain need to be rerun, and when the input job is triggered to start, it will wait for the input to be executed, and then force the downstream job to continue to execute. Figure 3 As shown, a schematic diagram of the re-running process of a batch job chain in an embodiment of the present application is provided.

[0091] In some embodiments of the present application, rerunning the batch processing job chain to be rerun and obtaining the batch processing result includes: calling the job chain stop interface to stop the running job chain and the queued job chain in the batch processing job chain to be rerun; starting the input job of the batch processing job chain to be rerun to rerun and obtain the rerun result of the batch processing job chain to be rerun; writing the rerun result of the batch processing job chain to be rerun into a batch processing result temporary table; when all batch processing job chains to be rerun have been rerun, locking the business data tables associated with all batch processing job chains to be rerun and suspending the business operations of the associated systems; writing the rerun result in the batch processing result temporary table into the business data table, and releasing the lock of the business data table after writing is completed.

[0092] After determining the batch job chain to be rerun, you can further call the job chain stop interface to stop the running batch job chain from continuing to run and stop the queued batch job chain from continuing to queue, and then rerun the job chain to be rerun starting from the input job in sequence.

[0093] Based on the component composition of the batch job chain, the re-running process of the batch job chain is introduced in detail:

[0094] 1) Input component: Responsible for extracting data from the business database to the Hive table for subsequent batch components to perform logical processing. After reading the data table in an unprocessed state, it will find the matching job chain according to the table name and rerun it. The rerun will call the interface to start the input component, and the parameters passed include whether to increase the number of extraction, the interception time for data table operations, business hours, etc. In addition, the input component will set the status of the associated data table to processing to prevent repeated processing.

[0095] 2) Upsert component: generally used to represent the tail job of a batch job chain, responsible for writing the results of logical processing back to the business table. The upsert component in the embodiment of the present application does not directly operate the business library. A temporary table is set up in Hive as a data table for pre-writing the business library. The upsert component will write the result data into the temporary table.

[0096] 3) Commit pre-component: used to submit the business database pre-component. In the scheduling system configuration, it is the post-component of the tail job of all job chains, and it will be triggered when all job chains are completed. The commit pre-component will lock the business tables related to all tail jobs of the currently configured job chain to notify the interception component to suspend business operations and prepare to write batch data into the business database.

[0097] 4) The commit component is responsible for actually writing the result data into the business database. It is configured to be executed after the commit pre-component in the scheduling system. The commit component will release the business table lock after writing into the business database. After all locks are released, the batch running status will be modified and the business date will be added, marking the end of the batch running task for that business date period.

[0098] In some embodiments of the present application, the step of rerunning the input job of the batch job chain to be rerun and obtaining the rerun result of the batch job chain to be rerun includes: determining the data extraction method of the batch job chain to be rerun; extracting incremental data according to the business cache table when the data extraction method of the batch job chain to be rerun is an incremental method; rerunning according to the incremental data and the extracted data of the batch job chain to be rerun, and obtaining the rerun result of the batch job chain to be rerun.

[0099] The input component supports parameter passing and can extract data in an incremental manner. The so-called incremental method is to read from the business data table cached by the interception component. If incremental extraction is required, the input will search the business data table saved by the interception component for all incremental data under the current business date that is less than the interception time. Of course, in addition to the incremental extraction method, full extraction can also be used. The specific configuration of the data extraction method can be flexibly set by technicians in this field according to actual needs, and no specific limitation is made here.

[0100] In some embodiments of the present application, the scheduling manager component can provide the related functions involved in the above embodiments based on the scheduling system, for example, it can include querying, parsing, and operating the batch processing job chain of the scheduling system, and the specific functions are as follows:

[0101] 1) Job chain query function: Search for the corresponding job chain information according to the table name. First, find the matching input job according to the table name, and then recursively search for all dependent jobs based on the input job. The job information includes job status, job name, and last start time. Then the interface will analyze and generate job chain information, including the current running job list and queued job list in the job chain.

[0102] 2) Job chain stop function: stop the job chain according to the job name. The interface will first obtain the status of the incoming job. If the job has been completed, it will recursively find the queued or running jobs along the job chain and stop them, ensuring that all jobs in the entire job chain are stopped.

[0103] 3) Job start function: start the job according to the job name. Mainly used to call when the processing component reruns the job chain.

[0104] 4) Job status query function: query the job status according to the job name. Mainly used to monitor the input job status to trigger downstream jobs.

[0105] The present application embodiment also provides a batch data processing device 400, such as Figure 4 As shown, a schematic diagram of the structure of a batch data processing device in an embodiment of the present application is provided, wherein the batch data processing device comprises: an interception component 410 and a processing component 420, wherein:

[0106] An interception component 410 is used to intercept a business operation request, wherein the business operation request is a request for changing data in a business database. When the business database is in a batch running state, a business operation cache table is updated according to the business operation request, and the business operation cache table is used to cache data related to the business operation;

[0107] The processing component 420 is used to determine the batch processing job chain to be re-run according to the business operation cache table and the batch processing job chain information, and re-run the batch processing job chain to be re-run to obtain a batch processing result.

[0108] In some embodiments of the present application, the interception component 410 is specifically used to: when the business database is in a batch running state, determine whether the business database is in a forced batch running state; when the business database is not in a forced batch running state, determine whether a data table of the business database has a write lock; when there is no write lock or the write lock is released, parse the business operation request to obtain a parsed result of the business operation request, the parsed result including the business operation content, the business operation type, and the data affected by the business operation; update the business operation cache table according to the parsed result of the business operation request; wherein the business operation cache table stores the table name of the business data table affected by the business operation, the business operation type, the interception time, and the data processing status.

[0109] In some embodiments of the present application, the interception component 410 is specifically used for: if the business operation type is modification, searching for data of other fields according to the field data contained in the modification statement, and splicing the field data contained in the modification statement and the other found field data into complete field data and saving them in the business operation cache table; if the business operation type is insertion, saving the field data contained in the insert statement directly into the business operation cache table; if the business operation type is deletion, deleting the data inserted into the business operation cache table according to the field data contained in the delete statement.

[0110] In some embodiments of the present application, the processing component 420 is specifically used to: when the business database is in a batch running state, read the business operation cache table to obtain the table name of the business data table in the unprocessed state; determine the batch job chain to be re-run based on the table name of the business data table in the unprocessed state and the batch job chain information.

[0111] In some embodiments of the present application, the batch data processing device 400 also includes: a scheduling manager component, used to obtain the batch processing job chain information, the batch processing job chain information includes the execution status of the job chain and the input job information of the job chain, the execution status includes the job chain that has been completed, the running job chain and the queued job chain; the processing component 420 is specifically used to: match the table name of the business data table in the unprocessed state with the input job information of the job chain to determine the batch processing job chain to be re-run.

[0112] In some embodiments of the present application, the processing component 420 is specifically used to: call the job chain stop interface to stop the running job chain and the queued job chain in the batch job chain to be rerun; start the input job of the batch job chain to be rerun to rerun, and obtain the rerun result of the batch job chain to be rerun; write the rerun result of the batch job chain to be rerun into a batch result temporary table; when all batch job chains to be rerun have been rerun, lock the business data tables associated with all batch job chains to be rerun and suspend the business operations of the associated systems; write the rerun results in the batch result temporary table into the business data table, and release the lock of the business data table after writing is completed.

[0113] In some embodiments of the present application, the processing component 420 is specifically used to: determine the data extraction method of the batch job chain to be re-run; when the data extraction method of the batch job chain to be re-run is an incremental method, extract incremental data according to the business cache table; re-run the batch job chain to be re-run according to the incremental data and the extracted data, and obtain the re-run result of the batch job chain to be re-run.

[0114] It can be understood that the above-mentioned batch data processing device can implement each step of the batch data processing method provided in the above-mentioned embodiment, and the relevant explanations about the batch data processing method are applicable to the batch data processing device, which will not be repeated here.

[0115] In summary, the key points and technical effects of the batch data processing method and device of the present application are mainly:

[0116] 1) A new batch data processing solution is proposed. It uses the hierarchical architecture of batch processing jobs and realizes the full process control of batch jobs through unified scheduling management, eliminating a whole set of business temporary tables, account replenishment tables, and account reconciliation tables, reducing data redundancy and storage space occupation, and reducing the complexity of the system to achieve uninterrupted service. It is especially suitable for systems with low concurrency such as management systems;

[0117] 2) Through the full process control of batch job processing, the interception component is used to obtain conflict data, and the processing component and the scheduling manager component are used to control the operation process of the batch job chain. The functional components are pluggable, do not invade related systems, and are independent of the main business system process, which greatly reduces the code complexity of the management end.

[0118] The batch data processing method and device of the present application can be widely used in the construction and transformation of uninterrupted services of bank management systems, without the need for complex business debt collection functions and excessive database design, which greatly reduces the system construction cost and subsequent operation and maintenance costs and improves the system's operating efficiency.

[0119] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 5 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0120] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0121] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0122] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a batch data processing device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0123] Intercepting a business operation request, wherein the business operation request is a request to change data in a business database;

[0124] When the business database is in a batch running state, updating a business operation cache table according to the business operation request, wherein the business operation cache table is used to cache data related to the business operation;

[0125] Determine the batch processing job chain to be re-run according to the business operation cache table and the batch processing job chain information;

[0126] Re-run the batch processing job chain to be re-run to obtain a batch processing result.

[0127] The above application Figure 1The method performed by the batch data processing device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0128] The present application also provides a computer program product, including a computer program or an instruction, which, when executed by a processor, implements any of the aforementioned batch data processing methods, and is specifically used to execute:

[0129] Intercepting a business operation request, wherein the business operation request is a request to change data in a business database;

[0130] When the business database is in a batch running state, updating a business operation cache table according to the business operation request, wherein the business operation cache table is used to cache data related to the business operation;

[0131] Determine the batch processing job chain to be re-run according to the business operation cache table and the batch processing job chain information;

[0132] Re-run the batch processing job chain to be re-run to obtain a batch processing result.

[0133] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0137] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0138] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0139] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0140] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0141] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0142] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A batch data processing method, characterized in that: The batch data processing method comprises: Intercepting a business operation request, wherein the business operation request is a request to change data in a business database; When the business database is in a batch running state, updating a business operation cache table according to the business operation request, wherein the business operation cache table is used to cache data related to the business operation; Determine the batch processing job chain to be re-run according to the business operation cache table and the batch processing job chain information; Re-run the batch processing job chain to be re-run to obtain a batch processing result.

2. The batch data processing method according to claim 1, characterized in that: When the business database is in a batch running state, updating the business operation cache table according to the business operation request includes: When the business database is in a batch run state, determining whether the business database is in a forced batch run state; When the business database is not in a forced batch running state, determining whether a data table of the business database has a write lock; If the write lock does not exist or the write lock is released, the business operation request is parsed to obtain a parsing result of the business operation request, wherein the parsing result includes business operation content, business operation type, and data affected by the business operation; Updating the business operation cache table according to the parsing result of the business operation request; The business operation cache table stores the table name of the business data table affected by the business operation, the business operation type, the interception time and the data processing status.

3. The batch data processing method according to claim 2, characterized in that: The updating of the business operation cache table according to the parsing result of the business operation request includes: If the business operation type is modification, then search for data of other fields according to the field data contained in the modification statement, and splice the field data contained in the modification statement and the other found field data into complete field data and save them in the business operation cache table; If the business operation type is insert, the field data contained in the insert statement is directly saved in the business operation cache table; If the business operation type is deletion, the data inserted into the business operation cache table is deleted according to the field data contained in the deletion statement.

4. The batch data processing method according to claim 1, characterized in that: Determining the batch processing job chain to be re-run according to the business operation cache table and the batch processing job chain information includes: When the business database is in a batch running state, read the business operation cache table to obtain the table name of the business data table in an unprocessed state; The batch processing job chain to be re-run is determined according to the table name of the unprocessed business data table and the batch processing job chain information.

5. The batch data processing method according to claim 4, characterized in that: The determining the batch processing job chain to be rerun according to the table name of the unprocessed business data table and the batch processing job chain information comprises: Acquire the batch processing job chain information, the batch processing job chain information includes the execution status of the job chain and the input job information of the job chain, the execution status includes the job chain that has been completed, the job chain that is in operation, and the job chain in queue; The table name of the unprocessed business data table is matched with the input job information of the job chain to determine the batch job chain to be re-run.

6. The batch data processing method according to claim 1, characterized in that: Re-running the batch processing job chain to be re-run to obtain the batch processing result includes: For the running job chains and queued job chains in the batch job chain to be rerun, call the job chain stop interface to stop running and queuing; Starting the input job of the batch processing job chain to be rerun to rerun it, and obtaining the rerun result of the batch processing job chain to be rerun; Writing the rerun results of the batch processing job chain to be rerun into the batch processing result temporary table; When all batch processing job chains to be re-run have been re-run, the business data tables associated with all batch processing job chains to be re-run are locked and the business operations of the associated systems are suspended; The rerun results in the batch result temporary table are written into the business data table, and the lock of the business data table is released after the writing is completed.

7. The batch data processing method according to claim 6, characterized in that: The step of starting the input job of the batch processing job chain to be rerun to rerun the batch processing job chain to be rerun and obtaining the rerun result of the batch processing job chain to be rerun includes: Determine a data extraction method for the batch job chain to be rerun; When the data extraction mode of the batch job chain to be rerun is an incremental mode, extracting incremental data according to the business cache table; Re-running is performed based on the incremental data and the extracted data of the batch processing job chain to be re-run, so as to obtain the re-run result of the batch processing job chain to be re-run.

8. A batch data processing device, characterized in that: The batch data processing device comprises: An interception component, used to intercept a business operation request, wherein the business operation request is a request for changing data in a business database, and when the business database is in a batch running state, update a business operation cache table according to the business operation request, wherein the business operation cache table is used to cache data related to the business operation; The processing component is used to determine the batch processing job chain to be re-run according to the business operation cache table and the batch processing job chain information, and re-run the batch processing job chain to be re-run to obtain a batch processing result.

9. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes the batch data processing method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the batch data processing method according to any one of claims 1 to 7 is implemented.

Images