Batch data processing system and method

Abstract

The invention belongs to the technical field of data processing, and particularly relates to a batch data processing system and method, and the method comprises the steps: obtaining and analyzing a data file of a single-batch task, and obtaining a single-batch task ID, a single-batch task state and a packaging object; writing the single-batch task ID and the single-batch task state into a result database, and writing the packaging object into a cache database; obtaining data in the packaging object, the concurrent verification packaging object and the parallel-serial combined verification packaging object, and obtaining a to-be-stored packaging object, successfully verified data and unsuccessfully verified data; writing the to-be-stored packaging object, the successfully verified data and the unsuccessfully verified data into a cache database; and obtaining and removing the to-be-stored packaged object, writing data in the to-be-stored packaged object into the result database, and modifying the state of the single-batch task into an end state. Based on a distributed architecture of a scheduling cluster, a verification processing cluster and a data submission cluster, parallel processing of multi-task and multi-type data is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a batch data processing system and method. Background Art

[0002] The fare publishing system is responsible for the collection, processing and storage of relevant data such as fares and applicable rules, and provides them to the fare search and calculation system to participate in air ticket sales. Figure 1 This is a structural diagram of the freight rate publishing system. As can be seen from the figure, the general freight rate data collection module is responsible for processing the freight rates entered from the front end and transmitting them in the form of interface messages; the batch freight rate data collection module is responsible for processing the data files generated from the front end and the airline freight rate management system, and realizes the real-time collection of freight rate data files through batch import.

[0003] The freight rate files processed by the batch freight rate data collection module include more than 10 types, including groups, major customers, agreements, etc., and there are problems such as low data processing efficiency, long time consumption, and blockage in the import process. As airlines have connected with the air information system to publish freight rate data, the batch freight rate data collection module has assumed about 80% of the market share. The current processing performance can no longer meet the needs of the market and users, and it is urgently needed to be transformed and upgraded. Summary of the invention

[0004] In view of the above problems, the present invention proposes a batch data processing system and method.

[0005] A batch data processing system of the present invention comprises:

[0006] Processing cluster, used to process and verify single batch tasks;

[0007] A storage module, used for storing the results of processing and verifying the single batch task;

[0008] A configuration module is used to configure the execution and tracking of the single batch task.

[0009] Further,

[0010] The processing cluster includes a scheduling cluster, a verification processing cluster and a data submission cluster;

[0011] The scheduling cluster is used to obtain and parse the data file of the single batch task, obtain the single batch task ID, the single batch task status and the packaged object, write the single batch task ID and the single batch task status into the result database, and write the packaged object into the cache database;

[0012] The verification processing cluster is used to obtain and verify the packaged object, obtain the packaged object to be stored, the data that succeeded the verification and the data that failed the verification, and write the packaged object to be stored, the data that succeeded the verification and the data that failed the verification into the cache database;

[0013] The data submission cluster is used to obtain the packaged objects to be stored in the cache database, remove the packaged objects to be stored in the cache database, submit the data in the packaged objects to be stored in the result database, and modify the single batch task status to end.

[0014] Further,

[0015] The scheduling cluster is also used to periodically poll the tasks in the result database with the end status, read the verified data and the verified failed data from the cache database according to the single batch task ID, and remove the packaged objects from the cache database according to the single batch task ID.

[0016] Further,

[0017] The storage module includes a cache database and a result database;

[0018] The cache database is used to store the packaged objects, the packaged objects to be stored, the data that have passed the verification, and the data that have failed the verification;

[0019] The result database is used to store the single batch task ID, single batch task status and packaged objects to be stored.

[0020] The present invention also provides a batch data processing method based on the batch data processing system, the method comprising:

[0021] Obtain and parse the data file of a single batch task to obtain the single batch task ID, single batch task status and packaged object;

[0022] Write the single batch task ID and the batch task status into the result database, and write the packaged object into the cache database;

[0023] Acquire the packaged object, concurrently verify the packaged object, verify the data in the packaged object in combination with parallel and serial, and acquire the packaged object to be stored, the data that succeeded the verification, and the data that failed the verification;

[0024] Writing the packaged objects to be stored, the data that have passed the verification, and the data that have failed the verification into the cache database;

[0025] The packaged object to be stored is obtained from the cache database, the packaged object to be stored is removed from the cache database, the data in the packaged object to be stored is written into the result database, and the single batch task status is modified to be ended.

[0026] Further,

[0027] Also includes:

[0028] The tasks with end status in the result database are polled periodically, the data of verification success and verification failure are obtained from the cache database according to the single batch task ID, and the data is fed back to the user.

[0029] Further,

[0030] Also includes:

[0031] The tasks in the end status in the result database are polled periodically, and the packaged objects are removed from the cache database according to the single batch task ID.

[0032] Further,

[0033] The acquisition and parsing of the data file of the single batch task to obtain the single batch task ID, single batch task status and packaging object specifically includes:

[0034] Import the single batch task through the timed scheduling function to obtain a data file, and record the single batch task ID and single batch task status;

[0035] Parsing the data file to obtain a serializable data object;

[0036] The serializable data object is packaged to obtain the packaged object.

[0037] Further,

[0038] Concurrently verifying the packaged object means that different nodes of the verification processing cluster concurrently verify each packaged object.

[0039] Further,

[0040] The combination of parallel and serial refers to: overall serial and partial parallel.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention is based on the distributed architecture of a scheduling cluster, a verification processing cluster, and a data submission cluster, and realizes parallel processing of multiple tasks, multiple data packets, and multiple types of data.

[0043] The scheduled scheduling function avoids the situation where there are too many tasks and multiple batches of tasks are scheduled and imported at the same time, causing blockage, and achieves the technical effect of processing large quantities of data files in multiple batches of tasks with high concurrency.

[0044] According to business characteristics, the dependency between data when processing different types of data is optimized, and a processing mode combining parallel and serial processing is implemented, so that the system can process data files efficiently and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 It is a structural diagram of the existing freight rate publishing system;

[0047] Figure 2 It is a structural schematic diagram of a batch data processing system of the present invention;

[0048] Figure 3 is a flow chart of the batch data processing method of the present invention;

[0049] Figure 4 This is a schematic diagram of the data verification serial mode;

[0050] Figure 5 This is a schematic diagram of the data verification mode combining parallel and serial. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Figure 2 It is a structural diagram of a batch data processing system provided by an embodiment of the present invention, including a processing cluster, a storage module and a configuration module.

[0053] 1. Processing Clusters 101

[0054] The processing cluster is the core of this solution. It adopts a distributed architecture. The processing and verification of a single batch of tasks are the responsibility of the scheduling cluster, verification processing cluster, and data submission cluster. The details are as follows:

[0055] 1) Scheduling cluster

[0056] The scheduling cluster includes four functions: timed scheduling, file parsing, task decomposition, and task and node status query.

[0057] The scheduling cluster is used to obtain and parse the data files of single-batch tasks, obtain the single-batch task ID, single-batch task status and packaged objects, write the single-batch task ID and single-batch task status to the result database, and write the packaged objects to the cache database. It is also used to periodically poll the tasks with end status in the result database, read the data of successful verification and the data of failed verification from the cache database according to the single-batch task ID, and feed back these two types of data to the user in the form of emails, etc. At the same time, the packaged objects are removed from the cache database according to the single-batch task ID.

[0058] 2) Verify the processing cluster

[0059] The verification processing cluster is used to obtain and verify the packaged objects, obtain the packaged objects to be stored, the data that succeeded the verification, and the data that failed the verification, and write the packaged objects to be stored, the data that succeeded the verification, and the data that failed the verification into the cache database.

[0060] 3) Data submission cluster

[0061] The data submission cluster is used to obtain the packaged objects to be stored in the cache database, remove the packaged objects to be stored in the cache database, submit the data in the packaged objects to be stored in the result database, and change the status of the single batch task to end.

[0062] It is worth mentioning that the above three processing clusters must all have a task tracking function, which is responsible for recording the task information and results of the tasks processed for subsequent task tracking.

[0063] By coordinating and allocating the processing resources of the above three clusters, pipeline parallel processing of imported tasks in different clusters is achieved, and complex and time-consuming data verification logic is concentrated in the verification processing cluster, achieving maximum processing efficiency.

[0064] 2. Storage module 102

[0065] The storage module includes a cache database and a result database, which are used to store the results of processing and verifying a single batch of tasks, specifically:

[0066] The cache database is used to store packaged objects, packaged objects to be stored, data that has successfully been verified, and data that has failed verification.

[0067] The result database is used to store single batch task IDs, single batch task statuses, and packaged objects to be stored.

[0068] 3. Configuration module 103

[0069] The configuration library is used to configure the execution and tracking of single batch tasks, and can manage execution priorities according to business needs, thereby achieving differentiated services for users.

[0070] Based on the above batch data processing system, an embodiment of the present invention further provides a batch data processing method. The core of this method is to realize multi-task, multi-data packet, and multi-type data parallel processing. The key design points are as follows:

[0071] 1. An airline has multiple batches of tasks. Each batch of tasks (hereinafter referred to as "single batch tasks") has multiple data files. Each data file has multiple data, and the multiple data are divided into different types.

[0072] 2. A data package can contain a single data file in the form of text or table, or it can contain multiple data files in the form of compressed packages.

[0073] 3. Multiple batches of tasks from multiple airlines are processed in parallel on different cloud nodes of the scheduling cluster, verification processing cluster, and data submission cluster. Multiple batches of tasks from an airline must be processed serially in the timing of the above three clusters to ensure data consistency.

[0074] 4. The data of a single batch task can be divided into multiple subtask packages according to data relevance, and processed in parallel on cloud nodes in different verification processing clusters.

[0075] 5. A checksum processes different types of data in the subtask packages on the cloud nodes of the cluster and processes them in parallel according to the data dependencies.

[0076] Figure 3 The flow chart of the batch data processing method provided in the embodiment of the present invention takes the single batch task processing process of an airline as an example to illustrate. This process, as a complete data processing process with the smallest granularity, can be applied to the multi-batch task processing process of the aforementioned airline, and can also be applied to the multi-batch task processing process of multiple airlines.

[0077] S201: Acquire and parse the data file of a single batch task to obtain the single batch task ID, single batch task status and packaging object.

[0078] Specifically include:

[0079] S201-1: Import a single batch task through the scheduled scheduling function to obtain a data file, and record the single batch task ID and the single batch task status.

[0080] Pre-configure the scheduling mechanism, and import single batch tasks according to the scheduling mechanism through the scheduled scheduling function to avoid blocking caused by scheduling and importing multiple batches of tasks at the same time when there are too many tasks. The scheduling mechanism includes random scheduling and scheduling according to airline priority. Random scheduling is controlled by competition and randomly obtains tasks from a certain airline. Scheduling according to airline priority means obtaining tasks in order according to the airline priority.

[0081] The format of the data file obtained after importing a single batch task is XML, XLS, or a file compressed by XML.

[0082] After importing a single batch of tasks, you also need to obtain and record the single batch task ID and single batch task status so that you can track and process the tasks based on this later.

[0083] S201-2: Parse the data file to obtain a serializable data object.

[0084] S201-3: Pack the serializable data object to obtain a packaged object.

[0085] All serializable data objects under a single batch task can generate multiple packaged objects according to data relevance. The data in each packaged object is cohesive and has dependencies, and the data between different packaged objects are unrelated.

[0086] The packaged object contains various types of data, such as key customers, rules, agreements, freight rates, etc.

[0087] S202: Write the single batch task ID and the single batch task status into the result database, and write the packaged object into the cache database.

[0088] The purpose of writing the single batch task ID and single batch task status into the result database is to facilitate subsequent task tracking and processing.

[0089] S203: Obtain the packaged objects, verify each packaged object concurrently, and verify the data in the packaged objects in parallel and serially to obtain the packaged objects to be stored, the data that succeeded the verification, and the data that failed the verification.

[0090] When obtaining packaged objects from the cache database, you can use a preemptive approach or pre-configure airline priorities and obtain them in order based on priority.

[0091] When verifying a packaged object, different nodes in the verification processing cluster concurrently verify each packaged object.

[0092] In order to improve the system processing performance and efficiency, the present invention optimizes the verification method of different types of data in the packaged object. The scheme before optimization is the overall serial mode, such as Figure 4As shown in the figure, the optimized solution is a combination of parallel and serial modes, such as Figure 5 As shown. The parallel-serial combination mode refers to overall serial and partial parallel, which is derived from the characteristics and interdependencies of the freight rate business. Specifically, the verification of the agreed freight rate and the regular freight rate must rely on the rules. The rules refer to other rules except the refund rules and the change rules, such as flight restriction rules and time restriction rules. The verification of the rules must rely on the refund rules and the change rules. The freight rate release (ie "release") must rely on the group. The agreed freight rate and the regular freight rate can be processed in parallel, and the refund rules and the change rules can also be processed in parallel. Therefore, a parallel-serial combination processing design is carried out within a verification thread. For the three types of data, namely, key customers, fuel surcharges and fixed agency fees, since there is no tight business coupling relationship with other types of data, these three types of data can be processed in parallel with other data in independent threads.

[0093] By verifying the packaged object, the data that has been successfully verified and needs to be written into the result database, the data that has been successfully verified and the data that has failed to be verified are obtained. At the same time, the data that has been successfully verified and needs to be written into the result database is used to generate a new packaged object to be stored.

[0094] S204: Write the packaged objects to be stored, the data that has passed the verification, and the data that has failed the verification into the cache database.

[0095] S205: Obtain the packaged objects to be stored in the warehouse from the cache data, remove the packaged objects to be stored in the warehouse from the cache database, write the data in the packaged objects to be stored in the warehouse into the result database, and modify the single batch task status to end.

[0096] When obtaining the packaged objects to be stored in the cache database, you can use the preemptive method, or you can pre-configure the airline priority and obtain them in order according to the priority.

[0097] In the process of executing the above steps S201 to S205, it is necessary to periodically poll the tasks with end status in the result database, obtain the data with successful verification and the data with failed verification from the cache database according to the single batch task ID, and feed back these two types of data to the user in the form of email, etc. At the same time, remove the packaged objects from the cache database according to the single batch task ID.

[0098] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A batch data processing system, characterized in that: include: Processing cluster, used to process and verify single batch tasks; A storage module, used for storing the results of processing and verifying the single batch task; A configuration module is used to configure the execution and tracking of the single batch task.

2. The system according to claim 1, characterized in that The processing cluster includes a scheduling cluster, a verification processing cluster and a data submission cluster; The scheduling cluster is used to obtain and parse the data file of the single batch task, obtain the single batch task ID, the single batch task status and the packaged object, write the single batch task ID and the single batch task status into the result database, and write the packaged object into the cache database; The verification processing cluster is used to obtain and verify the packaged object, obtain the packaged object to be stored, the data that succeeded the verification and the data that failed the verification, and write the packaged object to be stored, the data that succeeded the verification and the data that failed the verification into the cache database; The data submission cluster is used to obtain the packaged objects to be stored in the cache database, remove the packaged objects to be stored in the cache database, submit the data in the packaged objects to be stored in the result database, and modify the single batch task status to end.

3. The system according to claim 2, characterized in that The scheduling cluster is also used to periodically poll the tasks in the result database with the end status, read the successfully verified data and the failed verified data from the cache database according to the single batch task ID, and remove the packaged objects from the cache database according to the single batch task ID.

4. The system according to claim 3, characterized in that The storage module includes a cache database and a result database; The cache database is used to store the packaged objects, the packaged objects to be stored, the data that have passed the verification, and the data that have failed the verification; The result database is used to store the single batch task ID, single batch task status and packaged objects to be stored.

5. A batch data processing method, characterized in that: include: Obtain and parse the data file of a single batch task to obtain the single batch task ID, single batch task status and packaged object; Write the single batch task ID and the batch task status into the result database, and write the packaged object into the cache database; Acquire the packaged object, verify the packaged object concurrently, verify the data in the packaged object in parallel and serial combination, and acquire the packaged object to be stored, the data that succeeded the verification, and the data that failed the verification; Writing the packaged objects to be stored, the data that have passed the verification, and the data that have failed the verification into the cache database; The packaged object to be stored is obtained from the cache database, the packaged object to be stored is removed from the cache database, the data in the packaged object to be stored is written into the result database, and the single batch task status is modified to be ended.

6. The method according to claim 5, characterized in that Also includes: The tasks with end status in the result database are polled periodically, the data of verification success and verification failure are obtained from the cache database according to the single batch task ID, and the data is fed back to the user.

7. The method according to claim 5, characterized in that Also includes: The tasks in the end status in the result database are polled periodically, and the packaged objects are removed from the cache database according to the single batch task ID.

8. The method according to claim 5, characterized in that The acquisition and parsing of the data file of the single batch task to obtain the single batch task ID, single batch task status and packaging object specifically includes: Import the single batch task through the timed scheduling function to obtain a data file, and record the single batch task ID and single batch task status; Parsing the data file to obtain a serializable data object; The serializable data object is packaged to obtain the packaged object.

9. The method according to claim 5, characterized in that Concurrently verifying the packaged object means that different nodes of the verification processing cluster concurrently verify each packaged object.

10. The method according to claim 5, characterized in that The combination of parallel and serial refers to: overall serial and partial parallel.

Images