Batch table design method for solving high concurrency problem of big data system

Abstract

The invention discloses a batch table design method for solving the high concurrency problem of a big data system. The method comprises the following steps of: constructing a high-availability Hadoop cluster, a Redis active-active cluster, a DolphinScheduler cluster, a Spark, a Hive and a Mysql; storing the whole amount of business data generated by T + 1 scheduling every day into a Hive table; meanwhile, the full-amount data is stored in a Redis active-active cluster in a specific array format through Spark calculation, and a latest batch is formed; meanwhile, the batch information is recorded in Mysql; rechecking the Redis active-active cluster on the basis of batch information of the Mysql; periodically cleaning expired batches of data based on a Redis active-active cluster; and the task is scheduled on the basis of a DolphinScheduler cluster.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a batch table design method for solving the high concurrency problem of a big data system. Background Art

[0002] China is the country with the fastest Internet development. Due to its large population base, valuable Internet applications need to face massive user requests and humanized user experience, especially in Internet finance, e-commerce platforms, insurance, payment, education and other industries. Their systems are in urgent need of high concurrency, high throughput and low latency. Traditional big data focuses on the optimization of business processes and HBase databases in terms of high concurrency, high throughput and low latency. Although the pressure of big data systems in high concurrency scenarios can be temporarily alleviated by adding machines and expanding the cluster size, with the continuous growth of data volume and complexity, such expansions are gradually unbalanced in terms of input and effect, and it is difficult to continuously meet system requirements. Summary of the invention

[0003] In view of the problems existing in the prior art, the present invention provides a batch table design method for solving the high concurrency problem of big data systems. By using an excellent and reliable batch table as a data transfer station, multi-dimensional, multi-gradient, multi-batch, and multi-copy high fault tolerance, it ensures high-speed data flow and improves the processing capability and stability of the system.

[0004] Glossary:

[0005] 1. Hadoop: An excellent open source big data processing platform that enables value mining of massive data through distributed storage and computing technology.

[0006] 2. Redis: A high-performance in-memory database.

[0007] 3. Spark: An efficient distributed computing engine that can quickly process large-scale data sets and is suitable for tasks that require high-speed data processing.

[0008] 4. Hive: A data warehouse solution based on Hadoop, which provides SQL query functions to facilitate users to process and analyze large-scale data.

[0009] 5. DolphinScheduler: is a big data workflow scheduling system that provides rich functions to manage and coordinate complex data processing processes.

[0010] 6. MySQL: A relational database management system known for its high performance, reliability and flexibility.

[0011] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0012] A batch table design method for solving the high concurrency problem of a big data system includes the following steps:

[0013] Step 1: Build a highly available Hadoop cluster and Spark cluster as the computing cluster for the batch table and the batch data processing engine. Build the DolphinScheduler cluster to schedule intermediate tasks. Build Hive to save the original table data. Build a Reids active-active cluster to save the final data. Install MySQL to record batch data.

[0014] Step 2: The data that needs to be saved in the database in the business process is called the first-stage data set. The first-stage data set is cleaned and calculated by the Spark cluster and then saved in Hive. To ensure the timeliness of the data, arrange the tasks reasonably, avoid resource conflicts, and improve the utilization of cluster resources.

[0015] Step 3: Write the first-stage data set into the Hash structure of the Redis active-active cluster in a specific array format string, which is called the second-stage data set to ensure that the array is not out of order.

[0016] Step 4: After the second-stage data set is successfully stored in the database, the batch information is recorded in MySQL, a new batch number is generated, and its uniqueness is guaranteed.

[0017] Step 5: Verify the integrity of Redis active-active data based on the batch information in MySQL.

[0018] Step 6: Regularly clean up expired batch data in the Redis active-active cluster.

[0019] Step 7: Abnormal recovery mechanism: After the normal business process is completed, back up the business data in step 2 to Hive.

[0020] Preferably, the Redis active-active cluster needs to be used independently and not shared with other application systems.

[0021] Preferably, the Spark cluster is used to implement the calculation of the first-stage data set and the second-stage data, the batch is implemented based on the Hash structure of the Redis active-active cluster, and the task scheduling execution is guaranteed according to the DolphinScheduler cluster. The various components need to cooperate with each other to ensure the integrity of the data without loss.

[0022] Optimally, the Redis dual-active synchronization mechanism has independent scheduling and cross-backup. The synchronization process must be the same and the same set of codes. The Spark cluster must have sufficient computing resources to generate batch tables, Hive must have enough disks to store the first-stage data, and the Redis dual-active cluster memory must be large enough to store the second-stage multi-batch data. When the first-stage data set and the second-stage data set are abnormal, there must be a remediation and recovery mechanism.

[0023] Preferably, the Redis active-active cluster is required to serve as a backup. If any cluster data is lost or unavailable, it can be quickly restored through the Spark cluster. There must be a mechanism for detecting liveness and deadness of the Redis active-active cluster. The first-stage data must be backed up, and the data of any day in the recent period can be restored. When the data is abnormal or the cluster is abnormal, an alarm message must be sent to the enterprise micro robot for timely notification and manual intervention.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] First: it reduces system storage and improves transmission efficiency.

[0026] Second: The batch table design solves the difficulty of deleting historical data, avoids the high IO caused by full table scans, and improves the stability of the system. Under the same conditions, the concurrency and throughput are also greatly improved.

[0027] Third: Through systematic modularization, the system complexity is reduced and the system maintainability is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is an architectural diagram of a batch table design method for solving the high concurrency problem of a big data system according to the present invention.

[0029] Figure 2 It is a detailed flow chart of the present invention. DETAILED DESCRIPTION

[0030] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0031] like Figure 1 , 2 As shown in the figure, a batch table design method for solving the high concurrency problem of a big data system includes the following steps:

[0032] Step 1. Build a highly available Hadoop cluster and Spark cluster as the computing cluster for the batch table and act as the batch data processing engine. The memory and computing power that the Hadoop cluster can provide are particularly important and determine the computing power of the Spark cluster. Build a DolphinScheduler cluster to schedule intermediate tasks. Due to the large number of tasks, the DolphinScheduler cluster needs to deploy more worker nodes. Build Hive to save the original table data and build a Reids active-active cluster to save the final data. Since it is batch data, the capacity of the Redis active-active cluster must be 4-5 times the original business volume. Install MySQL to record batch data.

[0033] Step 2: The data that needs to be saved in the database in the business process is called the first-stage data set. The first-stage data set is cleaned and calculated by the Spark cluster and then saved in Hive. To ensure the timeliness of the data, arrange the tasks reasonably, avoid resource conflicts, and improve the utilization of cluster resources. The concurrency of tasks can be controlled by the DolphinScheduler cluster task group project.

[0034] Step 3: Write the first-stage data set into the Hash structure of the Redis active-active cluster in a specific array format string, which is called the second-stage data set to ensure that the array is not out of order. Use a specific Bean object to clarify the order of the fields, and put the newly added fields at the end. All tables must use the same set of codes to generate a specific array format to ensure the consistency of the Redis active-active cluster data. Use placeholders to separate fields. If the original table data is null or in other situations, placeholders must be present to ensure that the array is not out of order.

[0035] Step 4: After the second-stage data set is successfully stored in the database, the batch information is recorded in MySQL, and a new batch number is generated and guaranteed to be unique. The batch number is generated based on the date, accurate to the second, and guaranteed to be unique.

[0036] Step 5: Verify the integrity of Redis active-active data based on the batch information in MySQL. After the batch table is stored in the Redis active-active cluster, it will become 16,384 hash tables to evenly distribute the data. Verify the memory size, total number of keys, and fault tolerance of the batch table. The memory size and number of keys need to be recorded in the batch information.

[0037] Step 6. Regularly clean up expired batches of data in the Redis active-active cluster. The Redis active-active cluster will store multiple batches of data. When a new batch is written, the expired batches must be deleted to ensure that the batches in use are not deleted.

[0038] Step 7: Abnormal recovery mechanism. After the normal business process is completed, back up the business data in step 2 to Hive. The data for the past seven days needs to be backed up so that the data can be quickly restored when the Redis active-active cluster is abnormal or a batch of data is abnormal.

[0039] Batch data synchronization of Redis active-active clusters must be done with the same process, that is, the same code. The synchronization entry is different and runs in two tasks. The synchronization process needs to include the following processes:

[0040] 1. Both tasks are scheduled in the DolphinScheduler cluster and must have the same dependencies to ensure that the tasks are scheduled and run at the same time.

[0041] 2. The core process of the synchronization task should be encapsulated in an independent part, including conversion into a special array format and generation of batch data.

[0042] The batch table design method of the present invention solves the high concurrency problem of the big data system, reduces system storage and improves transmission efficiency. The batch table design solves the difficulty of deleting historical data, avoids the high IO caused by full table scanning, improves the stability of the system, and under the same conditions, the concurrency and throughput are greatly improved. The system complexity is reduced through systematic modularization, and the system maintainability is improved.

[0043] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A batch table design method for solving the high concurrency problem of a big data system, characterized in that: The steps include: Step 1: Build a highly available Hadoop cluster and Spark cluster as the computing cluster for the batch table and the batch data processing engine. Build the DolphinScheduler cluster to schedule intermediate tasks. Build Hive to save the original table data. Build a Reids active-active cluster to save the final data. Install MySQL to record batch data. Step 2: The data that needs to be saved in the database in the business process is called the first-stage data set. The first-stage data set is cleaned and calculated through the Spark cluster and then saved in Hive; Step 3: Write the first-stage data set into the Hash structure of the Redis active-active cluster in a specific array format string, which is called the second-stage data set; Step 4: After the second-stage data set is successfully stored in the database, the batch information is recorded in MySQL. A new batch number must be generated and its uniqueness must be guaranteed. Step 5: Verify the integrity of Redis active-active data based on the batch information in MySQL. Step 6: Regularly clean up expired batch data of the Redis active-active cluster; Step 7: After the normal business process is completed, back up the business data in step 2 to Hive.

2. A batch table design method for solving the high concurrency problem of a big data system according to claim 1, characterized in that: The Redis active-active cluster needs to be used independently and not shared with other application systems.

3. A batch table design method for solving the high concurrency problem of a big data system according to claim 1, characterized in that: The Spark cluster is used to implement the calculation of the first-stage data set and the second-stage data. The Hash structure based on the Redis active-active cluster is used to implement batches. The DolphinScheduler cluster is used to ensure task scheduling and execution.

4. A batch table design method for solving the high concurrency problem of a big data system according to claim 3, characterized in that: Redis dual-active synchronization mechanism, independent scheduling, cross-backup; the synchronization process must be the same and the same set of code.

5. The batch table design method for solving the high concurrency problem of a big data system according to claim 1, characterized in that: The Spark cluster has enough computing resources to generate batch tables, Hive has enough disks to store the data in the first phase, and the Redis active-active cluster has enough memory to store multiple batches of data in the second phase.

6. A batch table design method for solving the high concurrency problem of a big data system according to claim 1, characterized in that: When anomalies occur in the first-stage and second-stage datasets, there are remediation and recovery mechanisms.

7. The batch table design method for solving the high concurrency problem of a big data system according to claim 1, characterized in that: Redis active-active clusters are required to serve as mutual backup. If data in any cluster is lost or unavailable, it can be quickly restored through the Spark cluster.

8. The batch table design method for solving the high concurrency problem of a big data system according to claim 7, characterized in that: The Redis active-active cluster has a live and dead detection mechanism. The data in the first phase is backed up, and the data of any day in the recent period can be restored.

9. A batch table design method for solving the high concurrency problem of a big data system according to claim 8, characterized in that: When data or cluster anomalies occur, an alarm message must be sent to the enterprise micro-robot for timely notification and manual intervention.

Images