A distributed disaster recovery drill data optimization and early warning method based on data warehouse

Abstract

This invention discloses a distributed disaster recovery drill data optimization and early warning method based on a data warehouse. The method includes: deploying a real-time data warehouse based on Iceberg data lake technology on a central server, using Flink and Spark streaming data to manage data online and offline, and calculating process pool occupancy in real time; training a central early warning model in a distributed manner. When the process pool occupancy falls below a set value, it prioritizes cross-regional private disaster recovery drill data calculations, enabling real-time tracking and capture of memory anomalies and early warning based on the central early warning model; submitting model training results from various locations to a central server to update a headquarters drill result database, storing the drill results after each training session, and storing private data in the real-time data warehouse. This method can effectively overcome model bias and reduce computational complexity.

Description

Technical Field

[0001] The present invention belongs to the technical field of disaster recovery drills, and in particular relates to a distributed disaster recovery drill data optimization and early warning method based on a data warehouse. Background Art

[0002] With the deepening development of digitalization, cybersecurity has become a crucial element of social development, making disaster recovery drills even more valuable. The collection and processing of disaster recovery data is a crucial component of these drills. Comprehensive information collection and accurate data ensure the proper execution of all drill tasks. The drill process closely resembles that of a real disaster, ensuring that the drill is effective and making automated disaster recovery drills a valuable reference for data maintenance. However, existing IT business systems often lack comprehensive information on business processes during disaster recovery drills, and issues such as privacy concerns at local edge nodes impacting central model training exist.

[0003] As big data technology continues to evolve, more and more concepts are being proposed and applied in production. The data lake concept is one of these. A data lake is a centralized repository that can store structured and unstructured data of any scale, supporting big data and AI computing. As a core component of the cloud-native data lake architecture, the Data Lake Formation (DLF) service helps users quickly and easily build cloud-native data lake solutions. Data Lake Formation provides unified metadata management on the lake, enterprise-level permission control, and seamless integration with multiple computing engines, breaking down data silos and unlocking insights into business value. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned existing technologies and provide a distributed disaster recovery drill data optimization and early warning method based on a data warehouse. The method comprehensively adopts distributed data convergence and utilizes edge nodes in local networks to train a central model, thereby optimizing the central training model and significantly reducing the training load of the central model. Drill data is collected and analyzed through edge nodes in various locations. By classifying data according to event reporting, disaster assessment, and disaster declaration, big data calculations and central data uploads are performed. At the same time, Flink and Spark streaming data are used to transmit data, and the problem of offline and online data being out of sync is stored in memory and synchronized, thereby significantly reducing resource consumption for massive data calculations.

[0005] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0006] A distributed disaster recovery drill data optimization and early warning method based on a data warehouse, comprising:

[0007] Step 1: Deploy a real-time data warehouse based on Iceberg Data Lake technology on the central server, use Flink and Spark streaming data to manage data online and offline, and calculate the process pool occupancy rate in real time;

[0008] Step 2: Distributed training of the central warning model. When the process pool occupancy rate falls below the set value, priority is given to cross-regional privacy disaster recovery drill data calculations to achieve real-time tracking and capture of memory anomalies and early warning based on the central warning model.

[0009] Step 3: Submit the model training results from various locations to the central server to update the headquarters' exercise result database, store the exercise results after each training session, and store the private data in a real-time data warehouse.

[0010] To optimize the above technical solutions, specific measures taken also include:

[0011] The above step 1 includes:

[0012] Step 1: Deploy a real-time data warehouse based on Iceberg Data Lake technology on the central server. The real-time data warehouse includes the computing layer Flink, Spark, Hive, Presto, the storage layer ORC, Parquet, AVRO, and the middle layer TableFormat.

[0013] Step 2: Pass the file formats and binary data structures of the ORC, Parquet, and AVRO column storage in the storage layer to the middle layer Table Format for data format unification.

[0014] At the same time, a process pool is created to track the thread occupancy and mapping memory usage of the Flink, Spark, Hive, and Presto data streams in the computing layer in real time.

[0015] Step 3: The middle-layer Table Format passes the data in a unified format to Flink. Flink uses FlinkSQL to call the streaming data and cache the data with the front-end Redis.

[0016] The calculation formula for the above thread occupancy is as follows:

[0017]

[0018] Where N is the number of worker threads in the thread pool at runtime;

[0019] N max Is the maximum number of threads set;

[0020] Describes the saturation of worker threads;

[0021] T cur is the number of tasks in the current collection time window;

[0022] T pre is the number of tasks in the previous acquisition time window;

[0023] Q is the task buffer queue size;

[0024] Describe the current task saturation;

[0025] Describes the task buffer queue growth rate;

[0026] ξ is the weight coefficient.

[0027] The above step 2 includes:

[0028] Step 1: The central server deploys the disaster recovery drill result database and the drill association database, and the local server deploys the local drill database;

[0029] Step 2: The central server program deployed at the headquarters executes the first round of central early warning model training start instructions, and sends the instructions to the local disaster recovery drill edge node servers, referred to as edge node servers;

[0030] Step 3: After receiving the central warning model, the edge node server aggregates the privacy data and transmits the data and performs model calculations to the central server.

[0031] Step 4: Use the Markov chain algorithm to build a central early warning model to predict the probability of failure during memory computing.

[0032] The specific steps of step 3 above are as follows:

[0033] Step 31: Classify and aggregate local exercise data and the database related to the exercise to form real and effective privacy analysis data and store it in the real-time data warehouse;

[0034] Step 32: All disaster recovery drill data in the local drill database is classified according to the drill steps through the program execution deployed on the edge node servers in various locations. The obtained n drill steps are mapped into n links of Flink streaming data, and the memory usage of the n drill steps is put into Spark's memory calculation.

[0035] The above-mentioned drill steps include incident reporting, disaster assessment, and disaster declaration classification.

[0036] The central warning model constructed in step 4 above is: X(k+1)=X(k)×P

[0037] Where: X(k) represents the state vector of the trend analysis and prediction object at time t=k;

[0038] P represents the one-step transition probability matrix;

[0039] X(k+1) represents the state vector of the trend analysis and prediction object at time t=k+1.

[0040] The present invention has the following beneficial effects:

[0041] This invention deploys a real-time data warehouse based on Iceberg Data Lake technology on a central server, providing a foundational environment for performance optimization and adaptation of big data computing. Disaster recovery drill data is transmitted in the form of a Flink data stream, combined with a process pool algorithm. The process pool occupancy rate is less than 50%, prioritizing cross-regional disaster recovery drill data operations with privacy. This can improve the speed of massive data computing and transmission. At the same time, combined with the Spark engine's advantages in in-memory computing, it can track and capture memory anomalies in real time and provide early warnings through a central early warning model. This invention can effectively overcome model bias issues and reduce computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of the distributed disaster recovery drill data optimization and early warning principle based on the data warehouse of the present invention. DETAILED DESCRIPTION

[0043] The embodiments of the present invention are described in further detail below with reference to the accompanying drawings.

[0044] like Figure 1 As shown, the present invention provides a distributed disaster recovery drill data optimization and early warning method based on a data warehouse, comprising:

[0045] Step 1: Deploy a real-time data warehouse based on Iceberg Data Lake technology on the central server, use Flink and Spark streaming data to manage data online and offline, and calculate the process pool occupancy rate in real time;

[0046] Step 1: Deploy a real-time data warehouse based on the Iceberg data lake technology on the central server. The real-time data warehouse includes the computing layer flink, spark, Hive, Presto, the storage layer ORC, Parquet, AVRO and the middle layer TableFormat. The real-time data warehouse architecture is as follows: Figure 1 shown.

[0047] Formally, Iceberg is a table format. Specifically, it is an intermediate layer table format based on the computing layer (Flink, Spark, Hive, and Presto) and the storage layer (ORC, Parquet, and AVRO). It is synchronized with the created process pool and provides a basic environment for performance optimization and adaptation of big data computing.

[0048] Step 2: Pass the file formats, binary data structures, and other data stored in the storage layer, such as ORC, Parquet, and AVRO, to the middle layer Table Format for data format unification.

[0049] At the same time, a process pool is created to track the thread occupancy and mapping memory usage of the Flink, Spark, Hive, and Presto data streams in the computing layer in real time, and to predict memory anomalies through the [central early warning model] (detailed explanation in step 2 below).

[0050] Thread occupancy: that is, the thread pool load indicator ω.

[0051] The load degree is converted from data such as the number of working threads, the maximum number of threads, and the size of the task buffer queue when the thread pool is running, and a percentage value is calculated through different weighted proportions.

[0052] The thread occupancy formula is as follows:

[0053]

[0054] Where N is the number of worker threads in the thread pool at runtime;

[0055] N max Is the maximum number of threads set;

[0056] Describes the saturation of worker threads;

[0057] T cur is the number of tasks in the current collection time window;

[0058] T pre is the number of tasks in the previous acquisition time window;

[0059] Q is the task buffer queue size;

[0060] Describe the current task saturation;

[0061] Describes the task buffer queue growth rate;

[0062] ξ is the weight coefficient.

[0063] Compare the preset thread pool load ω′. If it is greater than ω′, the adaptive parameter adjustment calculation is triggered; otherwise, the current collection time window is skipped.

[0064] Step 3: The middle-layer Table Format passes the data in a unified format to Flink. Flink uses FlinkSQL to call the streaming data and cache the data with the front-end Redis.

[0065] Glossary:

[0066] Spark provides in-memory computing, storing intermediate results in memory, resulting in higher iterative computing efficiency. By supporting a programming framework for distributed parallel computing based on directed acyclic graphs (DAGs), Spark reduces the need to write data to disk during iterations, improving processing efficiency.

[0067] Everything in Flink is made up of streams. Offline data is a bounded stream, and real-time data is an unbounded stream.

[0068] Unbounded data stream: An unbounded data stream has a beginning but no end;

[0069] Bounded Data Stream: A bounded data stream has a well-defined beginning and end.

[0070] Step 2: Distributed training of the central warning model. When the process pool occupancy rate falls below the set value, priority is given to cross-regional privacy disaster recovery drill data calculations to achieve real-time tracking and capture of memory anomalies and early warning based on the central warning model.

[0071] Step 1: The central server deploys the disaster recovery drill result database and the drill association database, and the local server deploys the local drill database;

[0072] Step 2: The central server program deployed at the headquarters executes the first round of central early warning model training start instructions, and sends the instructions to the local disaster recovery drill edge node servers, referred to as edge node servers;

[0073] Step 3: After receiving the central warning model, the edge node server aggregates the privacy data and then transmits the data and performs model calculations to the central server. The specific steps are as follows:

[0074] Step 31: Classify and aggregate local exercise data and the database related to the exercise to form real and effective privacy analysis data and store it in the real-time data warehouse;

[0075] Step 32: All disaster recovery drill data in the local drill database are classified according to the drill steps through the program execution deployed by the edge node servers in various locations;

[0076] The drill steps include incident reporting, hazard assessment, and disaster declaration classification;

[0077] Map the above three drill steps into the three stages of Flink streaming data, and put the memory usage of the three drill steps into Spark's memory calculation;

[0078] The advantage of this approach is that disaster recovery drill data is transmitted as a Flink data stream, combined with a process pool algorithm, with a process pool occupancy rate below 50%, prioritizing cross-regional disaster recovery drill data operations. This improves the speed of massive data processing and transmission. Furthermore, by leveraging the Spark engine's advantages in in-memory computing, memory anomalies can be tracked and captured in real time, providing early warnings through a central warning model.

[0079] Step 4: Use the Markov chain algorithm to build a central warning model to predict the probability of failure during memory calculation; the central warning model formula is: X(k+1)=X(k)×P

[0080] Where: X(k) represents the state vector of the trend analysis and prediction object at time t=k;

[0081] P represents the one-step transition probability matrix;

[0082] X(k+1) represents the state vector of the trend analysis and prediction object at time t=k+1.

[0083] Initial probability of historical data transmission anomaly: 0.3 0.7;

[0084] The probability of data transmission abnormality turning into normality is: 0.6, 0.4;

[0085] The probability of data transmission going from normal to abnormal is 0.3 0.7;

[0086] Model calculation process and results:

[0087] S1: Probability of next data transmission anomaly: 0.3x0.6+0.3x0.7=0.39;

[0088] S2: Probability of the next data transmission occurring normally: 0.3x0.4+0.7x0.7=0.61;

[0089] S3: The probability of abnormal and normal data transmission in the next time is 0.39 0.61.

[0090] Step 3: Submit the model training results from various locations to the central server to update the headquarters disaster recovery drill result database, store the drill results after each training, and then the central server program re-executes a new round of model training to complete the entire process.

[0091] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A distributed disaster recovery drill data optimization and early warning method based on a data warehouse, characterized in that: include: Step 1: Deploy a real-time data warehouse based on Iceberg Data Lake technology on the central server, use Flink and Spark streaming data to manage data online and offline, and calculate the thread occupancy rate of the process pool in real time; The calculation formula of the thread occupancy is as follows: Where N is the number of worker threads in the thread pool at runtime; N max Is the maximum number of threads set; Describes the saturation of the worker thread; T cur is the number of tasks in the current acquisition time window; T pre is the number of tasks in the previous acquisition time window; Q is the size of the task buffer queue; Describe the current task saturation; Describes the growth rate of the task buffer queue; ξ is the weight coefficient; Step 2: Distributed training of the central warning model. When the process pool occupancy rate falls below the set value, priority is given to cross-regional privacy disaster recovery drill data calculations to achieve real-time tracking and capture of memory anomalies and early warning based on the central warning model, including: Step 1: The central server deploys the disaster recovery drill result database and the drill association database, and the local server deploys the local drill database; Step 2: The central server program deployed at the headquarters executes the first round of central early warning model training start instructions, and sends the instructions to the local disaster recovery drill edge node servers, referred to as edge node servers; Step 3: After receiving the central warning model, the edge node server aggregates the privacy data and transmits the data and performs model calculations to the central server, including: Classify and aggregate local exercise data and databases related to exercise relationships to form authentic and effective privacy analysis data, which is then stored in a real-time data warehouse; Through the execution of programs deployed on edge node servers across the region, all disaster recovery drill data in the local drill database is classified according to the drill steps. The resulting n drill steps are mapped into n links of Flink streaming data. The memory usage of the n drill steps is also stored in Spark's in-memory calculations. Step 4: Use the Markov chain algorithm to build a central warning model to predict the probability of failure during the memory calculation process. The central warning model is: X(k+1)=X(k)×P; Where: X(k) represents the state vector of the trend analysis and prediction object at time t=k; P represents the one-step transition probability matrix; X(k+1) represents the state vector of the trend analysis and prediction object at time t=k+1; Step 3: Submit the model training results from various locations to the central server to update the headquarters' exercise result database, store the exercise results after each training session, and store the private data in a real-time data warehouse.

2. A distributed disaster recovery drill data optimization and early warning method based on a data warehouse according to claim 1, characterized in that: The step one comprises: Step 1: Deploy a real-time data warehouse based on Iceberg Data Lake technology on a central server. The real-time data warehouse includes a computing layer, a storage layer, and an intermediate layer. Step 2: Pass the ORC, Parquet, and AVRO file formats and binary data structures of the storage layer to the middle layer Table Format to unify the data formats. At the same time, a process pool is created to track the thread occupancy and mapping memory usage of the Flink, Spark, Hive, and Presto data streams in the computing layer in real time. Step 3: The middle-layer Table Format passes the data in a unified format to Flink. Flink uses FlinkSQL to call the streaming data and cache the data with the front-end Redis.

3. The method for optimizing and warning distributed disaster recovery drill data based on a data warehouse according to claim 1, characterized in that: The drill steps include incident reporting, disaster assessment, and disaster declaration classification.

Images