A flink-based distributed disaster recovery rehearsal thread pool optimization and adaptation method

Abstract

This invention discloses a Flink-based method for optimizing and adapting thread pools in distributed disaster recovery drills. It comprehensively employs distributed data convergence and utilizes edge nodes within local networks to train a central model, thereby optimizing the central training model and significantly reducing its training load. Flink is used as the primary computing engine for the disaster recovery drill data warehouse to address a critical aspect of distributed disaster recovery drill solutions. Thread pool optimization and scheduling address the adaptation issues between local and cross-local privacy data storage and the computing engine. The shared slot feature unique to Flink's thread pool merges multiple computational optimization chains. Simultaneously, an AI-based central early warning model is used to predict the resource overhead, infrastructure, and performance overhead of each stage of the computational chain, providing an optimal balance between these factors.

Description

Technical Field

[0001] This invention belongs to the field of software development and IT technology, specifically involving a method for optimizing and adapting a distributed disaster recovery drill thread pool based on Flink. Background Technology

[0002] With the deepening of digital development, cybersecurity has become a crucial guarantee for social development, making disaster recovery drills even more valuable. The collection and processing of disaster recovery data is a vital component of these drills. Comprehensive and accurate information collection ensures the normal execution of all tasks within the drill. The drill's processing closely resembles the handling process during a real disaster, ensuring the drill's effectiveness and providing a valuable reference for data maintenance through automated disaster recovery exercises.

[0003] The disaster recovery drill process information of existing IT business systems is incomplete, and the privacy data of edge nodes in various regions affects the training of the central model. Distributed applications differ from traditional monolithic applications. Different computing nodes have different hardware configurations, and the operation of thread pools also varies. Parameters such as the number of worker threads, the maximum number of worker threads, and the size of the task queue in the thread pool are mainly determined by the experience of project developers, lacking mature and reliable methods for advance planning. When the configuration of the thread pool does not meet the actual business needs, it is easy to cause a series of problems such as resource waste and task blocking, reducing the availability of the software project or even causing the software project to crash. Summary of the Invention

[0004] This invention addresses the problems of existing technologies by providing a Flink-based method for optimizing and adapting distributed disaster recovery exercise thread pools. This method applies artificial intelligence to disaster recovery exercises and reduces the load on central model training through thread pool adaptation technology, while ensuring that the distributed disaster recovery exercise data does not differ between central and local data.

[0005] To achieve the above-mentioned objectives, the technical solution of the present invention is as follows: An optimization and adaptation method for a distributed disaster recovery drill thread pool based on Flink, including the following steps: Step 1: Build and deploy a central early warning model on the central server; the formula for the central early warning model is: Where: P(A) is the probability of a disaster assessment failure occurring each day, P(B) is the percentage of false alarms after a disaster assessment failure occurs, P(A|B) is the probability that this type of failure will still occur after the failure is recovered, and P(B|A) is the probability of a disaster assessment failure occurring each day. Step 2: Create a thread pool and map the local disaster recovery exercise data to operators according to the exercise stages, and map the entire exercise process to an operation chain; Step 3: Map each operation chain to a TaskManager process; The JobManager manages the data operation of the distributed disaster recovery exercise site through the ActorSystem and participates in the operation of the central early warning model to obtain the failure probability value of each operator, and allocates priority threads based on the probability value, so as to make full use of the Task Slots (process slots) resources and distribute larger operators evenly on the TaskManager. Step 4: Finally, manage the flow of disaster recovery drill data to JobManagers through the Client.

[0006] Furthermore, the exercise includes alarm detection, disaster assessment, and disaster confirmation.

[0007] Furthermore, the JobManager is responsible for coordinating the transmission of local privacy and public data to the central database in the form of distributed tasks via a thread pool; The TaskManager is responsible for mapping a process that transmits data from the ground to the central system. Task Slots are mapped to threads, and a process can contain multiple threads. The TaskManager connects memory Task Slots (process slots) to memory resources, maps the data generated in each stage of the disaster recovery exercise to operators and puts them into the central early warning model for calculation. The probability value of the predicted operator's anomaly is also put into the operator. Finally, the operators of each stage are combined into a calculation chain. The thread pool is responsible for prioritizing threads after receiving the operation chain containing operator prediction probability values. The Client is responsible for preparing and submitting data streams to JobManagers. After that, it can disconnect or keep the connection open to obtain task status information.

[0008] Furthermore, the priority thread allocation includes: Prioritize sending operation chain threads with an operator prediction probability value of no more than 50% to multiple shared TaskSlots in a sequential and even manner, thereby making full use of the Task Slots (process slots) resources so that larger operators can be evenly distributed on the TaskManager. After the second priority weighted average, no more than 50% of the operation chain is sent to multiple shared Task Slots in sequence and evenly, so as to make full use of the Task Slots (process slots) resources and enable larger operators to be evenly distributed on the TaskManager. If the prediction probability of an operator in the computation chain is greater than or equal to 50%, or the weighted average of the computation chain is greater than or equal to 50%, then it is placed into multiple Task Slots threads within a separately created TaskManager process. Based on the number of computation chains and the memory size they occupy, operators with prediction probabilities greater than 50% and those with higher costs are evenly distributed across TaskManagers. This allows operators with prediction probabilities greater than 50% to share a single TaskManager process, and to share TCP connections and heartbeat messages with other TaskManagers under the same JobManager. They can also share some datasets and data structures, thereby reducing task overhead.

[0009] The advantages and beneficial effects of this invention are as follows: This invention uses Flink as the primary computing engine for the disaster recovery drill database. Flink is a distributed processing engine for both streaming and batch data. It addresses a key aspect of distributed disaster recovery drill solutions: adapting local and cross-regional privacy data storage to the computing engine through thread pool optimization and scheduling. Flink's thread pool, with its unique shared slot feature, merges multiple computational optimization chains. Simultaneously, it employs an AI-based central early warning model to provide an optimal balance between monitoring metrics mapping resource overhead, infrastructure, and performance overhead across the prediction computation chain. This method leverages Flink's distributed execution characteristics to offer a novel model for distributed disaster recovery drill data computation. Specifically, thread pool adaptation technology reduces the load on the central model training while ensuring data integrity across central and local disaster recovery drills. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the basic architecture of an embodiment of the method of the present invention. Detailed Implementation

[0011] The present invention will be further described below with reference to specific embodiments.

[0012] Example: An optimization and adaptation method for a distributed disaster recovery drill thread pool based on Flink, including the following steps: Step 1: Build and deploy a central early warning model on the central server; the formula for the central early warning model is: Where: P(A) is the probability of a disaster assessment failure occurring each day, P(B) is the percentage of false alarms after a disaster assessment failure occurs, P(A|B) is the probability that this type of failure will still occur after the failure is recovered, and P(B|A) is the probability of a disaster assessment failure occurring each day. For example, first, access the disaster recovery database to obtain disaster assessment data from disaster recovery drills in various regions.

[0013] P(A): Probability of daily disaster assessment failure: 1 (0 = none, 1 = yes); P(B): Percentage of false alarms following a disaster assessment failure. Number of false alarms in disaster assessment / Total number of fault recovery attempts [0.02] P(A|B): The probability that this type of fault will still occur after the fault is recovered. 1 - (Number of false alarms for the current type of fault / Number of faults of the current type still occurring) = 0.9 P(B|A): The probability of a disaster assessment failure occurring daily can be calculated using the formula: P(B|A) = 0.9 * 0. 02 / 1 = 0.018; Step 2: Create a thread pool and map the local disaster recovery exercise data to operators according to the exercise stages, and map the entire exercise process to an operation chain; Step 3: Map each operation chain to a TaskManager process; The JobManager manages the data operation of the distributed disaster recovery exercise site through the ActorSystem and participates in the operation of the central early warning model to obtain the failure probability value of each operator, and allocates priority threads based on the probability value, so as to make full use of the Task Slots (process slots) resources and distribute larger operators evenly on the TaskManager. Step 4: Finally, manage the flow of disaster recovery drill data to JobManagers through the Client.

[0014] The exercise included alarm detection, disaster assessment, and disaster confirmation.

[0015] The JobManager is responsible for coordinating the transmission of local privacy and public data to the central database in the form of distributed tasks via a thread pool. The TaskManager is responsible for mapping a process that transmits data from the local area to the central government. Task Slots are mapped to threads, which are used for local and cross-local privacy data exchange and transmission. A process can contain multiple threads. The TaskManager connects memory Task Slots (process slots) with memory resources, maps the data generated in each stage of the disaster recovery exercise into operators and puts them into the central early warning model for calculation. The probability value of the predicted operator's anomaly is also put into the operator. Finally, the operators of each stage are combined into a calculation chain. The thread pool is responsible for prioritizing threads after receiving the operation chain containing operator prediction probability values. The Client is responsible for preparing and submitting data streams to JobManagers. After that, it can disconnect or keep the connection open to obtain task status information.

[0016] The priority thread allocation includes: The algorithm prioritizes sending operation chain threads with prediction probability values ​​of no operator greater than 50% to multiple shared Task Slots in a sequential and even manner, thereby making full use of the Task Slots (process slots) resources and enabling larger operators to be evenly distributed on the TaskManager.

[0017] After the second-priority weighted average, no more than 50% of the operation chains are sent to the shared multiple Task Slots in a sequential and even manner, thereby making full use of the Task Slots (process slots) resources and enabling larger operators to be evenly distributed on the TaskManager.

[0018] If the predicted probability value of an operator in the computation chain is greater than or equal to 50%, or the weighted average value of the computation chain is greater than or equal to 50%, then the operator is placed in the process of multiple Task Slots threads within a separately created TaskManager process. Based on the number of computation chains and the memory size they occupy, operators with a predicted probability value greater than 50% and those with higher costs are evenly distributed across the TaskManager. This allows operators with a predicted probability value greater than 50% to share a single TaskManager process, and to share TCP connections and heartbeat messages with other TaskManagers under the same JobManager. They can also share some datasets and data structures, thereby reducing task overhead.

[0019] The default number of Task Slots should ideally be equal to the number of CPU cores. The method of chaining operators into tasks can reduce data handover and caching between threads, improve overall throughput, and reduce processing latency. On a TaskManager with multiple slots, operators share a single process, can share TCP connections and heartbeat messages, and can also share some datasets and data structures, thereby reducing task overhead. The JobManagers, or masters, coordinate the execution of distributed tasks. They are used to schedule tasks, coordinate checkpoints, coordinate error recovery, etc. At least one JobManager is required, but a highly available system may have multiple JobManagers, one leader and the others as standby. The TaskManagers, or workers, are used to execute data stream tasks or operators, cache and interact with data streams, and at least one TaskManager is required. The concepts of threads and thread pools are as follows: A thread is a single sequential flow of control within a process. Multiple threads can run concurrently within a process, each executing different tasks in parallel. If an application needs to use threads multiple times, it means that threads need to be created and destroyed multiple times, which inevitably consumes memory. Thread pools can reuse threads, minimizing the need for repeated thread creation and destruction, thus reducing memory consumption.

Claims

1. A method for optimizing and adapting a distributed disaster recovery drill thread pool based on Flink, characterized in that: Includes the following steps: Step 1: Build and deploy a central early warning model on the central server; the formula for the central early warning model is: ; Where: P(A) is the probability of a disaster assessment failure occurring each day, P(B) is the percentage of false alarms after a disaster assessment failure occurs, P(A|B) is the probability that this type of failure will still occur after the failure is recovered, and P(B|A) is the probability of a disaster assessment failure occurring each day. Step 2: Create a thread pool and map the local disaster recovery drill data to operators according to the drill stages, and map the entire drill process to an operation chain; the drill stages include alarm detection, disaster assessment, and disaster confirmation; Step 3: Map each operation chain to a TaskManager process; The JobManager manages the data operation of the distributed disaster recovery exercise site through the Actor System and participates in the operation of the central early warning model to obtain the failure probability value of each operator, and allocates priority threads based on the probability value, so as to make full use of the resources of Task Slots and distribute larger operators evenly on TaskManager. The JobManager is responsible for coordinating the transmission of local privacy and public data to the central database in the form of distributed tasks via a thread pool. The TaskManager is responsible for mapping data from the ground to the central process. Task Slots are mapped to threads, and a process can contain multiple threads. The TaskManager connects memory Task Slots to memory resources, maps the data generated in each stage of the disaster recovery exercise to operators, puts them into the central early warning model for calculation, predicts the probability value of anomalies in the operators, and finally combines the operators of each stage into a calculation chain. The default number of Task Slots is the number of CPU cores. The thread pool is responsible for prioritizing threads after receiving the operation chain containing operator prediction probability values. The Client is responsible for preparing and submitting data streams to JobManagers. After that, it can disconnect or maintain the connection to obtain task status information. The priority thread allocation includes: Prioritize sending operation chain threads with prediction probability values ​​of no operator greater than 50% to multiple shared TaskSlots in a sequential and even manner, thereby making full use of TaskSlots resources and enabling larger operators to be evenly distributed on TaskManager. After the second priority weighted average, no more than 50% of the operation chain is sent to the shared multiple Task Slots in sequence and evenly, so as to make full use of the resources of Task Slots and enable larger operators to be evenly distributed on TaskManager. If the prediction probability of an operator in the operation chain is greater than or equal to 50%, or the weighted average of the operation chain is greater than or equal to 50%, then it is placed in the process of multiple Task Slots threads within a separately created TaskManager process. Based on the number of operation chains and the memory size occupied, operators with prediction probabilities greater than 50% and those with high costs are evenly distributed on the TaskManager. This allows operators with prediction probabilities greater than 50% to share a single TaskManager process, and to share TCP connections and heartbeat messages with other TaskManagers under the same JobManager. They can also share some datasets and data structures, thereby reducing task overhead. Step 4: Finally, manage the flow of disaster recovery drill data to JobManagers through the Client.

Images