A system dynamic fault-tolerant method and system based on deep reinforcement learning

Abstract

This application belongs to the field of fault tolerance technology for distributed stream processing systems, and more specifically, relates to a dynamic fault tolerance method and system based on deep reinforcement learning. This application characterizes the deployment status of each task at each time step by obtaining the target task graph corresponding to each task deployment operation at that time step. During the encoding of the target task graph by the GCN, the GRU is used in conjunction with the GCN weight matrix of the previous time step to dynamically update the GCN weight matrix of the current time step, thereby accurately obtaining the output feature matrix including the dynamic temporal dependency features of the target task graph. Based on the resource consumption of each task, the current resource load status and resource load capacity of each processor, the target training reward function of the target reinforcement learning framework is obtained. Then, the target graph decoder can be trained based on the training penalty function, ensuring that the processors executing each task in the final allocation scheme will not fail due to resource overload.

Description

Technical Field

[0001] This application belongs to the field of fault tolerance technology for distributed stream processing systems, and more specifically, relates to a system and method for dynamic fault tolerance based on deep reinforcement learning. Background Technology

[0002] In recent years, with the rapid development of big data, cloud computing, and real-time computing technologies, distributed stream processing systems have become the core support for processing massive real-time data streams and are widely used in key areas such as financial risk control, IoT real-time analysis, industrial interconnection, and intelligent recommendation. As system business complexity increases and processing scale continues to expand, problems such as node failures, network fluctuations, and resource contention occur frequently. Ensuring the reliability and fault tolerance of the system during dynamic operation has become a core industry requirement. At the same time, the requirements for fault tolerance overhead, fault recovery efficiency, and resource utilization are becoming increasingly stringent. Traditional fault tolerance methods based on static backup and upstream backup are difficult to adapt to dynamically changing task scenarios. Fault tolerance methods for dynamically deployed systems have become a research focus and technical challenge in the field of distributed stream processing.

[0003] With the development of deep learning and reinforcement learning technologies, existing research has combined graph neural networks with reinforcement learning to transform the fault tolerance problem into a resource allocation and task deployment problem. However, these methods still have the following obvious shortcomings: 1. Existing technologies mostly use static stream processing topology graphs for modeling, extracting features only based on the task structure at a single time step. Static modeling cannot accurately express the deployment state and dynamic dependencies of the task at different time steps, leading to a disconnect between node resource allocation decisions and real-world scenarios. 2. Existing fault tolerance methods mostly use single graph convolutional networks (GCNs) for graph encoding, which can only extract spatial structure features and lacks temporal dynamic update capabilities. Although some solutions introduce temporal models such as LSTM (Long Short-Term Memory) to process the allocation sequence during the decoding stage, temporal modeling does not participate in the graph structure encoding process. Node features still come from the static graph convolution results, which cannot effectively capture the dynamic characteristics of the task graph evolving over time and is difficult to adapt to scenarios where node states continuously change. 3. Existing fault-tolerance methods, when using deep reinforcement learning models to obtain node resource allocation results, cannot guarantee the load safety of the processors allocated to each node, which can easily lead to processor overload and cause node task execution failure. Summary of the Invention

[0004] To address the aforementioned deficiencies in existing technologies, this application provides a system dynamic fault tolerance method and system based on deep reinforcement learning. The aim is to solve the problems of low fault tolerance efficiency, high fault tolerance overhead, and unreasonable processor allocation for tasks caused by the static node topology graph modeling, insufficient extraction of temporal dynamic features of the topology graph, lack of overload constraints in resource allocation results, and instability in the reinforcement learning training process of existing fault tolerance methods.

[0005] In a first aspect, this application provides a system dynamic fault-tolerance method based on deep reinforcement learning, characterized by comprising: S1. Based on the temporal execution relationship between tasks in the obtained distributed stream processing system, construct a spatiotemporal stream processing topology graph, where one node in the spatiotemporal stream processing topology graph corresponds to one task. S2. Based on the spatiotemporal flow processing topology graph, obtain the target task graph corresponding to each time step within the total task processing cycle. The target task graph includes the deployment status of each task at each time step. S3. Based on the target graph encoder, each target task graph is encoded sequentially according to the time step order to obtain the target output feature matrix corresponding to each target task graph; S4. The target graph decoder trained by the target reinforcement learning framework performs decoding operations on the output feature matrix of each target with the goal of minimizing the fault tolerance overhead of each task, and obtains the final allocation scheme of each task. The final allocation scheme includes the final allocation state of each task on each processor.

[0006] Furthermore, based on the spatiotemporal flow processing topology graph, the target task graph corresponding to each time step within the total task processing cycle is obtained, including: Get the deployment status of each task in the spatiotemporal flow processing topology map within the total task processing cycle after completing a task deployment operation. Each task deployment operation completes the deployment of at least one task. The deployment status includes the deployment status of each task and the processor allocation status corresponding to the deployment status. The deployment status includes deployed, not deployed, and being deployed. Based on the deployment status of each task at each time step, construct the target task graph for each time step.

[0007] Furthermore, step S3 includes: S31. Perform graph convolution operation on the target task graph corresponding to the current time step using GCN to obtain the initial output feature matrix corresponding to the current time step. Each row of the initial output feature matrix corresponds to the initial feature vector of a node. S32. Based on the initial feature vector and the weight matrix of the GCN corresponding to the previous time step, update the weight matrix of the GCN corresponding to the current time step through GRU, and perform graph convolution operation on the target task graph corresponding to the current time step through the GCN updated with the weight matrix to obtain the target output feature matrix.

[0008] Furthermore, based on the initial feature vector and the weight matrix of the GCN corresponding to the previous time step, the weight matrix of the GCN corresponding to the current time step is updated using GRU, including: The GRU-based reset gate filters the weight matrix of the GCN corresponding to the previous time step of the current time step element by element to obtain the historical weight matrix. Based on the initial feature vector, the learnable matrix corresponding to the initial feature vector, the historical weight matrix, and the learnable matrix corresponding to the historical weight matrix, the candidate weight matrix is ​​obtained; The fusion ratio between the candidate weight matrix and the weight matrix corresponding to the previous time step is adjusted through the update gate of GRU, and the weight matrix of GCN corresponding to the current time step is updated based on the fusion ratio.

[0009] The GRU (Gated Recurrent Unit) is a recurrent neural network composed of reset and update gates. The reset gate controls the degree to which historical feature information is retained in the current time step calculation, while the update gate controls the fusion ratio between historical and current feature information. Compared to the existing LSTM structure, it is simpler and has lower computational overhead. Since the method in this application needs to dynamically adjust the weight matrix of the GCN at each time step to improve the accuracy of the feature information in the encoded target output feature matrix, and the temporal dependencies between time steps are short, involving only the current time step and the previous time step, GRU is chosen to adjust the encoding process of the GCN for each target task graph. This effectively captures the dynamic evolution of the target task graph between consecutive time steps, thereby effectively improving the accuracy of the obtained target output feature matrix of each target task graph.

[0010] Furthermore, the target reinforcement learning framework is an Actor-Critic training algorithm framework.

[0011] Furthermore, the output feature matrix of each target is decoded, including: Obtain the computational resource graph corresponding to the spatiotemporal stream processing topology graph from the distributed stream processing system. The computational resource graph includes the processor allocation information for each task. The decoding process of the target graph decoder is constrained based on the computational resource graph, and the output feature matrix of each target is decoded based on the target graph decoder to obtain the processor-assignable probability vector for each task. Based on the processor-assignable probability vector corresponding to each task, the final allocation state of each task on each processor is obtained.

[0012] Furthermore, the process of training the object graph decoder using the object reinforcement learning framework includes: Based on the resource consumption of each task, the current resource load status and resource load capacity of each processor, the training penalty function of the target reinforcement learning framework is obtained, and the goal of the training reward function of the target reinforcement learning framework is to minimize the system fault tolerance overhead. Based on the training reward function and the training penalty function, the target training reward function is obtained, and the decoder is trained based on the target training reward function.

[0013] Secondly, this application provides a system dynamic fault-tolerant system based on deep reinforcement learning, characterized in that it includes: The topology graph acquisition module is used to construct a spatiotemporal stream processing topology graph based on the temporal execution relationships between tasks in the acquired distributed stream processing system. The task graph acquisition module is used to acquire the target task graph corresponding to each time step within the total task processing cycle based on the spatiotemporal flow processing topology graph. The graph encoding module is used to encode each target task graph sequentially according to the time step order based on the target graph encoder, so as to obtain the target output feature matrix corresponding to each target task graph; The graph decoding module is used to decode the target graph decoder trained by the target reinforcement learning framework, with the goal of minimizing the fault tolerance cost of each task, and to obtain the processor-assignable probability vector corresponding to each task. The allocation scheme acquisition module is used to obtain the final allocation status of each task on each processor based on the processor allocability probability vector corresponding to each task.

[0014] Thirdly, this application also provides an electronic device, characterized in that it comprises: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, such that, when the program is executed, the processor performs the method described in the first aspect or any possible implementation thereof.

[0015] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0016] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art: 1. This application characterizes the deployment status of each task at each time step by obtaining the target task graph corresponding to each time step of the task deployment operation. During the encoding of the target task graph by the GCN, the GRU is used in conjunction with the GCN weight matrix of the previous time step to dynamically update the GCN weight matrix of the current time step. This allows for the accurate acquisition of the output feature matrix, including the dynamic temporal dependency features of the target task graph, thus making the final allocation scheme more consistent with the needs of real-world system operation scenarios. 2. By obtaining the target training reward function of the target reinforcement learning framework based on the resource consumption of each task, the current resource load status of each processor, and the resource load capacity, the target graph decoder can be trained based on the training penalty function. This ensures that the processors executing each task in the final allocation scheme will not fail due to resource overload. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced one by one below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the system dynamic fault tolerance method provided in the embodiments of this application.

[0019] Figure 2 This is a diagram illustrating the evolution of the target task diagrams provided in the embodiments of this application.

[0020] Figure 3 This is another flowchart illustrating the system dynamic fault tolerance method provided in the embodiments of this application.

[0021] Figure 4 This is a schematic diagram of the structure of a dynamic fault-tolerant system based on deep reinforcement learning provided in an embodiment of this application.

[0022] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0024] In the following description, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The following description provides multiple embodiments of this application, which can be substituted or combined with each other. Therefore, this application can also be considered to include all possible combinations of the same and / or different embodiments described. Thus, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then this application should also be considered to include embodiments containing one or more other possible combinations of A, B, C, and D, even if such embodiments are not explicitly described in the following text.

[0025] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the described elements without departing from the scope of this application. Various processes or components may be appropriately omitted, substituted, or added to the examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined into other examples.

[0026] like Figure 1 As shown, the system dynamic fault-tolerance method based on deep reinforcement learning provided in this application includes at least the following steps: S1. Based on the temporal execution relationship between tasks in the obtained distributed stream processing system, construct a spatiotemporal stream processing topology graph, where one node in the spatiotemporal stream processing topology graph corresponds to one task.

[0027] In the embodiments of this application, the execution subject of the method is the central control processor of the distributed stream processing system. The system dynamic fault tolerance problem is transformed into a resource allocation and task deployment problem. By dynamically and reasonably deploying each task within the total task cycle to each resource processor for execution, the system fault tolerance overhead is reduced and the system fault tolerance efficiency is improved.

[0028] S2. Based on the spatiotemporal flow processing topology graph, obtain the target task graph corresponding to each time step within the total task processing cycle. The target task graph includes the deployment status of each task at each time step.

[0029] In one possible implementation, step S2 includes: obtaining the deployment status of each task in the spatiotemporal flow processing topology map within the total task processing cycle after completing a task deployment operation, wherein each task deployment operation completes the deployment of at least one task. Based on the deployment status of each task at each time step, construct the target task graph for each time step.

[0030] In the embodiments of this application, such as Figure 2As shown in the figure, the temporal changes in the task deployment status of the target task graph at each time step are illustrated. The data flow direction of each edge in the target task graph represents the temporal execution relationship of each task node in the spatiotemporal stream processing topology graph. The target task graph is based on the spatiotemporal stream processing topology graph, with the task deployment status of each node added. The deployment status includes the deployment state of each task and the corresponding processor allocation state. The deployment state includes deployed, not deployed, and being deployed. The processor allocation state corresponds to the processor assigned to each deployed or being deployed node in the graph. For example, the first node in the graph is assigned to processor number 0. t1-t4 in the graph represent different time steps. Furthermore, each task deployment operation is executed according to the temporal relationship between tasks. Only after the deployment of the preceding task is completed will the deployment of the next group of tasks be executed. Sometimes a single task deployment operation deploys a single task, and sometimes it deploys multiple tasks.

[0031] S3. Based on the target graph encoder, each target task graph is encoded sequentially according to the time step order to obtain the target output feature matrix corresponding to each target task graph.

[0032] In one possible implementation, step S3 includes: By performing graph convolution on the target task graph corresponding to the current time step using GCN, the initial output feature matrix corresponding to the current time step is obtained. Each row of the initial output feature matrix corresponds to the initial feature vector of a node. The GRU-based reset gate filters the weight matrix of the GCN corresponding to the previous time step of the current time step element by element to obtain the historical weight matrix. Based on the initial feature vector, the learnable matrix corresponding to the initial feature vector, the historical weight matrix, and the learnable matrix corresponding to the historical weight matrix, the candidate weight matrix is ​​obtained; The fusion ratio between the candidate weight matrix and the weight matrix corresponding to the previous time step is adjusted through the update gate of GRU, and the weight matrix of GCN corresponding to the current time step is updated based on the fusion ratio.

[0033] The updated GCN with weight matrix is ​​used to perform graph convolution on the target task graph at the current time step to obtain the target output feature matrix.

[0034] In the embodiments of this application, such as Figure 3 As shown, this embodiment of the application selects GCN as the target graph encoder. Unlike encoding a single static graph, this invention encodes each time step. Construct the corresponding task graph Furthermore, graph convolution calculations are performed separately to characterize the structural features of the task at different time stages. Specifically, for each time step... The corresponding task diagram When performing graph convolution calculations, graph convolution operations are performed on each layer of the task graph. The operation functions for each convolution layer of the graph convolution are defined as follows:

[0035] in, Represents graph convolution operations. The first representing GCN Convolutional layers, Represents the time step t. The feature matrix output by the convolutional layer, This represents the input at time step t. The input feature matrix of each convolutional layer Represents the task graph at time step t. The adjacency matrix is ​​used to represent the task graph. The connection characteristics between the nodes, Represents the task graph at time step t. The adjacency matrix after adding self-loops to each node. , It is the identity matrix. yes The degree matrix, For activation function, Indicates at time step Next The weight matrix of a layered graph convolutional network.

[0036] Building upon this foundation, to model the dynamic changes in graph structure over time, this invention introduces a parameter update mechanism on top of the static GCN. Specifically, gated recurrent units (GRUs) are used to dynamically update the weight parameters of the graph convolutional network.

[0037] GCN in time step t No. l The layer weight update methods, weight update, update gate, reset gate, and candidate weight formulas are as follows:

[0038] in," " indicates an element-wise addition operation. This indicates a parameter update operation. 't' represents the time step index. ; This represents the layer index of a graph convolutional network; Indicates at time step Next The feature matrix of the layer graph convolutional network corresponds to the initial feature vector, where each row corresponds to the feature vector of a task node; Indicates time step Next Layer weight matrix; Indicates time step -1st The weight matrix of the layer graph convolutional network corresponds to the historical weight matrix in this application; The candidate weight matrix can be represented by the current node and preset together with the historical weights. Indicates at time step Next The update gate output of the layer graph convolutional network is adjustable. and The fusion ratio, This indicates a reset of the gate output. Representative candidate weight matrix China indicates Learnable matrix, This represents the historical weight matrix corresponding to the candidate weights. The learnable matrix.

[0039] S4. The target graph decoder trained by the target reinforcement learning framework performs decoding operations on the output feature matrix of each target with the goal of minimizing the fault tolerance overhead of each task, and obtains the final allocation scheme of each task. The final allocation scheme includes the final allocation state of each task on each processor.

[0040] In one possible implementation, decoding the output feature matrix of each target includes: Obtain the computational resource graph corresponding to the spatiotemporal stream processing topology graph from the distributed stream processing system. The computational resource graph includes the processor allocation information for each task. The decoding process of the target graph decoder is constrained based on the computational resource graph, and the output feature matrix of each target is decoded based on the target graph decoder to obtain the processor-assignable probability vector for each task. Based on the processor-assignable probability vector corresponding to each task, the final allocation state of each task on each processor is obtained.

[0041] In the embodiments of this application, such as Figure 3 As shown, the target graph decoder is built on an attention model and includes at least an attention layer and a normalization layer. The graph decoder performs matching and scoring operations on the target output features in the form of node embeddings extracted from the GCN at each time step with each processor. In the figure, X1-X4 represent different time steps. t i The target output feature, E s1 -E s4 Indicates different time steps t i The task-processor matching score is calculated. The task-processor allocation problem can be described as:

[0042] Represents the given task graph and computational resource map Under the condition of complete allocation scheme The conditional probability; It represents the complete task allocation scheme, including the mapping relationship between all task nodes and processor resources; This represents a task graph, where nodes represent tasks and edges represent data dependencies between tasks. This represents a resource graph (such as CPU cores, computing units, etc.), where each node represents available processor resources. The symbol for multiplication indicates that the probabilities of all task nodes i are multiplied, reflecting that the allocation process is generated sequentially. This represents the conditional probability of the assignment of the i-th task under given conditions; Indicates the sequence number of the task currently being assigned; Indicates the first The time steps corresponding to each task; Indicates at time step The following are the resource allocation decisions for task nodes; in addition, in order to handle new allocations... In relation to previous assignments, reinforcement learning models learn a state representation to encode associated information, set. This is used to represent the result of the mean pool for the data backup task embedding. The above vector is concatenated with the data backup task embedding of its parent node and used as input to the attention layer.

[0043] Finally, to achieve prediction for each node, we use single-head attention to simulate all nodes, with each node receiving an attention score. Then use softmax on all Normalization is performed to obtain the processor-assignable probability vector.

[0044] Furthermore, the reinforcement learning framework proposed in this application adopts the Actor-Critic training algorithm framework. The reward objective of the loss function during the training process is the system fault tolerance overhead. By generating resource allocation decisions through the Actor network and evaluating the value of the decisions in real time through the Critic network, the training variance is effectively reduced, the sample utilization rate and policy stability are improved, and the optimal resource allocation strategy can be learned more accurately and efficiently, significantly reducing the system fault tolerance overhead and improving dynamic fault tolerance performance.

[0045] In one possible implementation, the process of training the target graph decoder using a target reinforcement learning framework includes: Based on the resource consumption of each task, the current resource load status and resource load capacity of each processor, the training penalty function of the target reinforcement learning framework is obtained, and the goal of the training reward function of the target reinforcement learning framework is to minimize the system fault tolerance overhead. Based on the training reward function and the training penalty function, the target training reward function is obtained, and the decoder is trained based on the target training reward function.

[0046] In this embodiment of the application, to avoid processor overload during resource allocation during reinforcement learning training, the present invention introduces an evaluation rule based on resource constraints to determine the current task allocation scheme. For example... Figure 3 As shown, the decoder introduces an evaluation module to constrain the decoding process of the target output feature matrix, specifically by using the resource load state of each processor as a decoding constraint. This is done at each time step. Below, for the current task allocation result, first calculate the resource load of each processor, where, let the processor... The set of tasks already assigned is ,Task The resource requirements are Then the processor The current load is .

[0047] Furthermore, the processor is set up The maximum resource load capacity is When satisfied When this occurs, the processor is determined to be overloaded. Based on this, a negative reward is introduced to the current allocation scheme, with the penalty term defined as follows: ,in This is the penalty coefficient, used to control the intensity of the penalty. In a multi-tasking scenario, when multiple processors are overloaded simultaneously, the penalty terms for all overloaded processors are summed to obtain the total penalty term. ,in This represents the set of processors that experienced overload. Finally, the penalty term is fused with the original reward to obtain the final reward function. This allows the model to be influenced by both the prediction results and resource constraints when making task allocation decisions, achieving dynamic constraints on multi-task resource contention scenarios and avoiding unreasonable task concentration.

[0048] Figure 4 A schematic diagram of the structure of a system dynamic fault-tolerant based on deep reinforcement learning is provided for an embodiment of this application, as shown below. Figure 4 As shown, the system includes at least: The topology graph acquisition module is used to construct a spatiotemporal stream processing topology graph based on the temporal execution relationships between tasks in the acquired distributed stream processing system. The task graph acquisition module is used to acquire the target task graph corresponding to each time step within the total task processing cycle based on the spatiotemporal flow processing topology graph. The graph encoding module is used to encode each target task graph sequentially according to the time step order based on the target graph encoder, so as to obtain the target output feature matrix corresponding to each target task graph; The graph decoding module is used to decode the target graph decoder trained by the target reinforcement learning framework, with the goal of minimizing the fault tolerance cost of each task, and to obtain the processor-assignable probability vector corresponding to each task. The allocation scheme acquisition module is used to obtain the final allocation status of each task on each processor based on the processor allocability probability vector corresponding to each task.

[0049] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0050] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0051] It is understood that the processor in the embodiments of this application can be a CPU (Central Processing Unit), or other general-purpose processors, DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0052] like Figure 5 As shown, Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: a processor 501, a communications interface 502, a memory 503, and a communication bus 504. The processor 501, communications interface 502, and memory 503 communicate with each other via the communication bus 504. The processor 501 can call software instructions in the memory 503 to execute the methods described in the above embodiments.

[0053] Furthermore, the logical instructions in the aforementioned memory 503 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application.

[0054] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, ROM (Read-only Memory), PROM (Programmable ROM), EPROM (Erasable PROM), EEPROM (Electrically Erasable EPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0055] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line DSL) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD (Solid State Disk)).

[0056] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0057] Those skilled in the art will readily understand that the above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A system dynamic fault-tolerant method based on deep reinforcement learning, characterized in that, include: S1. Based on the temporal execution relationship between tasks in the obtained distributed stream processing system, a spatiotemporal stream processing topology graph is constructed, wherein one node of the spatiotemporal stream processing topology graph corresponds to one task; S2. Based on the spatiotemporal flow processing topology map, obtain the target task map corresponding to each time step within the total task processing cycle. The target task map includes the deployment status of each task at the time step. S3. Based on the target graph encoder, each target task graph is encoded sequentially according to the time step order to obtain the target output feature matrix corresponding to each target task graph; S4. The target graph decoder trained by the target reinforcement learning framework performs decoding operations on the output feature matrix of each target with the goal of minimizing the fault tolerance overhead of each task, to obtain the final allocation scheme of each task. The final allocation scheme includes the final allocation state of each task on each processor.

2. The system dynamic fault tolerance method of claim 1, wherein, Based on the aforementioned spatiotemporal flow processing topology graph, the target task graph corresponding to each time step within the total task processing cycle is obtained, including: The deployment status of each task in the spatiotemporal flow processing topology map within the total task processing cycle is obtained after completing a task deployment operation. Each task deployment operation completes the deployment of at least one task. The deployment status includes the deployment status of each task and the processor allocation status corresponding to the deployment status. The deployment status includes deployed, not deployed, and being deployed. Based on the deployment status of each task corresponding to each time step, a target task graph corresponding to each time step is constructed.

3. The system dynamic fault tolerance method of claim 1, wherein, The target map encoder is a GCN, and step S3 includes: S31. Perform graph convolution operation on the target task graph corresponding to the current time step using GCN to obtain the initial output feature matrix corresponding to the current time step. Each row of the initial output feature matrix corresponds to the initial feature vector of a node. S32. Based on the initial feature vector and the weight matrix of the GCN corresponding to the previous time step of the current time step, update the weight matrix of the GCN corresponding to the current time step through GRU, and perform graph convolution operation on the target task graph corresponding to the current time step through the GCN updated with the weight matrix to obtain the target output feature matrix.

4. The system dynamic fault tolerance method of claim 3, wherein, The step of updating the weight matrix of the GCN corresponding to the current time step using GRU based on the initial feature vector and the weight matrix of the GCN corresponding to the previous time step of the current time step includes: The GRU reset gate is used to filter the weight matrix of the GCN corresponding to the previous time step of the current time step element by element to obtain the historical weight matrix. Based on the initial feature vector, the learnable matrix corresponding to the initial feature vector, the historical weight matrix, and the learnable matrix corresponding to the historical weight matrix, a candidate weight matrix is ​​obtained; The fusion ratio between the candidate weight matrix and the weight matrix corresponding to the previous time step is adjusted through the update gate of the GRU, and the weight matrix of the GCN corresponding to the current time step is updated based on the fusion ratio.

5. The system dynamic fault tolerance method of claim 1, wherein, The target reinforcement learning framework is the Actor-Critic training algorithm framework.

6. The system dynamic fault tolerance method of claim 1, wherein, The decoding operation for each of the target output feature matrices includes: Obtain the computing resource graph corresponding to the spatiotemporal stream processing topology graph from the distributed stream processing system. The computing resource graph includes the processor allocation information for each task. The decoding process of the target graph decoder is constrained based on the computing resource graph, and the output feature matrix of each target is decoded based on the target graph decoder to obtain the processor-allocated probability vector corresponding to each task. Based on the processor-assignable probability vector corresponding to each task, the final allocation state of each task on each processor is obtained.

7. The system dynamic fault tolerance method of claim 1, wherein, The process of training the target graph decoder using the target reinforcement learning framework includes: Based on the resource consumption of each task, the current resource load status and resource load capacity of each processor, the training penalty function of the target reinforcement learning framework is obtained, and the objective of the training reward function of the target reinforcement learning framework is to minimize the system fault tolerance overhead. Based on the training reward function and the training penalty function, a target training reward function is obtained, and the decoder is trained based on the target training reward function.

8. A deep reinforcement learning based system dynamic fault tolerance system for performing the method of any one of claims 1-7, characterized in that, include: The topology graph acquisition module is used to construct a spatiotemporal stream processing topology graph based on the temporal execution relationships between tasks in the acquired distributed stream processing system. The task graph acquisition module is used to acquire the target task graph corresponding to each time step within the total task processing cycle based on the spatiotemporal flow processing topology graph. The graph encoding module is used to encode each of the target task graphs sequentially according to the time step order based on the target graph encoder, so as to obtain the target output feature matrix corresponding to each of the target task graphs. The graph decoding module is used to decode the output feature matrix of each target by a target graph decoder trained by the target reinforcement learning framework, with the goal of minimizing the fault tolerance cost of each task, to obtain the processor-assignable probability vector corresponding to each task. The allocation scheme acquisition module is used to obtain the final allocation status of each task on each processor based on the processor allocability probability vector corresponding to each task.

9. An electronic device, comprising: include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, the computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the method as claimed in any one of claims 1-7.

Images