Distributed component flow online dynamic reconstruction method based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based distributed component flow online dynamic reconstruction method, which is applied to a complex and high-dynamic distributed system or cloud computing environment, and is used for carrying out component scheduling in real time by using a reinforcement learning method aiming at a distributed component flow scene, dynamically adjusting a task flow and processing complex load change and fault recovery requirements. According to the method, firstly, the component stream state of the system is sensed and constructed, future state prediction is carried out according to CPU, memory and disk storage of each computing node, if the component stream is blocked after several hours of the system, the distributed component stream of the system needs to be reconstructed at the moment, a reinforcement learning agent is used for carrying out reconstruction strategy recommendation, and the reconstruction strategy recommendation efficiency is improved. The system executes the reconstruction strategy, and when the execution of the component flow reconstruction strategy is completed, the system perceives the state of each node again and iteratively updates the reinforcement learning model, and then predicts the future state according to the condition of each node, if the system is stable, the process is ended, and if the system is not stable, the intelligent agent is continuously used for strategy recommendation.

Description

Technical Field

[0001] The present invention belongs to the field of distributed system management and optimization, and in particular relates to a component flow reconstruction technology in a distributed system. Background Art

[0002] The technical background of the online dynamic reconstruction method of distributed component flows mainly stems from the complexity and dynamic challenges faced by modern distributed systems. With the rapid development of cloud computing, big data and Internet of Things technologies, distributed systems have become the key infrastructure supporting various large-scale applications and services. However, these systems are usually composed of a large number of heterogeneous components, which are interconnected through complex data flows and control flows, forming a highly dynamic and complex interactive network.

[0003] During the operation of a distributed system, the interaction process between components may change due to various factors such as changes in system load, network latency, component failure, etc. In order to maintain the efficient operation and stability of the system, a method is needed that can monitor and analyze the system status in real time and dynamically adjust and optimize the interaction process between components according to demand.

[0004] With the development of deep learning, the online dynamic reconstruction method of distributed component flows has been significantly promoted and innovated. Deep learning technology not only enhances the monitoring and analysis capabilities of distributed systems on the status of component flows, but also provides smarter and more efficient decision support, making the dynamic reconstruction process more accurate and reliable. In addition, the distributed characteristics of the deep learning framework also provide strong support for the application of this method in large-scale distributed systems, promoting its widespread development and application in practice.

[0005] Although deep learning technology has been maturely applied in some fields, in the distributed reconstruction scenario of component flows, deep learning models may have insufficient generalization capabilities when generating scheduling strategies. Especially when encountering new and unknown load patterns, how to maintain the robustness and adaptability of the model and avoid strategy failure is an important challenge. Summary of the invention

[0006] To solve the above technical problems, the present invention proposes an online dynamic reconstruction method of distributed component flow based on reinforcement learning. The reinforcement learning algorithm is used to dynamically adjust and optimize the component flow in the distributed system online to adapt to the ever-changing task requirements and system environment, thereby improving resource utilization and ensuring the stability and reliability of the system, providing a new solution for the intelligent management and optimization of distributed systems.

[0007] The technical solution adopted by the present invention is: a distributed component flow online dynamic reconstruction method based on reinforcement learning, the distributed system based on which includes: deploying the server end on the server, connecting each computing node and the server through a core switch, deploying multiple containers in the computing node, and deploying component services in the container; the method includes the following steps:

[0008] S1. In a distributed system, regularly sense and monitor the resource usage of each computing node;

[0009] S2, the distributed system predicts resource demand in the future based on the resource usage of each computing node through an autoregressive model based on machine learning;

[0010] S3, judging the system state in the future period of time according to the prediction result obtained in step S2; if the distributed system has redundant computing resources in the future period of time, then the process ends; otherwise, the process goes to step S4;

[0011] S4. Construct component flow scenarios corresponding to distributed systems;

[0012] S5. Building a reinforcement learning agent based on the component flow scenario of step S4, and training the reinforcement learning agent based on the resource usage of the computing nodes obtained in step S1;

[0013] S6, recommending strategies based on the reinforcement learning agent trained in step S5;

[0014] S7. The distributed system performs actual operations based on the policy recommendation result obtained in step S6, and then returns to step S1.

[0015] Beneficial effects of the present invention: The present invention realizes efficient resource utilization, load balancing and fault recovery through adaptive scheduling and intelligent decision-making. The system automatically adjusts the scheduling strategy, reduces manual intervention, and reduces operation and maintenance costs. Through continuous learning and optimization, the system can flexibly respond in complex environments, improve overall performance and intelligence, and is applicable to a variety of distributed application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of the hardware architecture of the distributed component flow of the present invention.

[0017] Figure 2 Schematic diagram of the overall process of the online dynamic reconstruction method of distributed component flow based on reinforcement learning

[0018] Figure 3 Schematic diagram of the initialization reinforcement learning agent training method designed for the present invention.

[0019] Figure 4It is a schematic diagram of the component flow structure of the present invention. DETAILED DESCRIPTION

[0020] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0021] The present invention provides a distributed component flow online dynamic reconstruction method based on reinforcement learning, and its overall process is as follows: Figure 1 As shown, including:

[0022] Step 1: According to the distributed component flow scenario of the CloudSim cloud computing simulation platform, create a virtual server and multiple virtual computing nodes and deploy distributed components on the nodes to build a virtual environment, covering multiple factors such as computing nodes, servers, resource consumption, and load changes. The simulation scenario provides real-world dynamic data for subsequent reinforcement learning model training, helping the system understand load fluctuations and resource requirements in different scenarios. Assuming that there are N computing nodes in the system, the resource usage of each node (such as CPU, memory, disk, etc.) can be represented by the following vector:

[0023] r i =[r i1 ,r i2 ,…,r im ] T

[0024] Among them, r i Represents the resource state vector of the i-th computing node. There are m computing nodes in total. ij Indicates the usage percentage of the jth resource (such as CPU, memory, etc.) of the computing node.

[0025] By simulating different load changes, the system obtains a load function f(r i ), which describes the running status of the system component flow under a specific resource distribution.

[0026] f(r i )=max(r i1 ,r i2 …,r im )

[0027] Step 2: When building a reinforcement learning agent, design appropriate reward and punishment strategies to guide the agent to optimize resource scheduling and task migration decisions. The reinforcement learning model will continuously test and adjust strategies, iterate training in a simulated environment, gradually master the optimal scheduling strategy under different load conditions, and finally form an efficient agent decision model;

[0028]

[0029] Where s is the current state, a is the action to be performed, R(s,a) is the immediate reward after the action is performed, γ is the discount factor, and the value range of γ is between 0 and 1. In this embodiment, the value is 0.1; s′ is the new state transferred to after the action is performed, a′ represents the action to be performed in the future, and max a′ Q(s′, a′) is an estimate of future rewards. In this embodiment, Q is set to 1 when the system does not crash, and Q is set to 0 when the system crashes. The formula for R(s, a) is as follows, which is expressed by the difference between the evaluation load before the state execution and the execution load.

[0030] R(s,a)=max(f(r i )| t0 )-max(f(r i )| t )

[0031] The reinforcement learning agent gradually learns the optimal scheduling strategy by exploring and updating the Q value.

[0032] Step 3: In the actual system, regularly sense and monitor the resource usage of each computing node in the distributed system, including CPU, memory, disk storage, etc.; in this embodiment, the parameters (CPU, memory, disk storage, etc.) included in the current node resource usage can be sensed once a minute. In this embodiment, the node resource usage is specifically the percentage of CPU, memory, disk storage and other parameters. By collecting the data of the previous 4 hours, 240 data points can be obtained for each parameter to predict the resource usage of the distributed system in the next 2 hours, that is, 120 data points for a single parameter in the next 2 hours; in this embodiment, the prediction is made every 5 minutes, and it can be set according to specific needs in actual applications. Through real-time monitoring, the system can timely build and update the running status of the component flow, provide a basis for real-time decision-making, and design the computing node resource state vector as r i :

[0033] r i =[r i1 ,r i2 , ..., r im ] T

[0034] Vector r i is the global resource status of the system, including the resource usage of all nodes at the current moment; i1 、r i2 ,……r im , which correspond to the percentage of CPU, memory, disk storage, etc. in the distributed system. The system updates the resource status of each node through regular perception and provides data support for subsequent predictions.

[0035] Step 4: The system predicts resource demand and system status in the future based on the resource usage of each computing node (such as CPU, memory, disk storage, etc.) through the ARIMA time series analysis method. This step helps the system identify potential bottlenecks or resource shortages in advance and promptly discover possible blockages or failures; those skilled in the art should know that LSTM time series prediction methods, CNN-LSTM time series prediction methods, etc. can also be used to predict resource demand and system status in the future.

[0036] Step 5: Use the machine learning-based autoregressive model to predict each resource sequence of the computing node; based on the prediction results of the future state, the system will make a stability judgment. If the prediction shows that the distributed system will have redundant computing resources in the future, that is, the predicted future resource usage is less than 100%, it means that the distributed system resource demand remains balanced and the component flow can run stably, then the current cycle ends; if the prediction shows that component flow blockage, resource bottleneck or system overload may occur in a few hours, the system will determine that reconstruction is needed to avoid performance degradation or failure.

[0037] Step 6: When the system needs to be refactored, the system will call the trained reinforcement learning agent to recommend strategies. The agent will recommend a series of optimal refactoring strategies based on the current system status and load forecast. These strategies may include resource migration, task rescheduling, component flow reconfiguration, etc., aiming to avoid system congestion or overload and improve overall performance.

[0038] Assuming the current state is s, the agent will choose the optimal strategy based on the Q-value function of the state and action:

[0039]

[0040] Among them, a * The best reconfiguration strategy recommended for the agent includes operations such as task migration, resource adjustment or component flow reconfiguration. In this embodiment, action a is component reconfiguration, which specifically includes adding or deleting components.

[0041] Step 7: The system performs actual operations according to the reconstruction strategy recommended by the intelligent agent. This includes dynamically migrating computing tasks, reconfiguring service deployment, adjusting resource allocation, etc. By executing these strategies, the system can quickly adjust when load imbalance or resource bottlenecks occur to ensure the stable operation of component flows;

[0042] Step 8: After the reconstruction strategy is executed, the system continues to regularly sense and monitor the resource status of each computing node, and updates the state model of the component flow. Through the strategy effect of the model, the reinforcement learning model will iteratively update and optimize according to the new real-time data to improve the accuracy of future predictions and the intelligence of scheduling decisions. Go to step 4, where

[0043] Assuming the new state is R(t+1), the system will continue to iteratively train the model using the updated data to optimize the prediction and scheduling strategies:

[0044]

[0045] It should be noted that the hardware structure of the distributed component flow online dynamic reconstruction method based on reinforcement learning is as follows Figure 1 As shown, its structure is a one-to-many form. The server is deployed on a high-performance server. Each computing node and the server are connected through a core switch. Multiple containers can be deployed in the computing node, and component services can be deployed in the container.

[0046] The overall process of the present invention is as follows Figure 2 As shown, the method first perceives and constructs the component flow state of the system, and predicts the future state based on the CPU, memory, and disk storage of each computing node. If the component flow of the system is blocked after a few hours, the distributed component flow of the system needs to be reconstructed, and the reinforcement learning agent is used to recommend the reconstruction strategy. The system executes the reconstruction strategy. When the component flow reconstruction strategy is executed, the system re-perceives the state of each node and iteratively updates the reinforcement learning model. Then, the future state is predicted based on the situation of each node. If the system is stable, it ends, otherwise it continues to use the agent to recommend strategies.

[0047] Among them, the initialization reinforcement learning agent training method designed by the present invention is as follows Figure 3As shown, firstly, a reasonable component flow scenario is randomly constructed; the model outputs the component flow reconstruction strategy according to the various parameters in the scenario; the system reconstructs the component flow according to the strategy; the result status is monitored and evaluated by the monitoring system. If the required accuracy is not met, the status result is returned to the agent to reward and punish it, the agent model parameters are updated and iterated, and a new recommended reconstruction strategy is generated to continue to complete the current task; through multiple rounds of continuous iterations, the recommended strategy effect of the agent model is strengthened and improved. When the execution strategy meets the status requirements, the model output judgment will be performed. For example, if it is recommended 10,000 times and each time meets the status requirements, the model can be output. The status requirement here means that after the execution strategy, the system resource utilization rate is less than 100%. If the output conditions are not met, reasonable component flow scenarios will continue to be randomly constructed, and the agent will continue to learn until the output conditions are met. Among them, the component flow random scenario is as follows: Figure 4 As shown, each component in the component flow may be distributed in different nodes or different containers.

[0048] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A distributed component flow online dynamic reconstruction method based on reinforcement learning, characterized in that: The distributed system based on it includes: deploying the server end on the server, connecting each computing node and the server through a core switch, deploying multiple containers in the computing node, and deploying component services in the container; the method includes the following steps: S1. In a distributed system, regularly sense and monitor the resource usage of each computing node; S2, the distributed system predicts resource demand in the future based on the resource usage of each computing node through an autoregressive model based on machine learning; S3, judging the system state in the future period of time according to the prediction result obtained in step S2; if the distributed system has redundant computing resources in the future period of time, then the process ends; otherwise, the process goes to step S4; S4. Construct component flow scenarios corresponding to distributed systems; S5. Building a reinforcement learning agent based on the component flow scenario of step S4, and training the reinforcement learning agent based on the resource usage of the computing nodes obtained in step S1; S6, recommending strategies based on the reinforcement learning agent trained in step S5; S7. The distributed system performs actual operations based on the policy recommendation result obtained in step S6, and then returns to step S1.

2. According to the method of online dynamic reconstruction of distributed component flow based on reinforcement learning in claim 1, it is characterized in that: The autoregressive model based on machine learning in step S2 specifically adopts the ARIMA time series analysis method.

3. According to claim 1, the distributed component flow online dynamic reconstruction method based on reinforcement learning is characterized in that: The reinforcement learning agent expression in step S5 is: Among them, s is the current state, a is the action to be performed, R(s,a) is the immediate reward after the action is performed, γ is the discount factor; s′ is the new state transferred to after the action is performed, max a′ Q(s′, a′) is an estimate of future rewards.

4. According to claim 3, the distributed component flow online dynamic reconstruction method based on reinforcement learning according to claim 1 is characterized in that: The state space of the reinforcement learning agent is the set of resource usage of each computing node.

5. According to claim 4, the distributed component flow online dynamic reconstruction method based on reinforcement learning according to claim 1 is characterized in that: The formula for R(s,a) is as follows: R(s,a)=max(f(r i )|t0)-max(f(r i )| t ) Among them, f(r i ) represents the load function, t0 represents the current time, t represents the time after execution, f(r i )=max(r i1 ,r i2 …,r im ).

6. According to claim 5, the distributed component flow online dynamic reconstruction method based on reinforcement learning according to claim 1 is characterized in that: aSpecifically refers to the addition or deletion of components.

Images