Raid card cache capacity management method and device based on deep reinforcement learning algorithm

Abstract

The invention relates to the field of intelligent cache management, in particular to a Raid card cache capacity management method and device based on a deep reinforcement learning algorithm. The method comprises the steps of collecting historical operation data of the Raid card, screening key feature data influencing a cache capacity strategy according to the historical operation data of the Raid card, and preprocessing the key feature data influencing the cache capacity strategy; constructing a deep reinforcement learning model, and training the deep reinforcement learning model by using the preprocessed cache capacity change features; and deploying the trained deep reinforcement learning prediction model to an actual operation environment of the Raid card, monitoring the operation condition of the Raid card in real time, and outputting a real-time optimized cache capacity strategy. According to the method, the cache demand change of the Raid card under different load conditions can be effectively coped with, the cache capacity can be dynamically adjusted to optimize the resource utilization rate, the cache hit rate and the response time can be remarkably improved, the system delay is reduced, and the performance and the efficiency of an overall storage system of the Raid card are optimized.

Description

Technical Field

[0001] The present invention relates to the field of cache capacity technology, and in particular to a RAID card cache capacity management method and device based on a deep reinforcement learning algorithm. Background Art

[0002] With the popularity of emerging applications such as big data and cloud computing, the access mode of computer disk data has become increasingly complex and changeable, and the I / O load of the system often has strong timing and uncertainty. Traditional cache management usually adopts a fixed capacity strategy, that is, the cache size is set when the system is initialized and is not adjusted during operation. Under high load conditions, the fixed disk cache capacity may not be enough to meet system requirements, resulting in I / O bottlenecks and affecting overall performance; while under low load conditions, excessive cache capacity may lead to resource waste and increased energy consumption. Therefore, how to dynamically adjust the cache capacity of the Raid card according to the real-time load and optimize resource utilization has become an urgent problem to be solved.

[0003] In recent years, with the development of artificial intelligence technology, especially the advancement of deep learning algorithms, new solutions have been provided for intelligent cache management. Methods based on deep learning technology can predict future load conditions and dynamically adjust cache capacity by analyzing historical I / O data and current system status, thereby achieving more efficient cache management. At the same time, intelligent cache elimination strategies can also give priority to retaining data blocks with a higher probability of future access when cache space is limited, further improving the response speed and reliability of the system. However, traditional deep learning methods rely on static loss function training and cannot adjust the model in real time to cope with the dynamic environmental changes of the Raid card, resulting in poor cache management capabilities. Summary of the invention

[0004] In order to address the shortcomings of traditional cache capacity management algorithms in the dynamic management of RAID card cache capacity and the opaque decision-making process of deep learning algorithms, the present invention provides a RAID card cache capacity dynamic management method and device based on a deep reinforcement learning algorithm, which effectively responds to the changes in cache requirements of RAID cards under different load conditions and can help engineers dynamically adjust cache capacity to optimize resource utilization.

[0005] To achieve the above object, the technical solution of the present invention is: The Raid card cache capacity management method based on deep reinforcement learning algorithm includes the following steps: S1. Collect historical operation data of the Raid card, filter key feature data that affect the cache capacity strategy based on the historical operation data of the Raid card, and pre-process key status data that affect the cache capacity strategy; S2. Build a multi-layer perceptron model based on a deep reinforcement learning algorithm, train the model using the pre-processed cache capacity change features, and obtain a trained multi-layer perceptron prediction model; S3, deploy the trained multi-layer perceptron prediction model to the actual operation environment of the Raid card, monitor the operation of the Raid card in real time, and output the cache capacity strategy for optimizing the performance of the Raid card in real time; S4. Regularly retrain and adjust the deep reinforcement learning prediction model to adapt to new workload patterns and environmental changes after the Raid card version is updated.

[0006] Specifically, the step S1 specifically includes: Collect relevant data of I / O load of Raid card system in real operation environment of Raid card in preset time period, and store the collected relevant data of I / O load of Raid card system; The collected I / O load related data of the Raid card system is cleaned and missing data is filled, the key feature data of the cache capacity change during the operation of the Raid card system is screened, and the key feature data is converted into a coding vector format.

[0007] Specifically, the key characteristic data affecting the cache capacity strategy include cache capacity and I / O load. The cache capacity is used to indicate the capacity currently allocated to the Raid card cache, and the I / O load is used to indicate the current Raid card I / O request amount.

[0008] Specifically, the step S2 includes: S21. Define the state space required by the deep reinforcement learning model. The state space is the input of the deep reinforcement learning model and is used to describe the current operating status of the Raid card system. S22. Define the action space required by the deep reinforcement learning model. The action space is the output of the deep reinforcement learning model and is used to represent the possible actions that may be taken in the current state. S23, interacting with the Raid card cache management environment through the deep reinforcement learning model, selecting a cache management action to adjust the cache capacity according to the current state, and updating the cache strategy of the deep reinforcement learning model according to the cache management action taken and the cache management environment feedback; S24, defining an overall reward function of the deep reinforcement learning model, using the overall reward function to evaluate the effect of each action to obtain a reward signal, and optimizing the cache decision strategy of the deep reinforcement learning model according to the obtained reward signal; S25. Based on the overall reward function, a multi-layer perceptron neural network is constructed for Raid card cache management. The pre-processed key feature data is used to train the multi-layer perceptron neural network to obtain a trained deep reinforcement learning model.

[0009] Specifically, the state space required by the deep reinforcement learning model includes cache capacity and I / O load conditions; the action space required by the deep reinforcement learning model includes: dynamically increasing cache capacity and dynamically reducing cache capacity.

[0010] Specifically, the step S23 includes: The deep reinforcement learning algorithm agent in the deep reinforcement learning model performs an optimal cache management action based on the soft strategy according to the cache capacity and I / O load. The cache management action includes increasing or decreasing the cache capacity to adapt to the current load demand. The calculation formula of the soft strategy is: ; Among them, π(ɑ|s) represents the probability of selecting an action in state s, Q(s, ɑ) is the action value function of executing action ɑ in state s, τ is the temperature parameter, and ∑b represents the summation of all possible actions b.

[0011] Specifically, the overall reward function of the deep reinforcement learning model defined in step S24 includes: Set the reward target of the deep reinforcement learning model, and define the reward function of the model for response time according to the change of the response time of the Raid card system to I / O requests. If an action leads to a reduction in the system response time, the deep reinforcement learning algorithm agent will receive a positive reward; if the response time increases, a negative reward will be given; The cache hit rate reward function of the deep reinforcement learning model is defined based on the change in cache hit rate. If an action leads to an increase in the cache hit rate, the deep reinforcement learning algorithm agent will receive a positive reward; if the hit rate decreases, a negative reward will be given; The response time reward function and the cache hit rate reward function are weighted together to obtain the overall reward function.

[0012] A computer device, comprising a processor and a memory for storing a program executable by the processor, characterized in that when the processor executes the program stored in the memory, the above-mentioned Raid card cache capacity management method based on deep reinforcement learning algorithm is implemented Compared with the prior art, the present invention has the following advantages and beneficial effects: The present invention provides a method and device for dynamically adjusting the cache capacity of a Raid card based on deep reinforcement learning. By analyzing the cache capacity and I / O load data under the operation state of the Raid card, a deep reinforcement learning model is trained to dynamically adjust the cache capacity to adapt to changes in the system load. The constructed deep reinforcement learning model is used to make cache adjustment decisions, and the cache capacity is dynamically optimized to adapt to different load conditions by real-time monitoring of system performance indicators. In addition, the deep reinforcement learning model is periodically retrained to adapt to the new workload after the Raid card system is upgraded to ensure continuous optimization of the cache strategy. This technology significantly improves the cache hit rate, reduces system latency, and optimizes the performance and efficiency of the overall storage system of the Raid card. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0014] Figure 1 This is a flow chart of a Raid card cache capacity management method based on a deep reinforcement learning algorithm in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the deep reinforcement learning model training framework in Example 1 of the present invention; Figure 3 This is a schematic diagram of the composition of the RAID card cache capacity management device based on the deep reinforcement learning algorithm in Example 2 of the present invention. DETAILED DESCRIPTION

[0015] The technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is obvious that the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and the implementation of the present invention is not limited to this. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0016] Example 1 The present invention screens out key feature data that affect the cache strategy by analyzing the dynamic changes in cache capacity and I / O load conditions in the Raid card system, and uses this as a basis for training and constructing a reinforcement learning prediction model for cache management. The present invention uses a deep reinforcement learning model to make cache adjustment decisions, and dynamically optimizes cache capacity to adapt to different load conditions by real-time monitoring of system performance indicators. By introducing a periodic retraining mechanism, the reinforcement learning prediction model for cache management can flexibly respond to changes in Raid card system upgrades or workload patterns, ensuring continuous optimization of cache management strategies.

[0017] like Figure 1 As shown, a flow chart of a method for dynamically adjusting the cache capacity of a Raid card based on deep reinforcement learning is provided. The method for dynamically adjusting the cache capacity of a Raid card based on deep reinforcement learning provided in this embodiment includes the following steps: S1. Collect historical operation data of the Raid card, filter key feature data that affect the cache capacity strategy according to the historical operation data of the Raid card, and pre-process the key feature data that affect the cache capacity strategy.

[0018] Collect historical I / O load related data of the Raid card system, set preprocessing operations to convert the historical I / O load related data into key features in vector format, and initialize the state of the deep reinforcement learning agent according to the key features in vector format.

[0019] S11. Through the I / O data monitoring script of the Raid card, the operation data of the Raid card in the real operation environment can be collected within a preset time period, and the key characteristic data affecting the cache capacity strategy can be screened according to the collected operation data of the Raid card system, and the collected operation data of the Raid card system can be stored. It can be stored in a specific database or file system to ensure the integrity and accessibility of the data.

[0020] Preferably, the key characteristic data that affect the cache capacity strategy include cache capacity and I / O load. Cache capacity indicates the current allocated cache capacity of the Raid card, which helps the algorithm determine whether capacity adjustment is needed; I / O load reflects the current I / O request volume and system load intensity of the Raid card, which helps the algorithm evaluate whether cache capacity needs to be dynamically adjusted to optimize system performance.

[0021] S12. Clean and fill in missing data on the collected I / O load data of the Raid card system, and filter key feature data of cache capacity changes during the operation of the Raid card system to meet the input requirements of the deep reinforcement learning model.

[0022] When processing the collected raw Raid card system operation data, the data is first cleaned. For some missing outlier data that may exist, the data sequence is filled with 0 to ensure the integrity of the system operation. Then, the key features in the raw data are extracted and converted to extract the key features of the cache capacity change during the operation of the Raid card system. The one-hot encoding method is used to encode these key features of cache capacity change into a vector format to meet the input requirements of the deep reinforcement learning model.

[0023] Initialize the state of the deep reinforcement learning agent according to the key features in vector format. Through this series of preprocessing operations, the accuracy and consistency of the deep reinforcement learning model training data are guaranteed, laying the foundation for the effectiveness of the cache-managed reinforcement learning prediction model.

[0024] Specifically, the deep reinforcement learning algorithm agent will continue to dynamically interact with the cache management environment of the Raid card. The deep reinforcement learning algorithm agent constantly obtains current state information from the environment, and through the state information agent will learn how to adopt appropriate capacity management strategies under different load conditions to optimize system performance.

[0025] S2. Build a deep reinforcement learning model, use the preprocessed cache capacity change features as model input, and construct a trained multi-layer perceptron prediction model. Figure 2 As shown, a schematic diagram of the deep reinforcement learning model training framework.

[0026] S21. Define the state space required by the deep reinforcement learning model. The state space is the input of the deep reinforcement learning model and is used to describe the current operating status of the Raid card system. For the cache management of the Raid card, the state space required by the deep reinforcement learning model should include cache capacity and I / O load.

[0027] The deep reinforcement learning model is a technology that combines deep learning and reinforcement learning. It uses neural networks to approximate value functions or strategies, allowing agents to optimize decisions in complex, high-dimensional environments. Specifically, the deep reinforcement learning model includes the following steps: first, the agent perceives the current state of the environment and selects actions based on the strategy; second, after executing the action, it obtains rewards and new states from the environment; then, the state, action, and reward are modeled through neural networks to optimize the strategy or value function; finally, through continuous exploration and feedback adjustment, the agent gradually learns the optimal decision-making strategy and maximizes the cumulative reward. In the cache capacity dynamic management model, the agent is the subject that executes decisions and takes actions. It continuously optimizes the cache management strategy by monitoring the system status (I / O load).

[0028] The cache capacity dynamic management model is used to dynamically adjust the cache allocation of the Raid card in real time according to the load changes of the current Raid card system to improve system performance and data storage efficiency. The specific steps include: (1) real-time monitoring of the load changes of the Raid card system; (2) the cache capacity dynamic management model predicts the future cache capacity requirements in real time according to the Raid load changes; (3) the Raid card system adjusts the cache capacity according to the cache capacity dynamic management model strategy; (4) the Raid card system provides performance feedback based on the adjustment results. The cache capacity dynamic management model is an overall algorithm, which includes training a simple neural network (multi-layer perceptron) for cache capacity management decisions. The multi-layer perceptron neural network is trained and optimized through a deep reinforcement learning model.

[0029] In the cache capacity dynamic management model, the deep reinforcement learning model is a key technology, which is mainly used to achieve intelligent cache resource allocation optimization. The deep reinforcement learning model perceives system status information (I / O load request) through a neural network, and combines the reinforcement learning mechanism to continuously adjust the cache strategy based on real-time environmental feedback. Its main purpose is to dynamically optimize cache allocation and improve system performance (including improving throughput and cache hit rate, reducing response latency, etc.), and ultimately achieve automated and efficient cache management to adapt to work requirements under different loads.

[0030] S22. Define the action space required by the deep reinforcement learning model. The action space is the output of the deep reinforcement learning model and is used to represent the possible actions to be taken in the current state. For the cache management of the Raid card, specifically, the action space includes: dynamically increasing the cache capacity and dynamically reducing the cache capacity.

[0031] Dynamically increase cache capacity: Dynamically increase cache capacity according to the current load to accommodate higher I / O request requirements and avoid system performance degradation. Dynamically reduce cache capacity: Dynamically reduce cache capacity when the load is light to save system resources and reduce energy consumption. The action space of the deep reinforcement learning model is defined as the possible operations in the Raid card cache management: increase cache capacity to cope with higher I / O requests and avoid performance degradation; reduce cache capacity to save resources and reduce energy consumption when the load is light to optimize system performance.

[0032] S23. Interact with the RAID card cache management environment through the model, select cache management actions to adjust the cache capacity according to the current system status, and update the cache strategy according to the cache management actions taken and the cache management environment feedback (RAID card system performance changes).

[0033] In this example, the deep reinforcement learning algorithm agent dynamically optimizes the cache strategy through continuous interaction with the actual environment. Specifically, the deep reinforcement learning algorithm agent first performs an optimal cache management action based on the soft strategy according to the current system state, which includes cache capacity and I / O load. These actions include increasing or decreasing cache capacity to adapt to the current load requirements. The calculation formula of the soft strategy is: ; where π(ɑ|s) denotes the probability of selecting an action in state s, Q(s, ɑ) is the action-value function of performing action ɑ in state s, τ is a temperature parameter that controls the trade-off between exploration and exploitation, and ∑b denotes the sum over all possible actions b.

[0034] The action-value function Q(s, ɑ) is a core concept in reinforcement learning, and its specific value needs to be estimated through the learning process of the agent. In the process of interacting with the environment, the agent continuously updates and optimizes Q(s, ɑ) through experience and feedback in order to accurately evaluate the expected return of choosing an action in a specific state. Therefore, although Q(s, ɑ) is a standard component in reinforcement learning, its actual calculation and accuracy are derived through a specific learning and training process.

[0035] When the agent performs an optimal cache management action, the Raid card system will generate corresponding performance feedback. By continuously monitoring these feedbacks, the impact of the actions taken on system performance is evaluated, and the feedback information is used to update its strategy. Over time, the agent gradually optimizes its decision-making ability through continuous trial and learning, ensuring that the Raid card system always maintains the best performance under various load conditions. This interactive process not only improves the system's adaptive ability, but also can achieve precise management and optimization of the Raid card cache capacity through the self-adjustment mechanism of the deep reinforcement learning model, thereby effectively improving the overall efficiency of the system.

[0036] S24. Define the overall reward function of the deep reinforcement learning model, use the overall reward function to evaluate the effect of each action to obtain a reward signal, and optimize the cache decision strategy of the deep reinforcement learning model based on the obtained reward signal.

[0037] Specifically, the overall reward function that defines the deep reinforcement learning model includes: S241. Set the reward target of the deep reinforcement learning model, and define the response time reward function of the deep reinforcement learning model according to the change of response time, where the system response time is an important indicator for evaluating the performance of the Raid card.

[0038] Specifically, the response time reward function can be defined based on the change in response time. If an action results in a reduction in the system response time, the deep reinforcement learning algorithm agent will receive a positive reward; if the response time increases, a negative reward will be given. The size of the reward can be proportional to the magnitude of the change in response time. The more the response time is reduced, the higher the reward. The reward function can be defined as: ; Among them, ɑ is a positive coefficient, which indicates the change of response time (negative for decrease and positive for increase), and is used to control the magnitude of reward.

[0039] S242. Define the cache hit rate reward function of the deep reinforcement learning model according to the change of cache hit rate. The cache hit rate directly reflects the effectiveness of the cache management strategy.

[0040] The cache hit rate reward function can be defined based on the change in cache hit rate: if an action leads to an increase in cache hit rate, the agent will receive a positive reward; if the hit rate decreases, a negative reward will be given. Similar to response time, the size of the reward can be proportional to the magnitude of the change in hit rate, which can be defined as: ; Among them, β is a positive coefficient, which indicates the change of cache hit rate (increase is positive, decrease is negative).

[0041] S243, performing a weighted sum of the response time reward function and the cache hit rate reward function to obtain an overall reward function. The overall reward function is used to evaluate the effect of each action to obtain a reward signal, and the deep reinforcement learning agent optimizes its decision strategy according to the obtained reward signal.

[0042] In the actual operation of the Raid card, the system performance can be represented by the above two characteristic indicators. In order to comprehensively consider the response time and cache hit rate, the reward function of the overall algorithm can be designed as the weighted sum of the two indicators. By adding the above two reward functions and reasonably setting the weights of the two reward functions, the overall reward function can balance the importance of different indicators. The overall reward function is used to evaluate the effect of each action to obtain a reward signal. The agent optimizes its decision-making strategy based on the reward signal obtained and makes decisions that are more in line with the overall performance optimization of the system.

[0043] S25. Based on the overall reward function, a four-layer multilayer perceptron neural network is constructed for Raid card cache management. The multilayer perceptron includes an input layer, two hidden layers and an output layer.

[0044] The deep reinforcement learning model is trained with the key feature data after the training, and the trained deep reinforcement learning model is used for cache management prediction. The multilayer perceptron neural network is a feedforward neural network. Each layer consists of a series of neurons, and the neurons are connected by weights. Specifically, the input layer of the multilayer perceptron neural network receives the key feature data of the Raid card (cache capacity and I / O load changes), and the single hidden layer uses a nonlinear activation function to process and learn these key feature data. The output layer generates cache management action probabilities according to task requirements. The cache capacity change characteristics during the operation of the Raid card are preprocessed, and the preprocessing is used Reinforcement learning is used to optimize the cache management strategy of the multi-layer perceptron through interaction with the environment. The reinforcement learning agent obtains system performance feedback after the model executes an action and calculates the value of different actions to update the model parameters so that it can more accurately execute appropriate management decisions in future environments. During the training process, the experience replay mechanism is used to stabilize the training process, thereby improving the decision-making ability and system performance of the reinforcement learning agent under various load conditions.

[0045] S3. Deploy the trained deep reinforcement learning prediction model to the actual operating environment of the Raid card, monitor the operation of the Raid card in real time, and output a real-time optimized cache capacity strategy.

[0046] Specifically, when the trained deep reinforcement learning prediction model is performing dynamic management of cache capacity, the algorithm will first predict and output the cache capacity management strategy to be executed based on the real-time cache capacity and I / O load conditions during the operation of the Raid card.

[0047] The deep reinforcement learning model selects the optimal action (increase or decrease cache capacity) by analyzing the current state, and continuously adjusts and optimizes the strategy based on the system's performance feedback (response time, changes in cache hit rate). When the system detects an increase in I / O load, the algorithm decides to dynamically increase the cache capacity to accommodate higher request requirements; when the load is reduced, the cache capacity is reduced to save resources and reduce energy consumption. In this process, the algorithm not only optimizes the immediate performance of the Raid card system, but also improves the Raid card system's adaptability under various load conditions through continuous learning and adjustment, ultimately achieving efficient cache management and ensuring system stability and long-term efficient operation.

[0048] It optimizes the real-time performance of the Raid card system and improves the adaptability of the Raid card system under various load conditions, ultimately achieving efficient cache management and ensuring system stability and long-term efficient operation.

[0049] S4. Regularly retrain and adjust the deep reinforcement learning prediction model to adapt to new workload patterns and environmental changes after the Raid card version is updated, and further optimize the cache management strategy of the deep reinforcement learning prediction model output.

[0050] Specifically, in order to improve the adaptability of the Raid card intelligent cache management method based on deep reinforcement learning to the replacement of new versions of cache, a certain period of time is preset to regularly collect the latest data access records of the Raid card, and the latest data access records are used to train the deep reinforcement learning prediction model to update the deep reinforcement learning prediction model. Over time, this dynamic adaptation mechanism ensures that the system can continuously optimize the storage performance and overall efficiency of the Raid card under different workloads and usage conditions, thereby effectively responding to the ever-changing usage environment.

[0051] Before each Raid card update cycle begins, the system collects all data access records since the last update and evaluates whether the existing deep reinforcement learning model has achieved expected cache performance optimization in the latest Raid card cycle. If the algorithm performance fails to meet expectations, the system will combine historical data access records with the new version of data records through incremental learning to retrain the deep reinforcement learning prediction model to update its parameters. In this way, the deep reinforcement learning prediction model can be continuously optimized to ensure that the Raid card maintains the best cache management effect under changing workload conditions.

[0052] The present invention provides a Raid card intelligent cache management method based on deep reinforcement learning, aiming to overcome the limitations of traditional cache management through a deep reinforcement learning model. The method uses a deep reinforcement learning model to dynamically adjust the cache capacity strategy by analyzing the historical access data and current system status of the Raid card. In addition, the method supports a periodic update mechanism, and can regularly update the model according to the latest data access situation of the Raid card to ensure that the cache strategy is continuously optimized as the system is upgraded and the workload changes. By introducing a deep reinforcement learning model and a periodic update mechanism, the present invention can significantly improve the cache hit rate, reduce system latency, and optimize the performance and efficiency of the overall storage system of the Raid card. It effectively responds to the changes in the cache requirements of the Raid card under different load conditions, which can not only help engineers dynamically adjust the cache capacity to optimize resource utilization, but also avoid the complex computing overhead brought by the deep learning algorithm while maintaining stable system performance.

[0053] Example 2 like Figure 3As shown, a schematic diagram of the composition of a RAID card cache capacity management device based on a deep reinforcement learning algorithm is provided. This embodiment also provides a computer device, including a processor and a memory for storing a processor executable program. The computer device may be a server, a computer, etc., and includes a processor, a memory, an input device, a display, and a network interface connected via a system bus. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the processor executes the computer program stored in the memory, the RAID card cache capacity management method based on the deep reinforcement learning algorithm of the above-mentioned embodiment 1 is implemented as follows: S1. Collect historical operation data of the Raid card, filter key feature data that affect the cache capacity strategy according to the historical operation data of the Raid card, and pre-process the key feature data that affect the cache capacity strategy; S2. Build a deep reinforcement learning model, and use the preprocessed cache capacity change characteristics to train the deep reinforcement learning model to obtain a trained deep reinforcement learning prediction model; S3, deploy the trained deep reinforcement learning prediction model to the actual operation environment of the Raid card, monitor the operation of the Raid card in real time, and output a real-time optimized cache capacity strategy; S4. Regularly retrain and adjust the deep reinforcement learning prediction model to adapt to new workload patterns and environmental changes after the Raid card version is updated.

[0054] Specifically, the step S1 specifically includes: Through the I / O data monitoring script of the Raid card, the relevant data of the I / O load of the Raid card system in the real operating environment of the Raid card is collected within a specified time period, and the collected relevant data of the I / O load of the Raid card system is stored; The collected data related to the I / O load of the Raid card system is cleaned and missing data is filled, the key feature data of the cache capacity change when the Raid card system is running is screened, and the key feature data is converted into a coding vector format.

[0055] The step S2 comprises: S21. Define the state space required by the deep reinforcement learning model. The state space is the input of the deep reinforcement learning model and is used to describe the current operating status of the Raid card system. S22. Define the action space required by the deep reinforcement learning model. The action space is the output of the deep reinforcement learning model and is used to represent the possible actions that may be taken in the current state. S23, interacting with the Raid card cache management environment through the deep reinforcement learning model, selecting a cache management action to adjust the cache capacity according to the current state, and updating the cache strategy of the deep reinforcement learning model according to the cache management action taken and the cache management environment feedback; S24, defining an overall reward function of the deep reinforcement learning model, using the overall reward function to evaluate the effect of each action to obtain a reward signal, and optimizing the cache decision strategy of the deep reinforcement learning model according to the obtained reward signal; S25. Based on the overall reward function, a multi-layer perceptron neural network is constructed for Raid card cache management. The pre-processed key Raid card cache data is used to train the multi-layer perceptron neural network to obtain a trained deep reinforcement learning model.

[0056] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A Raid card cache capacity management method based on a deep reinforcement learning algorithm, characterized in that: The steps include: S1. Collect historical operation data of the Raid card, filter key feature data that affect the cache capacity strategy according to the historical operation data of the Raid card, and pre-process the key feature data that affect the cache capacity strategy; S2. Build a deep reinforcement learning model, and use the preprocessed cache capacity change characteristics to train the deep reinforcement learning model to obtain a trained deep reinforcement learning prediction model; S3, deploy the trained deep reinforcement learning prediction model to the actual operation environment of the Raid card, monitor the operation of the Raid card in real time, and output a real-time optimized cache capacity strategy; S4. Regularly retrain and adjust the deep reinforcement learning prediction model to adapt to new workload patterns and environmental changes after the Raid card version is updated.

2. The RAID card cache capacity management method based on deep reinforcement learning algorithm as claimed in claim 1, characterized in that: The step S1 specifically includes: Through the I / O data monitoring script of the Raid card, the relevant data of the I / O load of the Raid card system in the real operating environment of the Raid card is collected within a preset time period, and the collected relevant data of the I / O load of the Raid card system is stored; The collected data related to the I / O load of the Raid card system is cleaned and missing data is filled, the key feature data of the cache capacity change when the Raid card system is running is screened, and the key feature data is converted into a coding vector format.

3. The Raid card cache capacity management method based on deep reinforcement learning algorithm as claimed in claim 2, characterized in that: The key characteristic data affecting the cache capacity strategy include cache capacity and I / O load. The cache capacity is used to indicate the capacity currently allocated to the Raid card cache, and the I / O load is used to indicate the current Raid card I / O request amount.

4. The Raid card cache capacity management method based on deep reinforcement learning algorithm as claimed in claim 3, characterized in that: The step S2 comprises: S21. Define the state space required by the deep reinforcement learning model. The state space is the input of the deep reinforcement learning model and is used to describe the current operating status of the Raid card system. S22. Define the action space required by the deep reinforcement learning model. The action space is the output of the deep reinforcement learning model and is used to represent the possible actions that may be taken in the current state. S23, interacting with the Raid card cache management environment through the deep reinforcement learning model, selecting a cache management action to adjust the cache capacity according to the current state, and updating the cache strategy of the deep reinforcement learning model according to the cache management action taken and the cache management environment feedback; S24, defining an overall reward function of the deep reinforcement learning model, using the overall reward function to evaluate the effect of each action to obtain a reward signal, and optimizing the cache decision strategy of the deep reinforcement learning model according to the obtained reward signal; S25. Based on the overall reward function, a multi-layer perceptron neural network is constructed for Raid card cache management. The pre-processed key feature data is used to train the multi-layer perceptron neural network to obtain a trained deep reinforcement learning model.

5. The Raid card cache capacity management method based on deep reinforcement learning algorithm as claimed in claim 4, characterized in that: The state space required by the deep reinforcement learning model includes cache capacity and I / O load conditions; the action space required by the deep reinforcement learning model includes: dynamically increasing cache capacity and dynamically reducing cache capacity.

6. The Raid card cache capacity management method based on deep reinforcement learning algorithm as claimed in claim 5, characterized in that: The step S23 comprises: The deep reinforcement learning algorithm agent in the deep reinforcement learning model performs an optimal cache management action based on the soft strategy according to the cache capacity and I / O load. The cache management action includes increasing or decreasing the cache capacity to adapt to the current load demand. The calculation formula of the soft strategy is: ； Among them, π(ɑ|s) represents the probability of selecting an action in state s, Q(s, ɑ) is the action value function of executing action ɑ in state s, τ is the temperature parameter, and ∑b represents the summation of all possible actions b.

7. The Raid card cache capacity management method based on deep reinforcement learning algorithm as claimed in claim 6, characterized in that: The overall reward function of the deep reinforcement learning model defined in step S24 includes: Set the reward target of the deep reinforcement learning model and define the response time reward function of the deep reinforcement learning model based on the change in response time. If an action leads to a decrease in the system response time, the deep reinforcement learning algorithm agent will receive a positive reward; if the response time increases, a negative reward will be given; The cache hit rate reward function of the deep reinforcement learning model is defined based on the change in cache hit rate. If an action leads to an increase in the cache hit rate, the deep reinforcement learning algorithm agent will receive a positive reward; if the hit rate decreases, a negative reward will be given; The response time reward function and the cache hit rate reward function are weighted together to obtain the overall reward function.

8. The Raid card cache capacity management method based on deep reinforcement learning algorithm as claimed in claim 6, characterized in that: The input layer of the multi-layer perceptron neural network receives the key feature data of the Raid card, a single hidden layer processes and learns the key feature data using a nonlinear activation function, and an output layer generates cache management action probabilities according to task requirements.

9. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, the Raid card cache capacity management method based on the deep reinforcement learning algorithm described in any one of claims 1-8 is implemented.

Images