Multi-agent reinforcement learning database intelligent operation and maintenance method based on multi-agent markov decision process

Abstract

The application discloses a multi-agent reinforcement learning database intelligent operation and maintenance method based on a multi-agent Markov decision process. Database operation data are collected, and candidate suggestions are generated through five special agents, i.e., natural language query, SQL optimization, log analysis, fault diagnosis and resource prediction; an operation and maintenance scene is modeled as a multi-agent Markov decision process, and a global optimal action is generated through conflict arbitration and joint reward maximization collaborative decision-making; a final instruction is obtained by calling a large language model and correcting a RAG knowledge base, and strategy parameters are updated after feedback after execution. The application solves problems such as strong dependence on traditional operation and maintenance experience and poor heterogeneous adaptation, realizes automatic decision-making and active prevention and control, and improves system stability and resource utilization.

Description

Technical Field

[0001] This invention relates to the field of database operation and maintenance and artificial intelligence, and discloses a multi-agent reinforcement learning-based intelligent database operation and maintenance method based on multi-agent Markov decision process. Background Technology

[0002] As digital transformation deepens, enterprises' core business relies increasingly on databases, leading to an exponential increase in the complexity of database operations and maintenance. Traditional database operations and maintenance primarily rely on manual experience, resulting in the following significant drawbacks: manual troubleshooting and location are required after a fault occurs, leading to long average fault resolution times and potential business interruptions; operations and maintenance personnel struggle to simultaneously consider multiple objectives such as query performance, resource consumption, and system stability, potentially resulting in locally optimal decisions; experienced operations and maintenance personnel are scarce, knowledge transfer is difficult, and standardization is low; and the operational logic for different types of databases varies significantly, making it difficult to establish a unified operations and maintenance system.

[0003] Existing intelligent operation and maintenance solutions mostly adopt a single-agent reinforcement learning architecture, which can only handle single-dimensional operation and maintenance tasks, cannot achieve multi-task collaborative decision-making, and is prone to action conflicts. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multi-agent reinforcement learning database intelligent operation and maintenance method based on multi-agent Markov decision process. It achieves global optimal operation and maintenance through multi-agent collaborative decision-making, and improves the security and accuracy of operation and maintenance actions by combining large language model and retrieval enhancement generation technology. At the same time, it supports unified operation and maintenance of heterogeneous databases.

[0005] To achieve the aforementioned technical effects, the technical solution adopted by this invention is: a multi-agent reinforcement learning database intelligent operation and maintenance method based on multi-agent Markov decision processes, comprising: Collect database operation data, process the operation data through multiple dedicated intelligent agents, and obtain candidate operation and maintenance suggestions output by each intelligent agent; The database operation and maintenance scenario is modeled as a multi-agent Markov decision process. The multi-agent Markov decision process is defined, and collaborative decision-making is carried out on each candidate operation and maintenance suggestion based on the multi-agent Markov decision process to generate the globally optimal operation and maintenance action. The global optimal operation and maintenance action is verified by calling the large language model, and the global optimal operation and maintenance action is corrected by combining the search enhancement to generate the RAG knowledge base, so as to obtain the final operation and maintenance instruction. The final operation and maintenance instruction is executed, and the execution result is fed back to the multi-agent Markov decision process to update the policy parameters.

[0006] As a preferred embodiment, the multi-agent Markov decision process is defined, specifically including: Define the state space to obtain a set of system states that include query characteristics and resource load; Define the action space to obtain a set of operation and maintenance operations that include index adjustment and parameter configuration; Define a reward function to obtain the reward calculation results with query response time and resource cost as variables; Define a discount factor to obtain a decay coefficient used to balance immediate rewards and long-term returns.

[0007] As a preferred embodiment, the processing of the operational data specifically includes: The system uses a natural language query agent to parse user input and generate candidate SQL statements. By analyzing the execution plan using a SQL performance optimization agent, suggestions for index adjustment can be obtained. Log features are extracted by a log analysis agent to obtain a log anomaly map; By matching historical cases with the fault diagnosis agent, the root cause diagnosis results are obtained; By predicting load trends through a resource prediction agent, resource scaling suggestions are obtained.

[0008] As a preferred implementation, collaborative decision-making is performed on each candidate operation and maintenance suggestion based on the multi-agent Markov decision process, specifically including: Determine if there are any conflicts among the candidate maintenance suggestions. If so, arbitrate according to the preset priority rules to obtain the arbitration result. The arbitration result is calculated to obtain the action combination corresponding to the maximum combined reward; Based on the action combination, the globally optimal operation and maintenance action is generated.

[0009] As a preferred implementation, the arbitration according to preset priority rules specifically includes: Determine the action type; if it is a fault self-healing action, assign it the highest priority. Determine the action objective; if it is a resource-saving action, it has higher priority than a cost-optimization action. If actions have the same priority, select the action combination that maximizes the combined reward.

[0010] As a preferred implementation, the globally optimal operation and maintenance action is verified, specifically including: Using a loosely coupled approach, natural language requirements are transformed into quantifiable objectives, which are then input into the multi-agent Markov decision process to obtain the optimization objective. Alternatively, a tightly coupled approach can be adopted to extract high-level semantic features from monitoring data and dynamically adjust the weights of the reward function to obtain an optimization strategy. Alternatively, collaboration modes can be dynamically switched based on task complexity to handle routine and complex tasks separately, resulting in appropriate collaboration outcomes.

[0011] As a preferred implementation, the globally optimal operation and maintenance action is modified, specifically including: Build an operations and maintenance knowledge base, slice and vectorize historical work orders, and generate vector indexes; Retrieve relevant knowledge fragments of the current alarm, inject them into the context of the large language model, and obtain the constrained output range; The globally optimal operation and maintenance action is verified to obtain the corrected final operation and maintenance instructions.

[0012] As a preferred embodiment, after executing the final maintenance instruction, the method further includes: Monitor the execution results and determine whether the strategy confidence level is lower than the preset threshold. If so, stop the execution and transfer it to manual review to obtain the manual confirmation result. Performance data before and after execution is collected and stored in the experience replay pool. The policy network of the multi-agent Markov decision process is incrementally updated to obtain the updated policy parameters.

[0013] As a preferred implementation, the method is adapted to heterogeneous databases, specifically including: Identify the target database type, fine-tune the large language model, and generate adapted model parameters; Configure the feature extraction operator for the corresponding database to obtain the standardized state input; Configure the corresponding database action execution interface to obtain standardized operation and maintenance instructions.

[0014] As a preferred implementation, the step of extracting log features through a log analysis agent to obtain a log anomaly map specifically includes: Identify error keywords and component identifiers in the logs and generate structured events; Align cross-node logs by transaction ID and time window to obtain the chain of error paths; Construct a log exception graph to automatically categorize error types and identify frequently triggered causes.

[0015] Compared with existing technologies, this invention achieves collaborative decision-making by multiple specialized agents through multi-agent Markov decision process modeling, effectively solving the problems of one-sided decision-making and conflicting actions of single agents, and generating globally optimal operation and maintenance actions that take into account performance, cost and stability.

[0016] By combining the semantic understanding capabilities of the large language model with the domain knowledge of the RAG knowledge base, the operation and maintenance actions are double-checked and corrected, which significantly reduces the risk of misoperation and improves the interpretability of the system.

[0017] Through the execution result feedback and incremental update mechanism, the system can continuously learn from operational experience, constantly optimize decision-making strategies, and adapt to ever-changing business scenarios.

[0018] By identifying database types, fine-tuning models, and configuring standardized interfaces, unified operation and maintenance of various heterogeneous databases has been achieved, reducing enterprise operation and maintenance costs.

[0019] It covers the entire process from data collection, fault diagnosis, decision generation to execution feedback, significantly reducing manual intervention, shortening fault response time, and improving operation and maintenance efficiency. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the logical flow of the intelligent database operation and maintenance method based on multi-agent Markov decision process of the present invention. Figure 2 This is a schematic diagram of the Markov decision process interaction mechanism in the multi-agent reinforcement learning database intelligent operation and maintenance method based on the multi-agent Markov decision process of the present invention. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the embodiments and accompanying drawings. However, this should not be construed as limiting the scope of the above-described subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0022] refer to Figure 1 , 2 Example 1: This example provides a multi-agent reinforcement learning-based intelligent operation and maintenance method for databases, based on multi-agent Markov decision processes. The specific steps are as follows: Step S1: Data Acquisition and Multi-Agent Processing First, collect all runtime data from the database, including but not limited to: query statements, execution plans, system logs, and performance metrics, including: CPU, memory, disk I / O, network latency, transaction information, and resource usage.

[0023] The collected operational data is distributed to multiple dedicated intelligent agents for parallel processing, with each agent focusing on a specific operational task: It consists of an intent parsing sub-agent, a schema understanding sub-agent, an SQL generation sub-agent, and an execution verification sub-agent. The intent parsing sub-agent uses LLM fine-tuning combined with a rule engine to achieve entity recognition and intent classification; the schema understanding sub-agent uses graph neural networks to construct schema embeddings and dynamically perceive changes in database metadata; the SQL generation sub-agent uses beamsearch sampling to generate a set of candidate SQL statements and uses syntax tree verification and security interception mechanisms to prevent errors and high-risk operations; the execution verification sub-agent predicts execution time and selects the optimal execution plan.

[0024] Analyze the execution plan of SQL statements, check index usage, identify missing or redundant indexes, and generate index adjustment suggestions; at the same time, analyze database parameter configurations and propose optimization suggestions for key parameters such as lock contention, connection count, and data caching.

[0025] Identify error keywords and component identifiers in the logs to generate structured events; align cross-node logs by transaction ID and time window to trace cascading error paths; construct a log anomaly graph to automatically classify error types and label high-frequency triggering reasons.

[0026] By combining error logs, performance data, and user feedback, an improved KNN algorithm is used to retrieve historical similar failure cases, perform root cause diagnosis, and generate failure self-healing suggestions.

[0027] Analyze historical resource usage data, identify resource usage patterns and trends, predict CPU, memory, disk, and network resource requirements for a future period, and generate resource scaling recommendations.

[0028] Step S2: Multi-agent Markov Decision Process Modeling and Collaborative Decision Making The database operation and maintenance scenario is modeled as a multi-agent Markov decision process (MAMDP), and its standard quintuple definition is as follows: State space S: contains query features, including a set of system states such as the number of joins, predicate complexity, data distribution, statistical histogram, and resource status; among which, data distribution includes table size and index existence, and resource status includes CPU / IO load and network latency.

[0029] Action Space A: Contains a collection of operation and maintenance operations such as index adjustment, parameter configuration, resource scaling, and fault repair.

[0030] State transition probability P: describes the probability distribution of the system transitioning from the current state to the next state after performing a certain operation and maintenance action.

[0031] The reward function R is designed with query response time and resource overhead as the core variables, while also considering factors such as SLA achievement rate and energy consumption. R = -α × response time - β × resource cost + γ × SLA achievement bonus; Where α, β, and γ are weighting coefficients.

[0032] The discount factor γ has a value range of [0,1] and is used to balance immediate rewards and long-term returns. It is usually set to 0.95.

[0033] Based on the MAMDP model described above, collaborative decision-making is performed on the candidate operation and maintenance suggestions output by each agent: ① First, determine whether there are any conflicts among the candidate operation and maintenance suggestions. If there are conflicts, arbitrate according to the preset priority rules: fault self-healing actions have the highest priority, followed by resource guarantee actions, and finally cost optimization actions; for actions with the same priority, select the action combination that maximizes the joint reward.

[0034] ② Employ multi-agent reinforcement learning algorithms such as QMIX or MADDPG to calculate the joint reward value corresponding to different action combinations.

[0035] ③ Select the action combination corresponding to the maximum combined reward to generate the globally optimal operation and maintenance action.

[0036] Step S3: Large Language Model Validation and RAG Correction The system utilizes domestically developed large-scale language models such as DeepSeekR1, Qwen3.5, and GLM5 to validate the generated globally optimal operation and maintenance actions. The validation process supports three collaborative modes: ① Loosely coupled mode: The large language model is used as an independent front-end module to transform the natural language requirements of operation and maintenance personnel into quantitative targets such as maximizing throughput and latency <200ms, and input them into the MAMDP system for optimization.

[0037] ② Tightly coupled mode: The large language model is deeply embedded in the MAMDP system to extract high-level semantic features from the raw monitoring data, dynamically adjust the weight of the reward function, and provide domain knowledge support for the agent.

[0038] ③ Hybrid Collaboration Mode: The collaboration mode is dynamically switched according to the complexity of the task. The loosely coupled mode is used for routine operation and maintenance tasks to ensure stability, while the tightly coupled mode is switched for complex fault diagnosis and optimization tasks to improve decision-making capabilities.

[0039] The globally optimal operational actions are corrected by combining search-enhanced generation (RAG) technology. Build a knowledge base for operations and maintenance, and slice and vectorize documents such as historical work orders, operation and maintenance manuals, and fault cases, and use Milvus vector database to generate vector indexes.

[0040] For the current operation and maintenance scenario, relevant knowledge fragments are retrieved from the knowledge base, injected into the context window of the large language model, and the output range of the model is constrained.

[0041] The large language model combines the retrieved knowledge to verify and correct the globally optimal operation and maintenance actions, and generates the final operation and maintenance instructions.

[0042] Step S4: Instruction Execution and Policy Update The final operation and maintenance instructions are executed on the target database through a standardized action execution interface. The system status and execution results are continuously monitored during the execution process. Confidence assessment and manual review: The confidence level of the strategy is calculated in real time. If the confidence level is lower than the preset threshold (0.7), the execution is immediately stopped and the process is transferred to manual review. The execution will continue only after the confirmation result is obtained from the operation and maintenance personnel.

[0043] Collect system performance data before and after execution, including response time, throughput, and resource utilization, and store it in the experience replay pool. Periodically sample data from the experience replay pool to incrementally update the MAMDP policy network and optimize policy parameters.

[0044] The method of this invention supports unified operation and maintenance of heterogeneous databases. The specific adaptation process is as follows: ① Automatically identifies the type of the target database, such as MySQL, Oracle, MongoDB, TiDB, etc.

[0045] ② Large model fine-tuning: Use the operation and maintenance data of the corresponding database to fine-tune the large language model and generate adapted model parameters.

[0046] ③ Feature extraction operator configuration: Configure the dedicated feature extraction operator for the corresponding database to convert the running data in different formats into standardized state input.

[0047] ④ Action execution interface configuration: Configure the action execution interface for the corresponding database to convert standardized operation and maintenance instructions into specific operations that can be executed by the database.

[0048] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A multi-agent reinforcement learning-based intelligent database operation and maintenance method based on multi-agent Markov decision processes, characterized in that, include: Collect database operation data, process the operation data through multiple dedicated intelligent agents, and obtain candidate operation and maintenance suggestions output by each intelligent agent; The database operation and maintenance scenario is modeled as a multi-agent Markov decision process. The multi-agent Markov decision process is defined, and collaborative decision-making is carried out on each candidate operation and maintenance suggestion based on the multi-agent Markov decision process to generate the globally optimal operation and maintenance action. The global optimal operation and maintenance action is verified by calling the large language model, and the global optimal operation and maintenance action is corrected by combining the search enhancement to generate the RAG knowledge base, so as to obtain the final operation and maintenance instruction. The final operation and maintenance instruction is executed, and the execution result is fed back to the multi-agent Markov decision process to update the policy parameters.

2. The intelligent operation and maintenance method for multi-agent reinforcement learning databases based on multi-agent Markov decision processes according to claim 1, characterized in that, The multi-agent Markov decision process is defined, specifically including: Define the state space to obtain a set of system states that include query characteristics and resource load; Define the action space to obtain a set of operation and maintenance operations that include index adjustment and parameter configuration; Define a reward function to obtain the reward calculation results with query response time and resource cost as variables; Define a discount factor to obtain a decay coefficient used to balance immediate rewards and long-term returns.

3. The intelligent operation and maintenance method for multi-agent reinforcement learning databases based on multi-agent Markov decision processes according to claim 1, characterized in that, The processing of the aforementioned operational data specifically includes: The system uses a natural language query agent to parse user input and generate candidate SQL statements. By analyzing the execution plan using a SQL performance optimization agent, suggestions for index adjustment can be obtained. Log features are extracted by a log analysis agent to obtain a log anomaly map; By matching historical cases with the fault diagnosis agent, the root cause diagnosis results are obtained; By predicting load trends through a resource prediction agent, resource scaling suggestions are obtained.

4. The intelligent operation and maintenance method for multi-agent reinforcement learning databases based on multi-agent Markov decision processes according to claim 1, characterized in that, Based on the aforementioned multi-agent Markov decision process, collaborative decision-making is performed on each candidate operation and maintenance suggestion, specifically including: Determine if there are any conflicts among the candidate maintenance suggestions. If so, arbitrate according to the preset priority rules to obtain the arbitration result. The arbitration result is calculated to obtain the action combination corresponding to the maximum combined reward; Based on the action combination, the globally optimal operation and maintenance action is generated.

5. The intelligent operation and maintenance method for multi-agent reinforcement learning databases based on multi-agent Markov decision processes according to claim 1, characterized in that, The arbitration based on preset priority rules specifically includes: Determine the action type; if it is a fault self-healing action, assign it the highest priority. Determine the action objective; if it is a resource-saving action, it has higher priority than a cost-optimization action. If actions have the same priority, select the action combination that maximizes the combined reward.

6. The intelligent operation and maintenance method for multi-agent reinforcement learning databases based on multi-agent Markov decision processes according to claim 1, characterized in that, The verification of the globally optimal operation and maintenance action specifically includes: Using a loosely coupled approach, natural language requirements are transformed into quantifiable objectives, which are then input into the multi-agent Markov decision process to obtain the optimization objective. Alternatively, a tightly coupled approach can be adopted to extract high-level semantic features from monitoring data and dynamically adjust the weights of the reward function to obtain an optimization strategy. Alternatively, collaboration modes can be dynamically switched based on task complexity to handle routine and complex tasks separately, resulting in appropriate collaboration outcomes.

7. The intelligent operation and maintenance method for a multi-agent reinforcement learning database based on a multi-agent Markov decision process according to claim 1, characterized in that, The globally optimal operation and maintenance action is modified, specifically including: Build an operations and maintenance knowledge base, slice and vectorize historical work orders, and generate vector indexes; Retrieve relevant knowledge fragments of the current alarm, inject them into the context of the large language model, and obtain the constrained output range; The globally optimal operation and maintenance action is verified to obtain the corrected final operation and maintenance instructions.

8. The intelligent operation and maintenance method for multi-agent reinforcement learning databases based on multi-agent Markov decision processes according to claim 1, characterized in that, After executing the final maintenance instruction, the process also includes: Monitor the execution results and determine whether the strategy confidence level is lower than the preset threshold. If so, stop the execution and transfer it to manual review to obtain the manual confirmation result. Performance data before and after execution is collected and stored in the experience replay pool. The policy network of the multi-agent Markov decision process is incrementally updated to obtain the updated policy parameters.

9. The intelligent operation and maintenance method for multi-agent reinforcement learning databases based on multi-agent Markov decision processes according to claim 1, characterized in that, The method is adapted to heterogeneous databases, specifically including: Identify the target database type, fine-tune the large language model, and generate adapted model parameters; Configure the feature extraction operator for the corresponding database to obtain the standardized state input; Configure the corresponding database action execution interface to obtain standardized operation and maintenance instructions.

10. The intelligent operation and maintenance method for multi-agent reinforcement learning databases based on multi-agent Markov decision processes according to claim 1, characterized in that, The step of extracting log features through a log analysis agent to obtain a log anomaly map specifically includes: Identify error keywords and component identifiers in the logs and generate structured events; Align cross-node logs by transaction ID and time window to obtain the chain of error paths; Construct a log exception graph to automatically categorize error types and identify frequently triggered causes.

Images