An online reinforcement learning method and system based on a virtual sandbox
By constructing a virtual sandbox for online reinforcement learning, the agent learns autonomously in an isolated environment and receives real feedback, solving the problems of dependence on manual annotation and disconnection from the training environment, and realizing efficient and secure migration and learning of operation and maintenance strategies.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Filing Date
- 2026-03-02
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies rely on manually labeled data when training operational agents, the reward function design is subjective, and the training environment is disconnected from the real environment, resulting in insufficient model generalization ability and poor policy performance in the real world.
A virtual sandbox is constructed that is consistent with the real environment but physically isolated. An online reinforcement learning algorithm is used to simulate operation and maintenance challenges in the sandbox. The agent learns autonomously in the sandbox and receives objective rewards, including simulating hardware failures, software anomalies and user behaviors in the sandbox, and updating the policy network in real time.
It achieves autonomous learning without manual data labeling, has an objective and fair reward mechanism, ensures safe training, and allows policies to be directly transferred to real environments. It solves the problems of data scarcity and the simulation-reality gap, and improves the operational capabilities of intelligent agents.
Smart Images
Figure CN122133743A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and IT operations and maintenance, and in particular to an online reinforcement learning method and system based on a virtual sandbox. Background Technology
[0002] With the increasing complexity of information systems, automated operations and maintenance (AIOps) using artificial intelligence technology has become a mainstream trend. Among them, reinforcement learning-based agents are considered an ideal solution for solving complex and dynamic cluster fault diagnosis and recovery tasks.
[0003] However, existing technologies face the following main challenges when training operational agents: Reliance on manually labeled data: Mainstream methods typically rely on historical operation and maintenance logs, monitoring data, etc., for offline supervised learning or imitation learning. These datasets are often scarce, costly to label, and cover limited fault scenarios, resulting in insufficient model generalization ability.
[0004] Reward function design is difficult and subjective: Reward functions in reinforcement learning usually require careful design by domain experts. This not only injects a large amount of human prior knowledge and bias, but also makes it difficult to quantify the long-term value of all operations, limiting the potential of agents to discover super-human solutions.
[0005] Sim-to-Real Gap: For security reasons, agents in the training phase are typically unable to interact with real production environments. This prevents the agent from learning dynamic feedback from the real environment, and policies learned in the simulation environment may experience significantly reduced performance when deployed in the real world.
[0006] Therefore, there is an urgent need for a new training paradigm that can break free from data dependence, provide objective rewards, and allow agents to interact with high-fidelity environments in real time in a secure environment. Summary of the Invention
[0007] In view of the above problems, the present invention is proposed to provide an online reinforcement learning method and system based on a virtual sandbox that overcomes or at least partially solves the above problems.
[0008] According to one aspect of the present invention, an online reinforcement learning method based on a virtual sandbox is provided, the online reinforcement learning method comprising: Construct a virtual sandbox that is functionally identical to the real environment but physically isolated, including: Dynamically clone or synchronize in real time the network topology, hardware configuration, software stack, service deployment and data status of the real production cluster to build a virtual sandbox that is functionally identical to the real environment but physically isolated. Simulate real-world operational challenges in a sandbox environment; The reinforcement learning agent to be trained; Environmental status monitoring and reward assessment; It receives the experience trajectories generated by the interaction between the agent and the sandbox, and uses an online reinforcement learning algorithm to update the policy network of the agent module in real time.
[0009] Optionally, simulating real-world operational challenges in a sandbox environment specifically includes: Based on preset scripts or random strategies, various common hardware failures, software anomalies, and performance bottlenecks are simulated in the sandbox. Simulate real user access traffic, API calls, and business load to provide realistic background pressure for the agent's decision-making.
[0010] Optionally, the hardware failure includes disk corruption and network partitioning; the software anomaly includes program crashes and configuration errors.
[0011] Optionally, the reinforcement learning agent to be trained specifically includes: The agent is granted full operational privileges within the sandbox environment to execute maintenance actions output by its policy network.
[0012] Optionally, the full operation permissions specifically include: executing Shell commands, calling K8s API, and modifying configuration.
[0013] Optionally, the environmental status monitoring and reward assessment specifically includes: Key performance indicators of the sandbox environment are collected in real time to form the state of reinforcement learning. The effectiveness of the agent's operation is judged based on preset system health standards; A significant positive reward is generated when the agent's behavior restores the system from a faulty state to a healthy state; otherwise, a negative reward or zero reward is given depending on the degree of state deterioration or no change.
[0014] Optionally, the key performance indicators specifically include: CPU / memory utilization, service response latency, and error rate.
[0015] This invention also provides an online reinforcement learning system based on a virtual sandbox, applying the aforementioned online reinforcement learning method based on a virtual sandbox. The online reinforcement learning system includes: The Mirror Sandbox Environment module is used to build a virtual sandbox that is functionally identical to the real environment but physically isolated. An automated scenario generation module is used to simulate real-world operational challenges in a sandbox environment; The operations and maintenance intelligent agent module is used for the reinforcement learning intelligent agent to be trained; The environmental status monitoring and reward assessment module is used for environmental status monitoring and reward assessment. The online reinforcement learning training module receives experience tuples generated by the agent's interaction with the sandbox and updates the agent module's policy network in real time using an online reinforcement learning algorithm.
[0016] This invention provides an online reinforcement learning method and system based on a virtual sandbox. The online reinforcement learning method includes: constructing a virtual sandbox that is functionally identical to the real environment but physically isolated; simulating real-world operational challenges within the sandbox environment; a reinforcement learning agent to be trained; environmental state monitoring and reward evaluation; receiving experience trajectories generated by the agent's interaction with the sandbox; and using an online reinforcement learning algorithm to update the agent's policy network in real time. This enables the agent to learn autonomously through direct interaction with the environment in a safe and high-fidelity environment, without the need for manually labeled data, and to obtain objective rewards based on feedback from the real environment.
[0017] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description
[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart illustrating an online reinforcement learning method based on a virtual sandbox, as provided in an embodiment of the present invention. Detailed Implementation
[0020] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0021] The terms "comprising" and "having," and any variations thereof, in the specification, embodiments, claims, and drawings of this invention are intended to cover non-exclusive inclusion, such as including a series of steps or units.
[0022] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0023] This invention aims to address the problems of existing training methods for operational intelligence agents, such as reliance on manual data, strong subjectivity of reward functions, and disconnect between the training environment and the real environment. It provides an online reinforcement learning system and method based on a mirrored sandbox, enabling the agent to learn autonomously through direct interaction with the environment in a safe, high-fidelity environment, without the need for manually labeled data, and to obtain objective rewards based on feedback from the real environment.
[0024] like Figure 1 As shown, this invention provides an online reinforcement learning method based on a virtual sandbox, the online reinforcement learning method comprising: Build a virtual sandbox that is functionally identical to the real environment but physically isolated; Simulate real-world operational challenges in a sandbox environment; The reinforcement learning agent to be trained; Environmental status monitoring and reward assessment; It receives the experience trajectories generated by the interaction between the agent and the sandbox, and uses an online reinforcement learning algorithm to update the policy network of the agent module in real time.
[0025] This invention provides an online reinforcement learning system for operation and maintenance agents. This system constructs a mirror sandbox that is completely consistent with the real production environment, and combines automated scenario generation and online reinforcement learning to achieve closed-loop self-evolution training of the agents.
[0026] The system includes: MirroredSandboxEnvironment: The core of the system. This module is responsible for dynamically cloning or synchronizing in real time the network topology, hardware configuration, software stack, service deployment, and data status of the real production cluster, building a virtual sandbox that is functionally identical to the real environment but physically isolated.
[0027] Automated scenario generation module: Responsible for simulating real-world operational challenges in a sandbox environment. This module includes: Fault injection unit: It can simulate various common hardware failures (such as disk failure, network partition), software anomalies (such as program crashes, configuration errors) and performance bottlenecks in the sandbox according to preset scripts or random strategies.
[0028] User behavior simulation unit: It can simulate the access traffic, API calls and business load of real users, providing realistic background pressure for the decision-making of intelligent agents.
[0029] Operations Agent Module: A reinforcement learning agent to be trained. This agent is granted full operational privileges within the sandbox environment (such as executing shell commands, calling K8s APIs, modifying configurations, etc.) to execute the operations actions output by its policy network.
[0030] Environmental Status Monitoring and Reward Assessment Module: State monitoring unit: responsible for collecting key performance indicators (KPIs) of the sandbox environment in real time, such as CPU / memory utilization, service response latency, error rate, etc., which constitute the state of reinforcement learning.
[0031] The reward evaluation unit assesses the agent's performance based on preset system health standards (Service Level Objectives, SLOs). When the agent's actions restore the system from a faulty state to a healthy state (e.g., the error rate drops below a threshold), a significant positive reward is generated. Otherwise, a negative reward or zero reward is given depending on the degree of state deterioration or no change. This reward mechanism is based on real-world feedback from the environment, rather than manually defined rules.
[0032] The online reinforcement learning training module is responsible for receiving experience tuples (states, actions, rewards, and next states) generated by the interaction between the agent and the sandbox, and updating the agent module's policy network in real time using online reinforcement learning algorithms (such as PPO, SAC, etc.).
[0033] The method steps of the present invention are as follows: Step S1: Environment Initialization: When the system starts, the mirror sandbox module clones the current production environment and generates a high-fidelity initial sandbox.
[0034] Step S2: Scene Trigger: The automated scene generation module injects a fault or simulates a stress scenario into the sandbox, causing the environmental state to deviate from normal.
[0035] Step S3: Intelligent Agent Interaction: The operation and maintenance intelligent agent module observes the abnormal state and selects and executes one or a series of operation and maintenance actions according to the current strategy.
[0036] Step S4: Reward Calculation and Feedback: The environment status monitoring and reward evaluation module continuously monitors the sandbox status. Once the system KPIs are detected to have returned to normal, the problem is considered resolved, and a positive reward is issued to the agent.
[0037] Step S5: Model Iteration: The online reinforcement learning training module uses this successful interaction experience to update the agent's policy. If the agent's actions are ineffective or worsen the situation, it will receive a negative reward, which is also used to correct the policy.
[0038] Step S1: Cyclic Training: The system automatically resets the sandbox environment or injects new faults, continuously repeating steps S2 to S5, so that the agent can continuously learn and experiment in massive and diverse simulated combat, and continuously improve its operation and maintenance capabilities.
[0039] Beneficial effects: Achieving true autonomous learning: Freed from dependence on expensive and sparse manually labeled data, intelligent agents learn through direct interaction with the environment, and training data can be generated infinitely, with efficiency and potential far exceeding traditional methods.
[0040] The reward mechanism is objective and fair: the rewards come directly from the real-world feedback of the environment, eliminating biases imposed by human design, and enabling the agent to explore and learn better solutions that may surpass the experience of human operation and maintenance experts.
[0041] Absolute training safety: All exploratory and even destructive operations are conducted in an isolated sandbox, ensuring the absolute safety of the production environment and solving the core problem of "not daring to try and make mistakes" in agent training.
[0042] Seamless policy transfer (Sim-to-Real): Since the sandbox is a high-fidelity mirror of the production environment, the policies learned by the agent in it can be directly applied to the real environment with extremely high confidence, effectively solving the "simulation-reality gap" problem.
[0043] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. An online reinforcement learning method based on a virtual sandbox, characterized in that, The online reinforcement learning method includes: Construct a virtual sandbox that is functionally identical to the real environment but physically isolated, including: Dynamically clone or synchronize in real time the network topology, hardware configuration, software stack, service deployment and data status of the real production cluster to build a virtual sandbox that is functionally identical to the real environment but physically isolated. Simulate real-world operational challenges in a sandbox environment; The reinforcement learning agent to be trained; Environmental status monitoring and reward assessment; It receives the experience trajectories generated by the interaction between the agent and the sandbox, and uses an online reinforcement learning algorithm to update the policy network of the agent module in real time.
2. The online reinforcement learning method based on a virtual sandbox according to claim 1, characterized in that, The simulation of real-world operational challenges in the sandbox environment specifically includes: Based on preset scripts or random strategies, various common hardware failures, software anomalies, and performance bottlenecks are simulated in the sandbox. Simulate real user access traffic, API calls, and business load to provide realistic background pressure for the agent's decision-making.
3. The online reinforcement learning method based on a virtual sandbox according to claim 1, characterized in that, The hardware failures include disk corruption and network partitioning; the software anomalies include program crashes and configuration errors.
4. The online reinforcement learning method based on a virtual sandbox according to claim 1, characterized in that, The reinforcement learning agent to be trained specifically includes: The agent is granted full operational privileges within the sandbox environment to execute maintenance actions output by its policy network.
5. The online reinforcement learning method based on a virtual sandbox according to claim 4, characterized in that, The full operational permissions specifically include: executing Shell commands, calling K8s API, and modifying configurations.
6. The online reinforcement learning method based on a virtual sandbox according to claim 1, characterized in that, The environmental status monitoring and reward assessment specifically includes: Key performance indicators of the sandbox environment are collected in real time to form the state of reinforcement learning. The effectiveness of the agent's operation is judged based on preset system health standards; A significant positive reward is generated when the agent's behavior restores the system from a faulty state to a healthy state; otherwise, a negative reward or zero reward is given depending on the degree of state deterioration or no change.
7. The online reinforcement learning method based on a virtual sandbox according to claim 6, characterized in that, The key performance indicators specifically include: CPU / memory utilization, service response latency, and error rate.
8. An online reinforcement learning system based on a virtual sandbox, employing the online reinforcement learning method based on a virtual sandbox as described in any one of claims 1-7, characterized in that, The online reinforcement learning system includes: The Mirror Sandbox Environment module is used to build a virtual sandbox that is functionally identical to the real environment but physically isolated. An automated scenario generation module is used to simulate real-world operational challenges in a sandbox environment; The operations and maintenance intelligent agent module is used for the reinforcement learning intelligent agent to be trained; The environmental status monitoring and reward assessment module is used for environmental status monitoring and reward assessment. The online reinforcement learning training module receives experience tuples generated by the agent's interaction with the sandbox and updates the agent module's policy network in real time using an online reinforcement learning algorithm.