Memory adaptive adjustment method and system based on reinforcement learning

Abstract

The invention discloses a memory self-adaptive adjustment method based on reinforcement learning, which comprises the following steps: modeling a JVM memory management problem into a Markov decision process, taking a Q-learning reinforcement learning algorithm as an intelligent decision core, and through an autonomous closed loop of state awareness, action decision, security execution, reward feedback and model updating, realizing self-adaptive adjustment of the memory. The adaptive and refined optimization of the JVM memory is realized, and the dynamic load change can be coped with without manual intervention. Compared with the prior art, autonomous optimization of the JVM memory is achieved through the Q-learning algorithm, dynamic load changes can be coped with without manual intervention, and the problems that a traditional method is poor in flexibility and high in hysteresis quality are solved; the multi-target reward function gives consideration to the memory utilization rate, the application performance, the resource cost and the system stability, and the comprehensive optimization requirement in a complex scene is met; according to the invention, a security constraint rule and a progressive execution mechanism are built in, performance jitter or OOM risk caused by an adjustment action is avoided, and stable service operation is guaranteed.

Description

Technical Field

[0001] This invention relates to the fields of computer system performance optimization and artificial intelligence application technology, and in particular to a memory adaptive adjustment method based on reinforcement learning. Background Technology

[0002] In a Java application runtime environment, JVM memory management plays a decisive role in application performance. Current mainstream JVM memory tuning methods have significant shortcomings and struggle to meet the dynamically changing application workload requirements. These shortcomings, as shown in the diagram, include:

[0003] 1. Static configuration method: The memory size is set by fixing parameters such as -Xmx / -Xms. It cannot be adjusted according to the real-time changes in application load. When the load surges, it is easy to cause memory overflow (OOM) failure, while when the load is low, it will cause memory resources to be wasted.

[0004] 2. Rule-based adjustment method: This method relies on fixed thresholds to trigger adjustment actions. It has poor adaptability in complex application scenarios. For example, when facing sudden traffic or periodic load fluctuations, the threshold setting is difficult to balance performance and resource utilization, and is prone to adjustment lag.

[0005] 3. High dependence on manual intervention: Traditional methods rely heavily on the experience of operations and maintenance personnel for parameter configuration and optimization, lack a data-driven decision-making mechanism, and usually only make passive adjustments after an OOM event occurs, which has already caused service interruption and makes it impossible to achieve early prevention and autonomous optimization.

[0006] To address the aforementioned issues, there is an urgent need for a method and system capable of real-time sensing of JVM runtime status, autonomous decision-making on memory adjustment strategies, and consideration of multiple optimization objectives to solve existing technical problems. Summary of the Invention

[0007] This invention aims to solve at least one of the technical problems existing in the prior art, and proposes a memory adaptive adjustment method and system based on reinforcement learning.

[0008] In a first aspect, embodiments of the present invention provide a memory adaptive adjustment method based on reinforcement learning, comprising:

[0009] Real-time collection of runtime performance metrics data of the target JVM instance, including heap memory utilization, young generation memory ratio, garbage collection frequency, average garbage collection pause time, and number of active threads;

[0010] The performance index data is preprocessed and features are extracted to construct a state vector S_t representing the current memory state of the JVM. The continuous index is discretized to convert the continuous state space into a discrete state space.

[0011] The state vector S_t is input into a pre-trained Q-learning decision model, which selects the optimal memory adjustment action A_t from a predefined action space based on the built-in Q-value table and using an ε-greedy strategy.

[0012] The action A_t is executed using preset safety constraint rules, which include prohibiting dangerous adjustment actions, limiting the magnitude of a single adjustment, and performing the adjustment in batches in a gradual manner.

[0013] Collect the new JVM state S_{t+1} after executing action A_t, and calculate the immediate reward R_t based on a multi-objective reward function that integrates memory utilization, performance, cost and stability indicators.

[0014] Based on the state S_t, action A_t, reward R_t, and new state S_{t+1}, the Q-value table is updated using the Bellman equation to complete one online learning iteration.

[0015] Furthermore, the actions in the action space include at least adjusting the heap memory size, adjusting the ratio of the young generation to the old generation, and switching the garbage collector algorithm type.

[0016] Furthermore, the security constraint rules include:

[0017] When the heap memory usage in the state vector S_t is higher than the first safety threshold, the action of reducing the heap memory size is prohibited.

[0018] When the heap memory utilization rate in the state vector S_t is lower than the second safety threshold, the action of increasing the heap memory size is prohibited.

[0019] The upper limit is set for the magnitude of a single memory adjustment. The gradual adjustment method includes: decomposing the target adjustment amount into multiple micro-adjustments with a step size no greater than the upper limit, and executing them in batches over multiple monitoring periods.

[0020] Furthermore, the immediate reward R_t is calculated based on a multi-objective reward function, which is expressed as follows:

[0021] R=w1*ΔMemory+w2*ΔPerformance+w3*ΔCost+w4*ΔStability

[0022] Among them, ΔMemory is the reward component calculated based on the change in heap memory usage before and after the action is executed; ΔPerformance is the reward component calculated based on the change in application response time or throughput before and after the action is executed; ΔCost is the reward component calculated based on the saving of memory resource costs; ΔStability is the penalty or reward component calculated based on the change in system stability indicators; w1, w2, w3, w4 are the weight coefficients of each component, and w1+w2+w3+w4=1.

[0023] Furthermore, the method also includes a distributed collaboration service, the specific steps of which include:

[0024] The local decision-making model of multiple JVM instances periodically synchronizes their Q-value tables to a central coordinator;

[0025] The central coordinator aggregates and merges the received multiple Q-value tables to generate a globally optimized Q-value table.

[0026] The global Q-value table is then distributed to each local decision model to achieve knowledge sharing and collaborative optimization.

[0027] Furthermore, in the initial deployment phase of the Q-learning decision model, a rule engine is used to provide initial decisions. The rule engine is pre-configured with experience-based "if-then" rules, which are used to replace the model in action selection during the cold start phase when the Q-value table data is sparse. The transformation relationships generated by the rule engine are stored as initial experience samples in the experience replay pool to accelerate the initial training of the Q-learning model.

[0028] Secondly, the present invention also discloses a memory adaptive adjustment system based on reinforcement learning, comprising:

[0029] The monitoring agent module is deployed in the host environment where the target JVM is located, and is used to collect performance indicator data in real time.

[0030] The state processing module is used to receive the performance index data and perform preprocessing, feature extraction, and state vector S_t construction on the performance index data.

[0031] The Q-learning decision engine module has a built-in Q-value table and decision algorithm, which is used to receive the state vector and output the optimal memory adjustment action A_t;

[0032] The safety action executor module is used to receive the recommended action A_t, perform security verification on it, and then execute it in a progressive manner by calling the JVM management interface.

[0033] The reward calculator module is used to collect the new JVM state S_{t+1} after executing action A_t, and calculate the immediate reward R_t based on the state changes before and after the action is executed and the multi-objective reward function.

[0034] The learning update module updates the Q-value table in the decision engine using the Bellman equation based on the (S_t, A_t, R_t, S_t+1) tuple.

[0035] Furthermore, the deployment configuration of the system can be any of the following:

[0036] It attaches to the target JVM process in the form of a JavaAgent;

[0037] It is deployed in the same Kubernetes Pod as the application container running the target JVM, in the form of a Sidecar container, and communicates with the application container through a shared volume or local network;

[0038] It runs as a cluster-level controller in the form of KubernetesOperator, and enables batch memory tuning by managing the resource definitions of multiple application containers.

[0039] Furthermore, the system also includes a distributed collaboration service module for synchronizing and sharing the Q-value table among multiple JVM instances; the specific working steps include:

[0040] The local decision-making model of multiple JVM instances periodically synchronizes their Q-value tables to a central coordinator;

[0041] The central coordinator aggregates and merges the received multiple Q-value tables to generate a globally optimized Q-value table.

[0042] The global Q-value table is then distributed to each local decision model to achieve knowledge sharing and collaborative optimization.

[0043] Thirdly, the present invention also discloses an electronic device, characterized in that it comprises:

[0044] One or more processors;

[0045] Memory, used to store one or more programs;

[0046] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 6.

[0047] This invention discloses a memory adaptive adjustment method based on reinforcement learning. It models the JVM memory management problem as a Markov decision process and uses the Q-learning reinforcement learning algorithm as the core of intelligent decision-making. Through an autonomous closed loop of "state perception - action decision - safe execution - reward feedback - model update", it realizes the adaptive and refined optimization of JVM memory, which can cope with dynamic load changes without manual intervention.

[0048] The beneficial effects of this invention include at least the following:

[0049] 1. Intelligent and Adaptive: The Q-learning algorithm enables autonomous optimization of JVM memory, which can cope with dynamic load changes without manual intervention, solving the problems of poor flexibility and strong lag in traditional methods;

[0050] 2. Multi-objective optimization: The multi-objective reward function takes into account memory utilization, application performance, resource cost and system stability, and meets the comprehensive optimization needs in complex scenarios;

[0051] 3. High security: Built-in security constraint rules and progressive execution mechanism avoid performance jitter or OOM risk caused by adjustment actions, ensuring stable service operation;

[0052] 4. High scalability: Supports various deployment scenarios such as single machine, containerization, distributed and hybrid cloud, and can adapt to different application needs by expanding the action space and optimizing the reward function;

[0053] 5. Cost and performance balance: While ensuring application performance, it significantly reduces memory resource consumption and cloud resource costs, making it particularly suitable for large-scale microservices and cloud computing environments. Attached Figure Description

[0054] Figure 1 A flowchart illustrating a memory adaptive adjustment method based on reinforcement learning provided in an embodiment of the present invention;

[0055] Figure 2 The following is a flowchart illustrating the operational logic of a memory adaptive adjustment method based on reinforcement learning, as provided in an embodiment of the present invention.

[0056] Figure 3 A structural block diagram of a memory adaptive adjustment system based on reinforcement learning provided in an embodiment of the present invention;

[0057] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0058] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0059] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0060] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0061] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0062] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0063] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0064] To address at least one of the technical problems existing in the aforementioned related technologies, the present invention provides a memory adaptive adjustment method and system based on reinforcement learning.

[0065] This invention provides a memory adaptive adjustment method based on reinforcement learning, such as... Figure 1 and 2 ,include:

[0066] S100. Real-time collection of runtime performance metrics data of the target JVM instance, including heap memory utilization, young generation memory ratio, garbage collection frequency, average garbage collection pause time, and number of active threads;

[0067] Among them, Heap Usage: range [0,1], represents the current resource usage of heap memory;

[0068] Young Generation Ratio: Reflects the allocation ratio of JVM memory generation structure;

[0069] Garbage collection frequency (GCFrequency): The number of garbage collection events that occur per unit of time;

[0070] Garbage collection average pause time (GCPauseTime): A key metric affecting application response performance;

[0071] ActiveThreadCount: Reflects the current load intensity of the application.

[0072] S200. The performance index data is preprocessed and features are extracted to construct a state vector S_t representing the current memory state of the JVM, and the continuous index is discretized to convert the continuous state space into a discrete state space.

[0073] Specifically, a series of key performance indicators (KPIs) are collected in real time during JVM runtime. These KPIs collectively constitute the state of the environment perceived by the agent. The state vector S_t at time step *t* can be represented as:

[0074] S_t=(H,Y,F,P,T)

[0075] in:

[0076] H: Heap memory usage, range [0,1].

[0077] Y: Young Generation Ratio (YoungGenRatio)

[0078] F: Garbage collection frequency per unit time (GCF frequency).

[0079] P: Average pause time for garbage collection events (GCPauseTime).

[0080] T: The number of active threads in the JVM (ActiveThreadCount).

[0081] To adapt to tabular Q-learning, continuous states need to be discretized. For example, the heap memory utilization rate can be divided into multiple intervals such as [0-10%), [10-20%), ..., [90-100%] in 10% increments.

[0082] S300. The state vector S_t is input into the pre-trained Q-learning decision model. The decision model selects the optimal memory adjustment action A_t from the predefined action space according to the built-in Q-value table and using an ε-greedy strategy. In this embodiment, the actions in the action space include at least adjusting the heap memory size, adjusting the ratio of the young generation to the old generation, and switching the garbage collector algorithm type.

[0083] For example, a predefined actionspace A contains all executable memory adjustment operations:

[0084] A={IncreaseHeap(δ),DecreaseHeap(δ),IncreaseYoungGen(δ),DecreaseYoungGen(δ),ChangeGCAlgorithm,NoOp}

[0085] Where δ represents the adjustment margin (e.g., 10%). The Q-learning decision engine queries its internally maintained Q-table based on the current state S_t. The Q-value Q(s,a) represents the expected long-term cumulative reward obtained by choosing action *a* in state *s*. The agent uses an ε-greedy strategy to select actions: it randomly explores new actions with probability ε, and selects the action with the highest current Q-value with probability *1-ε* (utilizing known knowledge).

[0086] S400. The action A_t is executed using preset safety constraint rules, which include prohibiting dangerous adjustment actions, limiting the magnitude of a single adjustment, and performing the adjustment in batches in a gradual manner;

[0087] In this embodiment, the security constraint rules include:

[0088] When the heap memory usage in the state vector S_t is higher than the first safety threshold, the action of reducing the heap memory size is prohibited.

[0089] When the heap memory utilization rate in the state vector S_t is lower than the second safety threshold, the action of increasing the heap memory size is prohibited.

[0090] The upper limit is set for the magnitude of a single memory adjustment. The gradual adjustment method includes: decomposing the target adjustment amount into multiple micro-adjustments with a step size no greater than the upper limit, and executing them in batches over multiple monitoring periods.

[0091] The action A_t output by the decision engine will be implemented by a SafeExecutor. This module embeds safety constraint rules, such as:

[0092] Dangerous operations are prohibited: When heap memory utilization is >85%, the "reduce heap memory" action is prohibited.

[0093] Gradual adjustments: Each memory adjustment is limited to a certain range (e.g., ±20%) to prevent drastic performance fluctuations caused by sudden configuration changes. These configuration changes can be dynamically applied via the JVM's HotSpotAttach API or the API of a container platform (such as Kubernetes).

[0094] S500. Collect the new JVM state S_{t+1} after executing action A_t, and calculate the immediate reward R_t based on the multi-objective reward function, which integrates memory utilization, performance, cost and stability indicators;

[0095] Specifically, the immediate reward R_t is calculated based on a multi-objective reward function, which is expressed as follows:

[0096] R=w1*ΔMemory+w2*ΔPerformance+w3*ΔCost+w4*ΔStability

[0097] Among them, ΔMemory is the reward component calculated based on the change in heap memory usage before and after the action is executed; ΔPerformance is the reward component calculated based on the change in application response time or throughput before and after the action is executed; ΔCost is the reward component calculated based on the saving of memory resource costs; ΔStability is the penalty or reward component calculated based on the change in system stability indicators; w1, w2, w3, w4 are the weight coefficients of each component, and w1+w2+w3+w4=1.

[0098] S600. Based on the state S_t, action A_t, reward R_t, and new state S_{t+1}, update the Q-value table using the Bellman equation to complete one online learning iteration.

[0099] Specifically, after receiving the reward R_t, the Q-value table is updated using the Bellman Equation:

[0100] *Q(S_t,A_t)←Q(S_t,A_t)+α[R_t+γmaxQ(S_{t+1},a)-Q(S_t,A_t)]*

[0101] Here, α is the learning rate, and γ is the discount factor. Through continuous iteration, the Q-value table eventually converges, meaning the agent learns which actions to take in which states to obtain the maximum long-term reward.

[0102] In some preferred embodiments, the method further includes a distributed collaboration service, the specific steps of which include:

[0103] The local decision-making model of multiple JVM instances periodically synchronizes their Q-value tables to a central coordinator;

[0104] The central coordinator aggregates and merges the received multiple Q-value tables to generate a globally optimized Q-value table.

[0105] The global Q-value table is then distributed to each local decision model to achieve knowledge sharing and collaborative optimization.

[0106] In this embodiment, during the initial deployment phase of the Q-learning decision model, a rule engine is used to provide initial decisions. The rule engine is pre-configured with experience-based "if-then" rules, which are used to replace the model in action selection during the cold start phase when the Q-value table data is sparse. The transformation relationships generated by the rule engine are stored as initial experience samples in the experience replay pool to accelerate the initial training of the Q-learning model.

[0107] This embodiment discloses a reinforcement learning-based adaptive memory adjustment method. It models the JVM memory management problem as a Markov decision process, using the Q-learning reinforcement learning algorithm as the core of intelligent decision-making. Through an autonomous closed loop of "state awareness - action decision - safe execution - reward feedback - model update," it achieves adaptive and refined optimization of JVM memory, responding to dynamic load changes without manual intervention. Compared to existing technologies, this invention achieves autonomous optimization of JVM memory through the Q-learning algorithm, responding to dynamic load changes without manual intervention, solving the problems of poor flexibility and strong lag in traditional methods. The multi-objective reward function of this invention balances memory utilization, application performance, resource cost, and system stability, meeting the comprehensive optimization needs in complex scenarios. This invention incorporates safety constraint rules and a progressive execution mechanism to avoid performance fluctuations or OutOfMemory errors caused by adjustment actions, ensuring stable service operation.

[0108] To better understand the methods disclosed in the above embodiments, this invention applies the methods to specific cases. For example, taking a single Tomcat Web application as an example, the core implementation process of this embodiment includes the following:

[0109] 1. Hardware and software environment

[0110] Hardware configuration: CPU: 4 cores 2.5GHz, Memory: 16GB DDR4, Hard drive: 100GB SSD.

[0111] Software environment: Operating system: Ubuntu 20.04; Java runtime: OpenJDK 17; Target application: Apache Tomcat 9.0 with Spring Boot 2.7 web application.

[0112] 2. System Deployment and Integration

[0113] This invention is preferably deployed in the form of a JavaAgent. The system is packaged into a JAR file and loaded via the -javaagent parameter when the target JVM starts.

[0114] java-javaagent:qlearning-memory-agent.jar=config.properties\

[0115] -Xmx4g-Xms1g\

[0116] -jarmy-springboot-app.jar

[0117] The config.properties file is the system's configuration file.

[0118] 3. Configuration and Implementation of Core Components

[0119] (1) Collection of monitoring indicators

[0120] The agent collects metrics through the JVM's JMX (Java Management Extensions) interface. It uses com.sun.management.GarbageCollectorMXBean to obtain GC information and java.lang.management.MemoryMXBean to obtain memory pool information.

[0121] (2) Definition and discretization of state space

[0122] Define the state dimensions and their discretization intervals in the configuration file config.properties:

[0123] The state S_t will be encoded as a discretized key, for example (0.7,20,500) indicates that the heap memory utilization is in the range of 70%-85%, the GC frequency is 10-20 times / minute, and the GC pause is 500-1000 milliseconds.

[0124] (3) Definition of action space

[0125] (4) Reward function configuration

[0126] (5) Q-learning algorithm parameter configuration

[0127] 4. Work Process

[0128] Startup: The Agent starts with the JVM, loads the configuration, and initializes the Q-value table (initially empty or loading a pre-trained model).

[0129] Training phase:

[0130] The status S_t is collected every 5 minutes (configurable).

[0131] The decision engine selects action A_t based on the current exploration rate ε (e.g., increase heap memory by 10%).

[0132] The safety executor verifies this action: the current heap utilization is 75%, allowing the execution of `increase_heap_10`. It dynamically sets `-XX:MaxHeapSize` by calling the `com.sun.tools.attach.VirtualMachineAPI`.

[0133] Wait for the next cycle, collect the new state S_{t+1} and performance data (such as the P99 response time in this cycle).

[0134] The reward calculator calculates the reward R_t (e.g., response time decrease, reward +15; memory increase, cost penalty -2; net reward +13).

[0135] The learning update module updates the Q-value table based on the (S_t, A_t, R_t, S_{t+1}) tuple.

[0136] The exploration rate decreases over time, and decision-making gradually shifts from exploration to utilization.

[0137] Stable operation phase: After several days of training, the exploration rate drops to a low level, and the system can stably manage memory according to the learned optimal strategy.

[0138] 5. Expected Results

[0139] After applying simulated traffic load to the Tomcat application and training for 72 hours, the results compared to a configuration with a fixed 4GB heap memory are as follows:

[0140] Memory cost: Average heap memory requirement has been reduced from 4G to 2.8G, saving 30% of memory resources.

[0141] Performance: P99 latency stabilized at around 150ms from 250ms, with reduced glitches.

[0142] Stability: Successfully prevented two OOM (Out of Memory) errors that could have been caused by sudden traffic spikes during the training period.

[0143] The following example uses a cloud-native Kubernetes cluster to demonstrate large-scale implementation in a containerized, microservices environment.

[0144] 1. System Architecture

[0145] This invention is preferably deployed in the form of KubernetesOperator.

[0146] Deploy a Deployment named memory-optimizer-operator to the cluster.

[0147] Operators listen to Pods that contain specific annotations, such as optimize-memory:"true".

[0148] No agent needs to be installed in each monitored Pod; the Operator collects metrics through the K8sMetricsAPI and Prometheus.

[0149] 2. Custom Resource Definition (CRD)

[0150] Define a MemoryOptimizationPolicyCRD object to configure differentiated policies for different applications.

[0151] 3. Distributed learning

[0152] The Operator acts as a central coordinator, maintaining a local copy of the Q-value table for each managed Pod (i.e., each JVM instance). Periodically (e.g., hourly), it performs federated averaging of the Q-values ​​from all copies to generate a global Q-value table, which is then synchronized back to all Pods. This allows all instances of a service to share learning experiences, accelerating overall convergence.

[0153] 4. Action execution

[0154] Operators perform tweaks by dynamically updating the resource limits of Pods. Kubernetes kubelet then takes over and safely applies these changes.

[0155] The following is an example of an optimized implementation method using a hybrid cloud scenario as an example of cost:

[0156] This embodiment further expands upon the previous one, demonstrating how to achieve global optimization of cost and performance in a hybrid cloud environment.

[0157] 1. Expanding the action space

[0158] Building upon cluster management actions, cloud service provider API calls are introduced. The action space is expanded to:

[0159] Local adjustment: Adjust Pod memory limits (same as Example 2).

[0160] Vertical scaling up / down: Change the virtual machine instance type of the node where the Pod resides (e.g., upgrade from m5.large to m5.xlarge).

[0161] Horizontal scaling: Adjusts the number of replicas of a Deployment.

[0162] Migration: Migrate a Pod from an on-premises data center to a public cloud node pool, or vice versa.

[0163] 2. Cost-Perceived Reward Function

[0164] The ΔCost component in the reward function R_t requires complex calculations:

[0165] 3. Implementation Results

[0166] During periods of low traffic at night, the system may automatically perform the following sequence of actions:

[0167] Reduce the memory usage of Pods for some non-critical services.

[0168] These Podconsolidations are merged onto fewer nodes.

[0169] Shut down any vacant nodes to save on electricity and resource costs.

[0170] Before the daytime traffic peak arrives, reverse the above operation.

[0171] The entire process is fully automated, requiring no maintenance intervention, and achieves significant cost savings while ensuring SLA.

[0172] Based on the same inventive concept, embodiments of the present invention also provide a memory adaptive adjustment system based on reinforcement learning, comprising:

[0173] The monitoring agent module is deployed in the host environment where the target JVM is located, and is used to collect performance indicator data in real time.

[0174] The state processing module is used to receive the performance index data and perform preprocessing, feature extraction, and state vector S_t construction on the performance index data.

[0175] The Q-learning decision engine module has a built-in Q-value table and decision algorithm, which is used to receive the state vector and output the optimal memory adjustment action A_t;

[0176] The safety action executor module is used to receive the recommended action A_t, perform security verification on it, and then execute it in a progressive manner by calling the JVM management interface.

[0177] The reward calculator module is used to collect the new JVM state S_{t+1} after executing action A_t, and calculate the immediate reward R_t based on the state changes before and after the action is executed and the multi-objective reward function.

[0178] The learning update module updates the Q-value table in the decision engine using the Bellman equation based on the (S_t, A_t, R_t, S_t+1) tuple.

[0179] In this embodiment, the system can be deployed in any of the following ways:

[0180] It attaches to the target JVM process in the form of a JavaAgent;

[0181] It is deployed in the same Kubernetes Pod as the application container running the target JVM, in the form of a Sidecar container, and communicates with the application container through a shared volume or local network;

[0182] It runs as a cluster-level controller in the form of KubernetesOperator, and enables batch memory tuning by managing the resource definitions of multiple application containers.

[0183] The system also includes a distributed collaboration service module for synchronizing and sharing the Q-value table among multiple JVM instances; the specific working steps include:

[0184] The local decision-making model of multiple JVM instances periodically synchronizes their Q-value tables to a central coordinator;

[0185] The central coordinator aggregates and merges the received multiple Q-value tables to generate a globally optimized Q-value table.

[0186] The global Q-value table is then distributed to each local decision model to achieve knowledge sharing and collaborative optimization.

[0187] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 4 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the memory adaptive adjustment methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0188] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0189] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0190] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0191] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the memory adaptive adjustment methods described in the above embodiments. The computer-readable storage medium may be volatile or non-volatile.

[0192] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described memory adaptive adjustment method.

[0193] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0194] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0195] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0196] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0197] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0198] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0199] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0200] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0201] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0202] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A memory self-adaptive adjustment method based on reinforcement learning, characterized in that, The method comprises the following steps: Collecting real-time performance index data of a target JVM instance, the indexes including heap memory usage, young generation memory proportion, garbage collection frequency, average garbage collection pause time, and active thread number; Preprocessing and feature extraction are performed on the performance index data to construct a state vector S_t representing the current memory state of the JVM, and continuous state space is converted into discrete state space through discretization processing of continuous indexes; The state vector S_t is input into a pre-trained Q-learning decision model, which selects an optimal memory adjustment action A_t from a predefined action space according to a built-in Q value table and an ε-greedy strategy; The action A_t is executed by using preset safety constraint rules, including prohibition of dangerous adjustment actions, limitation of single adjustment amplitude, and gradual batch execution of adjustment; The new state S_{t+1} of the JVM after the execution of the action A_t is collected, and an immediate reward R_t is calculated based on a multi-objective reward function, which comprehensively considers memory utilization, performance, cost, and stability indexes; According to the state S_t, the action A_t, the reward R_t, and the new state S_{t+1}, the Q value table is updated using the Bellman equation to complete one online learning iteration.

2. The method of claim 1, wherein, The actions in the action space include at least adjusting the size of the heap memory, adjusting the ratio of the young generation memory to the old generation memory, and switching the garbage collector algorithm type.

3. The method of claim 1, wherein, The safety constraint rules include: When the heap memory usage in the state vector S_t is higher than a first safety threshold, the action of reducing the size of the heap memory is prohibited; When the heap memory usage in the state vector S_t is lower than a second safety threshold, the action of increasing the size of the heap memory is prohibited; An upper limit is set for the amplitude of single memory adjustment, and the gradual batch adjustment includes decomposing the target adjustment amount into multiple micro-adjustments with a step size not greater than the upper limit, and performing the micro-adjustments in multiple monitoring periods.

4. The method of claim 1, wherein, The multi-objective reward function for calculating the immediate reward R_t is as follows: R = w1*ΔMemory + w2*ΔPerformance + w3*ΔCost + w4*ΔStability Where ΔMemory is a reward component calculated based on the change in heap memory usage before and after the action execution; ΔPerformance is a reward component calculated based on the change in application response time or throughput before and after the action execution; ΔCost is a reward component calculated based on the saving of memory resource cost; ΔStability is a penalty or reward component calculated based on the change in system stability index; w1, w2, w3, and w4 are weight coefficients of each component, and w1 + w2 + w3 + w4 = 1.

5. The method of claim 1, wherein, The method further comprises a distributed collaborative service, and the specific steps include: Local decision models of multiple JVM instances periodically synchronize their Q value tables to a central coordinator; The central coordinator aggregates and fuses the received multiple Q value tables to generate a globally optimized Q value table; The global Q-value table is re-deployed to each local decision model to realize knowledge sharing and collaborative optimization.

6. The method of claim 1, wherein, In the initial deployment stage of the Q-learning decision model, an initial decision is provided by a rule engine pre-configured with experience-based "if-then" rules for action selection in the cold start stage when the Q-value table data is sparse; the conversion relationship generated by the rule engine is stored in the experience replay pool as initial experience samples to accelerate the initial training of the Q-learning model.

7. A memory self-adaptive adjustment system based on reinforcement learning, employing any of the memory self-adaptive adjustment methods of claims 1-6, characterized in that, The system comprises: a monitoring agent module deployed in a host environment where the target JVM is located, for collecting performance indicator data in real time; a state processing module for receiving the performance indicator data and pre-processing, feature extraction and state vector S_t construction of the performance indicator data; a Q-learning decision engine module with a built-in Q-value table and decision algorithm, for receiving a state vector and outputting an optimal memory adjustment action A_t; a safe action executor module for receiving the recommended action A_t, performing security verification and executing in a gradual manner by calling a JVM management interface; a reward calculator module for collecting a new JVM state S_{t+1} after the execution of the action A_t, calculating an immediate reward R_t according to a multi-objective reward function based on the state change before and after the action execution; a learning update module for updating the Q-value table in the decision engine according to the (S_t, A_t, R_t, S_{t+1}) tuple using the Bellman equation.

8. The system of claim 7, wherein, The system can be deployed in any of the following forms: in the form of a JavaAgent attached to the inside of the target JVM process; in the form of a Sidecar container deployed in the same KubernetesPod as the application container running the target JVM and communicating through a shared volume or local network; in the form of a KubernetesOperator running as a cluster-level controller to achieve batch memory adjustment through resource definition (ResourceDefinition) management of multiple application containers.

9. The system of claim 7, wherein, The system further comprises a distributed collaboration service module for synchronizing and sharing Q-value tables among multiple JVM instances; the specific working steps include: the local decision model of each JVM instance periodically synchronizes its Q-value table to a central coordinator; the central coordinator aggregates and fuses the multiple Q-value tables received to generate a globally optimized Q-value table; the global Q-value table is re-deployed to each local decision model to realize knowledge sharing and collaborative optimization.

10. An electronic device, comprising: The system comprises: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 6.

Images