Hyper-parameter dynamic optimization method and system based on adaptive meta-reinforcement learning

Abstract

The invention discloses a hyper-parameter dynamic optimization method and system based on adaptive meta reinforcement learning, and the method comprises the steps: carrying out the hyper-parameter initialization through an optimizer based on a tree structure Pasen estimator algorithm, so as to obtain the initial hyper-parameter configuration of an intelligent agent; enabling the intelligent agent to interact with the environment based on the initial hyper-parameter configuration to carry out internal circulation of reinforcement learning training; on the basis of inner circulation of reinforcement learning training, an outer circulation element learning mechanism is executed, and in the outer circulation element learning mechanism, an element learner collects and monitors aggregation performance indexes every preset time step length; causing the meta learner to dynamically adjust one or more hyper-parameters of the agent based on the aggregated performance metric; and continuing the internal circulation of the reinforcement learning training by using the adjusted hyper-parameters. According to the method, the meta-reinforcement learning does not need complex infrastructures, the hyper-parameter optimization is easier to obtain, and meanwhile, the cost and the environmental influence are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning. More particularly, the present application relates to a hyperparameter dynamic optimization method and system based on adaptive meta-reinforcement learning. BACKGROUND

[0002] Reinforcement learning has driven the development of artificial intelligence by enabling agents to master complex games, robot control, and resource management with minimal human guidance. However, hyperparameter tuning remains a significant challenge: learning rates, discount factors, and exploration strategies have a significant impact on performance and can vary depending on the task or even the training process. Existing methods, often based on costly grid search or manual adjustments requiring deep domain knowledge, have limitations, and this challenge is more severe in multi-task scenarios where agents need to perform well in multiple environments. The training requirements of modern reinforcement learning continue to consume a large amount of computing resources (Xiao, 2024). For example, OpenAI Five required 128,000 CPU cores to run for 10 months, while the training of Alpha Star was equivalent to 200 years of game time (Z. Li et al., 2025). The huge demand for computing resources limits reinforcement learning research to institutions with abundant resources, negatively impacting environmental sustainability and widespread accessibility. Researchers, students, and practitioners in resource-constrained economies face significant barriers to entry. The popularization of machine learning has become increasingly urgent (Barbierato & Gatti, 2024). It is predicted that the reinforcement learning market size will grow to $32 trillion by 2037, with a compound annual growth rate (CAGR) of 65% (Sahu et al., 2023). Despite rapid growth, access to related technologies still relies on expensive computing resources. SUMMARY

[0003] An object of the present application is to solve at least the above problems and provide at least the advantages to be described later.

[0004] Another object of the present application is to provide a hyperparameter dynamic optimization method and system based on adaptive meta-reinforcement learning, which provides advanced reinforcement learning strategies for resource-constrained researchers and promotes adaptive hyperparameter optimization as a paradigm shift for more accessible, more robust, and more efficient reinforcement learning systems.

[0005] To achieve these objects and other advantages in accordance with the present application, a hyperparameter dynamic optimization method based on adaptive meta-reinforcement learning is provided, comprising the following steps: S1, initializing hyperparameters using an optimizer based on the tree structure Parsons estimator algorithm to obtain the initial hyperparameter configuration of the agent; S2. An inner loop that enables the agent to perform reinforcement learning training by interacting with the environment based on the initial hyperparameter configuration; S3. Based on the inner loop of reinforcement learning training, an outer loop meta-learning mechanism is executed. In the outer loop meta-learning mechanism, at predetermined time steps, the meta-learner collects and monitors aggregate performance indicators. S4. The meta-learner dynamically adjusts one or more hyperparameters of the agent based on the aggregated performance index; S5. Continue the inner loop of reinforcement learning training using the adjusted hyperparameters.

[0006] Preferably, the optimizer for the tree-structured Parsons estimator algorithm is Optuna.

[0007] Preferably, in step S1, a two-stage hyperparameter initialization process is adopted: In the first phase, Optuna optimization is performed independently for each task and algorithm combination to determine the optimal hyperparameter configuration for each task. In the second stage, cross-task meta-optimization is performed to evaluate the average performance of each hyperparameter configuration across all tasks in order to find hyperparameter configurations that have good generalization ability to task distribution.

[0008] Preferably, the meta-learner is a recurrent neural network based on Long Short-Term Memory (LSTM).

[0009] Preferably, the aggregated performance metrics include at least one of the following: average reward, reward trend, round length, policy loss, value loss, gradient magnitude, exploration metric, and training progress.

[0010] Preferably, the dynamically adjusted hyperparameters include the learning rate, discount factor, and exploration parameters.

[0011] Preferably, the learning rate is adjusted using a multiplicative update rule: , where Δα is the adjustment amount of the meta-learner output; The adjustment of the discount factor and exploration parameters adopts an additive update rule, and the numerical range is pruned after the update. Discount factor: Exploration parameters: .

[0012] Preferably, the training of the meta-learner is driven by a meta-reward signal, which is defined as the difference in the agent's average reward between two consecutive adaptive windows. in, It is the moving average reward over time t. It is an adaptive interval.

[0013] Preferably, the method further includes a stability control mechanism, including at least one of the following: Set upper and lower limits for the hyperparameter adjustment amount of the meta-learner output; The range of hyperparameter values ​​is cropped to ensure that they are within the preset range; Set an upper limit on the cumulative drift of hyperparameters relative to their initial values; A meta-learner warm-up period is set up during the initial N time steps of training, during which fixed hyperparameters are used.

[0014] This invention also provides a hyperparameter dynamic optimization system based on adaptive meta-reinforcement learning, comprising: The hyperparameter initialization module uses an optimizer based on a tree-structured Parsons estimator algorithm to initialize hyperparameters and obtain the initial hyperparameter configuration of the agent. The reinforcement learning training module enables the agent to perform reinforcement learning training in an inner loop based on the initial hyperparameter configuration and interact with the environment. The meta-learning control module, based on the inner loop of reinforcement learning training, introduces an outer loop meta-learning mechanism. The meta-learning control module includes: The performance monitoring unit enables the meta-learner to collect and monitor aggregate performance metrics at predetermined time steps. The dynamic adjustment unit enables the meta-learner to dynamically adjust one or more hyperparameters of the agent based on the aggregated performance index, and provides the adjusted hyperparameters to the reinforcement learning training module for continued training.

[0015] The present invention has at least the following beneficial effects: Meta-reinforcement learning without the need for complex infrastructure. Experiments were run on the free version of Google Colab, with each configuration requiring 8-11 hours of training on a single T4 GPU. This makes hyperparameter optimization more readily available, while reducing cost and environmental impact. The results were validated on three algorithms (PPO, A2C, DQN) and four environments (CartPole, MountainCar, BSuite catch / memory task). Five independent runs showed that the average reward was improved by 6%-18%, the variance was reduced by 25%-30%, the convergence speed was faster, and the memory ability (+28%), exploration ability (+44%), generalization ability (+31%), and noise resistance (+25%) were all improved. Cross-task evaluations show that meta-learners trained in one environment can achieve 65%-80% of their full performance in other environments without retraining. This suggests that transfer adaptation is a general meta-skill, rather than an environment-specific optimization.

[0016] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0017] Figure 1 This is a flowchart of the adaptive meta-reinforcement learning system architecture described in this invention, illustrating the dual-loop structure; Figure 2 For the exploration-utilization of the evolution graph during training, the exploration parameters of DQN(ε) and PPO / A2C(entropy) are scheduled, and AMRL dynamically adjusts these parameters based on observed performance. Figure 3 This is a learning rate sensitivity analysis plot, showing the average reward (solid line) and standard deviation (shaded area) of five independent runs. The optimal performance is approximately 10. -3 nearby; Figure 4 The hyperparameter interactive heatmap displays the validation reward of the combination of learning rate and discount factor, presenting a complex optimization landscape; Figure 5 This is a progress graph of the experiment. Each point in the graph represents an experiment and is coded by reward color. The graph shows that Optuna gradually optimizes to the high reward region over time. Figure 6 The average reward and error band plot (bootstrap confidence interval) during the training process highlights the variability across multiple runs; Figure 7A1 The learning rate adaptive trajectory plots of AMRL with baseline fixed values ​​in CartPole-v1, MountainCar-v0, and BSuite / memory environments are shown. Figure 7A2 The discount factor adaptive trajectory plots for AMRL and baseline fixed values ​​in CartPole-v1, MountainCar-v0, and BSuite / memory environments. Figure 7A3 Adaptive trajectory plots of exploration parameters for AMRL with baseline fixed values ​​in CartPole-v1, MountainCar-v0, and BSuite / memory environments; Figure 7B1 This illustrates different training stages of the learning rate adaptive trajectory plot in the CartPole-v1 environment. Figure 7B2 This illustrates different training stages of the discount factor adaptive trajectory plot in the CartPole-v1 environment. Figure 7B3 A schematic diagram illustrating different training stages for exploring parameter-adaptive trajectory plots in the CartPole-v1 environment; Figure 8The convergence graphs of AMRL in CartPole-v1, MountainCar-v0, memory_len, and catch environments show that AMRL converges faster and has better asymptotic performance than the baseline. Figure 9 The training stability graph of AMRL in the catch environment shows that AMRL achieves a standard deviation reduction of approximately 25% across all algorithms. Figure 10 The image is a violin plot to show the reward distribution for different reward outcomes. In the catch task, the reward distribution of AMRL is more concentrated at high values ​​than the baseline, indicating higher consistency. Figure 11 The BSuite capability comparison chart (radar chart) shows the comparison of AMRL with the baseline in terms of memory, exploration, generalization and noise resistance. Figure 12 The cross-task transfer performance graph shows that the meta-learner can achieve 65%-80% of the full performance of AMRL in the new environment without retraining, indicating that the learned adaptive policy has generalization ability. Figure 13 Aggregated performance plots for each task, with horizontal bar charts showing the average rewards for PPO, A2C, and DQN in each environment, compared to baselines and AMRL. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings, so that those skilled in the art can implement it based on the description.

[0019] This invention proposes an Adaptive Meta-Reinforcement Learning (AMRL) framework. This framework comprises five components: system architecture, environment design, agent design, Optuna-based hyperparameter optimization, and meta-learning adaptation. All experiments were run on a single T4 GPU using the free version of Google Colab to demonstrate its accessibility to resource-constrained researchers.

[0020] <System Architecture> like Figure 1 As shown, the AMRL system employs a dual-loop structure. Unlike standard reinforcement learning, its policy update and hyperparameter tuning occur simultaneously. The inner loop performs standard reinforcement learning: the agent interacts with the environment, performs actions, receives rewards, and updates the policy. The outer loop is AMRL's core innovation—a meta-learning mechanism that monitors aggregated performance metrics every 10,000 time steps and dynamically adjusts hyperparameters based on observed learning dynamics.

[0021] AMRL's architecture differs from previous methods that optimized hyperparameters offline during training intervals or used population-based parallelization during training. AMRL supports adaptive and online resource efficiency during training, making it suitable for resource-constrained configurations. It combines a meta-learning mechanism with Optuna's Time-Based Optimization (TPE) initialization, obtaining a good initial configuration through TPE and continuously optimizing it during training. This collaborative architecture enables the system to adapt to constantly changing learning conditions—for example, promoting exploration when training stagnates, reducing the learning rate when the policy is stable, and adjusting the time discount factor according to the reward structure—without human intervention or additional computational agents. The dual-loop structure achieves adaptive hyperparameter optimization through a single agent, eliminating the 10-30x resource multiplication cost of population-based methods.

[0022] <Environment Settings> AMRL is evaluated in four different reinforcement learning environments to test different capabilities of the agent: basic control, sparse reward exploration, reactive decision-making, and temporal memory. Environment details are as follows: CartPole-v1: Testing stability control capabilities (the agent receives a dense reward of +1 for each time step it keeps the pole upright). MountainCar-v0: Requires the agent to accumulate momentum through a policy (the agent receives a reward of -1 at each time step, representing the number of steps remaining to the target; the higher the reward value, the more effective the policy). bsuite / catch: Assess sensorimotor coordination (0 points for missing the target, +1 point for hitting the target); bsuite / memory_len: Assess time credit allocation ability (rewards are sparse, +1 point for correct task completion).

[0023] Table 1 summarizes the key features of each environment. All environments employ standard preprocessing (normalization, reward pruning, and frame stacking where necessary) and correctly handle terminal states. The diversity of environments ensures that the evaluation covers different reward structures, temporal dependencies, and action spaces, avoiding overfitting to the characteristics of a single environment. The diverse suite of environments validates AMRL's generalization ability across basic reinforcement learning challenges, from dense reward control to sparse reward exploration and memory-dependent reasoning.

[0024] Table 1. Environmental characteristics and evaluation diversity. "Maximum number of steps" indicates the training segment length limit; "Reward structure" indicates the reward type and density; MountainCar's negative reward indicates the number of steps remaining to complete the goal (the smaller the absolute value, the better the performance).

[0025] <Agent Architecture> The algorithm-agnostic design of AMRL enables it to be evaluated in multiple reinforcement learning paradigms to demonstrate its broad applicability.

[0026] This invention employs three state-of-the-art algorithms: PPO and A2C (online policy gradient methods) and DQN (offline value-based method). All algorithms use the same neural network architecture based on Stable-Baselines3: the input layer receives normalized state observations, two fully connected hidden layers (64 neurons each, ReLU activation function), and algorithm-specific output heads. PPO and A2C use a shared feature extractor with separate policy and value heads, while DQN directly maps states to Q-values. The network uses orthogonal initialization (gain √2), Adam optimizer, and gradient clipping (max norm 0.5). Figure 2 shows the evolution of the exploration-exploitation trade-off, with DQN's ε decaying from 1.0 to 0.1 and the entropy coefficient of PPO / A2C decaying relatively slowly. The AMRL meta-learner can actively adjust these schedules based on performance feedback. The shallow architecture (two layers of 64 neurons) is designed to improve computational efficiency and adapt to the memory limitations of Google Colab while providing sufficient representational power in the test environment. The algorithm-agnostic approach ensures that the improvements in AMRL reflect the core principles of meta-learning rather than optimizations specific to a particular algorithm.

[0027] <Hyperparameter Optimization> AMRL uses Optuna - a hyperparameter optimization library that uses the Tree-structured Parzen Estimator (TPE) algorithm. TPE models the hyperparameter distribution using two density functions: p(x | y < y*) (corresponding to well-performing configurations) and p(x | y ≥ y*) (corresponding to poorly-performing configurations), where y* is the performance threshold. The acquisition function balances exploration and exploitation: (1) where the Tree-structured Parzen Estimator models the probability distribution, and the search space includes the learning rate (1e-5 to 1e-2, logarithmic scale), discount factor γ (0.9 - 0.999), batch size (16 - 256), network architecture (32 - 512 hidden units), and algorithm-specific parameters (including PPO's clip ε (0.1 - 0.3) and DQN's exploration rate (0.01 - 1.0)).

[0028] The objective function maximizes the average cumulative reward over 10 validation training episodes, following the standard reinforcement learning evaluation protocol: (2) Using Optuna's Median Pruner, poorly performing trials are eliminated every 1000 steps, saving 40%-60% of computation. Figures 3-5 illustrate the learning landscape during the optimization process: Figure 3's hyperparameter sensitivity landscape shows the optimal learning rate is approximately 1e-3; Figure 4's co-localization landscape reveals a complex non-convex dependency between the learning rate and the discount factor; Figure 5 shows that Optuna continuously optimizes the learning settings after 100 trials to reach the high-reward region. To achieve multi-task generalization, a two-stage approach is adopted: Stage 1 involves conducting Optuna studies independently for each environment-algorithm pair (50-100 trials each) to determine the task-specific optimal solution; Stage 2 performs cross-task meta-optimization while evaluating the performance of each configuration across all K tasks, extending the multi-task learning principle to hyperparameter optimization through curriculum-aware evaluation. (3) This encourages the exploration of hyperparameters that are beneficial to the entire task distribution. Through TPE-based initialization, the number of trials is reduced by 40%-60%, while providing an effective initial configuration, enabling us to perform meta-learning efficiently in low-resource scenarios.

[0029] <Dynamic Meta-Learning Adaptive Mechanism> The meta-learner is implemented as a recurrent neural network, maintaining the hidden state that encodes historical performance trends, and performing periodic checkpoints (every T). adapt = 10000 time steps) Output hyperparameter tuning. Meta-learner observation 8-dimensional performance metric vector: (1) average reward, (2) reward trend, (3) training segment length, (4) policy loss, (5) value loss, (6) gradient magnitude, (7) exploration metric, (8) training progress. Table 2 lists the meta-learner network architecture in detail.

[0030] Table 2. Meta-learner network architecture and hyperparameter adaptive specifications. All output heads generate bounded adjustments to ensure training stability.

[0031] Component Specification Output dimension Activation function Use Input layer 8-dimensional normalized performance indicator 8 — Aggregated training state representation LSTM layer 32 hidden units 32 Tanh (internal) Sequence processing of historical performance Fully connected layer 32→16 neurons 16 ReLU Feature transformation and dimensionality reduction Learning rate head 16→1 mapping 1 Tanh Output Δα ∈ [−0.3, +0.3] for multiplicative adjustment Discount factor head 16→1 mapping 1 Tanh Output Δγ ∈ [−0.02, +0.02] for additive adjustment Exploration head 16→1 mapping 1 Tanh Output Δβ ∈ [−0.05, +0.05] for ε of DQN or entropy of PPO / A2C 1) Hyperparameter selection Hyperparameter selection involves tuning the learning rate (α), discount factor (γ), and exploration parameter (β) because they have the most significant and direct impact on training dynamics while maintaining meta-learner stability. The learning rate is the most direct controller of the gradient step size, regulating convergence speed and stability. The discount factor determines the depth of temporal credit allocation, which is particularly important in delayed reward environments. Exploration parameters (ε in DQN, entropy coefficients in PPO / A2C) control the tradeoff between exploration and exploitation, crucial for finding the optimal policy.

[0032] Other hyperparameters (such as the pruning ε of PPO, GAE λ, and value function coefficients) affect training, but their effects are usually algorithm-specific and more subtle. Including all hyperparameters increases the output dimension of the meta-learner from 3 to 6-8, which increases instability and overfitting risk when the training sample size is small. This invention focuses on general parameters applicable to each algorithm while maintaining dimensionality tractability for the meta-learning problem. The scaling methods for different types of hyperparameters are as follows: Learning rate: ; Discount factor: ; Exploration parameters: .

[0033] The learning rate follows the concept of adaptive optimization, growing or decaying exponentially over time, while the discount factor remains within a theoretically reasonable range. The exploration parameters adapt to the algorithm's characteristics, namely the ε-greedy strategy of DQN and the entropy regularization of PPO / A2C.

[0034] 2) Meta-learner training protocol The meta-learner is trained through reinforcement learning, treating hyperparameter tuning as an action and performance improvement as a reward. Training requires no additional steps, only 100,000 time steps of the agent's operation—the meta-learner observes performance metrics every 10,000 time steps (each run offering 10 adaptive opportunities) and learns from the meta-reward signal. (4) It is the moving average reward over time t. It uses an adaptive margin. The meta-learner uses PPO to optimize the expected meta-reward: (5) in Indicates the parameters of the meta-learner. It refers to the hyperparameter adjustment trajectory and the resulting performance changes.

[0035] 3) Meta-reward design and stability mechanism By measuring the performance improvement between consecutive adaptive windows, we can directly provide feedback on whether hyperparameter adjustments help or hinder learning, thereby establishing a causal relationship between meta-learner actions and training results (as shown in Equation 5).

[0036] Noise environment treatment: (Based on 1000 time steps, approximately 5-10 training segments, depending on the environment) Smooth random reward fluctuations. Without averaging, the variance of random training segments may produce misleading meta-rewards.

[0037] Alternative indicators: Dominance paradigm indicators (such as...) can be used. (Where the baseline is the average performance across all time steps). However, the formulation of this invention focuses on local improvements between consecutive windows, rather than global performance, enabling the meta-learner to respond to recent training dynamics. Preliminary experiments show that the dominance paradigm metric produces similar final performance, but with slower initial adaptation.

[0038] To prevent harmful early adjustments, this invention also implements four stability mechanisms: Bounded adjustment rate: The maximum change for each adaptive adjustment is ±30% of the learning rate, ±2% of the discount factor, and ±5% of the exploration parameters. These constraints prevent catastrophic updates that could disrupt training. Effective range pruning: The discount factor and exploration parameter scaling formula enforce hyperparameter boundaries (γ ∈ [0.90,0.99], β ∈ [0.0, 1.0]) to avoid invalid configurations; Maximum cumulative drift: The deviation of the learning rate from the initial value does not exceed 10 times; the cumulative change of the discount factor is limited to ±0.05. This prevents deviation from Optuna's carefully selected initial value; Meta-learner warm-up: Fixed hyperparameters are used for the first 20,000 time steps to allow the base agent to establish initial performance before meta-learning begins. This avoids destructive adjustments during the most unstable phase of training.

[0039] Observed stability: Analysis of 60 training runs (5 random seeds × 12 configurations) revealed no catastrophic failures. In some cases, early decisions by the meta-learner may temporarily degrade performance (a 5%-10% decrease within 1-2 adaptive windows, which always recovers), but no catastrophic failures were observed. The bounded adjustment space is well-calibrated: 98% of the meta-learner's learning rate changes were <20%, and 98% of the discount factor changes were <1%, indicating that the meta-learner has learned a more conservative and stable policy. Each adaptive step of the meta-learner inference takes only 15-25 milliseconds (0.8%-1.2% of total training time), and meta-learner updates are based solely on accumulated empirical data, occurring only 10 times per run, making the overhead negligible.

[0040] LSTM-based meta-learners achieve continuous adaptation through bounded adjustment, drift constraints, and a warm-up period, while providing strong stability guarantees to prevent harmful early decisions and enabling responsive optimization throughout the training process.

[0041] <Training and Assessment> A three-phase protocol was adopted to progressively verify the capabilities of AMRL within the computational constraints of Google Colab.

[0042] Phase 1: Hyperparameter search using Optuna Optuna is initialized using a TPE sampler, with 50-100 trials run per algorithm-environment pair. The agent is trained for 5000 time steps per trial, with a median pruning process terminating poorly performing trials after 1000-2000 time steps. This phase on a Tesla T4 GPU takes 4-6 hours per algorithm-environment pair to identify potentially valuable hyperparameter regions. The optimal configuration is used for initialization of the baseline (fixed) and AMRL (adaptive) implementations.

[0043] Phase 2: Adaptive Meta-Learning Training The agent is trained for 100,000 time steps, with meta-learning adaptation performed every 10,000 time steps. The meta-learner checks aggregate performance metrics and dynamically updates the learning rate, discount factor, and exploration parameters. The baseline agent uses the hyperparameters from Phase 1 throughout the training process. Each configuration is run five times independently using different random seeds (1, 42, 123, 456, 789), each run taking 1.5–2 hours. Due to Colab's single GPU and session limitations, execution must be sequential. Table 3 shows the complete training configuration.

[0044] Table 3. Training configuration and reproducibility specifications. All experiments used the same computational resources and evaluation process to ensure fair comparison. Parameter Specification Explanation Hardware NVIDIA Tesla T4 GPU Google Colab free version, 16 GB video memory Software Python 3.8, PyTorch 1.12, SB3 1.6, Optuna 3.0 Random seed [1, 42, 123, 456, 789] Five independent runs per configuration Configuration Total of 12 3 algorithms × 4 environments Training time steps 100,000 Uniform across all experiments Batch size PPO: 64, A2C: 32, DQN: 32 Optuna optimization, same baseline / AMRL Meta-adaptive frequency Every 10,000 time steps 10 opportunities per run Evaluation training episodes 20 training episodes at the end of training Tracking every 5,000 time steps Meta-learner warm-up First 20,000 time steps Fixed hyperparameters during warm-up Statistical test Paired t-test, Bonferroni correction adj = 0.0042 (12 comparisons) ​

[0045] Phase 3: Cross-task evaluation and generalization testing The meta-learner parameters for each source environment are applied to all other target environments, and the meta-learner weights are frozen (no additional training). Each evaluation runs for 50,000 time steps, and the transfer performance is calculated as a percentage of the full AMRL performance. (6) AMRL performance is based on 20 training segments ( The evaluation is conducted using six indicators: a. Average cumulative reward (7) b. Learning efficiency: Quantitative convergence speed, i.e., the time steps required to reach 90% of the final performance. (8) c. Training stability: Consistency is measured by the variance of the reward. (9) d. Performance Improvement: Normalized Comparison (10) e: Hyperparameter trajectory: Analysis of the training process.

[0046] f: BSuite reliability score: measures memory, exploration, generalization and noise resistance, normalized score 0-1.

[0047] <Experimental Results> For technical details regarding training setup and reproducibility (crucial for any experimental design), please refer to Table 3 above for experimental conditions and procedures. Experimental results are divided into six key analyses: overall performance score; learning dynamics and convergence; training stability; cognitive skills; cross-task generalization; and comparison with algorithm-related methods.

[0048] In this invention, a static hyperparameter set is used as the baseline, all determined through Optuna's TPE search. Therefore, the hyperparameters do not change during the learning process. AMRL uses the same initial values ​​(the hyperparameters may change every 10,000 time steps). This setup maintains fairness across four dimensions while isolating the effects of adaptive optimization. Same initialization: Both methods use the same Optuna optimization hyperparameters (e.g., CartPole-v1 PPO: α = 3.2 × 10⁻⁶). -4 (γ=0.98, β=0.01). The main difference lies in whether continuous adaptive optimization leads to better performance than stable offline hyperparameter optimization. Same training duration: Each repeated run executes the same 100,000 timesteps setting, without extending the training time; AMRL only increases the overhead by 0.3% (15-25 milliseconds for each adaptive run). Same resources: All experiments used the same Tesla T4 GPU and the same software stack (Table 3). Different seeds were used alternately for each run to minimize system variability; Optimal available baseline: Choose a baseline with Optuna’s TPE (50-100 trials, 4-6 hours) to match or exceed human expert tuning, representing a strong static configuration rather than a weak baseline.

[0049] The design aims to verify whether continuous adaptation outperforms the best static configuration obtained through thorough offline optimization. The performance difference is attributed to adaptive optimization rather than initialization quality, extended training, or increased resources.

[0050] 1. Statistical verification: Each configuration supports paired t-tests (α=0.05) and Bonferroni correction (α for 12 comparisons) in five independent runs. adj=0.0042). Effect sizes were calculated using Cohen's d, with 95% confidence intervals determined by a bias-corrected bootstrap method (10,000 resamplings). Table 4 presents the combined results, showing that AMRL achieved statistically significant improvements (p≤0.005) in all 12 configurations, with effect sizes ranging from d=0.87 (large) to d=4.05 (large).

[0051] Table 4. Comprehensive performance metrics and statistical validation. The average reward for the last 100 training segments is based on the average of the 5 seeds. For MountainCar, negative reward represents the time steps to complete the objective (smaller values ​​are better; maximum = 200 steps).

[0052] Note: Paired t-test combined with Bonferroni correction (α) adj =0.0042). Cohen d: 0.2 (small), 0.5 (medium), 0.8 (large), >1.2 (extremely large).

[0053] Figure 6 illustrates the average reward and error band (bootstrapping confidence interval) during training, highlighting the variability across multiple runs. All three AMRL implementations outperform the baseline in all environments. In the CartPole-v1 analysis, AMRL significantly outperforms the baseline across all algorithms, with PPO showing the highest absolute performance. In the sparse reward environment MountainCar-v0, all algorithms show significant improvements over the baseline. In the memory_len task, performance improvements range from 15% to 18%, indicating that AMRL effectively implements time-based credit allocation. In the catch task, all algorithms achieve approximately a 12% improvement, demonstrating the flexibility of AMRL under different cognitive requirements.

[0054] 2. Learning dynamics and convergence: AMRL achieves accelerated learning and asymptotic performance improvements across multiple tasks. In CartPole-v1, AMRL reaches the 400 reward milestone at the 50th training segment (baseline at the 75th segment), converges faster (AMRL at 200 training segments, baseline agent at 300), and ultimately performs better (AMRL 540-550, baseline 490-500). Furthermore, AMRL consistently performs well between the 200th and 500th training segments—even slightly improving upon the average of that segment, while the baseline slightly declines (averaging 348 after 400 training segments). Similar training segment improvements were observed in other tasks: (MountainCar-v0) AMRL achieves a performance breakthrough before the 150th training segment (baseline at 200); in the memory_len task, the algorithm fully converges at the 200th training segment (baseline at 300); and in the catch task, AMRL consistently outperforms the baseline throughout training. As shown in Figure 8, AMRL's self-vanishing hyperparameter adaptation improves the learning speed for all tasks (from simple cognitive tasks to complex tasks).

[0055] 2.1 Hyperparameter Trajectory Analysis: Figures 7A1–7A3 illustrate the hyperparameter evolution of AMRL in three environments, comparing the adaptive trajectory with a fixed baseline configuration. The trajectory shows environment-specific patterns, reflecting learned adaptations rather than random variations. In CartPole-v1 (dense reward), the learning rate starts from 3 × 10⁻⁶. -4 It steadily decreased to 2.3 × 10⁻⁶. -4 The baseline remains unchanged. The discount factor intentionally fluctuates around γ = 0.99 ± 0.005 to achieve time credit adjustment. The exploration parameter (ε) decreases rapidly in the early stages to ε ≈ 0.15, then maintains residual exploration, while the baseline converges to near zero, posing a risk of premature convergence. In MountainCar-v0 (sparse reward), the learning rate remains significantly high over a longer period, reaching 6 × 10⁻⁶ around the 150th training segment. -4 The peak value then increases further during plateau periods (the 200th and 300th training segments), exceeding the value fixed at 5×10. -4The baseline was maintained. The discount factor also remained high over time, starting from γ≈0.95 and eventually stabilizing at γ=0.99, consistent with the momentum policy. The exploration degree of action selection was significantly higher than the baseline (ε>0.25 vs. baseline ε=0.15) to overcome the effect of sparse rewards. For example, in the memory_len task (time credit), the policy was conservatively adaptive, with the learning rate starting from 2×10. -4 Reduced to 1.3×10 -4 To maintain accurate updates to the time-based assessment, the discount factor is significantly increased to γ≈0.99 to accommodate long-term dependencies. The entropy coefficient decays rapidly in the early stages of training while periodically increasing to maintain a certain level of exploration while ensuring the credit allocation process.

[0056] CartPole-v1 training analysis is divided into three phases: early exploration (0-150 training segments), optimization (150-350 training segments), and convergence (350-500 training segments). The early exploration phase utilizes a high learning rate and steep exploration decay to help the meta-learner quickly reach a certain level of proficiency. The optimization phase corresponds to a gradual decrease in the learning rate and stable adjustment of the discount factor. The convergence phase involves adjusting hyperparameter configurations that gradually stabilize and a gentle but continuous exploration. Figures 7B1-7B3 illustrate the hyperparameter scheduling according to these phases. The non-random initiation of adjustments is statistically validated; the cross-correlation σ_Δα between adjacent hyperparameter adjustments in all environments is close to r=0.7, indicating coherent policy adaptation in the continuous phases. The decrease in parameter adjustment variability over time is also statistically characterized; σ_Δα decreases from 0.08 in the early stages to 0.03 in the later stages, indicating increasing confidence in the learned scheduling. From CartPole to other environments, the tuning trajectory patterns of progressive meta-learners are similar in quality, scaling only according to task size. These trajectory patterns help explain key components of the performance improvements observed in AMRL: exploration gain contributes a 44% performance advantage (partly based on appropriate scheduling of tasks and policies that maintain high ε and entropy during critical attention periods), memory improvement contributes 28% (partly based on discounting factors that promote temporal credit in task training), and finally, artificial expansion variability is reduced by 25%–30% through learning rate scheduling that prevents premature convergence and avoids persistent instability.

[0057] The learning curves for different training segments show the convergence speed of AMRL and the baseline model. As shown in Figure 8, AMRL achieves superior performance earlier, converges faster, and has higher asymptotic performance.

[0058] 3. Training stability analysis Stability metrics provide important information about learning reliability and are independent of average performance. As shown in Figure 9, AMRL achieves reliable learning across all environments and algorithms, with a 25%-30% reduction in variance.

[0059] In the CartPole-v1 environment, the standard deviation of all methods decreased by 25%-30%, with Proximal Policy Optimization (PPO) showing the most significant stability improvement (23.75→17.81). In MountainCar-v0, the variance decreased by an average of 25%, with DQN showing the smallest relative improvement (14.00→10.50), possibly due to DQN's offline learning and experience replay characteristics, which make it inherently more stable. In Memory_len, the variance improvement remained stable at 25%-30%, but the relative value was small due to the reward scaling of 0-1. In the catch task, the overall variance of all methods decreased by approximately 25%. This stability effect was more pronounced in sparse reward spaces, and the high baseline variance indicated that the method was sensitive to the exploration trajectory. Under AMRL, the stability metrics of all methods showed comparable convergence (standard deviation range 15.4-19.5), while the baseline standard deviation ranged from 20.5-23.8, indicating that systematically tuning dynamic hyperparameters can effectively normalize the algorithm-related unique variance.

[0060] This stability is crucial for production machine learning systems, where consistent performance is just as important as high average performance. Reduced variance decreases the number of training runs required and increases the reliability of post-deployment performance.

[0061] 3.1 Ablation Study: Component Contribution Analysis To demonstrate the contributions of each AMRL component, ablation tests were performed on CartPole-v1 (PPO), removing each AMRL component one by one. Table 5 shows the results of five independent test runs. First, removing the TPE initialization (replacing it with random hyperparameter initialization) resulted in an 8.2% decrease in average reward and a 35% increase in variance, indicating the important role of TPE in providing a good initial configuration. Second, replacing the LSTM meta-learner with a simple multilayer perceptron (MLP, two layers, 32 neurons per layer, no recurrent layers) resulted in a 6.5% decrease in performance and a 22% increase in variance, indicating that historical performance should be processed sequentially rather than individually when evaluating the quality of adaptive learning. Third, training the meta-learner only in a single task (CartPole only), rather than a multi-task approach, resulted in a 31% decrease in generalization ability when applied to MountainCar, indicating that AMRL's multi-task training learns a broadly applicable adaptive method rather than a task-specific optimization pattern. Overall, the complete framework achieved an average reward of 503.5 ± 17.8, significantly lower than this value for all ablation AMRL conditions (p < 0.001, paired t-test). In fact, the average reward under any condition could be lower than the baseline (475.0 ± 23.8), with the lowest reward (438.2 ± 32.1) under the no-TPE condition. In summary, AMRL performance relies on the collaborative work of three components: TPE provides efficient initialization, LSTM enables time-adaptive learning, and multi-task training achieves generalization capability; the absence of any one of these components would prevent this performance from being achieved.

[0062] Table 5: Results of CartPole-v1 (PPO) Ablation Study Configuration Mean reward Standard deviation Comparison to full AMRL Comparison to baseline Full AMRL 503.5 17.8 — +6.0% No TPE (random initialization) 438.2 32.1 -13.0% -7.7% MLP instead of LSTM 471.8 21.7 -6.3% -0.7% Single-task training 485.3 19.5 -3.6% +2.2% Baseline (fixed hyperparameters) 475.0 23.8 -5.7% — Note: Except for "single-task training" (which mainly affects cross-task generalization rather than intra-task performance), all differences were significant at the p<0.001 level.

[0063] 4. Characteristics of return distribution The violin plots of the last 100 training segments reveal rich distributional features beyond simple statistics, showcasing the distribution and dispersion of performance. Figure 10 illustrates the reward distribution for different reward outcomes in the catch task, comparing baseline and AMRL performance. The violin plots show that AMRL produces a distribution more concentrated on high reward values, indicating more consistent high performance than the baseline method. Furthermore, AMRL's distribution is more compact and symmetrical around the mean, especially in high reward outcomes. This suggests that the framework produces more reliable results in repeated training segments, and the variability of high reward outcomes between training segments is lower than that of the baseline method. AMRL's narrower distribution of results near the peak performance level indicates that the framework still produces reliable performance results despite task variability.

[0064] 5. Cognitive Ability Assessment BSuite evaluates four cognitive dimensions: memory, exploration, generalization, and robustness. As shown in Figure 11, AMRL outperforms the baseline in all dimensions, with the radar plot exhibiting a more balanced circular distribution. Memory capability improved from ~0.70 to ~0.90 (≈28%); exploration capability improved from ~0.45 to ~0.65 (≈44%); generalization capability improved from ~0.65 to ~0.85 (≈31%); and robustness improved from ~0.60 to ~0.75 (≈25%). These improvements stem from adaptive discounting, learning rate control, and entropy / ε calibration, enhancing temporal credit allocation, exploration-exploitation balance, cross-task transfer, and robustness to random observations. The uniform improvement across all dimensions (Figure 11) indicates that the meta-learning hyperparameters do not sacrifice other capabilities during generalization.

[0065] 6. Analysis of cross-task generalization ability AMRL tested the performance of meta-learners trained in one environment to transfer to other environments without retraining. Meta-learner weights were frozen, and the program ran for 50,000 time steps. Transfer performance was evaluated as a percentage of the overall AMRL performance. Results showed that meta-learners trained in CartPole achieved 65%–75% performance in MountainCar, despite significant differences in dynamic characteristics (continuous state space, sparse reward, and momentum). Meta-learners trained in memory_len also transferred well to the catch task, achieving approximately 70%–80% performance. Transfer performance from dense reward tasks (CartPole and catch) to sparse reward exploration tasks (such as MountainCar) was poor, averaging only 40%–50%. The overall range of cross-task transfer was 65%–80%, indicating good generalization and significant diversity in adaptive spatial structures despite apparent asymmetric transfer between tasks. Figure 12 shows some representative transfer results: the transfer performance from CartPole to MountainCar is approximately 80%, from CartPole to memory_len is approximately 75%, and from MountainCar to catch is approximately 65%. These results demonstrate that the hyperparameter adaptive policy learned by AMRL can generalize to scenarios beyond a single task, revealing interesting diversity in adaptive spatial structures and providing possibilities for future interesting and predictable multi-task meta-learning designs.

[0066] 7. Cross-task aggregation performance analysis By summarizing the performance of different algorithms in each environment, the consistent advantages of AMRL can be evaluated at a higher level, and it can be verified that the improvements are not driven by specific algorithm-environment interactions. Figure 13 shows the actual runtime of each method, highlighting AMRL's more efficient use of computational resources.

[0067] <Discussion> Even under highly resource-constrained conditions (i.e., a single T4 GPU configured on Google Colab, training for 8-11 hours), AMRL still demonstrates significant performance improvements. In summary, average reward increases by 6%-18%, variance decreases by 25%-30%, convergence speed is accelerated, and cognitive abilities (memory +28%, exploration +44%, generalization +31%, noise robustness +25%) are improved. Furthermore, cross-task transfer across different environments (e.g., out-of-distribution environments) achieves 65%-80% of the overall performance, initially demonstrating the effectiveness of a general adaptive strategy, rather than task-specific parameter tuning. In addition to task-specific improvements, the AMRL framework is algorithm-independent and has been tested and validated in PPO, A2C, and DQN—evidence that performance improvements stem from the core principles of meta-learning, rather than task-specific algorithms. Stability metrics converge across different methods, providing initial support for AMRL's production deployment and potential reduction in training runs. BSuite analysis reveals several economic advantages in terms of performance: a 44% exploration gain compared to the baseline through adaptive ε and entropy tuning; a 28% memory gain compared to the baseline through optimized discount factors and learning rates in specific environment configurations; a 31% generalization gain compared to the baseline through adaptive hyperparameters; and finally, a 25% noise robustness gain through learning rate control during training. Overall, these results demonstrate that AMRL possesses robust and transferable meta-learning capabilities.

[0068] Table 6 systematically compares AMRL with related methods across six key dimensions. AMRL integrates three components that other methods have not fully implemented: online adaptation within training segments (adjusting hyperparameters every 10,000 time steps during training); single-agent efficiency (significantly lower than the 10-30 times overhead of population-based methods); and multi-task generalization (learned adaptive policies can be transferred across environments). The combination of these three components is AMRL's main contribution.

[0069] Table 6 Comparison of AMRL with other methods Method Adaptive type Online / offline Resource demand Multi-task Algorithm-agnostic In-training episode adaptation MAML [1] Task-level (initial parameters) Offline High (meta-training) Yes Yes No Meta-SGD [2] Task-level (learning rate) Offline High (meta-training) Yes Yes No Bayesian optimization [3] Inter-run Offline Medium (sequential trials) No Yes No PBT [5] During training Online Very high (10-30 times the number of agents) No Yes Yes Optuna / TPE

[14] Inter-run Offline Medium (50-100 trials) No Yes No AMRL (this study) In-training episode Online Low (single agent) Yes Yes Yes AMRL achieves real-time adaptation by updating hyperparameters every 10,000 time steps, responding to constantly changing learning dynamics. It implements complex meta-learning through a single agent, avoiding population-based parallelization and making it suitable for resource-constrained scenarios. The learned adaptive policies exhibit multi-task generalization capabilities, can be transferred to new environments, and achieve 65%-80% of the full performance. This method runs on a single T4 GPU on the free version of Google Colab, demonstrating resource accessibility. AMRL combines efficient initialization based on TPE (4-6 hours), continuous online adaptation via LSTM meta-learners, and cross-task learning to surpass task-specific tuning. This integration generates synergistic advantages, with multi-task training facilitating generalizable optimization principles within a practical single-agent framework.

[0070] Based on the same inventive concept, this invention also provides a hyperparameter dynamic optimization system based on adaptive meta-reinforcement learning, comprising: The hyperparameter initialization module uses an optimizer based on a tree-structured Parsons estimator algorithm to initialize hyperparameters and obtain the initial hyperparameter configuration of the agent. The reinforcement learning training module enables the agent to perform reinforcement learning training in an inner loop based on the initial hyperparameter configuration and interact with the environment. The meta-learning control module, based on the inner loop of reinforcement learning training, introduces an outer loop meta-learning mechanism. The meta-learning control module includes: The performance monitoring unit enables the meta-learner to collect and monitor aggregate performance metrics at predetermined time steps. The dynamic adjustment unit enables the meta-learner to dynamically adjust one or more hyperparameters of the agent based on the aggregated performance index, and provides the adjusted hyperparameters to the reinforcement learning training module for continued training.

[0071] All relevant content of each step involved in the aforementioned embodiments of the hyperparameter dynamic optimization method based on adaptive meta-reinforcement learning can be referenced to the functional description of the corresponding functional module of the hyperparameter dynamic optimization system based on adaptive meta-reinforcement learning in the embodiments of this application, and will not be repeated here.

[0072] The module division in this embodiment is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of this invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0073] The present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the aforementioned hyperparameter dynamic optimization method based on adaptive meta-reinforcement learning. This electronic device can be any terminal device including mobile phones, laptops, desktop computers, tablets, PDAs (Personal Digital Assistants), POS (Point of Sales) terminals, in-vehicle computers, etc.

[0074] The present invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described hyperparameter dynamic optimization method based on adaptive meta-reinforcement learning.

[0075] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, dedicated CPUs, dedicated memory, dedicated components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present invention, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, portable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0076] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A hyperparameter dynamic optimization method based on adaptive meta-reinforcement learning, characterized in that, Includes the following steps: S1. Use an optimizer based on a tree-structured Parsons estimator algorithm to initialize hyperparameters and obtain the initial hyperparameter configuration of the agent. S2. An inner loop that enables the agent to perform reinforcement learning training by interacting with the environment based on the initial hyperparameter configuration; S3. Based on the inner loop of reinforcement learning training, an outer loop meta-learning mechanism is executed. In the outer loop meta-learning mechanism, at predetermined time steps, the meta-learner collects and monitors aggregate performance indicators. S4. The meta-learner dynamically adjusts one or more hyperparameters of the agent based on the aggregated performance index; S5. Continue the inner loop of reinforcement learning training using the adjusted hyperparameters.

2. The hyperparameter dynamic optimization method based on adaptive meta-reinforcement learning as described in claim 1, characterized in that, The optimizer for the tree-structured Parsons estimator algorithm is Optuna.

3. The hyperparameter dynamic optimization method based on adaptive meta-reinforcement learning as described in claim 1, characterized in that, In step S1, a two-stage hyperparameter initialization process is adopted: In the first phase, Optuna optimization is performed independently for each task and algorithm combination to determine the optimal hyperparameter configuration for each task. In the second stage, cross-task meta-optimization is performed to evaluate the average performance of each hyperparameter configuration across all tasks in order to find hyperparameter configurations that have good generalization ability to task distribution.

4. The hyperparameter dynamic optimization method based on adaptive meta-reinforcement learning as described in claim 1, characterized in that, The meta-learner is a recurrent neural network based on Long Short-Term Memory (LSTM).

5. The hyperparameter dynamic optimization method based on adaptive meta-reinforcement learning as described in claim 1, characterized in that, The aggregated performance metrics include at least one of the following: average reward, reward trend, round length, policy loss, value loss, gradient magnitude, exploration metric, and training progress.

6. The hyperparameter dynamic optimization method based on adaptive meta-reinforcement learning as described in claim 1, characterized in that, The dynamically adjusted hyperparameters include the learning rate, discount factor, and exploration parameters.

7. The hyperparameter dynamic optimization method based on adaptive meta-reinforcement learning as described in claim 6, characterized in that, The learning rate is adjusted using a multiplicative update rule: , where Δα is the adjustment amount of the meta-learner output; The adjustment of the discount factor and exploration parameters adopts an additive update rule, and the numerical range is pruned after the update. Discount factor: Exploration parameters: .

8. The hyperparameter dynamic optimization method based on adaptive meta-reinforcement learning as described in claim 1, characterized in that, The training of the meta-learner is driven by a meta-reward signal, which is defined as the difference in the agent's average reward between two consecutive adaptive windows. in, It is the moving average reward over time t. It is an adaptive interval.

9. The hyperparameter dynamic optimization method based on adaptive meta-reinforcement learning as described in claim 1, characterized in that, The method also includes a stability control mechanism, including at least one of the following: Set upper and lower limits for the hyperparameter adjustment amount of the meta-learner output; The range of hyperparameter values ​​is cropped to ensure that they are within the preset range; Set an upper limit on the cumulative drift of hyperparameters relative to their initial values; A meta-learner warm-up period is set up during the initial N time steps of training, during which fixed hyperparameters are used.

10. A hyperparameter dynamic optimization system based on adaptive meta-reinforcement learning, characterized in that, include: The hyperparameter initialization module uses an optimizer based on a tree-structured Parsons estimator algorithm to initialize hyperparameters and obtain the initial hyperparameter configuration of the agent. The reinforcement learning training module enables the agent to perform reinforcement learning training in an inner loop based on the initial hyperparameter configuration and interact with the environment. The meta-learning control module, based on the inner loop of reinforcement learning training, introduces an outer loop meta-learning mechanism. The meta-learning control module includes: The performance monitoring unit enables the meta-learner to collect and monitor aggregate performance metrics at predetermined time steps. The dynamic adjustment unit enables the meta-learner to dynamically adjust one or more hyperparameters of the agent based on the aggregated performance index, and provides the adjusted hyperparameters to the reinforcement learning training module for continued training.

Images