A continuous control method and device based on adaptive deep reinforcement learning

Abstract

This invention discloses a continuous control method and device based on adaptive deep reinforcement learning, belonging to the field of intelligent control and robotics. The invention proposes a dynamic policy optimization framework integrating five innovative technologies: a dual-modal target network that fuses the outputs of the primary and secondary target networks through loss-driven dynamic weight allocation to suppress target value oscillations; an adaptive soft update algorithm that dynamically adjusts the target network update rate based on the reward change rate; a three-level dynamic experience storage architecture that combines value bias-driven priority sampling to improve the utilization rate of key samples; multi-scale feature calibration technology that performs differentiated normalization based on the differences in feature distribution at each level of the network; and an environment-aware exploration policy generator that drives adaptive exploration noise by real-time sensing of system dynamic parameters. This invention achieves an average policy performance improvement of 35%, a convergence speed improvement of over 40%, a control accuracy improvement of 25%, and a sample efficiency improvement of 2.1 times in 11 complex physical control tasks.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control and robotics technology, specifically relating to a continuous control method and device based on adaptive deep reinforcement learning. Background Technology

[0002] In the field of intelligent control, continuous control of complex physical systems faces challenges such as high-dimensional state spaces, dynamic environmental disturbances, and high control accuracy requirements. Although traditional deep reinforcement learning methods such as DQN have made breakthroughs in discrete action spaces, they have three core defects in continuous control scenarios: (1) the curse of dimensionality and loss of control accuracy caused by the discretization of the action space; (2) fixed exploration strategies are difficult to adapt to dynamic environmental changes; and (3) low efficiency in utilizing empirical data leads to slow policy convergence. Although existing policy optimization methods based on dual-network collaboration attempt to solve the continuous control problem, their fixed noise exploration, single-target network update mechanism, and uniform experience playback strategy still suffer from problems such as policy oscillation, insufficient exploration efficiency, and low sample utilization in highly dynamic physical systems.

[0003] The limitations of existing continuous control methods are as follows: traditional dual-network methods use random process noise with fixed parameters, which cannot dynamically adjust the exploration behavior according to the system dynamics, leading to overexploration in high-inertia systems or underexploration in precision control tasks; the target network uses hard updates or fixed-rate soft updates, making it difficult to balance convergence speed and stability; experience replay uses uniform sampling, ignoring the value of key state-action pairs, resulting in low data utilization efficiency; and the network normalization method is singular and cannot adapt to the dynamic changes of feature distribution at different levels. These shortcomings together restrict the application effect of deep reinforcement learning in the control of complex physical systems. Summary of the Invention

[0004] This invention provides a continuous control method based on adaptive deep reinforcement learning, implemented using a dynamic policy optimization framework (DPOF). The method includes the following steps: The control problem of complex physical systems is modeled as a stochastic time-varying Markov decision process; the environment E possesses time-varying dynamic characteristics (mass matrix). Damping coefficient Stiffness coefficient The agent receives multidimensional observations. Take continuous actions And receive comprehensive rewards ,in As a reward for task completion, As a penalty for energy consumption, , These are dynamic weighting coefficients. Assuming the environment is completely observable, the state... And includes estimated values ​​of environmental dynamic parameters. .

[0005] Agent behavior is determined by adaptive policies Defined as mapping state and environmental parameters to action distribution. The environment is modeled as a stochastic time-varying Markov decision process with a state space. Action space Initial state distribution Time-varying transition dynamics and dynamic reward function .

[0006] This invention defines the dynamic reward function as a weighted sum of future discounted rewards and introduces a task urgency factor. :

[0007] Among the discount factors Adaptive adjustment (reduced to 0.95 when the task difficulty is high). Task urgency factor ( d Let be the Euclidean distance between the state and the target. α , β (To adjust parameters). The goal of reinforcement learning is to maximize the policy. Expected dynamic returns:

[0008] State access distribution Dynamically dependent on environmental parameters .

[0009] To address the issue of target value oscillation caused by traditional single-target network updates, a main target network is designed. With auxiliary target network A parallel bimodal architecture. The outputs of the two networks are weighted and fused through a dynamic weight allocation mechanism to obtain the bimodal action value function:

[0010] Dynamic weights are driven in real time by policy loss:

[0011] in α Basic weight coefficient , β As a regulatory factor ,for t The policy loss value at time step [time]. When the policy loss is large, Automatically reducing the weights of the target network and increasing them provides a more conservative target estimate, effectively suppressing target value fluctuations.

[0012] The target network parameters are updated using a rate-adaptive soft update strategy.

[0013] Update rate Based on the average reward change rate over the last 100 steps Dynamic adjustment:

[0014] When the strategy is in a rapid improvement phase ( And when it is relatively large, Increase to accelerate the target network's follow-up; when the policy enters the oscillation region... Automatic reduction to stabilize training. This mechanism reduces target value fluctuations by approximately 60%, effectively balancing convergence speed and training stability.

[0015] This invention designs a three-level dynamic experience storage system. Perform off-policy learning, where This is the standard experience zone (60%). This is the critical area of ​​experience (high-value estimation bias, 25%). This is the area of ​​failure experience (negative reward sample) (negative reward, 15%).

[0016] Transformation tuples generated by the exploration strategy The samples are dynamically allocated to each sub-region based on their value density. During training, a dynamic priority sampling strategy is employed, with the sampling probability of each sample determined through a priority scoring function.

[0017] in Balance coefficient , For value estimation bias, r To normalize the reward value, the sampling probability and Positive correlation. The three sub-regions were sampled in a 7:2:1 ratio, which significantly improved the utilization of key data while reducing sample correlation.

[0018] To address the heterogeneity of feature distribution caused by the coexistence of multiple physical dimensions in complex physical systems, a differentiated normalization strategy is adopted for each layer of the neural network, with a unified normalization formula as follows: Implementation details of multi-scale feature calibration technology: (1) The input layer uses sliding window normalization with a window size of 500 to 1000 samples, and dynamically tracks the mean of the state distribution based on historical data. With variance Effectively address non-stationary distributions; (2) The hidden layer of the policy network adopts intra-layer batch normalization, with a batch size of 32-128; (3) The value network feature layer adopts cross-layer collaborative normalization, which integrates the mean and variance of the previous layer output to enhance the consistency of feature expression.

[0019] This technique enables the algorithm to converge stably without manual parameter tuning for different physical unit systems.

[0020] By real-time estimation of system dynamic parameters (mass matrix) M Damping coefficient D Stiffness coefficient K ), construct adaptive exploration noise that matches physical properties:

[0021] in The state correlation coefficient is dynamically adjusted according to the task completion rate; For an improved Ornstein-Uhlenbeck process; It is Gaussian white noise.

[0022] The improved OU process update formula is:

[0023] in Memory coefficient Mean term With noise amplitude Based on the system's current kinetic energy Dynamic adjustment:

[0024] in The maximum kinetic energy threshold, α , β To adjust the parameters. The dynamic calibration rules for the parameters are as follows: (1) High inertial system Increase To enhance noise inertia to match the system response; (2) High damping system Increase Improve random exploration coverage; (3) When approaching the target state , Linear decay reduces the intensity of exploration, enabling precise control.

[0025] Ultimately, the environment-aware exploration strategy is expressed as:

[0026] in For the current policy function, This is environmentally perceptible noise.

[0027] The bimodal value function satisfies the improved Bellman equation:

[0028] Dynamic target value Fusion of dual-modal network output:

[0029] Value network parameters Optimization is achieved by minimizing the loss function that incorporates dynamic L2 regularization:

[0030] Dynamic regularization coefficient ,in When the loss is large, the regularization coefficient automatically decreases, allowing for more flexible parameter adjustment; when training tends to stabilize, the regularization is enhanced to prevent overfitting.

[0031] Strategy parameters The update adopts a fusion task urgency factor Dynamic policy gradient:

[0032] Task urgency factor Increasing the gradient update strength as the target state approaches accelerates policy convergence and effectively alleviates the gradient vanishing problem in traditional deterministic policy gradient (DPG).

[0033] An adaptive feature fusion network is used as a function approximator: Actor Network: Input Layer (State / Image) → Multimodal Feature Extraction Module (Low-dimensional input: 2 fully connected hidden layers, 400 / 300 units; Pixel input: 3 convolutional layers with 32 filters + 2 fully connected layers with 200 units) → Dynamic Activation Layer (based on state entropy) Automatic switching between ReLU / LeakyReLU / Swish) → Output layer (tanh function with dynamic scaling factor): ,in ; Critics Network: A feature extraction module that shares the actor network; action input is received in the second fully connected layer, and bimodal output is used. .

[0034] Parameter initialization strategy: The weights of the actor network output layer adopt a truncated normal distribution. Initialization, the deviation is uniformly distributed. Initialization; the feature layer of the value network is initialized using Xavier, and the output layer weights are initialized using the Kaiming normal distribution; the weights of the remaining hidden layers are initialized using... (f is the input dimension) Initialization. The above strategy ensures the initial stability of the policy output and value estimation, avoiding gradient explosion or vanishing due to improper parameters in the early stages of training.

[0035] A segmented decay learning rate scheduling mechanism is adopted:

[0036] Among them, the actor's initial learning rate Initial learning rate of the value network , Here, represents the decay exponent, t represents the current iteration step, and T represents the total number of training steps. The learning rate is larger in the early stages of training to accelerate convergence, and gradually decreased in the later stages for fine-tuning, balancing exploration and utilization dynamically. Discount factor .

[0037] Compared with the prior art, the present invention has the following significant advantages: (1) Improved strategy convergence efficiency: The three-level dynamic experience storage and dynamic priority sampling mechanism improves data utilization efficiency by 50%, and increases strategy convergence speed by more than 40% in robotic arm control tasks (adaptive lever control requires only 600 steps, while traditional methods require 1200 steps). (2) Enhanced control accuracy: The synergistic effect of multi-scale feature calibration and environment-aware exploration strategy reduces control error by 25% and energy consumption by 30% in high-inertia systems (such as bipedal robots); (3) Significantly improved training stability: The bimodal target network and adaptive soft update mechanism reduced the fluctuation of the target value by 60%, and ablation experiments showed that the bimodal target network improved training stability by 60%; (4) Strong environmental adaptability: The dynamic target network and adaptive optimization mechanism can stably adapt to more than 10 different physical parameter configurations and support full-scene learning from low-dimensional state to 64×64 RGB pixel input.

[0038] (5) High exploration efficiency: The environment-aware exploration strategy improves exploration efficiency by 45%, multi-scale feature calibration enhances cross-task adaptability by 50%, and removing any innovative module leads to an overall performance decrease of more than 40%.

[0039] This invention also provides an intelligent control device, including: an edge computing unit (including an NPU acceleration module), a multimodal sensor interface, a real-time data buffer, and a memory storing an adaptive reinforcement learning algorithm. The edge computing unit executes the continuous control method based on adaptive deep reinforcement learning as described above through a parallel processing architecture. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of an adaptive deep reinforcement learning control framework of the present invention; Figure 2 The training loss curves of an example of the present invention are compared with those of different algorithms. Figure 3 This is a training reward curve comparing an example of the present invention with different algorithms. Detailed Implementation

[0041] The technical solutions in the embodiments of the present invention will be clearly and completely described below.

[0042] Experimental environment configuration: Adaptive reward function for tasks Basic task rewards Adaptive lever task Dynamic target capture task Action punishment Safety Rewards Bipedal walking task Energy consumption reward ( E (System kinetic energy). Fall detection threshold. The process terminates when the height is less than 0.45m. Non-Gaussian noise (uniformly distributed) is added to each step of every task. Simulate sensor error.

[0043] Performance standardization methods: Let the reward of the random policy be... MPC returns are The algorithm's reward is Then the standardized score is:

[0044] in, For the theoretical minimum energy consumption, This represents the actual energy consumption of the algorithm. Statistical analysis shows that the proposed method scores significantly higher than TD3 within the 95% confidence interval. Especially under dynamic disturbance scenarios, the energy efficiency reaches more than 85% of the theoretical optimal value (average 0.87±0.04).

[0045] Table 1. Experimental parameters in multiple environments

[0046] Table 1 systematically compares the key parameters of 11 environments, including the state space dimension dim(s), action space dimension dim(a), observation space dimension dim(o) (low dimension / pixel), and environmental dynamic properties (contact dynamics characteristics, time-varying parameter characteristics), to verify the adaptive capability of the proposed method to environments of different complexities.

[0047] Table 2 compares the control objectives, environmental constraints, and core technical challenges of different physical tasks, verifying the generalization ability of the proposed method in complex dynamic scenarios.

[0048] Table 2 Performance Comparison Table

[0049] Table 3 shows the overall performance (mean ± standard deviation of 5 runs) of the proposed method after 2000 steps of training in 11 novel dynamic environments. All scores are calculated as follows: Standardization (random policy = 0, MPC controller = 1), where Eᵢ is the theoretical minimum energy consumption. Comparison algorithms include TD3 and improved DPG; the proposed method performs optimally for both low-dimensional (lowd) and pixel inputs.

[0050] Table 3 Performance comparison of the proposed method and the comparison algorithm in multiphysics environments.

[0051] Experiments show that the proposed method significantly outperforms existing methods in terms of data efficiency: convergence is achieved in only 600 steps for the adaptive pendulum control task (compared to 1200 steps for traditional methods), and in 900 steps for the dynamic target grasping task (compared to 1200 steps for TD3). All tasks achieve stable performance within 2000 steps, with an average sample efficiency improvement of 2.1 times, demonstrating the algorithm's practicality in complex physical systems.

[0052] The dynamic optimization framework proposed in this invention is scalable: by increasing the depth of the convolutional layers (from 3 to 5 layers), it can handle 128×128 pixel inputs; after introducing the attention mechanism, the completion time of multi-robotic arm collaborative tasks is reduced by 40%. This lays the foundation for solving more complex real-world problems (such as multi-robot collaboration and flexible body control).

[0053] To verify the convergence performance and stability of the proposed method, comparative experiments were conducted under a standard physical control task environment. First, by recording the changes in the cumulative loss value during training, the number of training steps required for the algorithm to reach a stable control strategy was quantified, and the sample efficiency was evaluated. Comparative methods included improved DPG and TD3, and the experimental results are as follows: Figure 2 .

[0054] Then, by observing the fluctuations of the reward curve during training, the stability of the algorithm in a high-dimensional continuous action space is analyzed, verifying the effect of the bimodal target network and adaptive soft update mechanism on suppressing target value oscillations. Experimental results are as follows: Figure 3 As shown: Experimental results show that the proposed method significantly outperforms the comparative algorithms in both convergence speed and policy stability. Regarding convergence speed, the loss curves reveal that the proposed method exhibits a rapid decrease in loss value within the first 300 steps, reaching a stable convergence state earliest, completing the main convergence process around step 300. In contrast, the improved DPG requires approximately 700 steps, and TD3 takes even longer with significant initial fluctuations. Regarding policy stability, the reward curves further validate the superiority of the proposed method: the reward curve of the proposed method exhibits minimal fluctuation after convergence, indicating higher policy stability during environmental interactions. This is attributed to the effective suppression of policy oscillations during training by the bimodal target network and adaptive soft update mechanism introduced in this invention; while the reward curves of the improved DPG and TD3 still show significant fluctuations in the later stages.

[0055] Beyond the core innovations of this invention, the implementation details can be referred to in the prior art. However, its dynamic strategy optimization framework, multi-scale feature calibration mechanism, and adaptive exploration strategy constitute an inseparable technical whole, which is not a conventional combination or well-known technology for those skilled in the art. This embodiment does not limit this innovative combination, and the specific implementation method can be flexibly adjusted according to the application scenario.

Claims

1. A continuous control method based on adaptive deep reinforcement learning, characterized in that, Based on a dynamic policy optimization framework, the implementation includes the following steps: (1) The control problem of complex physical systems is modeled as a stochastic time-varying Markov decision process; the environment E has time-varying dynamic characteristics, including the mass matrix. Damping coefficient and stiffness coefficient The intelligent agent receives multidimensional observations. Take continuous actions Furthermore, to maximize the introduction of task urgency factors. Dynamic reward function For learning objectives; (2) Construct a dual-modal target network system, including a main target network and an auxiliary target network, and use a dynamic weight allocation mechanism. By fusing the outputs of the two networks, a stable bimodal action value function is obtained. ,in Basic weight coefficient , As a regulatory factor , Let be the policy loss value at time t; (3) Adopt an adaptive soft update strategy update rate Based on the average reward change rate over the last 100 steps Dynamically adjust and balance the convergence speed and training stability of the target network; (4) Design a three-level dynamic experience storage system Perform off-policy learning, where This is the standard experience zone. This is the critical experience zone. For the failure experience zone, a dynamic priority sampling strategy is adopted, and the priority scoring function is: ,in This is the balance coefficient; (5) Implement multi-scale feature calibration for each layer of the network: For the input layer, use sliding window normalization based on historical data; for the hidden layer of the policy network, use intra-layer batch normalization; and for the feature layer of the value network, use cross-layer collaborative normalization. The normalization formula is unified as follows: ; (6) Develop an environment-aware exploration strategy generator, which collects system dynamic parameters (mass matrix) in real time. M Damping coefficient D Stiffness coefficient K Dynamic calibration explores noise; the noise generation formula is as follows: Constructing an environment-aware exploration strategy is expressed as .

2. The method according to claim 1, characterized in that, The reinforcement learning objective is to maximize the policy. The expected dynamic return is State access distribution Dynamically dependent on environmental parameters .

3. The method according to claim 1, characterized in that, The updated formula for the improved Ornstein-Uhlenbeck process is as follows: ,in Memory coefficient Mean term With noise amplitude From the current kinetic energy of the system Dynamic adjustment.

4. The method according to claim 1, characterized in that, The bimodal value function satisfies the improved Bellman equation. Dynamic target value The output of the fusion dual-modal network is .

5. The method according to claim 1, characterized in that, Value network parameters By minimizing the loss function that incorporates dynamic L2 regularization Optimization is performed, including the dynamic regularization coefficient. ,in .

6. The method according to claim 1, characterized in that, Incorporating task urgency factors into policy gradient updates By increasing the gradient update strength when approaching the target state, the convergence speed of the policy is improved.

7. The method according to claim 1, characterized in that, An adaptive feature fusion network is used as a function approximator, and the output layer of the actor network adopts a tanh function with a dynamic scaling factor. , , The state entropy is used; the dynamic activation unit automatically switches between ReLU / LeakyReLU / Swish based on the state complexity.

8. The method according to claim 1, characterized in that, A segmented decay learning rate scheduling mechanism is adopted: Actor Network Learning Rate Value network learning rate ,in , , This is the decay index.

9. The method according to claim 1, characterized in that, The total capacity of the dynamic experience storage is 10. 6 The conventional experience zone, critical experience zone, and failure experience zone are sampled in a 7:2:1 ratio; discount factor. ; Loss-sensitive regularization coefficient Sensitivity parameters Configure according to task characteristics.

10. An intelligent control device, comprising: An edge computing unit (including an NPU acceleration module), a multimodal sensor interface, a real-time data buffer, and a memory storing an adaptive reinforcement learning algorithm are characterized in that the edge computing unit executes the method described in any one of claims 1-9 through a parallel processing architecture to achieve low-latency continuous control of the physical system.

Images