Updating method and device of deep reinforcement learning model

Abstract

The invention discloses an updating method and device for a deep reinforcement learning model, and belongs to the technical field of deep reinforcement learning models.The updating method and device for the deep reinforcement learning model comprise the following specific steps that firstly, environment interaction data are asynchronously collected through a plurality of parallel execution units; 2, sampling data from an experience pool based on a dynamic priority strategy, wherein the priority is jointly determined by TD-error and strategy similarity; 3, setting different updating frequencies for a key layer and a non-key layer of the network by adopting a hierarchical updating strategy; and step 4, gradually synchronizing parameters of the target network and the online network through a soft update technology. According to the method, the waiting time is reduced through an asynchronous updating mechanism, the resource utilization rate is improved, the efficiency is improved, the training speed is improved through dynamic priority sampling, convergence is accelerated, the calculation overhead is reduced through a block updating strategy, resources are saved, the error rate of a model passing an adversarial test is reduced, and the safety is improved.

Description

Technical Field

[0001] The present invention belongs to the field of deep reinforcement learning model technology, and specifically relates to a method and device for updating a deep reinforcement learning model. Background Art

[0002] Deep Reinforcement Learning (DRL) is a branch of machine learning that combines the perception capabilities of deep learning and the decision-making capabilities of reinforcement learning to solve sequential decision-making problems in complex environments. The existing update methods of deep reinforcement learning models have the following problems: Low synchronous update efficiency: Traditional methods use synchronous update strategies (such as A3C's synchronous multithreading), which are prone to waiting delays under heterogeneous computing resources, resulting in low resource utilization. Insufficient experience replay mechanism: Experience replay with fixed priority or uniform sampling is difficult to balance exploration and utilization, affecting the convergence speed. Poor model stability: In a dynamic environment, directly updating the policy network may cause policy mutations and cause system oscillations. High resource consumption: Frequent global model updates have high requirements for computing and storage resources and are difficult to deploy on edge devices. Summary of the invention

[0003] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a method and device for updating a deep reinforcement learning model.

[0004] The technical solution adopted to solve the above technical problems is: a method for updating a deep reinforcement learning model, including the following specific steps: Step 1: Asynchronously collect environmental interaction data through multiple parallel execution units; Step 2: Sampling data from the experience pool based on a dynamic priority strategy, where the priority is determined jointly by TD-error and strategy similarity; Step 3: Adopt a layered update strategy and set different update frequencies for the key and non-key layers of the network; Step 4: Gradually synchronize the target network and online network parameters through soft update technology.

[0005] Through the above technical solutions, the efficiency of synchronous update is improved, convergence is accelerated, model stability is improved, and resource consumption is reduced.

[0006] Furthermore, the sampling priority in the dynamic priority strategy It consists of two parts: temporal difference error and policy difference measurement, and the formula is: in, It is empirical data The time series difference error (TD-error) measures the deviation of the current value estimate. The specific expression is: is the immediate reward of environmental feedback, The discount factor controls the importance of future rewards, is the target network pair downshift state and actions The estimated value of Online Networking for Current Status and actions The estimated value of The KL divergence between the new and old strategies quantifies the magnitude of strategy updates and prevents strategy mutations. is the action probability distribution output by the policy network before updating, is the working probability distribution output by the updated policy network, is the balance coefficient, which controls the weight of TD-error and strategy difference.

[0007] The above technical solution can solve the problem that high TD-error samples indicate that the current model prediction is inaccurate and needs to be learned first, while high KL divergence samples are used to indicate strategy changes to determine whether they deviate excessively from historical experience.

[0008] Furthermore, the layered update frequency control adopts the following formula: Divide the deep network into key layers and non-key layers and define the update frequency : in, is the base update frequency, , is the adjustment factor.

[0009] Through the above technical solution, the waiting time is reduced and the resource utilization is improved through the asynchronous update mechanism.

[0010] Furthermore, the asynchronous gradient update uses the following formula: in, is the learning rate, is the loss function.

[0011] Through the above technical solution, the update consumption is reduced.

[0012] Furthermore, the KL divergence of the strategies before and after the update is calculated to ensure that it does not exceed the safety threshold. ;like , then update, if , reject this update and roll back to the old policy.

[0013] The above technical solution improves the stability of the model.

[0014] Furthermore, it includes a data acquisition module, an experience pool management module, an asynchronous update module and a security verification module. The data acquisition module is responsible for acquiring the environmental status through multi-source sensors, the experience pool management module is responsible for storing and dynamically weighting the sampled experience data, the asynchronous update engine is responsible for updating the neural network parameters in blocks, and the security verification module is responsible for verifying the stability of the model through adversarial sample testing.

[0015] The beneficial effects of the present invention are as follows: the present invention reduces waiting time and improves resource utilization and efficiency through an asynchronous update mechanism; increases training speed and accelerates convergence through dynamic priority sampling; reduces computing overhead and saves resources through a block update strategy; reduces the model error rate through adversarial testing and improves security. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0018] like Figure 1 As shown, the updating method and device of the deep reinforcement learning model of this embodiment include the following specific steps: Step 1: Asynchronously collect environmental interaction data through multiple parallel execution units; Step 2: Sampling data from the experience pool based on a dynamic priority strategy, where the priority is determined jointly by TD-error and strategy similarity; Step 3: Adopt a layered update strategy and set different update frequencies for the key and non-key layers of the network; Step 4: Gradually synchronize the target network and online network parameters through soft update technology, reduce waiting time through asynchronous update mechanism, improve resource utilization and efficiency, increase training speed and accelerate convergence through dynamic priority sampling, reduce computing overhead and save resources through block update strategy, reduce model error rate through adversarial testing and improve security.

[0019] The sampling priority in the dynamic priority strategy It consists of two parts: temporal difference error and policy difference measurement, and the formula is: in, It is empirical data The time series difference error (TD-error) measures the deviation of the current value estimate. The specific expression is: is the immediate reward of environmental feedback, The discount factor controls the importance of future rewards, is the target network pair downshift state and actions The estimated value of Online Networking for Current Status and actions The estimated value of The KL divergence between the new and old strategies quantifies the magnitude of strategy updates and prevents strategy mutations. is the action probability distribution output by the policy network before updating, is the working probability distribution output by the updated policy network, It is a balance coefficient that controls the weight of TD-error and policy difference. It solves the problem that high TD-error samples indicate that the current model prediction is inaccurate and needs to be learned first. High KL divergence samples indicate policy changes to determine whether they deviate too much from historical experience.

[0020] The layered update frequency control uses the following formula: Divide the deep network into key layers and non-key layers and define the update frequency : in, is the base update frequency, , is the adjustment factor.

[0021] The asynchronous gradient update uses the following formula: in, is the learning rate, It is the loss function, which reduces the update cost.

[0022] Calculate the KL divergence of the strategy before and after the update to ensure that it does not exceed the safety threshold ,like , then update, if , reject this update and roll back to the old strategy to improve model stability.

[0023] It includes a data acquisition module, an experience pool management module, an asynchronous update module and a security verification module. The data acquisition module is responsible for acquiring the environmental status through multi-source sensors, the experience pool management module is responsible for storing and dynamically weighting the sampled experience data, the asynchronous update engine is responsible for updating the neural network parameters in blocks, and the security verification module is responsible for verifying the model stability through adversarial sample testing.

[0024] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A method for updating a deep reinforcement learning model, characterized in that: The specific steps include: Step 1: Asynchronously collect environmental interaction data through multiple parallel execution units; Step 2: Sampling data from the experience pool based on a dynamic priority strategy, where the priority is determined jointly by TD-error and strategy similarity; Step 3: Adopt a layered update strategy and set different update frequencies for the key and non-key layers of the network; Step 4: Gradually synchronize the target network and online network parameters through soft update technology.

2. The method for updating a deep reinforcement learning model according to claim 1, characterized in that: The sampling priority in the dynamic priority strategy It consists of two parts: temporal difference error and policy difference measurement, and the formula is: in, It is empirical data The time series difference error (TD-error) measures the deviation of the current value estimate. The specific expression is: is the immediate reward of environmental feedback, The discount factor controls the importance of future rewards, is the target network pair downshift state and actions The estimated value of Online Networking for Current Status and actions The estimated value of The KL divergence between the new and old strategies quantifies the magnitude of strategy updates and prevents strategy mutations. is the action probability distribution output by the policy network before updating, is the working probability distribution output by the updated policy network, is the balance coefficient, which controls the weight of TD-error and strategy difference.

3. The method for updating a deep reinforcement learning model according to claim 2, characterized in that: The layered update frequency control uses the following formula: Divide the deep network into key layers and non-key layers and define the update frequency : in, is the base update frequency, , is the adjustment factor.

4. The method for updating the deep reinforcement learning model according to claim 3, characterized in that: The asynchronous gradient update uses the following formula: in, is the learning rate, is the loss function.

5. The method for updating a deep reinforcement learning model according to claim 4, characterized in that: Calculate the KL divergence of the strategy before and after the update to ensure that it does not exceed the safety threshold ,like , then update, if ， Reject this update and roll back to the old policy.

6. A device for the method for updating the deep reinforcement learning model as claimed in claim 5, characterized in that: It includes a data acquisition module, an experience pool management module, an asynchronous update module and a security verification module. The data acquisition module is responsible for acquiring the environmental status through multi-source sensors, the experience pool management module is responsible for storing and dynamically weighting the sampled experience data, the asynchronous update engine is responsible for updating the neural network parameters in blocks, and the security verification module is responsible for verifying the model stability through adversarial sample testing.

Images