A reinforcement learning asynchronous distributed architecture optimization method
By employing an asynchronous distributed architecture optimization method within a CPU+GPU heterogeneous parallel architecture, and utilizing the A3C algorithm and baseline parameter updates to optimize the decision model, the performance issues of deep reinforcement learning in adversarial game environments are resolved, achieving more efficient processing and stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING INST OF REMOTE SENSING EQUIP
- Filing Date
- 2023-04-12
- Publication Date
- 2026-07-07
AI Technical Summary
In existing technologies, deep reinforcement learning based on CPU+GPU heterogeneous parallel architecture has poor performance in game-based adversarial environments.
An asynchronous distributed architecture-based optimization method is adopted, which utilizes the Asynchronous Advantage Actor Commentator (A3C) algorithm combined with the baseline parameter update method to design an online execution and offline training framework. The decision model is optimized through the interaction between the decision execution system and the decision training system.
It improves the processing performance and stability of deep reinforcement learning in game-theoretic environments.
Smart Images
Figure CN116542318B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence reinforcement learning, and in particular to an optimization method based on asynchronous distributed architecture of reinforcement learning. Background Technology
[0002] Reinforcement learning (RL) is an artificial intelligence algorithm specifically designed for solving decision-making problems. Based on the Markov Decision Process (MDP) theory, RL refers to an agent continuously executing decisions and interacting with the environment to obtain a "reward" function that evaluates its own state, iteratively optimizing its strategy to maximize expected returns. This method does not rely on sufficient prior knowledge and can achieve globally optimal decisions through online learning. In 2015, Mnih V et al. proposed the Deep Q-network (DQN) method, a pioneering work in deep reinforcement learning, combining deep learning with reinforcement learning to achieve an end-to-end learning algorithm from perception to action. In 2016, AlphaGo, combining DQN and Monte Carlo tree search, defeated top human professional Go players in the classic dynamic game model of Go. This technology has been applied in multiple fields, such as drone control, communication security, radar countermeasures, and autonomous driving. However, the deep neural network computational complexity involved in deep reinforcement learning is far higher than that of traditional algorithms, making it difficult to meet the high timeliness requirements of online radar strategy optimization. Therefore, it is necessary to focus on how to improve the computational performance of the algorithm.
[0003] While traditional Central Processing Units (CPUs) possess powerful control capabilities and versatility for handling diverse data types, they also require logical judgments, branching, and interrupts. Their complex internal structure and relatively few computational units make them unsuitable for large-scale parallel data computation. Implementing deep learning algorithms using CPUs cannot meet the real-time requirements of radar jamming countermeasures. Compared to CPUs, Field-Programmable Gate Arrays (FPGAs) offer higher performance and lower power consumption, allowing for the implementation of customized deep learning hardware acceleration circuits. However, as a "universal chip," FPGAs require a large number of gate arrays pre-deployed to meet user needs, trading area for speed and consuming more FPGA core resources. Compared to CPUs and FPGAs, Graphics Processing Units (GPUs) have numerous computational units, enabling complex mathematical operations on large-scale, uniform, and dependency-free data. They typically work in conjunction with CPUs to complete computational tasks; therefore, the CPU+GPU heterogeneous parallel architecture is currently the mainstream hardware platform supporting intelligent algorithms. However, based on this architecture, deep reinforcement learning currently exhibits poor performance in adversarial game scenarios. Summary of the Invention
[0004] The purpose of this invention is to provide an optimization method for asynchronous distributed architecture based on reinforcement learning, which solves the problem of poor performance of deep reinforcement learning in game adversarial environments based on CPU+GPU heterogeneous parallel architecture.
[0005] An optimization method based on reinforcement learning asynchronous distributed architecture, the method being applied to decision execution systems and decision training systems, the method comprising:
[0006] Step 1: The decision execution system acquires environmental feedback information and constructs the offline environment:
[0007] The second step is for the decision execution system to make decisions based on environmental feedback information and the offline environment: The decision execution system takes environmental feedback information as input, inputs it into the online decision model, obtains the online decision results, applies the decision results to the real environment, and interacts with the real environment;
[0008] The third step is for the decision training system to optimize the offline decision model based on the decisions made by the decision execution system: The decision training system adopts the A3C distributed architecture and interacts quickly with the constructed offline environment. It leverages the advantages of multi-threading and quickly optimizes the offline decision model based on feedback and decision results.
[0009] Step 4: Update the online decision model based on the optimized decision model to complete the architecture optimization.
[0010] When the offline decision-making model meets certain conditions, its parameters are used to update the online decision-making model.
[0011] In one embodiment, the decision execution system acquires environmental feedback information and constructs an offline environment, including:
[0012] The decision execution system interacts with the real environment to obtain feedback information about the strategy in the environment, and uses the feedback information to build an offline environment for use by the decision training system.
[0013] In one embodiment, obtaining environmental feedback information includes: the decision execution system based on the decision... a t Interact with the real environment to obtain the current state of the strategy in response to the environment. s t .
[0014] In one embodiment, constructing the offline environment includes updating the offline environment at fixed intervals of a certain number of rounds, with the updated content being the current state of the agent. s t The actions taken and the next state obtained after interacting with the environment. s t+1 and benefits r t The above information is organized into a linked list, and the access count of each piece of information is marked. Every fixed number of rounds, the linked list is dynamically added to the offline environment, and the information record with the fewest access counts is deleted to avoid the problem of explosive data explosion caused by continuously increasing the amount of information fed back from the environment.
[0015] In one embodiment, the decision execution system makes decisions based on environmental feedback information and the offline environment, including:
[0016] The decision execution system takes environmental feedback information as input, feeds it into the online decision model, obtains the online decision results, applies the decision results to the real environment, and interacts with the real environment.
[0017] In one embodiment, the decision execution system's decision-making based on environmental feedback information and the offline environment further includes: employing the Actor network in the Actor-Critic (AC) network model as the online decision-making model, and processing the current state of the environmental feedback information. s t The input is fed into the online decision-making model Actor Network to obtain the decision. a t And will make decisions a tIt is used for interaction in real-world environments.
[0018] In one embodiment, the decision training system optimizes the offline decision model based on the decision made by the decision execution system, including:
[0019] The decision training system uses the A3C algorithm (Asynchronous Advantage Actor-Critic) to optimize the distributed architecture of the decision model in an offline environment. It interacts rapidly with the constructed offline environment, leverages the advantages of multi-threading, and quickly optimizes the offline decision model based on feedback and decision results.
[0020] In one embodiment, the A3C algorithm employs a global main thread responsible for updating Actor and Critic parameters, and multiple sub-threads (workers) responsible for independently interacting with the offline environment. After a certain number of iterations, the global main thread aggregates the learning results of the multiple sub-threads to update its own network parameters, and the multiple sub-threads then learn the jointly optimized results from the global main thread.
[0021] In one embodiment, updating the online decision model based on the optimized decision model to complete the architecture optimization includes:
[0022] When the offline decision-making model meets the predetermined conditions, the parameters of the offline decision-making model are used to update the online decision-making model.
[0023] In one embodiment, updating the online decision model with the parameters of the offline decision model when the offline decision model meets predetermined conditions includes:
[0024] The offline-trained decision-making model is updated to update the online decision-making model. A baseline is designed, and the training model that meets the baseline is used for the execution model that needs to be updated. Let the parameters of the execution network be... The parameters for training the network are: The value feedback of each interaction in the new self-updating offline environment is... Then the cumulative value Execute network parameters The cumulative value obtained is Training network parameters The cumulative value obtained is Updating the global model requires satisfying the following equation:
[0025]
[0026] in, (i=1, 2, ..., m) represents the cumulative value of each round i in the execution network. (i=1, 2, ..., m) represents the cumulative value of training the network in each round i. It is a threshold greater than 1. The cumulative value of the network after m rounds of training in a self-updating offline environment. The expected value is greater than the cumulative value of the execution network after m rounds in a self-updating offline environment. At this point, network parameters are executed. Update to training network parameters Otherwise, perform network parameter replacement. Training network parameters The network is retrained in a new self-updating offline environment based on the old execution network parameters.
[0027] This invention aims to provide an optimization method based on an asynchronous distributed architecture of reinforcement learning. Under a CPU+GPU heterogeneous parallel architecture, an online execution and offline training framework is designed. The Asynchronous Advantage Actor-Critic (A3C) algorithm is used to achieve rapid offline model optimization. Combined with a proposed baseline parameter update method, the online execution mode is updated, thereby improving the performance and stability of the optimized model in a game-theoretic environment. To improve the performance of deep reinforcement learning in game-theoretic environments, this paper proposes an optimization method based on an asynchronous distributed architecture of reinforcement learning, using a CPU+GPU heterogeneous parallel architecture. This method involves designing an online execution and offline training framework. The Asynchronous Advantage Actor-Critic (A3C) algorithm is used to achieve a fast offline multi-threaded asynchronous parallel optimization method. Combined with the proposed baseline parameter update method, the processing performance and stability of the algorithm are effectively improved. Attached Figure Description
[0028] Figure 1 This is a schematic diagram of an asynchronous distributed architecture optimization method based on reinforcement learning according to the present invention. Detailed Implementation
[0029] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of the present invention will become clearer from the following description and claims.
[0030] While the invention can be extended in many forms of modification and substitution, and the specification also provides some specific embodiments and detailed descriptions, it should be understood that the inventors' intention is not to limit the invention to the specific embodiments described. On the contrary, the inventors' intention is to protect all improvements, equivalent substitutions and modifications made within the spirit or scope defined by this claim.
[0031] An optimization method based on reinforcement learning asynchronous distributed architecture, the method being applied to decision execution systems and decision training systems, the method comprising:
[0032] Step 1: The decision execution system acquires environmental feedback information and constructs the offline environment:
[0033] The second step is for the decision execution system to make decisions based on environmental feedback information and the offline environment: The decision execution system takes environmental feedback information as input, inputs it into the online decision model, obtains the online decision results, applies the decision results to the real environment, and interacts with the real environment;
[0034] The third step is for the decision training system to optimize the offline decision model based on the decisions made by the decision execution system: The decision training system adopts the A3C distributed architecture and interacts quickly with the constructed offline environment. It leverages the advantages of multi-threading and quickly optimizes the offline decision model based on feedback and decision results.
[0035] Step 4: Update the online decision model based on the optimized decision model to complete the architecture optimization.
[0036] When the offline decision-making model meets certain conditions, its parameters are used to update the online decision-making model.
[0037] In one embodiment, the decision execution system acquires environmental feedback information and constructs an offline environment, including:
[0038] The decision execution system interacts with the real environment to obtain feedback information about the strategy in the environment, and uses the feedback information to build an offline environment for use by the decision training system.
[0039] In one embodiment, obtaining environmental feedback information includes: the decision execution system based on the decision... a t Interact with the real environment to obtain the current state of the strategy in response to the environment. s t .
[0040] In one embodiment, constructing the offline environment includes updating the offline environment at fixed intervals of a certain number of rounds, with the updated content being the current state of the agent. s t The actions taken and the next state obtained after interacting with the environment. s t+1 and benefits r t The above information is organized into a linked list, and the access count of each piece of information is marked. Every fixed number of rounds, the linked list is dynamically added to the offline environment, and the information record with the fewest access counts is deleted to avoid the problem of explosive data explosion caused by continuously increasing the amount of information fed back from the environment.
[0041] In one embodiment, the decision execution system makes decisions based on environmental feedback information and the offline environment, including:
[0042] The decision execution system takes environmental feedback information as input, feeds it into the online decision model, obtains the online decision results, applies the decision results to the real environment, and interacts with the real environment.
[0043] In one embodiment, the decision execution system's decision-making based on environmental feedback information and the offline environment further includes: employing the Actor network in the Actor-Critic (AC) network model as the online decision-making model, and processing the current state of the environmental feedback information. s t The input is fed into the online decision-making model Actor Network to obtain the decision. a t And will make decisions a t It is used for interaction in real-world environments.
[0044] In one embodiment, the decision training system optimizes the offline decision model based on the decision made by the decision execution system, including:
[0045] The decision training system uses the A3C algorithm (Asynchronous Advantage Actor-Critic) to optimize the distributed architecture of the decision model in an offline environment. It interacts rapidly with the constructed offline environment, leverages the advantages of multi-threading, and quickly optimizes the offline decision model based on feedback and decision results.
[0046] In one embodiment, the A3C algorithm employs a global main thread responsible for updating Actor and Critic parameters, and multiple sub-threads (workers) responsible for independently interacting with the offline environment. After a certain number of iterations, the global main thread aggregates the learning results of the multiple sub-threads to update its own network parameters, and the multiple sub-threads then learn the jointly optimized results from the global main thread.
[0047] In one embodiment, updating the online decision model based on the optimized decision model to complete the architecture optimization includes:
[0048] When the offline decision-making model meets the predetermined conditions, the parameters of the offline decision-making model are used to update the online decision-making model.
[0049] In one embodiment, updating the online decision model with the parameters of the offline decision model when the offline decision model meets predetermined conditions includes:
[0050] The offline-trained decision-making model is used to update the online decision-making model. A baseline is designed, and the training model that meets the baseline is used to update the execution model. Let the parameters of the execution network be... The parameters for training the network are: The value feedback of each interaction in the new self-updating offline environment is... Then the cumulative value Execute network parameters The cumulative value obtained is Training network parameters The cumulative value obtained is Updating the global model requires satisfying the following equation:
[0051]
[0052] in, (i=1, 2, ..., m) represents the cumulative value of each round i in the execution network. (i=1, 2, ..., m) represents the cumulative value of training the network in each round i. It is a threshold greater than 1. The cumulative value of the network after m rounds of training in a self-updating offline environment. The expected value is greater than the cumulative value of the execution network after m rounds in a self-updating offline environment. At this point, network parameters are executed. Update to training network parameters Otherwise, perform network parameter replacement. Training network parameters The network is retrained in a new self-updating offline environment based on the old execution network parameters.
[0053] This invention aims to provide an optimization method based on an asynchronous distributed architecture of reinforcement learning. Under a CPU+GPU heterogeneous parallel architecture, an online execution and offline training framework is designed. The Asynchronous Advantage Actor-Critic (A3C) algorithm is used to achieve rapid offline model optimization. Combined with a proposed baseline parameter update method, the online execution mode is updated, thereby improving the performance and stability of the optimized model in a game-theoretic environment. To improve the performance of deep reinforcement learning in game-theoretic environments, this paper proposes an optimization method based on an asynchronous distributed architecture of reinforcement learning, using a CPU+GPU heterogeneous parallel architecture. This method involves designing an online execution and offline training framework. The Asynchronous Advantage Actor-Critic (A3C) algorithm is used to achieve a fast offline multi-threaded asynchronous parallel optimization method. Combined with the proposed baseline parameter update method, the processing performance and stability of the algorithm are effectively improved.
[0054] In one embodiment, the following combination Figure 1 Specific embodiments of the present invention will be described in detail.
[0055] This invention proposes an optimization method for asynchronous distributed architecture based on reinforcement learning, characterized by the following specific steps:
[0056] The first step is to obtain environmental feedback information and build an offline environment.
[0057] It consists of two parts: obtaining environmental feedback information and building the offline environment.
[0058] Obtaining environmental feedback information: The decision execution system makes decisions based on... a t Interact with the real environment to obtain the current state of the strategy in response to the environment. s t .
[0059] Building an offline environment: The offline environment is updated at fixed intervals, and the updated content is the current state of the agent. s t The actions taken and the next state obtained after interacting with the environment. s t+1 and value feedback r t The above information is organized into a linked list, and the access count of each piece of information is marked. Every fixed number of rounds, the linked list is dynamically added to the offline environment, and the information record with the fewest access counts is deleted to avoid the problem of explosive data explosion caused by continuously increasing the amount of information fed back from the environment.
[0060] The second step involves the decision-making and execution system making a decision.
[0061] According to the method, the second step involves using the Actor network in the Actor-Critic (AC) network model as the online decision-making model to process the current state of environmental feedback information. s t The input is fed into the online decision-making model Actor Network to obtain the decision. a t And will make decisions a t It is used for interaction in real-world environments.
[0062] The third step involves optimizing the offline decision-making model through a decision-making training system.
[0063] According to the method, the key feature is that in the third step, a multi-threaded distributed asynchronous advantage actor-critic (A3C) algorithm is used to optimize the decision model in an offline environment. A3C employs a global main thread responsible for updating the Actor and Critic parameters, while multiple sub-threads (workers) are responsible for independently interacting with the offline environment. After a certain number of iterations, the global thread aggregates the learning results from the workers to update its own network parameters, and the workers then learn from the collective optimization results from the global thread.
[0064] Step 4: Update the online decision-making model
[0065] According to the method, the fourth step, updating the online decision model, refers to updating the online decision model with the offline-trained decision model, designing a baseline, and using the training model that satisfies the baseline to update the execution model: Let the parameters of the execution network be... The parameters for training the network are: The value feedback of each interaction in the new self-updating offline environment is... r t Then the cumulative value Execute network parameters The cumulative value obtained is R exe Training network parameters The cumulative value obtained is R train Updating the global model requires satisfying the following equation:
[0066]
[0067] in, R i exe ( i =1, 2, ..., m ) indicates that each round of the network is executed. i The cumulative value, R i train ( i =1, 2, ..., m ) indicates each round of training the network. i The cumulative value, It is a threshold greater than 1. The network is trained in a self-updating offline environment. m Cumulative value after rounds R train Expectations exceed execution network in self-updating offline environments m Cumulative value after roundsR exe At this point, network parameters are executed. Update to training network parameters Otherwise, perform network parameter replacement. Training network parameters The network is retrained in a new self-updating offline environment based on the old execution network parameters.
[0068] This invention aims to provide an optimization method based on an asynchronous distributed architecture of reinforcement learning. Under a CPU+GPU heterogeneous parallel architecture, an online execution and offline training framework is designed. The Asynchronous Advantage Actor-Critic (A3C) algorithm is used to achieve rapid offline model optimization. Combined with a proposed baseline parameter update method, the online execution mode is updated, thereby improving the performance and stability of the optimized model in a game-theoretic environment. To improve the performance of deep reinforcement learning in game-theoretic environments, this paper proposes an optimization method based on an asynchronous distributed architecture of reinforcement learning, using a CPU+GPU heterogeneous parallel architecture. This method involves designing an online execution and offline training framework. The Asynchronous Advantage Actor-Critic (A3C) algorithm is used to achieve a fast offline multi-threaded asynchronous parallel optimization method. Combined with the proposed baseline parameter update method, the processing performance and stability of the algorithm are effectively improved.
Claims
1. A reinforcement learning-based asynchronous distributed architecture optimization method, characterized in that, The method is applied to decision execution systems and decision training systems, and the method includes: The first step is for the decision-making and execution system to acquire environmental feedback information and construct the offline environment; The second step is for the decision-making and execution system to make decisions based on environmental feedback information and the offline environment. The third step involves the decision training system optimizing the offline decision-making model based on the decisions made by the decision execution system. Step 4: Update the online decision-making model based on the optimized decision-making model to complete the architecture optimization. The decision execution system acquires environmental feedback information and constructs an offline environment by: the decision execution system interacting with the real environment, acquiring feedback information of the strategy in the environment, and using the feedback information to construct an offline environment for use by the decision training system; The obtained environmental feedback information includes: the decision execution system based on the decision... a t Interact with the real environment to obtain the current state of the strategy in response to the environment. s t ; The construction of the offline environment includes: updating the offline environment at fixed intervals of a certain number of rounds, with the updated content being the current state of the agent. s t The actions taken and the next state obtained after interacting with the environment. s t+1 and value feedback r t The above information is formed into a linked list and the access count of each piece of information is marked. Every fixed number of rounds, the linked list is dynamically added to the offline environment, and the information record with the fewest access counts is deleted to avoid the problem of explosive big data caused by continuously increasing the feedback information from the environment. Furthermore, the step of updating the online decision model based on the optimized decision model to complete the architecture optimization includes: When the offline decision-making model meets predetermined conditions, the parameters of the offline decision-making model are used to update the online decision-making model, including: The offline-trained decision-making model is used to update the online decision-making model. A baseline is designed, and the trained model that meets the baseline is used to update the execution model. Let the parameters of the execution network be... The parameters for training the network are: The value feedback of each interaction in a self-updating offline environment is... Then the cumulative value Execute network parameters The cumulative value obtained is Training network parameters The cumulative value obtained is Updating the global model requires satisfying the following equation: in, Let i = 1, 2, ..., m, representing the cumulative value of each round i in the execution network. Let i = 1, 2, ..., m, representing the cumulative value of i in each round of training the network. It is a threshold greater than 1, representing the cumulative value of the network after m rounds of training in a self-updating offline environment. The expected value is greater than the cumulative value of the execution network after m rounds in a self-updating offline environment. At this point, network parameters are executed. Update to training network parameters Otherwise, perform network parameter replacement. Training network parameters The network is retrained in a new self-updating offline environment based on the old execution network parameters.
2. The method according to claim 1, characterized in that, The decision execution system makes decisions based on environmental feedback information and the offline environment, including: The decision execution system takes environmental feedback information as input, feeds it into the online decision model, obtains the online decision results, applies the decision results to the real environment, and interacts with the real environment.
3. The method according to claim 2, characterized in that, The decision execution system, based on environmental feedback information and the offline environment, further includes: employing the Actor network in the Actor-Critic (AC) network model as the online decision-making model, and incorporating the current state of the environmental feedback information. s t The input is fed into the online decision-making model Actor Network to obtain the decision. a t And will make decisions a t It is used for interaction in real-world environments.
4. The method according to claim 3, characterized in that, The decision training system optimizes the offline decision model based on the decisions made by the decision execution system, including: The decision training system uses the A3C algorithm (Asynchronous Advantage Actor-Critic) to optimize the distributed architecture of the decision model in an offline environment, and interacts quickly with the constructed offline environment. It leverages the advantages of multi-threading and rapidly optimizes the offline decision model based on feedback and decision results.
5. The method according to claim 4, characterized in that, The A3C algorithm employs a global main thread responsible for updating Actor and Critic parameters, while multiple sub-threads are responsible for independently interacting with the offline environment. After a certain number of iterations, the global main thread aggregates the learning results from the multiple sub-threads to update its own network parameters, and the multiple sub-threads then learn the jointly optimized results from the global main thread.