A training system and method for an action strategy model

By employing discrete action selection and hypernetwork-generated weights, the problem of inaccurate action selection in a hybrid action space is solved, improving the accuracy and interpretability of unmanned equipment control, while also enhancing computational efficiency and model adaptability.

CN118133006BActive Publication Date: 2026-11-17启元实验室
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410050942.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-12
Publication Date
2026-11-17
Estimated Expiration
2044-01-12

AI Technical Summary

Technical Problem

Existing hybrid motion space optimization methods such as PA-DDPG, PDQN and MPDQN have shortcomings in handling the coupling relationship between discrete motions and continuous parameters, resulting in inaccurate motion selection and poor interpretability, especially in the control of unmanned equipment.

Method used

Discrete actions are generated using a discrete action selection algorithm, and weights are generated for different discrete actions through a hypernetwork. Continuous parameters are then generated based on these weights, and the final action vector is obtained using a continuous action generation algorithm. A policy network and a value network are then constructed to improve the relevance and accuracy of action prediction.

Benefits of technology

It improves the accuracy and interpretability of actions in unmanned equipment control, enhances computational efficiency, and makes the model more flexible in adapting to different discrete actions, while simplifying the amount of data and computational complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118133006B_ABST
    Figure CN118133006B_ABST
Patent Text Reader

Abstract

This invention discloses a training system for an action strategy model. First, a discrete action selection algorithm is used to select discrete actions. Then, a hypernetwork is used to generate weights for different discrete actions based on their state and discrete action vectors. Based on these weights, continuous parameters are generated for the discrete actions. A continuous action generation algorithm is then used to obtain the final action vector from these continuous parameters. This solves the problems of low correlation and poor interpretability in the generation of control information for unmanned equipment, particularly in the generation of mixed action predictions. Different discrete actions generate different feature tensor weights based on the hypernetwork, making the continuous parameters more targeted and improving the accuracy of commands from unmanned control equipment. Compared to setting a dedicated feature extraction layer for each discrete action, this method is simpler, lighter, and requires less data, thus improving the accuracy and reliability of control data for unmanned equipment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, reinforcement learning, and automatic control, and particularly to a training system and method for action policy models. Background Technology

[0002] Typical reinforcement learning environments are usually discrete or continuous action spaces. Discrete action spaces can typically be solved using algorithms such as DQN (Deep Q-Leaning Network) and PPO (Proximal Policy Optimization), while continuous action spaces can be solved using algorithms such as DDPG (Deep Deterministic Policy Gradient) and TD3 (Twin Delayed Deep Deterministic Policy Gradient). Research on hybrid action spaces is relatively scarce, yet they are a very common scenario in current problems. Examples include the selection of discrete actions such as turning, accelerating, and braking, and the selection of continuous parameters such as turning angle, acceleration force, and braking force in autonomous driving problems; and the control problems of unmanned devices such as drones and robotic dogs, which also involve hybrid action spaces, such as forward, backward, and crouching actions, and the selection of parameters such as forward and backward speeds.

[0003] Existing methods for solving the hybrid action space include PA-DDPG, PDQN (Parameterized Deep Q-Network), and MPDQN (an improved version of the Parameterized Deep Q-Network). PA-DDPG follows the network structure of Deep Deterministic Policy Gradient (DDPG), taking the current state (state) as input to the policy network (Actor). This input passes through fully connected layers and a ReLU activation function, outputting a vector of discrete actions and continuous parameters. Then, the current state and the action vector from the previous Actor network are used as input to the value network (Critic), passing through fully connected layers and a ReLU activation function, outputting the specific Q-value corresponding to this action.

[0004] PA-DDPG has a significant problem: its actor part outputs discrete and continuous actions independently, without considering the relationship between the two parts. Following this line of thinking, PDQN provides a new algorithm.

[0005] The current state of PDQN is first used as the input of the Actor network. After passing through some fully connected layers and activation functions, it outputs the optimal vector of all continuous parameters in the action space.

[0006] The current state (state) and the action parameter vector output by the Actor network in the previous step are used as input to the discrete Q-network. After passing through some fully connected layers and activation functions, the output is Q(s,k,x). Note that this is similar to DQN, which outputs the Q values ​​of K discrete actions, where K is the size of the discrete action space. Then, the optimal discrete action is selected by means of argmax and other methods, and the corresponding continuous parameters are selected.

[0007] PDQN takes into account the relationship between action type and action parameters, but it has some hidden problems. For example, different discrete actions correspond to different continuous parameters. In PDQN, an irrelevant continuous parameter may also affect the discrete action type being optimized.

[0008] The aforementioned problems are all caused by the coupling of discrete actions and their incoherent continuous parameters in PDQN design. MPDQN was proposed to solve this problem.

[0009] MPDQN separates the optimization of K discrete actions, decoupling related optimization processes. Each discrete action only inputs its associated parameters. To maintain the consistent length of the parameter vectors for consecutive actions, irrelevant parts are masked. It also utilizes a shared network and parallel computation across multiple discrete actions. Specifically, a discrete network takes as input K tensors (composed of the state and the separated consecutive actions) and outputs a K*K Q-matrix. The first K represents the discrete action, and the second K represents the corresponding continuous parameters. Only the K elements on the diagonal are taken as the true Q-values. Then, when calculating the loss function and actions, the optimal Q-value is obtained through argmax.

[0010] In the algorithms above, PA-DDPG ignores the dependency relationship between continuous and discrete actions, while PDQN and MPDQN, although they take this into account, select discrete actions based on the continuous action values, and their dependency relationship does not conform to the actual logic.

[0011] At the same time, it is not reasonable to use the same weight for different discrete actions. Taking Honor of Kings as an example, the continuous parameters when performing movement operations and when releasing attack skills are often obtained from different logics, and the state parts they focus on are very different. Summary of the Invention

[0012] The purpose of this invention is to provide a training system for an action strategy model. First, a discrete action selection algorithm is used to select discrete actions. Then, a hypernetwork is used to generate weights for different discrete actions based on their states and discrete action vectors. Subsequently, based on these weights, continuous parameters are generated for the discrete actions. Finally, a continuous action generation algorithm is used to obtain the continuous parameters and thus the action vector.

[0013] Thus, this invention solves the problems of low correlation and poor interpretability in the generation of control information for unmanned equipment, thereby improving the accuracy and reliability of control data for unmanned equipment.

[0014] To address the aforementioned problems, a first aspect of the present invention provides a training system for an action policy model, comprising: a policy network and a value network. The policy network includes: a discrete action network, a supernetwork, a continuous action network, and a value network. Wherein:

[0015] The discrete action network is configured to take the current state as input, train the features of the current state, and generate multiple discrete actions corresponding to the current state. The super network is configured to take the current state and multiple discrete actions as input, and train the weights to obtain the weights corresponding to multiple discrete actions for multiple current states.

[0016] The continuous action network is configured to take the current state, multiple discrete actions, and the weights of multiple current states corresponding to multiple discrete actions as inputs, train the feature values ​​of the current state, multiple discrete actions, and the weights of multiple current states corresponding to multiple discrete actions, and generate continuous action parameters for the control of the agent.

[0017] A value network is configured to take the current state as input and obtain long-term value information of the current state, which is used to adjust the parameters during the training of discrete and continuous action networks.

[0018] A second aspect of the present invention provides a training method for an action policy model, comprising: taking the current state as input to a discrete action network, training the features of the current state, and generating a variety of discrete actions corresponding to the current state.

[0019] The current state and various discrete actions are used as inputs to the supernetwork, and the weights for various current states corresponding to various discrete actions are obtained through training.

[0020] The current state, multiple discrete actions, and the weights corresponding to multiple discrete actions in different current states are used as input to a continuous action network. The feature values ​​of these weights are trained to generate continuous action parameters for agent control.

[0021] The current state is used as input to the value network to obtain long-term value information of the current state, which can be used to adjust the parameters during the training of discrete and continuous action networks.

[0022] A third aspect of the present invention provides a method for controlling an agent using an action policy model, wherein the action policy model is trained using a method for training an action policy model according to the present invention, and the method for controlling an agent using an action policy model includes:

[0023] Obtain the agent's current state data. Based on the state data identified using the action policy model, obtain the action trajectory data corresponding to the agent's optimal long-term value information. Generate control information for the agent based on the action trajectory data.

[0024] A fourth aspect of the present invention provides an electronic device comprising: a processor and a memory storing computer instructions for an action policy model and / or computer instructions for processing state data of an agent using the action policy model, wherein when the computer instructions are executed by the processor, the electronic device performs a training method based on the action policy model and / or a method for controlling an agent using the action policy model according to the present invention.

[0025] A fifth aspect of the present invention provides a computer-readable storage medium comprising program instructions for an action policy model and / or program instructions for processing state data of an agent using the action policy model, wherein when the program instructions are executed by a processor, they enable the implementation of a training method based on the action policy model and / or a method for controlling an agent using the action policy model according to the present invention.

[0026] The above-described technical solution of the present invention has the following beneficial technical effects:

[0027] The action strategy model algorithm in this invention has stronger interpretability. Its action variables are generated first as discrete actions, and then their parameters are selected based on the discrete actions, successfully utilizing the correlation between the two. Furthermore, different discrete actions will also generate different feature tensors based on the hypernetwork to calculate weights, making the continuous parameters more targeted. This improves the accuracy of commands for unmanned control equipment.

[0028] Meanwhile, the action strategy model algorithm in this invention has high computational efficiency. It uses a hypernetwork to generate feature weights, making it adaptable to different discrete actions. Compared to calculating weights using fixed feature tensors, this is more flexible, and compared to setting a dedicated feature extraction layer for each discrete action, it is simpler and lighter, significantly reducing the amount of data. This ensures the efficiency of generating unmanned control commands.

[0029] The algorithmic framework of the action policy model in this invention is applicable to reinforcement learning algorithms under all policy network-value network (Actor-Critic) frameworks, and is easy to transfer and develop. This facilitates implementation in various unmanned control systems and makes it easier to integrate with other control systems. Attached Figure Description

[0030] Figure 1 This is a schematic diagram illustrating the composition of a training system for an action strategy model in one embodiment of the present invention.

[0031] Figure 2 This is a diagram illustrating the training algorithm architecture of the action strategy model in one embodiment of the present invention.

[0032] Figure 3 This is a flowchart illustrating a training method for an action strategy model in one embodiment of the present invention.

[0033] Figure 4 This is a network architecture diagram used to illustrate the training algorithm of the action strategy model in one embodiment of the present invention;

[0034] Figure 5 This is an execution flowchart illustrating the training algorithm of the action strategy model in one embodiment of the present invention.

[0035] Figure 6 This is a training data flow diagram used to illustrate the training algorithm of the action strategy model in one embodiment of the present invention.

[0036] Figure 7 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0038] Figure 1 This is a schematic diagram illustrating the composition of a training system for an action strategy model in one embodiment of the present invention. For example... Figure 1 As shown, the first aspect of the present invention provides a training system for an action policy model, comprising: a policy network 101 and a value network 201. The policy network 101 includes: a discrete action network 10, a supernetwork 20, and a continuous action network 30.

[0039] like Figure 1 ,2 As shown, the discrete action network 10 is configured to take the current state as input, train the features of the current state, and generate various discrete actions corresponding to the current state. Discrete Action Network 10: The discrete action (Actor) network 10 receives the current environment state s as input, performs feature extraction, and generates logits corresponding to all possible discrete actions through intermediate layer parameters. These logits represent the score or probability of each action.

[0040] Then, using an appropriate probability distribution, typically a softmax function, these logits are transformed into a probability distribution for each action. Sampling is performed on this probability distribution to obtain the selection of discrete action K, thus enabling appropriate behavioral decisions to be made in the current environment.

[0041] like Figure 1 , 2 As shown, the hypernetwork 20 in this invention is configured to take the current state and multiple discrete actions as input, and train to obtain weights for multiple current states corresponding to multiple discrete actions. In the training algorithm of the action policy model of this invention, the hypernetwork 20 receives the current state S and discrete actions K as input.

[0042] First, the discrete actions K are converted into one-hot encoded forms, where each selected discrete action is transformed into a vector with an action value index corresponding to a value of 1, and all other values ​​being 0, to highlight the selected specific action. Next, the one-hot encoded vector is merged with the state S to form a comprehensive input vector, the length of which is the length of the state features plus the number of discrete actions. This comprehensive input vector is then passed to the supernetwork 20, which may contain multiple layers of neural network structure, including fully connected layers.

[0043] The goal of SuperNetwork 20 is to learn and generate weight coefficients for state and discrete action feature extraction. This algorithm design allows for the fusion of state and action information, generating different weights based on different states and discrete action inputs. Discrete actions are treated as part of the input to the continuous action generation network, effectively utilizing the dependency between discrete and continuous actions and gaining a deeper understanding of situational information.

[0044] like Figure 1 , 2 As shown, the continuous action network 30 is configured to take the current state, multiple discrete actions, and the weights of multiple current states corresponding to multiple discrete actions as inputs, train the feature values ​​of the current state, multiple discrete actions, and the weights of multiple current states corresponding to multiple discrete actions, and generate continuous action parameters for the control of the agent.

[0045] In one embodiment of the present invention, the continuous action network 30, after generating the feature extraction weight matrix of the continuous network using the super network 20, uses the combined input vector of state and discrete action as the input to generate the weight matrix, and uses the weight matrix output by the super network 20 to perform feature processing to obtain the hidden layer vector.

[0046] The hidden layer vectors are then passed to fully connected layers and other neural network structures to further process and transform these features, ultimately outputting the mean and variance of the continuous parameters. The probability distribution is constructed using reparameterization techniques, and the specific continuous action parameters are obtained through sampling.

[0047] like Figure 1 , 2 As shown, the value network 201 is configured to take the current state as input to obtain long-term value information of the current state, which is used to adjust the parameters during training of the discrete action network 10 and the continuous action network 30.

[0048] The value network 201 (Critic) in one embodiment of this invention can receive the current state information of the agent as input, process this state information through a neural network structure, and finally output long-term value information for the current state. This value information is crucial for the agent, as it evaluates the quality of the current state and provides guidance for the training of the policy network.

[0049] The implementation algorithm flow of the training system for the action strategy model in this invention is as follows:

[0050] The state s is first input into the discrete action network 10, which produces a discrete action k. Then, the discrete action k and the state s are combined to form the inputs of the super network 20 and the continuous action network 30.

[0051] First, the inputs are fed into the supernetwork 20 to generate feature extraction weights for the continuous action network 30. Then, the weights are fed into the generated weight matrix for feature extraction, and the continuous actions are calculated. Subsequently, the action vectors are used to interact with the environment, continuously collecting data and storing it in a dynamically long experience replay pool.

[0052] During updates, samples are taken from the experience replay pool, and then the discrete action network, continuous action network, and value network 201 are updated iteratively according to the gradient update method of the policy network-value network (Actor-Critc) architecture. It is worth mentioning that since the two Actor networks select actions according to the policy in the same environment, the updates are also based on the same value network 201, that is, the two networks use the same advantage value to calculate the loss function.

[0053] This algorithm can be used for driving autonomous vehicles or unmanned equipment. It controls upper-level actions, such as acceleration, deceleration, and rotation, by selecting discrete actions. Then, it maps continuous parameters to the domain supported by the selected discrete actions of the device. For example, the steering of an autonomous vehicle can be mapped to the range of [-45°, 45°].

[0054] In another embodiment of the training system for the action policy model of the present invention, the discrete action network 10 is further configured to generate probabilities of multiple discrete actions corresponding to the current state. By utilizing the probabilities of multiple discrete actions, multiple discrete actions are obtained. This results in obtaining high-probability discrete actions, thereby making the data converge more easily during training.

[0055] In another embodiment of the training system for the action strategy model of the present invention, the discrete action network 10 is further configured to convert the probabilities of multiple discrete actions into probability distributions for each action using a normalized exponential function. Sampling is then performed on the probability distributions of each action to obtain multiple discrete actions. This ensures the uniformity and realism of the data distribution, thereby guaranteeing the recognition accuracy of the trained model.

[0056] In another embodiment of the training system for the action policy model of the present invention, the supernetwork 20 is further configured to encode multiple discrete actions using a label encoding method to obtain action codes for multiple discrete actions. The action codes for multiple discrete actions are then combined with the current state to form a combination, which is used as the input to the supernetwork 20.

[0057] In another embodiment of the training system for the action strategy model of the present invention, it further includes:

[0058] The environment interaction model is configured to receive discrete and continuous actions output by discrete action network 10 and continuous action network 30, and generate corresponding reward information.

[0059] The discrete action network 10 is further configured to output the discrete action probabilities corresponding to various discrete actions. The continuous action network 30 is further configured to output the probabilities of continuous parameters corresponding to continuous action parameters.

[0060] In another embodiment of the training system for the action strategy model of the present invention, it further includes:

[0061] The experience replay pool is configured to receive reward information output by the environmental interaction model, discrete action probabilities corresponding to various discrete actions, continuous parameter probabilities corresponding to continuous action parameters, and long-term value information output by the value network 201.

[0062] In another embodiment of the training system for the action strategy model of the present invention, it further includes: a training control unit configured to calculate the advantage value and reward value of the sample data in the experience revisit pool through the proximal policy optimization PPO algorithm.

[0063] Based on various discrete actions and their corresponding discrete action probabilities combined with advantage values, discrete policy gradients are obtained through discrete error calculation methods, and discrete action networks are trained using discrete policy gradients.

[0064] Based on the continuous action parameters and the corresponding probability combined with the advantage value, the continuous policy gradient is obtained through the continuous error calculation method, and the continuous action network is trained through the continuous policy gradient.

[0065] The value gradient is obtained by calculating the value error based on the long-term value information and return value output by the value network 201. The value network 201 is then trained using the value gradient.

[0066] Book Figure 3 This is a flowchart illustrating a training method for an action strategy model in one embodiment of the present invention.

[0067] One embodiment of the present invention provides a training method for an action policy model, wherein the network architecture of the action policy model is as follows: Figure 4 As shown, its action policy model is based on an Actor-Critic network architecture, with a super network 20 added on top to extract continuous network feature tensors based on discrete actions, emphasizing the dependency between discrete and continuous actions in the hybrid action space. Therefore, this network architecture mainly consists of four parts: discrete action network 10, continuous action network 30, super network 20, and value network 201.

[0068] like Figure 3 , 4 As shown, the second aspect of the invention is based on Figure 4 The present invention provides a method for training an action policy model, comprising the following steps:

[0069] Step S101: Use the current state as the input to the discrete action network 10, train the features of the current state, and generate a variety of discrete actions corresponding to the current state.

[0070] Step S102: Use the current state and multiple discrete actions as input to the supernetwork 20 to train and obtain the weights for multiple current states corresponding to multiple discrete actions.

[0071] like Figure 4As shown, the discrete action network 10 and the continuous action network 30 are used to select actions given a state and attempt to optimize long-term rewards. The output of the discrete action network 10 is the logits of each discrete action, and a distribution is generated based on this to sample the discrete action k that interacts with the environment.

[0072] The output of a continuous actor is the mean and variance of continuous parameters. Then, a normal distribution is formed based on these mean and variance, and samples are taken from it to obtain the action parameters x under the discrete action k that interacts with the environment.

[0073] Step S103: The current state, multiple discrete actions, and the weights of multiple current states corresponding to multiple discrete actions are used as inputs to the continuous action network 30 to train the feature values ​​of the current state, multiple discrete actions, and the weights of multiple current states corresponding to multiple discrete actions, thereby generating continuous action parameters for the control of the agent.

[0074] like Figure 4 As shown, the role of Hypernetwork 20 is to generate weights or architectures for other neural networks. It does this by learning to parameterize the weights of other neural networks, rather than directly learning the weights themselves. Hypernetwork 20 takes a low-dimensional vector or parameters as input and then generates weights for building another neural network.

[0075] The methods described above can help dynamically generate network parameters during training, allowing the structure and weights of the neural network to be adjusted and optimized based on input data or tasks. One of its advantages is that it can reduce the number of parameters in the neural network, generating network parameters during the training phase, making the model more flexible and efficient.

[0076] Step S104: The current state is used as the input to the value network 201 to obtain the long-term value information of the current state, which is used to adjust the parameters during the training of the discrete action network 10 and the continuous action network 30.

[0077] like Figure 4 As shown, the value network 201 receives the agent's current state information as input and processes this state information through a neural network structure, ultimately outputting long-term value information for the current state. This value information is crucial for the agent, as it evaluates the quality of the current state and provides guidance for training the policy network.

[0078] Its goal is to estimate the potential future rewards an agent can obtain in the current state. By learning this long-term value estimate, Value Network 201 helps the agent better understand its environment and guides it to make better decisions in different states. This value estimate can serve as an important reference during training, helping the agent optimize its policy network and thus improve its performance in various environments.

[0079] In another embodiment of the training method for the action policy model of the present invention, generating multiple discrete actions corresponding to the current state includes generating probabilities of multiple discrete actions corresponding to the current state. Multiple discrete actions are obtained using the probabilities of the multiple discrete actions.

[0080] In another embodiment of the training method for the action policy model of the present invention, obtaining multiple discrete actions by utilizing the probabilities of multiple discrete actions includes converting the probabilities of multiple discrete actions into probability distributions for each action using a normalized exponential function. Sampling is then performed on the probability distributions of each action to obtain multiple discrete actions.

[0081] In another embodiment of the training method for the action policy model of the present invention, the method further includes: encoding multiple discrete actions through a label encoding method to obtain action codes for multiple discrete actions. The action codes for multiple discrete actions are then combined with the current state to form a combination, which is used as the input to the supernetwork 20.

[0082] In another embodiment of the training method for the action strategy model of the present invention, it further includes:

[0083] Establish an environment interaction model. Through the environment interaction model, receive discrete and continuous actions output by discrete action network 10 and continuous action network 30, and generate corresponding discrete and continuous reward information.

[0084] Generating multiple discrete actions corresponding to the current state includes outputting the discrete action probabilities corresponding to each discrete action. Generating continuous action parameters includes outputting the continuous parameter probabilities corresponding to each continuous action parameter.

[0085] In another embodiment of the training method for the action strategy model of the present invention, it further includes:

[0086] Establish an experience replay pool. This pool receives reward information from the environmental interaction model, discrete action probabilities corresponding to various discrete actions, probabilities of continuous parameters corresponding to continuous action parameters, and long-term value information output by the value network 201.

[0087] In another embodiment of the training method for the action policy model of the present invention, it further includes: calculating the advantage value and reward value of the sample data in the experience revisit pool through the proximal policy optimization (PPO) algorithm.

[0088] Based on various discrete actions and their corresponding discrete action probabilities combined with advantage values, discrete policy gradients are obtained through discrete error calculation methods, and discrete action networks are trained using discrete policy gradients.

[0089] Based on the continuous action parameters and their corresponding probabilities combined with the advantage value, the continuous policy gradient is obtained using a continuous error calculation method. The continuous action network 30 is then trained using this continuous policy gradient. Based on the long-term value information and reward value output by the value network 201, the value gradient is obtained through value error calculation. The value network 201 is then trained using this value gradient.

[0090] Figure 5 This is a flowchart illustrating the execution of the training algorithm for the action strategy model in one embodiment of the present invention:

[0091] First, the state s is obtained from the environment. Then, the state is used as an input tensor and fed into the discrete network to calculate the discrete action k and the corresponding discrete action probability log_prob k. The execution of part 10 of the discrete action network ends.

[0092] The state s is then combined with the discrete action k to form a new input vector, which is fed into the supernetwork 20 to obtain the weights of the feature extraction layer of the continuous action network 30. This new input vector is then fed into the feature extraction layer to obtain the feature vector. Subsequently, it is fed into the backbone network of the continuous action network 30 for further calculations to obtain the continuous action x and the corresponding continuous action probability log_prob x.

[0093] The discrete action k and continuous parameter x are used to interact with the environment to obtain a new state s' and a reward r after the interaction. Then, the state s, discrete action k, discrete action probability log_prob k, continuous parameter x, continuous parameter probability log_prob x, and reward r are sent to the experience replay pool as training data for use during the training phase.

[0094] Figure 6 This is a training data flow graph used to illustrate the training algorithm of the action policy model in one embodiment of the present invention. Taking the PPO algorithm in the policy network-value network (Actor-Critic) architecture as an example, the advantage value and reward value of the sample data are first calculated from the experience replay pool using the GAE (Generalized Advantage Estimation) technique.

[0095] The GAE technique is an evaluation technique used in reinforcement learning, which mainly addresses the trade-off between bias and variance when calculating advantage estimation. GAE uses a parameter lambda (commonly denoted by λ, representing the decay factor) combined with the discounted cumulative reward of the current state and a value function-based prediction to calculate advantage estimation. Its formula is shown in equation (1):

[0096]

[0097] Where δ represents the TD error, which is the error between the current time estimate and the previous time estimate plus the current time reward. Its calculation formula (2) is as follows:

[0098] δ t =r+γV(t+1)-V(t) Formula (2)

[0099] γ represents the discount factor in reinforcement learning; a larger γ value indicates a greater emphasis on future rewards. The reward value is primarily the data that the value network 201 needs to fit; it represents the expected future reward of the current state, and its formula (3) is:

[0100]

[0101] Where V(t) represents the value function estimate at time t, i.e., the cumulative reward expected to be obtained in the state at time t. A(t) is the advantage function, representing the additional gain relative to the value function after taking a specific action.

[0102] The gradient calculation of value network 201 can be calculated based on the MSE loss of the estimated value and the return value, and its formula (4) is as follows:

[0103]

[0104] The gradient calculation of the Actor network also obtains the actions and probabilities from the experience replay pool, and then uses the PPO policy gradient update formula to update the gradient. The calculation formula (5) is as follows:

[0105]

[0106] in, This is the ratio of the new and old strategies in importance sampling, reflecting the degree of deviation between them. The clip operation filters out samples with a deviation greater than a threshold, as they offer little benefit to updating the current strategy and may even harm it.

[0107] It should be noted that the above formulas need to be used to calculate both discrete and continuous policies. Although the discrete action k is used as input to participate in the calculation of the continuous action, it is combined with the non-differentiable state vector s before being used as input. Therefore, if only the gradient of the continuous policy is calculated, its gradient cannot be propagated to the discrete action network 10, that is, the discrete action network 10 cannot be updated.

[0108] Therefore, the discrete policy gradient must be calculated and then backpropagated to affect the discrete action network 10. The diagram uses different colored lines to represent the networks affected by gradient propagation, making the network update process clearer. It can be seen that the value gradient only updates the value network 201, the discrete policy gradient only affects the discrete action network 10, while the continuous policy gradient affects both the super network 20 and the continuous network.

[0109] The advantage value used in continuous policy gradient calculation is the same as that used in discrete policy gradient calculation, because the advantage value represents the payoff under the current action choice, which is composed of both discrete and continuous actions.

[0110] A third aspect of the present invention provides a method for controlling an agent using an action policy model, wherein the action policy model is trained using a method for training an action policy model according to the present invention, and the method for controlling an agent using an action policy model includes:

[0111] Obtain the agent's current state data. Based on the state data identified using the action policy model, obtain the action trajectory data corresponding to the agent's optimal long-term value information. Generate control information for the agent based on the action trajectory data.

[0112] In another embodiment of the method for controlling an intelligent agent using an action strategy model according to the present invention, the intelligent agent includes: a drone and an autonomous vehicle. The continuous action parameters of the drone and the autonomous vehicle include: acceleration, deceleration, steering, and position information. The continuous action parameters of the drone include: the drone's flight direction information and flight altitude information.

[0113] A fourth aspect of the present invention provides an electronic device comprising: a processor and

[0114] A memory that stores computer instructions for an action policy model and / or computer instructions for processing state data of an agent using the action policy model, which, when executed by a processor, cause an electronic device to perform a training method based on the action policy model and / or a method for controlling an agent using the action policy model according to the present invention.

[0115] A fifth aspect of the present invention provides a computer-readable storage medium comprising program instructions for an action policy model and / or program instructions for processing state data of an agent using the action policy model, wherein when the program instructions are executed by a processor, they enable the implementation of a training method based on the action policy model and / or a method for controlling an agent using the action policy model according to the present invention.

[0116] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the action policy model training method in the embodiments of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, they execute the action policy model training method and the method of controlling an intelligent agent using the action policy model in any of the above method embodiments.

[0117] A non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required for at least one function; and the data storage area may store data created by the use of a processing device based on a training method according to an action policy model and a method for controlling an agent using the action policy model.

[0118] In addition, non-volatile computer-readable storage media may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0119] In some implementations, the non-volatile computer-readable storage medium may optionally include memory remotely configured relative to the processor. This remote memory can be connected via a network to a training method for the action policy model and a processing apparatus for a method that uses the action policy model to recognize language. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0120] The present invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform the training method of any of the above-mentioned action strategy models.

[0121] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of the present invention, such as... Figure 4 As shown, the device includes: one or more processors 710 and memory 720. Figure 7Taking a processor 710 as an example, the device for training the action policy model may further include an input device 730 and an output device 740. The processor 710, memory 720, input device 730, and output device 740 can be connected via a bus or other means.

[0122] Figure 7 Taking a bus connection as an example, the memory 720 is the aforementioned non-volatile computer-readable storage medium. The processor 710 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 720, thereby implementing the training method for the action strategy model described above. The input device 730 can receive input digital or character information and generate key signal inputs related to user settings and function control of the information delivery device. The output device 740 may include a display screen or other display device.

[0123] The above-described product can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0124] In one implementation, the above-described electronic device can be applied to an encryption / decryption platform, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable a training method processor for at least one action policy model to:

[0125] The current state is used as input to a discrete action network to train the features of the current state and generate various discrete actions corresponding to the current state.

[0126] The current state and various discrete actions are used as inputs to the supernetwork, and the weights for various current states corresponding to various discrete actions are obtained through training.

[0127] The current state, multiple discrete actions, and the weights corresponding to multiple discrete actions in different current states are used as input to a continuous action network. The feature values ​​of these weights are trained to generate continuous action parameters for agent control.

[0128] The current state is used as input to the value network to obtain long-term value information of the current state, which can be used to adjust the parameters during the training of discrete and continuous action networks.

[0129] The electronic device according to embodiments of the present invention exists in various forms, including but not limited to:

[0130] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, and feature phones.

[0131] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, etc.

[0132] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0133] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0134] (5) Other electronic devices with data interaction functions.

[0135] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of each embodiment or some parts of the embodiments.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training system for an action strategy model, characterized in that, It includes: Policy network, value network, and training control unit; wherein: The policy network includes: A discrete action network is configured to take the current state as input, train the features of the current state, and generate a variety of discrete actions corresponding to the current state. The discrete actions include acceleration, deceleration, and rotation, for use in driving autonomous vehicles or unmanned equipment. A hypernetwork, configured to take the current state and the various discrete actions as input, is trained to obtain weights for various current states corresponding to various discrete actions; and A continuous action network is configured to generate a feature extraction weight matrix for the continuous network based on the super network. The comprehensive input vector of the various current states and their corresponding discrete actions is used as the input of the feature extraction weight matrix generated by the super network. Feature processing is performed using the weight matrix output by the super network to obtain hidden layer vectors. The hidden layer vectors are passed to the neural network structure to output the mean and variance of the continuous parameters. A probability distribution is constructed using reparameterization techniques, and continuous action parameters are obtained by sampling. The neural network structure includes fully connected layers. The hypernetwork is further configured to encode the multiple discrete actions using an identifier encoding method to obtain action codes for the multiple discrete actions; The action codes of the various discrete actions are combined with the current state to form a combination, which is then used as the input of the supernetwork. The value network is configured to take the current state as input to obtain long-term value information of the current state, which is then used to adjust the parameters during training of the discrete action network and the continuous action network; and The training control unit is configured to optimize the advantage and reward values ​​of sample data in the experience replay pool by using a proximal strategy to calculate the PPO. Based on the various discrete actions and their corresponding discrete action probabilities, combined with the advantage value, a discrete policy gradient is obtained using a discrete error calculation method, and the discrete action network is trained using the discrete policy gradient. Based on the continuous action parameters and the corresponding probabilities of the continuous parameters, combined with the advantage value, the continuous policy gradient is obtained through the continuous error calculation method, and the continuous action network is trained through the continuous policy gradient. The value gradient is obtained by calculating the long-term value information output by the value network and the return value through value error; the value network is then trained using the value gradient.

2. The training system according to claim 1, characterized in that, The discrete action network is further configured to generate probabilities of multiple discrete actions corresponding to the current state; and to obtain multiple discrete actions using the probabilities of the multiple discrete actions.

3. The training system according to claim 2, characterized in that, The discrete action network is further configured to convert the probabilities of the various discrete actions into probability distributions of each action through a normalized exponential function; and to sample the probability distributions of each action to obtain various discrete actions.

4. The training system according to claim 1, characterized in that, Also includes: An environmental interaction model is configured to receive discrete and continuous actions output by the discrete action network and the continuous action network, and generate corresponding reward information. The discrete action network is further configured to output the discrete action probabilities corresponding to the various discrete actions; The continuous action network is further configured to output the probability of the continuous parameter corresponding to the continuous action parameter.

5. The training system according to claim 4, characterized in that, Also includes: The experience replay pool is configured to receive reward information output by the environment interaction model, discrete action probabilities corresponding to the various discrete actions, probabilities of continuous parameters corresponding to the continuous action parameters, and long-term value information output by the value network.

6. A method for training an action strategy model, used to control the system as described in any one of claims 1-5, characterized in that, It includes: The current state is used as input to a discrete action network to train the features of the current state and generate a variety of discrete actions corresponding to the current state. The current state and the various discrete actions are used as inputs to the supernetwork to train and obtain the weights for various current states corresponding to various discrete actions. The current state, the various discrete actions, and the weights of the various current states corresponding to the various discrete actions are used as inputs to a continuous action network. The feature values ​​of the current state, the various discrete actions, and the weights of the various current states corresponding to the various discrete actions are trained to generate continuous action parameters for the control of the agent. as well as The current state is used as input to the value network to obtain long-term value information of the current state, which is then used to adjust the parameters during training of the discrete action network and the continuous action network.

7. The training method according to claim 6, characterized in that, Generating multiple discrete actions corresponding to the current state includes generating the probability of multiple discrete actions corresponding to the current state; and using the probability of the multiple discrete actions to obtain multiple discrete actions.

8. The training method according to claim 7, characterized in that, Obtaining multiple discrete actions by utilizing the probabilities of the multiple discrete actions includes converting the probabilities of the multiple discrete actions into probability distributions for each action using a normalized exponential function; and sampling on the probability distributions of each action to obtain multiple discrete actions.

9. The training method according to claim 6, characterized in that, Also includes: The various discrete actions are encoded using an identifier encoding method to obtain the action codes for the various discrete actions; The action codes of the various discrete actions are combined with the current state to form a combination, which is then used as the input of the supernetwork.

10. The training method according to claim 6, characterized in that, Also includes: Establish an environmental interaction model; The environmental interaction model receives discrete and continuous actions output by the discrete action network and the continuous action network, and generates corresponding reward information. Generating multiple discrete actions corresponding to the current state includes outputting the discrete action probabilities corresponding to the multiple discrete actions; Generating continuous action parameters includes outputting the probability of the continuous parameter corresponding to the continuous action parameter.

11. The training method according to claim 10, characterized in that, Also includes: Establish an experience replay pool; The system receives reward information output by the environmental interaction model, discrete action probabilities corresponding to the various discrete actions, probabilities of continuous parameters corresponding to the continuous action parameters, and long-term value information output by the value network through the experience replay pool.

12. The training method according to claim 11, characterized in that, Also includes: The advantage and reward values ​​of the sample data in the experience replay pool are calculated by optimizing the PPO algorithm using a near-end strategy. Based on the various discrete actions and their corresponding discrete action probabilities, combined with the advantage value, a discrete policy gradient is obtained using a discrete error calculation method, and the discrete action network is trained using the discrete policy gradient. Based on the continuous action parameters and the corresponding probabilities of the continuous parameters, combined with the advantage value, the continuous policy gradient is obtained through the continuous error calculation method, and the continuous action network is trained through the continuous policy gradient. The value gradient is obtained by calculating the long-term value information output by the value network and the return value through value error; the value network is then trained using the value gradient.

13. A method for controlling an intelligent agent using an action policy model, characterized in that, The action policy model is trained using the method according to any one of claims 6 to 12, characterized in that the method of controlling the agent using the action policy model includes: Obtain the current state data of the agent; Based on the state data of the agent identified by the action strategy model, the action trajectory data corresponding to the agent's optimal long-term value information is obtained. The control information for the intelligent agent is generated based on the motion trajectory data.

14. An electronic device, characterized in that, include: processor; as well as A memory storing computer instructions for an action policy model and / or computer instructions for processing state data of an agent using the action policy model, wherein when the computer instructions are executed by the processor, the electronic device performs the method according to any one of claims 6 to 12 and / or the method according to claim 13.

15. A computer-readable storage medium, characterized in that, The program instructions include an action policy model and / or program instructions that process the agent's state data using the action policy model, which, when executed by a processor, cause the implementation of the method according to any one of claims 6 to 12 and / or the method according to claim 13.

Citation Information

Patent Citations

  • Hybrid power system energy management method based on P-DQN algorithm

    CN115099148A

  • Strategy gradient multi-agent reinforcement learning method based on optimistic projection

    CN116306898A