A deep reinforcement learning method for intelligent agents based on partner networks

Abstract

The present invention discloses a deep reinforcement learning method for an intelligent agent based on a partner network, comprising constructing a guidance signal function under different reward environments according to the control state of the intelligent agent, generating the experience tuple required by the partner network and saving it to the experience replay of the partner network; constructing a partner network model using a multi-layer long short-term memory network, calculating the guidance signal corresponding to the intelligent agent according to the historical control state and optional actions, and updating the partner network parameters; performing attenuation processing on the guidance signal corresponding to the intelligent agent; constructing a deep Q network reinforcement learning model based on a multi-baseline model, fusing the attenuated guidance signal with a set decision signal, and then training the deep Q network reinforcement learning model, generating the experience tuple required by the Q network and saving it to the experience replay of the intelligent agent, and updating the Q network parameters. The present invention reduces the action errors of the intelligent agent in the late exploration stage and improves the exploration stability of the intelligent agent by establishing a guidance signal that the intelligent agent can perceive.

Description

Technical Field

[0001] The present invention relates to, and in particular to, a deep reinforcement learning method for an intelligent agent based on a partner network. Background Art

[0002] Deep reinforcement learning, an adaptive artificial intelligence algorithm, has been widely used in fields such as autonomous driving, financial market trading, and intelligent robotic control. However, the trade-off between exploration and exploitation is a major challenge when using deep reinforcement learning algorithms. Many algorithms use epsilon-greedy methods to balance exploration and exploitation.

[0003] When applying this epsilon-greedy approach, the agent is more likely to choose random action values ​​in the early stages of exploration, allowing it to explore all possible environmental states and learn the optimal action-value function for all states. As exploration time passes, the randomness of the random policy gradually decreases. In the later stages of exploration (when the agent's reward fluctuates little over a long number of interaction episodes, i.e., when the reward reaches stability or the exploration period meets the task requirements), the agent relies more on its own experience to select action values, allowing it to fully utilize the learned action-value function. However, when the state dimension is large, the states are continuous variables, and exploration time is limited, it is difficult for the agent to fully explore all states. This means that in the later stages of exploration, the agent may still encounter new states. In this case, the agent is more likely to choose the learned policy. However, this policy may not be suitable for the new state, easily disrupting the stability of the agent's reward. In reinforcement learning algorithm implementations, researchers monitor the agent's reward to determine whether the algorithm has converged and succeeded. Therefore, unstable rewards are extremely detrimental to the convergence and application of the algorithm. In particular, in the fields of unmanned driving and intelligent robot control, if the intelligent agent cannot obtain stable rewards in the later stages of exploration, or if the intelligent agent cannot explore stably in the later stages of exploration, the risks caused are fatal.

[0004] Because deep reinforcement learning algorithms typically require long periods of exploration and training, and epsilon-greedy algorithms can lead to local optimality, many existing improved deep reinforcement learning techniques aim to add prior experience or improve epsilon-greedy strategies to improve algorithm exploration efficiency and the rewards obtained by the agent.

[0005] However, existing technologies often employ methods to enable agents to explore more and more efficiently in unknown environments during exploration, reducing the likelihood of encountering unknown environments in the later stages of exploration, thereby avoiding local optima or improving exploration efficiency. However, these methods cannot guarantee that new states will not appear in the later stages of exploration, nor that reward stability will be disrupted. Summary of the Invention

[0006] In response to the above-mentioned deficiencies in the prior art, the present invention provides a deep reinforcement learning method for an intelligent agent based on a partner network. By establishing guidance signals that the intelligent agent can perceive, it can reduce the action errors of the intelligent agent in the later stage of exploration and improve the exploration stability of the intelligent agent.

[0007] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0008] A deep reinforcement learning method for an intelligent agent based on a partner network, comprising the following steps:

[0009] S1. Construct guidance signal functions under different reward environments based on the agent's control state, generate the experience tuples required by the partner network, and save them to the experience replay of the partner network;

[0010] S2. Use a multi-layer long short-term memory network to build a partner network model, calculate the guidance signal corresponding to the agent based on the historical control state and optional actions, and update the partner network parameters;

[0011] S3, performing attenuation processing on the guidance signal corresponding to the intelligent agent;

[0012] S4. Construct a deep Q-network reinforcement learning model based on a multi-baseline model, fuse the attenuated guidance signal with the set decision signal, and then train the deep Q-network reinforcement learning model. Generate the experience tuples required by the Q-network and save them to the experience replay of the intelligent agent, and update the Q-network parameters.

[0013] Optionally, step S1 specifically includes the following sub-steps:

[0014] S11. Set the reward environment according to the agent's control state and construct the guidance signal function under different reward environments;

[0015] S12, determine whether an episode of interaction has ended; if so, execute step S13; otherwise, execute step S14;

[0016] S13, control state s at time step t t Optional action a t and the instantaneous reward r t+1 Generate experience tuple (s t ,a t ,r t+1 ) Save to temporary experience replay pool Execute step S2;

[0017] S14, for all time steps of the previous episode Calculate the corresponding target guidance signals respectively Execute step S15;

[0018] S15, control state s t′ Optional action a t′ , instantaneous reward r t+1 and the target guidance signal calculated in step S14 Generate experience tuples Experience replay saved to partner network Execute step S2.

[0019] Optionally, the guidance signal function under different reward environments constructed in step S11 is specifically:

[0020]

[0021] in, represents the target guidance signal at time step t, e represents the number of interaction steps in an episode, t represents the time step, j represents the temporary time step ranging from t to e, and r j represents the reward obtained by the agent at time step j, r t Indicates the control state s t Transfer to control state s t+1 The instantaneous reward obtained by the agent.

[0022] Optionally, step S2 specifically includes the following sub-steps:

[0023] S21. Use a two-layer long short-term memory network to build a partner network model;

[0024] S22, the sequential action tuple (s t-l+1 :s t ) and all optional actions Input the partner network model to obtain the guidance signal c t ;

[0025] S23. Experience Replay for Judging Partner Networks Is it greater than the batch update threshold B? C ; If yes, proceed to step S24; otherwise, no processing;

[0026] S24. Minimizing the mean square error between the network output and the target guidance signal is used as the target loss function of the partner network, and the batch gradient descent algorithm is used to update the partner network parameters.

[0027] Optionally, in step S24, minimizing the mean square error between the network output and the target guidance signal as the target loss function of the partner network is specifically:

[0028]

[0029] in, represents the objective loss function of the partner network, represents the expectation function of the partner network, represents the target guidance signal at time step t, c t represents the predicted guidance signal of the partner network at time step t.

[0030] Optionally, in step S24, the update formula for updating the partner network parameters using the batch gradient descent algorithm is specifically:

[0031]

[0032] Among them, ω′ represents the updated partner network parameters, ω represents the partner network parameters, α C represents the learning rate of the partner network, Represents the loss function L about the parameter ω C The gradient of L C represents the loss function of the partner network.

[0033] Optionally, step S3 specifically includes the following sub-steps:

[0034] S31. At the beginning of an interaction, initialize the decay factor λ = 1 and the number of interaction steps k = 0;

[0035] S32, the guidance signal c t Multiply by the attenuation factor λ to obtain the attenuated guidance signal c′ t ;

[0036] S33, update the attenuation factor to λ′=λ k+1 , the number of interaction steps is updated to k′=k+1.

[0037] Optionally, step S4 specifically includes the following sub-steps:

[0038] S41. Use multiple companion learning networks as baseline models to build a deep Q-network reinforcement learning model.

[0039] S42, fusing the attenuated guidance signal with the set decision signal to obtain a fused decision signal;

[0040] S43, generating an optimal action decision based on the fused decision signal according to the optimal action decision function;

[0041] S44, to control the state s t , the attenuated guidance signal c′ t Optional action a t , instantaneous reward r t+1 , control state s t+1 , the attenuated guidance signal c′ t+1And the interaction mark done generates the experience tuple (S t ,c′ t ,a t ,r t+1 ,s t+1 ,c′ t+1 ,done) Save to Q network experience replay

[0042] S44. Judging the Experience Replay of the Q Network Is it greater than the batch update threshold B? Q ; If yes, execute step S45; otherwise, do nothing;

[0043] S45. Minimize the mean square error between the evaluation network output and the target action value as the target loss function of the Q network, and use the batch gradient descent algorithm to update the Q network parameters.

[0044] Optionally, in step S45, minimizing the mean square error between the network output and the target action value as the target loss function of the Q network is specifically:

[0045]

[0046] Among them, L Q represents the target loss function of the Q network, represents the expected function of the Q network, Y t represents the target action value of the partner network, Q(s t ,a t ,c t ; θ) represents the fused decision signal.

[0047] Optionally, in step S45, the update formula for updating the Q network parameters using the batch gradient descent algorithm is specifically:

[0048]

[0049] Among them, θ′ represents the updated Q network parameters, θ represents the Q network parameters, α Q represents the learning rate of the Q network, Find the target loss function L with respect to the parameter θ Q The gradient of L Q represents the objective loss function of the Q network.

[0050] The present invention has the following beneficial effects:

[0051] (1) This paper, drawing on the human companionship learning model, constructs a deep reinforcement learning framework that can accompany intelligent agents in their learning. This framework mimics the process of human companionship and growth, and designs a time-varying, short-interval, full-sample learning, and continuously attenuated partner network to act as an "education expert" to accompany and guide the agent's exploration. The guidance signals from the partner network can effectively alleviate the unstable behavior of the agent, reduce the convergence and application difficulty of the deep reinforcement learning algorithm, reduce the agent's action errors in the later stages of exploration, and improve the agent's exploration stability.

[0052] (2) This invention exploits the fact that different deep reinforcement learning algorithms have different applicability in different environments. For example, RND and DQN+SR algorithms are more conducive to agent exploration in sparse reward environments, while algorithms such as DQN, DDQN, and Dueling DQN are more suitable for dense reward environments. Therefore, the partner network proposed in this invention can be loaded into many different baseline algorithms, enabling the agent to have more stable exploration results in the later stages of exploration and adapt to many different environments based on the selected baseline. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of a deep reinforcement learning method for an intelligent agent based on a partner network according to an embodiment of the present invention;

[0054] Figure 2 Schematic diagram of a deep reinforcement learning framework for an intelligent agent based on a partner network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0056] like Figure 1 and 2 As shown, the embodiment of the present invention provides a deep reinforcement learning method for an intelligent agent based on a partner network, comprising the following steps S1 to S4:

[0057] S1. Construct guidance signal functions under different reward environments based on the agent's control state, generate the experience tuples required by the partner network, and save them to the experience replay of the partner network;

[0058] In an optional embodiment of the present invention, step S1 specifically includes the following sub-steps:

[0059] S11. Set the reward environment according to the agent's control state and construct the guidance signal function under different reward environments;

[0060] S12, determine whether an episode of interaction has ended; if so, execute step S13; otherwise, execute step S14;

[0061] S13, control state s at time step t t Optional action a t and the instantaneous reward r t+1 Generate experience tuple (S t ,a t ,r t+1 ) Save to temporary experience replay pool Execute step S2;

[0062] S14, for all time steps of the previous episode Calculate the corresponding target guidance signals respectively Execute step S15;

[0063] S15, control state s t′ Optional action a t′ , instantaneous reward r t+1 and the target guidance signal calculated in step S14 Generate experience tuples Experience replay saved to partner network Execute step S2.

[0064] Specifically, this embodiment constructs different, perceptible guidance signal functions for the agent in sparse reward environments and dense reward environments. The guidance signal functions constructed in different reward environments are specifically:

[0065]

[0066] in, represents the target guidance signal at time step t, e represents the number of interaction steps in an episode, t represents the time step, j represents the temporary time step ranging from t to e, and r j represents the reward obtained by the agent at time step j, r t Indicates the control state s t Transfer to control state s t+1 The instantaneous reward obtained by the agent. The larger the value, the more likely the agent chooses action a. t The more suitable.

[0067] Since the rewards and actions obtained by the agent are related to the control state transfer, in order to enable the agent to perceive the guidance signal, the present invention converts the target guidance signal Y CDefined as a function of reward. In particular, in environments with sparse rewards or where the rewards are almost always the same, the paired effects of control state and action are often not reflected until the end of an episode. For example, in an environment where the agent scores only when it moves to the end point and does not score at other positions. Therefore, this embodiment designs guidance signal functions for two different situations: dense rewards and sparse rewards. For environments with sparse rewards, after the end of an episode, the long-term cumulative average reward after time t is calculated. For environments with dense rewards, the reward itself represents the paired effect of control state and action.

[0068] Taking the control of a landing robot as an example, the positive reward is that the robot stops at the specified position, and the negative reward includes that the robot is far away from the specified position, the engine is not completely shut down when the robot stops, the robot stops in the wrong posture, resulting in damage to the robot, and the robot operates the engine in the air. Starting the main engine will get a reward of -0.3, and starting the side engine will get a reward of -0.03. The state space of this example consists of 8 continuous variables: the position and speed of the landing robot on the vertical plane, the angle and angular velocity of the landing robot, and whether the legs on each side are in contact with the ground. The landing robot can take 4 actions: start the left engine, start the main engine, start the right engine, or take no action. An episode ends when the landing robot crashes or stops. This embodiment 1 is a dense reward embodiment, and the target guidance signal is expressed by r t express.

[0069] Taking autonomous driving as an example, a positive reward is when the autonomous vehicle reaches the designated location, and a negative reward is when the autonomous vehicle fails to reach the designated location. The state space of this example consists of two continuous variables: the position and speed of the autonomous vehicle. The autonomous vehicle can take three actions: drive left, drive right, or not use the engine. This embodiment 2 is an embodiment of sparse rewards, and the target guidance signal is used express.

[0070] S2. Use a multi-layer long short-term memory network to build a partner network model, calculate the guidance signal corresponding to the agent based on the historical control state and optional actions, and update the partner network parameters;

[0071] In an optional embodiment of the present invention, step S2 specifically includes the following sub-steps:

[0072] S21. Use a two-layer long short-term memory network to build a partner network model;

[0073] S22, the sequential action tuple (s t-l+1 :s t ) and all optional actions Input the partner network model to obtain the guidance signal Where (x l-p:x l ) means from x l-p to x l A set of sequential values. For example, (a t-l+1 :a i ) means {a t-l+1 , a t-l+2 ,…,a t-1 ,ai},(s t-l+1 :s t ) means {s t-l+1 , s t-l+2 ,…,s t-1 , s t}.

[0074] S23. Experience Replay for Judging Partner Networks Is it greater than the batch update threshold B? C ; If yes, proceed to step S24; otherwise, no processing;

[0075] S24. Minimizing the mean square error between the network output and the target guidance signal is used as the target loss function of the partner network, and the batch gradient descent algorithm is used to update the partner network parameters.

[0076] The objective loss function of the partner network is to minimize the mean square error between the network output and the target guidance signal:

[0077]

[0078] in, represents the objective loss function of the partner network, represents the expectation function of the partner network, represents the target guidance signal at time step t, c t represents the predicted guidance signal of the partner network at time step t.

[0079] The update formula for updating the partner network parameters using the batch gradient descent algorithm is as follows:

[0080]

[0081] Among them, ω′ represents the updated partner network parameters, ω represents the partner network parameters, α C represents the learning rate of the partner network, Represents the loss function L about the parameter ω C The gradient of L C represents the loss function of the partner network.

[0082] Specifically, this embodiment uses a two-layer long short-term memory network (LSTM) to build a partner network model. The input of the network is the historical control state and action, and the output is the predicted guidance signal c t LSTM adds three gates to RNN, namely forget gate, input gate and output gate, which can learn the long-term dependency of time series and make the partner network have long-term memory and guidance capabilities. t , a i ), the output of LSTM is as follows:

[0083] f t =σ(W f ·[h t-1 ,(s T , a i )]+b f )

[0084] u t =σ(W u ·[h t-1 ,(s T , a i )]+b u )

[0085] o t =σ(W o ·[h t-1 ,(s T , a i )]+b O )

[0086] m t =f t ⊙m t-1 +u t ⊙tanh(W m ·[h t-1 ,(s t , a i )]+b m )

[0087] h t =o t ⊙tanh(m t )

[0088] Among them, W and b represent the transfer matrix and offset respectively, f t Indicates discarding useless information in the past cell state, u t Indicates how much new information is added to the current cell state, o t Output information that determines the unit state, m tRepresents the cell activation vector of the current cell state, σ represents the sigmoid activation function, h t represents the hidden layer state at time t, ⊙ represents the multiplication of the corresponding elements, W f represents the weight of the forget gate, b f Represents the forget gate bias, W u represents the input gate weight, b u Represents the input gate bias, W o represents the output gate weight, b o Represents the output gate bias, W m represents the activation vector weight, b m represents the activation vector bias.

[0089] The sequential action-state tuple (s t-l+1 :s t , a t-l+1 :a i ) The output signal h′ is obtained through a two-layer LSTM network t After that, we get a i For state s t Impact as follows:

[0090]

[0091] Just a i With s t The guidance signals for all optional actions need to be calculated.

[0092] Taking the control of a landing robot as an example, the input of the partner network is an 8-dimensional historical state of 4 time steps and a one-dimensional optional action.

[0093] Taking autonomous driving as an example, the input of the partner network is a 2D historical state of 4 time steps and a 1D optional action.

[0094] The goal of the partner network is to minimize the mean square error between the network output and the target guidance signal, and the loss function is Defined as:

[0095]

[0096] in, represents the objective loss function of the partner network, represents the expectation function of the partner network, represents the target guidance signal at time step t, c t represents the predicted guidance signal of the partner network at time step t.

[0097] In order to maintain the high quality of the guidance signal of the partner network in the local interval, the present invention is a small-scale (only a few consecutive interaction sets) experience replay. The partner network is optimized using batch gradient descent algorithm. This method can make the partner network in a local continuous interval. Global learning, becoming an "education expert" with high-quality guidance capabilities. Changing as the exploration process changes, this enables the partner network to become "education experts" in different exploration intervals.

[0098] The update rule of the partner network parameter ω is:

[0099]

[0100] The control state and the guidance signal of each optional action can be obtained through the above steps.

[0101] S3, performing attenuation processing on the guidance signal corresponding to the intelligent agent;

[0102] In an optional embodiment of the present invention, similar to the human learning process, the agent requires greater guidance in the early stages of exploration. Later in the exploration, the agent's own experience is more extensive than that of the partner network, and less guidance is needed. Therefore, this embodiment attenuates the guidance signal output by the partner network.

[0103] Step S3 specifically includes the following sub-steps:

[0104] S31. At the beginning of an interaction, initialize the decay factor λ = 1 and the number of interaction steps k = 0;

[0105] S32, predict the guidance signal c t Multiply by the attenuation factor λ to obtain the attenuated guidance signal

[0106] S33, the attenuation factor decreases exponentially, and the attenuation factor is updated to λ′=λ k+1 , the number of interaction steps is updated to k′=k+1.

[0107] S4. Construct a deep Q-network reinforcement learning model based on a multi-baseline model, fuse the attenuated guidance signal with the set decision signal, and then train the deep Q-network reinforcement learning model. Generate the experience tuples required by the Q-network and save them to the experience replay of the intelligent agent, and update the Q-network parameters.

[0108] In an optional embodiment of the present invention, step S4 specifically includes the following sub-steps:

[0109] S41. Use multiple companion learning networks as baseline models to build a deep Q-network reinforcement learning model.

[0110] In this embodiment, with reference to DQN (Deep Q-Networks), DDQN (Double DQN), Dueling DQN, DQN+RND (Random Network Distillation) and DQN+SR (Successor Representation) networks, five companion learning networks, CDQN (Companion DQN), Double CDQN (Double Companion DQN), Dueling CDQN (Dueling Companion DQN), CDQN+RND (Companion DQN+RandomNetwork Distillation) and CDQN+SR (Companion DQN+Successor Representation), are proposed to construct a deep Q network reinforcement learning model based on five baseline models.

[0111] Specifically, if Figure 2 As shown, this embodiment selects five network models: DQN, DDQN, Dueling DQN, DQN+RND, and DQN+SR as baseline models. After loading the partner network, these baseline models become CDQN, CDDQN, DuelingCDQN, CDQN+RND, and CDQN+SR, respectively. The Q network of the five baseline models consists of four fully connected layers (Dense), a tensor concatenation layer (Concatenate), and a tensor summation layer (Add). The random network of DQN+RND consists of three fully connected layers, with 256, 256, and 64 neural units, respectively. The ratio of internal reward to external reward of DQN+RND is 1:1. The Succesor Representation Network of DQN+SR consists of three fully connected layers, with 64, 128, and 256 neural units, respectively. The input of the prediction network of DQN+SR is the output of the Succesor Representation Network and the current action. The two inputs pass through a fully connected layer with 128 neurons, are then element-wise multiplied, and then pass through a fully connected layer with 64 neurons, and finally pass through a fully connected layer to output the next state. The ratio of the external reward to the internal reward of DQN+SR is 1:0.25, and the internal reward is calculated by the inverse of the L2 norm of the subsequent feature of the current state. *Represents a fully connected layer of a neural network, Add represents a tensor addition layer of a neural network, and Concatenate represents a tensor concatenation layer of a neural network.

[0112] S42, fusing the attenuated guidance signal with the set decision signal to obtain a fused decision signal;

[0113] In this embodiment, the decision signal of the Q learning method without a guidance signal is q t Indicates that the attenuated guidance signal c′ t With q t Fusion is performed to obtain the fused decision signal Q(s t ,a t ,c t ; θ), expressed as:

[0114] Q(s t ,a t ,c t ;θ)=q t +c′ t

[0115] Where θ represents the Q network parameter.

[0116] S43, generating an optimal action decision based on the fused decision signal according to the optimal action decision function;

[0117] In this embodiment, the optimal action decision function is expressed as:

[0118]

[0119] Among them, ε represents the probability of randomly selecting an action under the ε-greedy exploration strategy.

[0120] S44, to control the state s t , the attenuated guidance signal c′ t Optional action a t , instantaneous reward r t+1 , control state s t+1 , the attenuated guidance signal c′ t+1 And the interaction mark done generates the experience tuple (s t , c′ t , a t , r t+1 , s t+1 , c′ t+1 , done) save to the Q network experience replay

[0121] S44. Judging the Experience Replay of the Q Network Is it greater than the batch update threshold B?Q ; If yes, execute step S45; otherwise, do nothing;

[0122] S45. Minimize the mean square error between the evaluation network output and the target action value as the target loss function of the Q network, and use the batch gradient descent algorithm to update the Q network parameters.

[0123] In this embodiment, the target action value of CDQN is calculated as follows:

[0124]

[0125] The target action value of Double CDQN is calculated as follows:

[0126]

[0127] The target action value of Dueling CDQN is calculated as follows:

[0128]

[0129] Where V(s t+1 , a t+1 , c t+1 ; θ′, β′) is the value function, β′ is the parameter of the value function, A(s t+1 , a t+1 , c t+1 ; θ′, δ′) is the advantage function, is the average value of the advantage function, and δ′ is the parameter of the advantage function of the target network.

[0130] The target action values ​​of CDQN+RND and CDQN+SR are the same as those of CDQN. t+1 It is a weighted combination of intrinsic and extrinsic rewards.

[0131] In this embodiment, the objective loss function of the Q network is to minimize the mean square error between the evaluation network output and the target action value, specifically:

[0132]

[0133] Among them, L Q represents the target loss function of the Q network, represents the expected function of the Q network, Y t represents the target action value of the partner network, Q(s t , a t , c t ; θ) represents the fused decision signal.

[0134] Considering that the agent is the role of exploring the environment, it needs to be trained on a large number of samples to stabilize the training process. It is represented as a large-scale (interaction sample containing dozens or hundreds of interaction sets) experience replay, and the Q network is The parameters are optimized by the minimum batch gradient descent algorithm. The update formula for updating the Q network parameters is as follows:

[0135]

[0136] Among them, θ′ represents the updated Q network parameters, θ represents the Q network parameters, α Q represents the learning rate of the Q network, Find the target loss function L with respect to the parameter θ Q The gradient of L Q represents the objective loss function of the Q network.

[0137] Specifically, when updating the Q network parameters, the present embodiment first sets the total number of interaction sets E, the greed factor ε, and the learning rate α. C , α Q , Experience Replay and their respective capacities R C , R Q , target network update steps C, decay factor λ, sampling batch B C , B Q , discount factor γ and action space length And randomly initialize the partner network parameters ω and Q network parameters θ;

[0138] Then let the target network parameter θ′=θ; for all episodes of episode={1, 2, ..., E}, randomly initialize the state s0, time step t=1, done=0, The interaction flag done is a sign of whether an interaction has ended. When it is 0, it means continuing the interaction, and when it is 1, it means stopping the interaction.

[0139] When the interaction flag done is equal to 0, execute step S2 to obtain all optional actions a t With control state s t The predicted guidance signal c t , and update the partner network parameter ω; when the interaction flag done is equal to 1, execute step S1, calculate and save the target guidance signal of the previous episode;

[0140] Then execute step S3 to obtain the attenuated guidance signal c′ t and a new attenuation factor λ;

[0141] Then execute step S4 to obtain the decision signal Q(s t, a t , c t ; θ), and generate the optimal action strategy based on the optimal action decision function;

[0142] Finally, the agent performs the optimal action Interact with the environment to get r t+1 , s t+1 , done.

[0143] After obtaining the interaction flag "done," continue to execute steps S12 to S15, S22 to S24, and S3 to S4 in sequence, updating the time step t′ = t + 1 and setting θ′ = θ every C interval time steps. The experiment is terminated when the interaction set is greater than the total number of interaction sets E.

[0144] Drawing on the human companionship learning model, this paper constructs a companion learning framework that can accompany intelligent agents in their learning. This framework mimics the human companionship and growth process, designing a time-varying, short-interval, full-sample learning, and decaying influence partner network to serve as an "education expert" to accompany and guide the agent's exploration. The partner network's guidance signals can effectively mitigate the agent's unstable behavior, reduce the convergence and application difficulty of deep reinforcement learning algorithms, reduce the agent's action errors in the later stages of exploration, and improve the agent's exploration stability.

[0145] This invention exploits the varying suitability of different deep reinforcement learning algorithms in different environments. For example, RND and DQN+SR algorithms are more conducive to agent exploration in sparsely rewarded environments, while algorithms like DQN, DDQN, and Dueling DQN are more suitable for densely rewarded environments. Therefore, the companion network proposed in this invention can be loaded onto many different baseline algorithms, enabling more stable exploration results in the later stages of exploration, and adapting to a variety of environments based on the selected baseline.

[0146] To mitigate the agent's instability in the later stages of exploration, this paper experimentally records the average reward and standard deviation of the reward for the last 30% of episodes. The experimental environment and hyperparameter settings are shown in Table 1. The standard deviation of the reward is used to observe the stability of exploration. Because the effectiveness of reinforcement learning algorithms is sensitive to random seeds, each experiment in this paper was repeated five times with different random seeds, and the results were averaged.

[0147] Table 1 Experimental environment and hyperparameter settings

[0148]

[0149] The mean and standard deviation of the rewards for the experimental results are shown in Tables 2-5. Bold text indicates that the proposed algorithm improves over the baseline. The experimental set in the table shows the mean and standard deviation of the rewards for the reciprocal experimental set.

[0150] Table 2 Statistics of experimental results of CDQN, Double CDQN, Dueling CDQN, DQN, DDQN, and Dueling DQN in Example 1

[0151]

[0152] Table 3 Statistics of experimental results of CDQN, Double CDQN, Dueling CDQN, DQN, DDQN, and Dueling DQN in Example 2

[0153]

[0154]

[0155] Table 4 Statistics of experimental results of CDQN+RND, CDQN+SR and CDQN+RND, CDQN+SR in Example 1

[0156]

[0157]

[0158] Table 5 Statistics of experimental results of CDQN+RND, CDQN+SR and CDQN+RND, CDQN+SR in Example 2

[0159]

[0160] From the experimental results in Table 2 to Table 5, it can be seen that the present invention can improve the exploration stability of the intelligent agent in the later stage of the experiment.

[0161] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0164] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

[0165] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A deep reinforcement learning method for intelligent agents based on a partner network, characterized in that: The following steps are involved: S1. Construct guidance signal functions under different reward environments based on the agent's control state, generate the experience tuples required by the partner network, and save them to the experience replay of the partner network; S2. Use a multi-layer long short-term memory network to build a partner network model, calculate the guidance signals corresponding to the agent based on the historical control state and optional actions, and update the partner network parameters; specifically, it includes the following sub-steps: S21. Use a two-layer long short-term memory network to build a partner network model; S22, the agent in time step t , time step l The sequential action tuple ( ) and all optional actions ( ) Input the partner network model to obtain the guidance signal ; S23. Experience Replay for Judging Partner Networks Is it greater than the batch update threshold? ; If yes, proceed to step S24; otherwise, no processing; S24, minimizing the mean square error between the network output and the target guidance signal as the target loss function of the partner network, and updating the partner network parameters using a batch gradient descent algorithm; S3, performing attenuation processing on the guidance signal corresponding to the intelligent agent; S4. Construct a deep Q-network reinforcement learning model based on a multi-baseline model, fuse the attenuated guidance signal with the set decision signal, and then train the deep Q-network reinforcement learning model. Generate the experience tuples required by the Q-network and save them to the experience replay of the intelligent agent, and update the Q-network parameters.

2. The method for deep reinforcement learning of an intelligent agent based on a partner network according to claim 1, characterized in that: Step S1 specifically includes the following sub-steps: S11. Set the reward environment according to the agent's control state and construct the guidance signal function under different reward environments; S12, determine whether an episode of interaction has ended; if so, execute step S13; Otherwise, execute step S14; S13, time step t Control status Optional actions and instant rewards Generate experience tuples ( ) Save to temporary experience replay pool , execute step S2; S14, for all time steps of the previous episode = { } Calculate the corresponding target guidance signals respectively , execute step S15; S15, control state Optional actions , instant rewards and the target guidance signal calculated in step S14 Generate experience tuples ( ) Experience replay saved to partner network , execute step S2.

3. The method for deep reinforcement learning of an intelligent agent based on a partner network according to claim 2, characterized in that: The guidance signal functions under different reward environments constructed in step S11 are specifically: in, Represents the time step The target guidance signal, e represents the number of interaction steps in an episode, t represents the time step, j Indicates the range t arrive e The temporary time step, Indicates that at time step The reward obtained by the agent when Indicates control status Transfer to control state The instantaneous reward obtained by the agent.

4. The method for deep reinforcement learning of an intelligent agent based on a partner network according to claim 1, characterized in that: In step S24, the objective loss function of the partner network is to minimize the mean square error between the network output and the target guidance signal, which is specifically: in, represents the objective loss function of the partner network, represents the expectation function of the partner network, Represents the time step The target guidance signal, represents the partner network at time step predictive guidance signals.

5. The method for deep reinforcement learning of an intelligent agent based on a partner network according to claim 1, characterized in that: The update formula for updating the partner network parameters using the batch gradient descent algorithm in step S24 is specifically: in, Indicates the updated partner network parameters, Represents partner network parameters, represents the learning rate of the partner network, Indicates that the parameters The loss function The gradient, represents the loss function of the partner network.

6. The method for deep reinforcement learning of an intelligent agent based on a partner network according to claim 1, characterized in that: Step S3 specifically includes the following sub-steps: S31. Initialize the decay factor at the beginning of an interaction , interaction steps ; S32, the guidance signal Multiply by the attenuation factor , get the attenuated guidance signal ; S33, update the attenuation factor to , the number of interaction steps is updated to .

7. The method for deep reinforcement learning of an intelligent agent based on a partner network according to claim 1, characterized in that: Step S4 specifically includes the following sub-steps: S41. Use multiple companion learning networks as baseline models to build a deep Q-network reinforcement learning model. S42, fusing the attenuated guidance signal with the set decision signal to obtain a fused decision signal; S43, generating an optimal action decision based on the fused decision signal according to the optimal action decision function; S44, control state , attenuated guidance signal Optional actions , instant rewards , control status , attenuated guidance signal and interactive identification Generate experience tuples ( ) Experience replay saved to Q network ; S44. Judging the Experience Replay of the Q Network Is it greater than the batch update threshold? ; If yes, execute step S45; otherwise, do nothing; S45. Minimize the mean square error between the evaluation network output and the target action value as the target loss function of the Q network, and use the batch gradient descent algorithm to update the Q network parameters.

8. The method for deep reinforcement learning of an intelligent agent based on a partner network according to claim 7, characterized in that: In step S45, the objective loss function of the Q network is to minimize the mean square error between the network output and the target action value, which is specifically: in, represents the target loss function of the Q network, represents the expected function of the Q network, represents the target action value of the partner network, represents the fused decision signal, Represents the Q network parameters.

9. The method for deep reinforcement learning of an intelligent agent based on a partner network according to claim 8, characterized in that: The update formula for updating the Q network parameters using the batch gradient descent algorithm in step S45 is specifically: in, represents the updated Q network parameters, represents the Q network parameters, represents the learning rate of the Q network, Ask about parameters The objective loss function The gradient, represents the objective loss function of the Q network.

Images