Reinforced learning system and method for intelligent agent exploration in high-dynamic decision-making scene

Abstract

The invention discloses a reinforcement learning system and method for high-dynamic decision-making scene agent exploration, and the system comprises a positive actor-commentator model, a negative actor-commentator model, an actor-commentator model, a strategy similarity model, and a strategy action selection model. Meanwhile, the closeness degree of the main strategy and the positive strategy is minimized, the closeness degree of the main strategy and the negative strategy is maximized, so that the main strategy generates high-value actions with high probability and generates low-value actions with low probability, and the overall performance of the method is improved; meanwhile, in the training process, main strategy actions, positive strategy actions and negative strategy actions are selected and executed in a probability form through a strategy action selection model, the diversity of action selection is increased, the diversity of exploration environments is indirectly improved, and finally performance improvement is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep reinforcement learning, and in particular to an online reinforcement learning algorithm based on an actor-critic framework, and mainly to a reinforcement learning system and method for intelligent agent exploration in highly dynamic decision-making scenarios. Background Art

[0002] In unmanned intelligent agent control scenarios, unmanned intelligent agents often need to operate in complex and dynamically changing environments, and the uncertainties in the environment increase the difficulty of perception and decision-making. In highly dynamic decision-making scenarios, unmanned intelligent agents need to face complex terrain, weather conditions, and dynamic battlefield situations, while dealing with enemy electronic interference and cyber attacks. In this case, unmanned intelligent agents need to autonomously complete basic actions such as movement, hovering, and interference, and then achieve complex tasks such as reconnaissance, patrolling, supply, and support.

[0003] Specifically, in special environments, unmanned intelligent agents may need to travel through diverse terrains such as mountains, forests, and deserts. These terrain characteristics make perception and navigation extremely challenging. At the same time, rapid changes in weather conditions, such as storms, rain and snow, will also have a significant impact on the stable flight of unmanned intelligent agents and the accuracy of sensors. In addition, enemy electronic interference and network attacks, such as signal shielding, camouflaged signals, and network attacks, further increase the uncertainty and risk of unmanned intelligent agents in the process of mission execution.

[0004] In the face of these challenges, unmanned agents not only need to have strong perception and environmental adaptability, but also need to make efficient decisions in the ever-changing special scenario environment. This includes autonomously selecting the best action path, dynamically adjusting task priorities, and being able to complete tasks independently in the event of communication interruption or enemy interference. Unmanned agents need to be highly flexible and robust to cope with various emergencies on the battlefield.

[0005] In addition, the complexity and diversity of special scenario tasks also put forward higher requirements for unmanned intelligent agents. For example, in reconnaissance missions, unmanned intelligent agents need to conduct covert operations in enemy fire coverage areas while collecting high-value intelligence; in patrol missions, unmanned intelligent agents need to cover a wide area, identify and respond to potential threats; in supply missions, unmanned intelligent agents need to navigate accurately to ensure accurate delivery of materials; in support missions, unmanned intelligent agents need to cooperate with ground forces to provide real-time intelligence and fire support.

[0006] In this scenario, how to efficiently explore the complex battlefield environment and develop fast-converging algorithms,

[0007] This has become a core challenge for autonomous control of unmanned intelligent agents in highly dynamic decision-making scenarios. The exploration algorithm needs to be able to adapt to environmental changes quickly and find the optimal path and decision quickly; the convergence algorithm needs to achieve

[0008] The best state ensures that the unmanned intelligent agent is efficient and accurate in performing tasks. In general, the autonomous control of unmanned intelligent agents in highly dynamic decision-making scenarios requires the comprehensive use of intelligent decision-making algorithms and powerful environmental exploration capabilities to cope with complex and changing battlefield environments and mission requirements.

[0009] Deep reinforcement learning is an important research direction in the field of artificial intelligence in recent years. It combines deep learning with reinforcement learning to enable intelligent agents to autonomously learn decision-making strategies in complex and high-dimensional environments.

[0010] Traditional reinforcement learning methods are often inefficient when faced with high-dimensional state and action spaces, and have difficulty processing large-scale data. Deep learning is good at extracting features and performing pattern recognition from high-dimensional data, and can effectively solve this problem. Therefore, deep reinforcement learning is widely used in games, robot control, drone control and other fields.

[0011] 40 The actor-critic based online reinforcement learning method is an important class of deep reinforcement learning algorithms.

[0012] The method separates the learning process of the policy function and the value function, where the "actor" is responsible for selecting actions and

[0013] The "critic" is responsible for evaluating the actions chosen by the actor and giving feedback. The actor network generates an action probability distribution based on the state, and the critic network evaluates the value of the current policy and uses the temporal difference error to update the value function.

[0014] The advantage of this approach is that it enables more stable policy updates and can effectively handle continuous action45 spaces.

[0015] In the actor-critic architecture-based algorithm, the TD3 algorithm adopts a dual network design and uses the minimum value of two independent target Q networks to estimate the Q value, which effectively alleviates the Q

[0016] The TD3 algorithm can solve the problem of overestimation of the value; at the same time, a delayed update strategy is used to stabilize the network update process.

[0017] The SAC algorithm performs well in the task and significantly improves the stability and performance of training. However, its training process is still slow and requires a large amount of environmental interaction data. The SAC algorithm introduces the maximum entropy framework to optimize the expected return of the strategy at the same time.

[0018] The entropy of the report and strategy is used to encourage the agent to explore; at the same time, the temperature parameter automatic adjustment mechanism is used to balance the exploration and utilization process. The SAC algorithm performs well in tasks that require a lot of exploration. However, the complex calculation and parameter adjustment process greatly increases the difficulty of algorithm convergence.

[0019] An important prerequisite for the convergence of actor-critic online reinforcement learning methods is that the agent needs to explore enough environmental states, that is, the strategy not only needs to allow the agent to obtain higher environmental rewards, but also

[0020] It is necessary to explore as diverse environments as possible so that the agent can escape from the local optimum and achieve better overall performance.

[0021] The ICM algorithm based on intrinsic motivation exploration promotes

[0022] The ICM algorithm performs well in sparse reward environments and significantly improves the performance of intelligent agents.

[0023] The RND algorithm proposes an exploration method based on random networks, which defines the intrinsic reward by comparing the output difference of two neural networks. Compared with the ICM algorithm, the RND algorithm is simple to implement and easy to

[0024] However, the definition of its intrinsic reward depends on the initialization of the random network, which leads to the instability of the exploration process. 65 Therefore, there is an urgent need for an effective environment exploration design to help the intelligent agent explore the environment more stably and efficiently.

[0025] Get better performance and improve the versatility of the algorithm. Summary of the invention

[0026] The present invention is to overcome the shortcomings of the existing actor-critic based online reinforcement learning method in terms of environment exploration ability, and proposes a reinforcement learning system and method for intelligent agent exploration in highly dynamic decision-making scenarios.

[0027] The proposed method includes positive actor-critic model, negative actor-critic model, actor-critic model, policy similarity model and policy action selection model, which guides the learning of the main policy by using positive policy and negative policy, while minimizing the closeness between the main policy and the positive policy and maximizing the closeness between the main policy and the negative policy.

[0028] The main strategy generates high-value actions with high probability and low-value actions with low probability, which improves the overall performance of method 75. At the same time, the strategy action selection model is used to select the main action in a probabilistic form during the training process.

[0029] Strategy actions, positive strategy actions, and negative strategy actions increase the diversity of action selection, indirectly improve the diversity of the exploration environment, and ultimately achieve performance improvement.

[0030] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a reinforcement learning system for intelligent agent exploration in highly dynamic decision-making scenarios, comprising at least a positive actor-critic model, a negative actor-critic model, an actor-critic model, a strategy similarity model and a strategy action selection model,

[0031] The positive actor-critic model: includes a positive actor network and a positive critic network. The positive actor network is used as a positive strategy to generate positive actions by maximizing the Q value. The positive critic network aims to minimize the temporal difference error between the generated Q value and the target Q value, and is used to give the value of the action generated by the positive actor network in the current state.

[0032] 85 The negative actor-critic model: comprises a negative actor network and a negative critic network, wherein the negative actor network is used as a negative strategy to generate negative actions by minimizing the Q value; the negative critic network takes minimizing the temporal difference error between the generated Q value and the target Q value as the goal, and is used to give the value of the action generated by the negative actor network in the current state;

[0033] The actor-critic model: includes an actor network and a critic network. The actor network is used as the main strategy to generate actions that are close to the positive strategy and far away from the negative strategy. The critic network is used to give the value of the action generated by the actor network in the current state.

[0034] The strategy similarity model is implemented by an autoencoder, which inputs the actions generated by two strategies and outputs the similarity of the two actions;

[0035] The strategy action selection model: according to the value of 95 corresponding to the action obtained by the main strategy, positive strategy and negative strategy, the action is selected probabilistically to perform the interaction with the environment.

[0036] As an improvement of the present invention, the actor-critic model, the positive actor-critic model, and the negative actor-critic model are all deep neural networks, which process vector signals through 3 fully connected layers and process image signals through 3 convolutional layers plus 2 fully connected layers.

[0037] In order to achieve the above purpose, the present invention also adopts a technical solution: a positive and negative network comparison reinforcement learning method for intelligent 100-agent exploration of high-dynamic decision-making scenarios, which includes at least two stages: training and execution.

[0038] For continuous action scenarios: In the training phase, the agent action is selected by the policy action selection model from the actions obtained by the main strategy, positive strategy and negative strategy. The main strategy, positive strategy and negative strategy are all determined by the Gaussian distribution generated by the corresponding actor network, and the action is obtained by resampling the Gaussian distribution;

[0039] In the execution phase, the mean of the Gaussian distribution corresponding to the main strategy is used as the agent action;

[0040] For discrete action scenarios: In the training phase, the agent actions are selected by the policy action selection model from the actions obtained by the main strategy, positive strategy and negative strategy. The main strategy, positive strategy and negative strategy are directly generated by the corresponding actor network to generate the probability of all actions being selected. The actions are sampled through the corresponding category distribution.

[0041] In the execution phase, the action with the highest probability of distribution of the category corresponding to the main strategy is used as the agent action.

[0042] As an improvement of the present invention, the training objective of the positive actor network is to maximize the expected Q value:

[0043]

[0044] in is the parameter of the positive actor network, s t is the environmental status information at time t, D is the experience replay pool, is the action obtained according to the positive actor network strategy, represents the jth positive critic network, represents the parameters of the jth positive critic network;

[0045] The training objective of the positive critic network is to minimize the gap between the value estimate and the target value:

[0046]

[0047] in represents all positive critic networks, φ + represents the parameters of all positive critic networks, s t , r t ,d t They represent the environment state information at time t, the actions taken by the agent, the rewards given by the environment, and the completion status of the environment, respectively. t+1represents the environmental status information at time t+1, y + represents the target value composed of the immediate reward and the target positive critic network's estimate of future value;

[0048] The training objective of the negative actor network is to minimize the expected Q value:

[0049]

[0050] in are the parameters of the negative actor network, s t is the environmental status information at time t, D is the experience replay pool, is the action obtained according to the negative actor network strategy, represents the jth negative critic network, represents the parameters of the jth negative critic network;

[0051] The training objective of the negative critic network is to minimize the gap between the value estimate and the target value:

[0052]

[0053] in represents all negative critic networks, φ - represents the parameters of all negative critic networks, s t , r t ,d t They represent the environment state information at time t, the actions taken by the agent, the rewards given by the environment, and the completion status of the environment, respectively. t+1 represents the environmental status information at time t+1, y - represents the target value composed of the immediate reward and the target negative critic network's estimate of the future value;

[0054] The training objective of the actor network is to minimize the following formula:

[0055]

[0056] in is the j-th critic network, φ j is the parameter of the j-th target negative critic network, a t is the action obtained according to the actor network strategy, where:

[0057]

[0058] Where sim is the strategy similarity model, τ is the temperature balance coefficient, N is the total number of negative action samples generated by the negative actor network strategy, According to s tThe strategy corresponding to the positive actor network Get all According to s t The strategy corresponding to the negative actor network get;

[0059] The training objective of the critic network is to minimize the gap between the value estimate and the target value:

[0060]

[0061] Where Q φ represents all critic networks, φ represents the parameters of all critic networks, s t , r t ,d t They represent the environment state information at time t, the actions taken by the agent, the rewards given by the environment, and the completion status of the environment, respectively. t+1 represents the state of the environment at time t+1, and y represents the target value composed of the immediate reward and the target critic network's estimate of the future value;

[0062] As another improvement of the present invention, the target value y + The specific calculation method is:

[0063]

[0064] Where γ is the discount factor, which discounts future rewards. is the jth target positive critic network, are the parameters of the j-th target positive critic network, According to s t+1 Obtained by the strategy corresponding to the positive actor network;

[0065] The target value y - The calculation method is as follows:

[0066]

[0067] in is the j-th target negative critic network, are the parameters of the j-th target negative critic network, According to s t+1 Obtained by the strategy corresponding to the negative actor network;

[0068] The target value y is calculated as follows:

[0069]

[0070] in is the jth target critic network, is the parameter of the jth target critic network, a t+1 According to s t+1 Obtained by the strategy corresponding to the actor network.

[0071] As another improvement of the present invention, the training goal of the autoencoder of the strategy similarity model is to minimize the gap between the prediction of the next environmental state and the actual next environmental state:

[0072]

[0073] Among them, AE is the autoencoder model, β is the parameter of the autoencoder model, and s t is the current time step environment state, action a t From the main strategy, MSE is the mean square error function, s t+1 is the environmental state at the next time step, The prediction of the environment state at the next time step obtained by the autoencoder;

[0074] The strategy similarity calculation method is as follows:

[0075]

[0076] in Indicates that in the environment state s t Next, actions from different strategies The similarity between them, d is the Euclidean distance function, cos is the cosine similarity function, The calculation method is as follows:

[0077]

[0078] Where Encoder is the encoder model, s t is the state of the environment at the current time step, Get actions for a policy.

[0079] As a further improvement of the present invention, in the strategy action selection model, the probability of an action being selected is obtained in a Softmax normalized form, wherein the probability of an action being selected corresponding to the main strategy is calculated as follows:

[0080]

[0081] where s t is the current environment state, action a t is the action obtained through the main strategy, is the action obtained by the forward strategy, is the action obtained by the negative strategy;

[0082] The probability of the action corresponding to the forward strategy being selected is calculated as follows:

[0083]

[0084] The probability of the action corresponding to the negative strategy being selected is calculated as follows:

[0085]

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] (1) The present invention uses the setting of positive strategy and negative strategy, and uses the action selection model to randomize and probabilistically select actions, thereby increasing the diversity of action selection of unmanned intelligent agents and improving the ability to explore complex environments, especially highly dynamic decision-making environments.

[0088] (2) The present invention uses the similarity between the positive strategy, the negative strategy and the main strategy as a guide for the learning of the main strategy, optimizes the learning process of the main strategy, helps the main strategy to escape from the local optimal situation and achieve the global optimal solution, thereby improving the actual performance of the main strategy.

[0089] (3) The present invention uses positive strategies and negative strategies to guide the learning of the main strategy, which enhances the directionality of strategy learning, helps the main strategy converge to the global optimal solution faster, and improves the convergence speed of the method in complex environments and diversified action spaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 It is a schematic diagram of the architecture of positive and negative network comparative learning in the method of the present invention;

[0091] Figure 2 It is a structural diagram of the autoencoder model in the system strategy similarity model of the present invention;

[0092] Figure 3 It is a schematic diagram of the architecture of the strategy similarity model calculation simulation in the method of the present invention;

[0093] Figure 4 It is a schematic diagram of the architecture of the action selection model in the system of the present invention;

[0094] Figure 5 It is the overall operation training flow chart of the present invention;

[0095] Figure 6 It is the main principle diagram of the method of the present invention. DETAILED DESCRIPTION

[0096] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0097] Example 1

[0098] A reinforcement learning system for intelligent agent exploration in highly dynamic decision-making scenarios includes at least a positive actor-critic model, a negative actor-critic model, an actor-critic model, a policy similarity model, and a policy action selection model. The positive actor-critic model includes a positive actor network and a positive critic network. The positive actor network, as a positive strategy, generates positive actions by maximizing the Q value. The positive critic network takes the temporal difference error between the generated Q value and the target Q value as the goal, and is used to give the value of the action generated by the positive actor network in the current state. The negative actor-critic model includes a negative actor network and a negative critic network. The negative actor network, as a negative strategy, generates negative actions by minimizing the Q value. The negative critic network takes the temporal difference error between the generated Q value and the target Q value as the goal, and is used to give the value of the action generated by the negative actor network in the current state. The actor-critic model includes an actor network and a critic network. The actor network is used as the main strategy to generate actions close to the positive strategy and far from the negative strategy. The critic network is used to give the value of the action generated by the actor network in the current state. The policy similarity model is implemented by an autoencoder, which inputs the actions generated by the two strategies and outputs the similarity of the two actions. The policy action selection model probabilistically selects the interaction between the action execution and the environment according to the values ​​corresponding to the actions obtained by the main strategy, the positive strategy, and the negative strategy.

[0099] In the positive actor-critic model, the positive actor network aims to maximize the Q value to obtain actions with higher value. In highly dynamic decision-making scenarios, the optimization goal of this network is particularly important because it can help the agent choose the best action strategy in a complex and changing environment, so as to efficiently complete various tasks. Through learning and optimization, the positive actor network enables the agent to always choose actions that can maximize the long-term cumulative rewards when performing tasks. These actions are not only optimal in the current situation, but also have high value in various changes and challenges that may arise in the future. For example, when performing strike missions, the positive actor network enables the agent to choose the best attack route and timing, so as to cause the greatest damage to the target in the shortest time; in reconnaissance missions, the positive actor network can guide the agent to avoid the enemy's defense system and collect key intelligence safely and efficiently. In support missions, the positive actor network can help the agent quickly respond to the friendly force's help signal, provide necessary fire support or resource supply, and ensure the friendly force's dominant position. In patrol missions, the positive actor network enables the agent to reasonably plan patrol routes, cover a larger area, and detect and respond to potential threats in a timely manner. In the escort mission, the positive actor network guides the agent to choose the best escort strategy to protect important targets from enemy attacks. By maximizing the Q value, the positive actor network not only improves the performance of the agent in a single task, but also enhances its comprehensive ability in a multi-task environment. In highly dynamic decision-making scenarios, the agent needs to face various emergencies and complex environmental changes. The optimization goal of the positive actor network enables the agent to quickly adapt and efficiently respond to these challenges. This not only improves the efficiency and accuracy of task completion, but also significantly increases the survivability and overall combat value of the agent on the battlefield. Therefore, in highly dynamic decision-making scenarios, the positive actor network maximizes the Q value, enabling the agent to take appropriate actions and efficiently complete tasks such as strike, reconnaissance, support, patrol, and escort, thereby obtaining higher simulation value and actual combat effects. Through continuous optimization and learning, the positive actor network enables the agent to perform well in various complex tasks, providing solid technical support for the realization of autonomous control and efficient combat of unmanned agents. The positive critic network aims to minimize the temporal difference error between the generated Q value and the target Q value to obtain better value estimation.

[0100] The overall goal of the negative actor-critic model is to generate low-quality actions. The negative actor network aims to minimize the Q value and aims to obtain actions with lower value. In highly dynamic decision-making scenarios, the optimization goal of this network is to provide negative examples by taking action strategies that are completely opposite to the positive strategy. This negative example helps the main strategy to learn and optimize better through negative guidance. Specifically, the negative actor network creates a set of suboptimal or inferior strategies by learning and selecting actions that may lead to lower returns. These strategies perform poorly in practical applications, but their existence has important reference value for the optimization of the main strategy. For example, when performing a strike mission, the negative actor network may choose actions that delay the attack, expose the position, or waste resources. Although these actions seem harmless in the short term, they may lead to mission failure or significant losses in the long run. By observing these negative strategies, the main strategy can identify and avoid these potential errors. In reconnaissance missions, the negative actor network may guide the agent to take paths and methods that are easily discovered by the enemy and cannot collect effective intelligence. By comparing with these inefficient strategies, the main strategy can better optimize its own reconnaissance path and concealment method, and improve the efficiency and security of intelligence collection. In the support mission, the negative actor network may choose actions that cannot respond to the needs of friendly forces in a timely manner and have unreasonable resource allocation. The main strategy can learn from them how to more effectively dispatch resources and respond to support. In the patrol mission, the negative actor network may plan those plans with insufficient coverage and unreasonable patrol routes. The main strategy can optimize its own patrol path and strategy by comparing these plans to ensure wider area coverage and more timely threat response. In the escort mission, the negative actor network may adopt those strategies with weak defense and exposed targets. The main strategy can develop a more robust and effective escort strategy through reverse learning to protect important targets from enemy attacks. By minimizing the Q value, the negative actor network not only provides negative strategy examples, but also helps the main strategy to identify and understand what is a suboptimal or inferior decision to a certain extent. This reverse guidance mechanism has unique value for the optimization of the main strategy, which can help it converge to the optimal solution more quickly, thereby improving the overall performance of the intelligent agent in highly dynamic decision-making scenarios.

[0101] In summary, the negative actor network aims to minimize Q-values ​​in highly dynamic decision-making scenarios, and guides the learning and optimization of the main policy in another way by providing negative examples of low-value actions. Through this reverse guidance mechanism, the main policy can better identify and avoid inefficient or wrong decisions, thereby achieving more efficient and accurate autonomous control. The negative critic network aims to minimize the temporal difference error between the generated Q-value and the target Q-value to obtain better value estimates.

[0102] The actor-critic model, based on the traditional optimization objectives, is guided by the positive and negative actor-critic models to obtain more accurate high-value actions. The goal of the actor model is divided into two parts. The first part is to maximize the Q value of the generated action, and the second part is to minimize the contrast loss between the generated action and the positive policy action and the negative policy action according to the policy similarity model; the critic model aims to minimize the temporal difference error between the generated Q value and the target Q value to obtain a better value estimate.

[0103] Traditional methods only use actor-critic models. In highly dynamic decision-making scenarios, situations change rapidly.

[0104] Traditional actor-critic models cannot adapt quickly when faced with frequent changes and complex interference, which leads to a lag in the model's policy update, affecting the timeliness and effectiveness of decision-making. Since the update of the actor-critic model depends on the policy iteration process, that is, the policy evaluation process and the policy improvement process, errors in any process will lead to iterative accumulation of errors. Therefore, in a complex decision space, the actor network may fall into a local optimal solution and find it difficult to explore the global optimal strategy. The value estimation of the critic network may also be inaccurate, thus affecting the overall policy optimization. At the same time, when exploring new strategies, traditional actor-critic models often rely on randomness or simple heuristic methods, resulting in low exploration efficiency, making it difficult to efficiently explore complex environments and quickly discover high-value strategies. Finally, when faced with uncertain and noisy environments, the performance of the actor-critic model may drop significantly, showing poor robustness. This is particularly evident in high-risk and adversarial environments.

[0105] The actor-critic model based on positive and negative network contrast learning optimizes both positive and negative strategies simultaneously,285 enabling the agent to identify and avoid low-value actions during exploration, thereby more efficiently discovering high-value strategies and enabling the agent to adapt to environmental changes more quickly. The negative examples provided by the negative actor network help the critic network to more accurately evaluate the value of actions, and the actor network converges to a better strategy more quickly. The introduction of the negative actor network enables the agent to identify potential wrong decisions and avoid them, improving the robustness of the overall strategy. In the face of uncertainty and noise, the agent can perform tasks more robustly. At the same time, through contrast learning, the positive actor network and the negative actor network work together to help the agent escape from the local optimal solution. The positive actor network focuses on maximizing the Q value, while the negative actor network minimizes the Q value. The comparison between the two makes the strategy optimization more comprehensive and balanced.

[0106] The policy similarity model is set up and implemented by an autoencoder. The encoder of the autoencoder takes the environment state and policy action as input and outputs a latent representation; the decoder takes the latent representation output by the encoder as input and outputs a prediction of the environment state at the next time step. The training goal of the autoencoder is to minimize the gap between the actual environment state at the next time step and the prediction of the environment state generated by the autoencoder. The policy similarity is obtained by the environment state and the two policy actions. Specifically, the environment state and the policy action are first passed through the encoder to obtain a latent representation, and then the final policy similarity is obtained based on the Euclidean distance and cosine similarity distance of the two latent representations.

[0107] Set up a policy action selection model. During the training process, under a given environment state, according to the value of the action obtained by the main strategy, positive strategy, and negative strategy, the action execution and the interaction with the environment are selected probabilistically. The action selection probability is obtained by normalizing the corresponding strategy Q value through Softmax. The action selection model adds randomness to the agent's exploration of highly dynamic environments, greatly improving the agent's ability to explore the environment.

[0108] Example 2

[0109] A reinforcement learning method for agent exploration in highly dynamic decision-making scenarios is designed to solve the defects of the traditional online deep reinforcement learning method based on the actor-critic architecture, such as insufficient adaptability to highly dynamic environments, insufficient environmental exploration capabilities, insufficient utilization capabilities, and slow convergence speed. Using the system described in Example 1, the online deep reinforcement learning based on the actor-critic architecture uses additional positive strategies and negative strategies for strategy learning guidance. The training of the positive strategy is consistent with the basic actor-critic algorithm, and the goal is to generate high-value actions. The training goal of the negative strategy is opposite to the basic actor-critic algorithm, and the goal is to generate low-value actions. The positive strategy, negative strategy and main strategy actions are calculated by the strategy similarity model to obtain the contrast loss as an additional training target for the main strategy. During the overall training process, the actions of the agent interacting with the environment are obtained in the form of probability by the action selection model according to the value of different strategies.

[0110] The overall training process of this method follows the Markov decision process. The agent accumulates experience data during the interaction with the environment, and randomly samples a certain batch size of past experience data for training during the training process. The agent obtains the corresponding action based on the policy network and action selection model, and executes it in the environment. The environment simulates the state transition based on the current environment state and the action taken by the agent, obtains the reward for this step and the environment state for the next step, and repeats the cycle. As the agent interacts with the environment, it accumulates sufficiently diverse and high-quality data, which assists the reinforcement learning algorithm to complete an efficient learning process and obtain a high-performance agent.

[0111] Reinforcement learning methods for agent exploration in highly dynamic decision-making scenarios, such as Figure 1 As shown, this method designs positive and negative strategies for strategy learning guidance, and the right part of each network is its learning goal. The positive actor-critic model includes a positive actor network and a positive critic network, and the training goal of the positive actor network is to maximize:

[0112]

[0113] in is the parameter of the positive actor network, s t is the environmental status information at time t, D is the experience replay pool, is the action obtained according to the positive actor network strategy, represents the jth positive critic network, represents the parameters of the jth positive critic network;

[0114] The training objective of the positive critic network is to minimize:

[0115]

[0116] in represents all positive critic networks, φ + represents the parameters of all positive critic networks, s t , r t ,d t They represent the environment state information at time t, the actions taken by the agent, the rewards given by the environment, and the completion status of the environment, respectively. t+1 Represents the environmental status information at time t+1. The above five data are all from the experience replay pool D, where y + The calculation method is as follows:

[0117]

[0118] Where γ is the discount factor, which discounts future rewards. is the jth target positive critic network, is the parameter of the j-th target positive critic network, where According to s t+1 Obtained by the strategy corresponding to the positive actor network;

[0119] The training objective of the negative actor network is to minimize:

[0120]

[0121] in are the parameters of the negative actor network, s tis the environmental status information at time t, D is the experience replay pool, is the action obtained according to the negative actor network strategy, represents the jth negative critic network, represents the parameters of the jth negative critic network;

[0122] The training objective of the negative critic network is to minimize:

[0123]

[0124] in represents all negative critic networks, φ - represents the parameters of all negative critic networks, s t , r t ,d t They represent the environment state information at time t, the actions taken by the agent, the rewards given by the environment, and the completion status of the environment, respectively. t+1 Represents the environmental status information at time t+1. The above five data are all from the experience replay pool D, where y - The calculation method is as follows:

[0125]

[0126] Where γ is the discount factor, which discounts future rewards. is the j-th target negative critic network, is the parameter of the j-th target negative critic network, where According to s t+1 Obtained by the strategy corresponding to the negative actor network;

[0127] The training objective of the actor network is to minimize:

[0128]

[0129] in is the jth critic network, φ j is the parameter of the j-th target negative critic network, a t is the action obtained according to the actor network strategy, where:

[0130]

[0131] Where sim is the strategy similarity model, τ is the temperature balance coefficient, N is the total number of negative action samples generated by the negative actor network strategy, According to s t The strategy corresponding to the positive actor network Get all According to st The strategy corresponding to the negative actor network get;

[0132] The training objective of the critic network is to minimize:

[0133]

[0134] Where Q φ represents all critic networks, φ represents the parameters of all critic networks, s t , r t ,d t They represent the environment state information at time t, the actions taken by the agent, the rewards given by the environment, and the completion status of the environment, respectively. t+1 Represents the environmental status information at time t+1. The above five data are all from the experience replay pool D, where y is calculated as follows:

[0135]

[0136] Where γ is the discount factor, which discounts future rewards. is the jth target critic network, is the parameter of the jth target critic network, where a t+1 According to s t+1 Obtained by the strategy corresponding to the actor network.

[0137] The actor-critic model, positive actor-critic model, and negative actor-critic model are all deep neural networks, and different architectures are used for processing according to the environmental state representation of different application scenarios. For vector signal representation, the network architecture is constructed using 3 layers of fully connected layers; for image signal representation, the network architecture is constructed using 3 layers of convolutional layers plus 2 layers of fully connected layers.

[0138] The target critic network parameters are not updated with the algorithm training. The target critic network uses soft update to update parameters. That is, each time the corresponding critic network parameters are updated, the critic network parameters are copied to the target critic network in a certain proportion. The soft update formula is as follows:

[0139]

[0140] Where σ is a soft update parameter, which is used to control the parameter update ratio and prevent unstable parameter updates.

[0141] The discount factor γ is set between 0 and 1 to measure the impact of future rewards on current decisions, prevent the accumulated rewards from growing infinitely, and make the strategy more adaptable to long-term planning.

[0142] The temperature balance coefficient τ controls the smoothness of the softmax function, affects the steepness of the similarity distribution, balances the contribution of positive strategy actions and negative strategy actions in the contrast loss, effectively distinguishes positive samples from negative samples, and thus improves the contrastive learning effect of the algorithm.

[0143] In the described UAV high-dynamic decision-making scenario, the environmental state includes two parts. The first part is the partial observation information of the UAV on the environment. The environmental state is sampled as an 84x84 grayscale image, and the pixel value is normalized to [0,1]. Four frames of continuous time-series images are stacked, and the final state is represented as an 84x84x4 stacked grayscale image. The second part is the agent state information, including agent coordinates, heading, radar switch state, radar frequency, jamming radar switch, jamming radar frequency, radar detected enemy unit list, and the number of remaining missiles. All agent state information constitutes a high-dimensional vector information signal.

[0144] like Figure 2 , 3 As shown, the method of the present invention designs a strategy similarity model, which accepts the environment state and the actions of different strategies as input, and outputs the similarity of the two strategy actions in the current state. The strategy similarity model consists of an encoder and a decoder of an autoencoder. The input of the encoder is the current environment state and the action obtained by a certain strategy, and the output is a potential representation. The input of the decoder is the potential representation output by the encoder, and the output is the prediction of the environment state at the next time step. The training goal of the autoencoder is to minimize:

[0145]

[0146] Among them, AE is the autoencoder model, β is the parameter of the autoencoder model, and s t is the current time step environment state, action a t From the main strategy, MSE is the mean square error function, s t+1 is the environmental state at the next time step, The prediction of the environment state at the next time step obtained by the autoencoder;

[0147] The strategy similarity is calculated as follows:

[0148]

[0149] in Indicates that in the environmental state s t Next, actions from different strategies The similarity between them, d is the Euclidean distance function, cos is the cosine similarity function, and the Euclidean distance is calculated as follows:

[0150]

[0151] Where n is the potential representation The dimension of Respectively In the i-th dimension, the cosine similarity is calculated as follows:

[0152]

[0153] in For vector The Euclidean norm of

[0154] in The calculation method is as follows:

[0155]

[0156] Where Encoder is the encoder model, s t is the state of the environment at the current time step, Get actions for a policy.

[0157] like Figure 4 As shown, the method of the present invention designs an action selection model. Under the current environment state, according to the values ​​corresponding to the actions obtained by the main strategy, positive strategy, and negative strategy, the action corresponding to the strategy is randomly selected in the form of probability. The probability of the action being selected is obtained in the form of Softmax normalization, where the probability of the action corresponding to the main strategy being selected is calculated as follows:

[0158]

[0159] The probability of the action corresponding to the forward strategy being selected is calculated as follows:

[0160]

[0161] The probability of the action corresponding to the negative strategy being selected is calculated as follows:

[0162]

[0163] In the training phase, the agent action is selected by the strategy action selection model from the actions obtained by the main strategy, positive strategy, and negative strategy. For continuous actions, the main strategy, positive strategy, and negative strategy are all determined by the Gaussian distribution generated by the corresponding actor network, and the action is obtained by resampling the Gaussian distribution. In the execution phase, the mean of the Gaussian distribution corresponding to the main strategy is directly used as the agent action. For discrete actions, the main strategy, positive strategy, and negative strategy are all directly generated by the corresponding actor network to generate the probability of all actions being selected, and the action is obtained by sampling the corresponding category distribution. In the execution phase, the action with the highest probability of the category distribution corresponding to the main strategy is directly used as the agent action.

[0164] The intelligent agent action of the UAV high-dynamic decision-making scenario includes two parts. The first part is the heading of the continuous value, which takes the value of [0,359]. The second part is the radar switch and frequency, the jammer switch and frequency, whether to launch a missile and the target to attack, all of which take the value of discrete integer.

[0165] like Figure 5 As shown in FIG, the entire algorithm operation process is divided into an environment interaction stage and a training stage. In the environment interaction stage 435, the environment gives the current state, the agent observes the environment state and selects an action to interact with the environment according to the action selection model. The environment obtains the reward of this step and the state of the next step according to the agent's action. The agent stores the current state, next state, action, and reward information in the experience replay pool, and determines whether to enter the training stage based on whether the update interval is reached. In the training stage, the agent extracts a certain batch of data from the experience replay pool, and uses this data to simultaneously update the positive actor-critic network and the negative actor-critic network, and at the same time updates the actor-critic network according to the contrast loss 440.

[0166] In summary, the method of the present invention designs a positive and negative actor-critic model structure, uses positive and negative strategies to guide the learning and training process of the main strategy; uses a strategy similarity model to measure the similarity between strategies, so that the main strategy is closer to the positive strategy and away from the negative strategy; at the same time, an action selection model is used to obtain diversified actions to interact with the environment, thereby improving the algorithm's ability to explore highly dynamic environments.

[0167] 445 It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, they can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A reinforcement learning system for agent exploration in highly dynamic decision-making scenarios, characterized by: At least including positive actor-critic model, negative actor-critic model, actor-critic model, policy similarity model and policy action selection model, The positive actor-critic model: includes a positive actor network and a positive critic network. The positive actor network is used as a positive strategy to generate positive actions by maximizing the Q value. The positive critic network aims to minimize the temporal difference error between the generated Q value and the target Q value, and is used to give the value of the action generated by the positive actor network in the current state. The negative actor-critic model: includes a negative actor network and a negative critic network. The negative actor network is used as a negative strategy to generate negative actions by minimizing the Q value. The negative critic network takes minimizing the temporal difference error between the generated Q value and the target Q value as the goal, and is used to give the value of the action generated by the negative actor network in the current state; The actor-critic model: includes an actor network and a critic network. The actor network is used as the main strategy to generate actions that are close to the positive strategy and far away from the negative strategy. The critic network is used to give the value of the action generated by the actor network in the current state. The strategy similarity model is implemented by an autoencoder, which inputs the actions generated by two strategies and outputs the similarity of the two actions; The strategy action selection model probabilistically selects the action execution and the interaction with the environment according to the value corresponding to the action obtained by the main strategy, positive strategy and negative strategy.

2. The reinforcement learning system for intelligent agent exploration in highly dynamic decision-making scenarios as claimed in claim 1, characterized in that: The actor-critic model, the positive actor-critic model, and the negative actor-critic model are all deep neural networks. The deep neural networks process vector signals through 3 fully connected layers and process image signals through 3 convolutional layers plus 2 fully connected layers.

3. A reinforcement learning method for intelligent agent exploration in highly dynamic decision-making scenarios using the system as claimed in claim 1, characterized in that: It includes at least two stages: training and execution. For continuous action scenarios: In the training phase, the agent action is selected by the policy action selection model from the actions obtained by the main strategy, positive strategy and negative strategy. The main strategy, positive strategy and negative strategy are all determined by the Gaussian distribution generated by the corresponding actor network, and the action is obtained by resampling the Gaussian distribution; In the execution phase, the mean of the Gaussian distribution corresponding to the main strategy is used as the agent action; For discrete action scenarios: In the training phase, the agent actions are selected by the policy action selection model from the actions obtained by the main strategy, positive strategy and negative strategy. The main strategy, positive strategy and negative strategy are directly generated by the corresponding actor network to generate the probability of all actions being selected. The actions are sampled through the corresponding category distribution. In the execution phase, the action with the highest probability of distribution of the category corresponding to the main strategy is used as the agent action.

4. The positive and negative network comparison reinforcement learning method for highly dynamic decision-making scene agent exploration as claimed in claim 3, characterized in that: The training goal of the positive actor network is to maximize the expected Q value: in is the parameter of the positive actor network, s t is the environmental status information at time t, D is the experience replay pool, is the action obtained according to the positive actor network strategy, represents the jth positive critic network, represents the parameters of the jth positive critic network; The training objective of the positive critic network is to minimize the gap between the value estimate and the target value: in represents all positive critic networks, φ + represents the parameters of all positive critic networks, s t , r t ,d t They represent the environment state information at time t, the actions taken by the agent, the rewards given by the environment, and the completion status of the environment, respectively. t+1 represents the environmental status information at time t+1, y + represents the target value composed of the immediate reward and the target positive critic network's estimate of future value; The training objective of the negative actor network is to minimize the expected Q value: in are the parameters of the negative actor network, s t is the environmental status information at time t, D is the experience replay pool, is the action obtained according to the negative actor network strategy, represents the jth negative critic network, represents the parameters of the jth negative critic network; The training objective of the negative critic network is to minimize the gap between the value estimate and the target value: in represents all negative critic networks, φ - represents the parameters of all negative critic networks, s t , r t ,d t They represent the environment state information at time t, the actions taken by the agent, the rewards given by the environment, and the completion status of the environment, respectively. t+1 represents the environmental status information at time t+1, y - represents the target value composed of the immediate reward and the target negative critic network's estimate of the future value; The training objective of the actor network is to minimize the following formula: in is the j-th critic network, φ j is the parameter of the j-th target negative critic network, a t is the action obtained according to the actor network strategy, where: Where sim is the strategy similarity model, τ is the temperature balance coefficient, N is the total number of negative action samples generated by the negative actor network strategy, According to s t The strategy corresponding to the positive actor network Get all According to s t The strategy corresponding to the negative actor network get; The training objective of the critic network is to minimize the gap between the value estimate and the target value: Where Q φ represents all critic networks, φ represents the parameters of all critic networks, s t , r t ,d t They represent the environment state information at time t, the actions taken by the agent, the rewards given by the environment, and the completion status of the environment, respectively. t+1 represents the state of the environment at time t+1, and y represents the target value composed of the immediate reward and the target critic network's estimate of the future value.

5. The positive and negative network comparison reinforcement learning method for highly dynamic decision-making scene agent exploration according to claim 4, characterized in that: The target value y + The specific calculation method is: Where γ is the discount factor, which discounts future rewards. is the jth target positive critic network, are the parameters of the j-th target positive critic network, According to s t+1 Obtained by the strategy corresponding to the positive actor network; The target value y - The calculation method is as follows: in is the j-th target negative critic network, are the parameters of the j-th target negative critic network, According to s t+1 Obtained by the strategy corresponding to the negative actor network; The target value y is calculated as follows: in is the jth target critic network, is the parameter of the jth target critic network, a t+1 According to s t+1 Obtained by the strategy corresponding to the actor network.

6. The positive and negative network comparison reinforcement learning method for intelligent agent exploration in highly dynamic decision-making scenarios as claimed in claim 4, characterized in that: The training goal of the autoencoder of the policy similarity model is to minimize the gap between the next environmental state prediction and the actual next environmental state: Among them, AE is the autoencoder model, β is the parameter of the autoencoder model, and s t is the current time step environment state, action a t From the main strategy, MSE is the mean square error function, s t+1 is the environmental state at the next time step, The prediction of the environment state at the next time step obtained by the autoencoder; The strategy similarity calculation method is as follows: in Indicates that in the environmental state s t Next, actions from different strategies The similarity between them, d is the Euclidean distance function, cos is the cosine similarity function, The calculation method is as follows: Where Encoder is the encoder model, s t is the state of the environment at the current time step, Get actions for a policy.

7. The positive and negative network comparison reinforcement learning method for highly dynamic decision-making scene agent exploration according to claim 6, characterized in that: In the strategy action selection model, the probability of an action being selected is obtained in the form of Softmax normalization, where the probability of the action corresponding to the main strategy being selected is calculated as follows: where s t is the current environment state, action a t is the action obtained through the main strategy, is the action obtained by the forward strategy, is the action obtained by the negative strategy; The probability of the action corresponding to the forward strategy being selected is calculated as follows: The probability of the action corresponding to the negative strategy being selected is calculated as follows:

Images