Reinforcement learning system and method for high dynamic decision scenario agent exploration

Abstract

The application discloses a reinforcement learning system and method for high dynamic decision scene agent exploration, which comprises a positive actor-critic model, a negative actor-critic model, an actor-critic model, a strategy similarity model and a strategy action selection model. The learning of a main strategy is guided by using a positive strategy and a negative strategy, and the closeness of the main strategy to the positive strategy is minimized and the closeness of the main strategy to the negative strategy is maximized, so that the main strategy generates high-value actions with a high probability and generates low-value actions with a low probability, and the overall performance of the method is improved. Meanwhile, in the training process, the strategy action selection model is used to select the main strategy action, the positive strategy action and the negative strategy action in the form of probability, the diversity of action selection is increased, the diversity of the exploration environment is indirectly improved, and finally the performance is improved.

Description

Technical Field

[0001] This invention belongs to the technical field of deep reinforcement learning, and particularly relates to an online reinforcement learning algorithm based on the actor-critic framework. It mainly relates to a reinforcement learning system and method for agent exploration in highly dynamic decision-making scenarios. Background Technology

[0002] In unmanned agent control scenarios, unmanned agents often need to operate in complex and dynamically changing environments, where uncertainties increase the difficulty of perception and decision-making. In highly dynamic decision-making scenarios, unmanned agents need to cope with complex terrain, weather conditions, and dynamic battlefield situations, while also dealing with enemy electronic interference and cyberattacks. Under these circumstances, unmanned agents need to autonomously complete basic actions such as movement, hovering, and interference to achieve complex tasks such as reconnaissance, patrolling, resupply, and support.

[0003] Specifically, in special environments, unmanned intelligent agents may need to traverse diverse terrains such as mountains, forests, and deserts, making perception and navigation extremely challenging. Meanwhile, rapid changes in weather conditions, such as storms, rain, and snow, can significantly impact the stable flight of unmanned intelligent agents and the accuracy of their sensors. Furthermore, enemy electronic interference and cyberattacks, such as signal jamming, camouflage, and cyberattacks, further increase the uncertainty and risks for unmanned intelligent agents during mission execution.

[0004] To meet these challenges, unmanned intelligent agents not only need strong perception and environmental adaptability, but also need to make efficient decisions in rapidly changing and unique scenarios. This includes autonomously selecting the best course of action, dynamically adjusting task priorities, and being able to complete tasks independently even in the event of communication disruptions or enemy interference. Unmanned intelligent agents need to possess high flexibility and robustness to cope with various unforeseen situations on the battlefield.

[0005] Furthermore, the complexity and diversity of special mission scenarios place higher demands on unmanned intelligent agents. For example, in reconnaissance missions, unmanned intelligent agents need to conduct covert operations within enemy fire coverage areas while collecting high-value intelligence; in patrol missions, unmanned intelligent agents need to cover a wide area to identify and respond to potential threats; in resupply missions, unmanned intelligent agents need precise navigation to ensure accurate delivery of supplies; and in support missions, unmanned intelligent agents need to coordinate with ground forces to provide real-time intelligence and fire support.

[0006] In this context, how can we efficiently explore complex battlefield environments and develop algorithms for rapid convergence?

[0007] This presents a core challenge for autonomous control of unmanned intelligent agents in highly dynamic decision-making scenarios. The exploration algorithm needs to adapt to environmental changes rapidly, quickly finding the optimal path and making decisions; the convergence algorithm needs to achieve a convergence result within a limited time.

[0008] Optimal performance is crucial to ensuring the efficiency and accuracy of unmanned intelligent agents in mission execution. Overall, the autonomous control of unmanned intelligent agents in highly dynamic decision-making scenarios requires the comprehensive application of intelligent decision-making algorithms and robust environmental exploration capabilities to cope with complex and ever-changing battlefield environments and mission requirements.

[0009] Deep reinforcement learning is an important research direction in the field of artificial intelligence in recent years. It combines deep learning with reinforcement learning, enabling intelligent agents to autonomously learn decision-making strategies in complex and high-dimensional environments.

[0010] (Omitted). Traditional reinforcement learning methods are often inefficient when dealing with high-dimensional state and action spaces, and struggle to handle large-scale data. Deep learning, on the other hand, excels at extracting features and performing pattern recognition from high-dimensional data, effectively solving this problem. Therefore, deep reinforcement learning is widely used in fields such as games, robot control, and drone control.

[0011] The 40 actor-critic-based online reinforcement learning method represents an important class of deep reinforcement learning algorithms.

[0012] The method separates the learning processes of the policy function and the value function, where the "actor" is responsible for selecting actions, while...

[0013] The "critic" network is responsible for evaluating the actor's chosen actions and providing feedback. The actor network generates an action probability distribution based on the state, while the critic network evaluates the value of the current strategy and updates the value function using temporal difference errors.

[0014] The advantage of this approach is that it can perform policy updates more stably and can effectively handle the continuous action space.

[0015] In the actor-critic architecture-based algorithm, the TD3 algorithm employs a dual-network design, using the minimum value in two independent target Q-networks to estimate the Q-value, effectively mitigating the Q-value problem in policy gradient methods.

[0016] The TD3 algorithm addresses the overestimation problem and employs a delayed update strategy to stabilize the network update process. It also addresses the continuous control issue.

[0017] It performs excellently on tasks, significantly improving training stability and performance. However, its training process remains slow, requiring a large amount of environmental interaction data. The SAC algorithm introduces a maximum entropy framework to simultaneously optimize the expected return of the policy.

[0018] The algorithm uses entropy values ​​for reporting and policy to encourage agent exploration; simultaneously, it employs an automatic temperature parameter adjustment mechanism to balance exploration and exploitation. The SAC algorithm performs well in tasks requiring extensive exploration; however, the complex computation and parameter tuning process significantly increases the difficulty of convergence.

[0019] A crucial prerequisite for the convergence of actor-critic-based online reinforcement learning methods is that the agent needs to explore a sufficient number of environmental states; that is, the policy needs to do more than just grant the agent higher environmental rewards.

[0020] It is necessary to explore as many diverse environments as possible so that the agent can break out of local optima and achieve better overall performance.

[0021] Intrinsic motivation-based exploration (IM) algorithms promote exploration by rewarding agents' exploratory behavior in the environment.

[0022] The learning process utilizes estimations from environmental dynamics models, using model prediction errors to define intrinsic rewards, thus encouraging agents to explore unknown states. The ICM algorithm performs exceptionally well in sparse reward environments, significantly improving intelligence.

[0023] The ability to explore potential entities. However, in complex environments, training environmental dynamics models can incur additional computational burdens. The RND algorithm proposes an exploration method based on random networks, defining intrinsic rewards by comparing the output differences of two neural networks. Compared to the ICM algorithm, the RND algorithm is simpler to implement and easier to use.

[0024] Training is possible, but the definition of its intrinsic reward depends on the initialization of the random network, leading to instability in the exploration process. Therefore, an effective environmental exploration design is urgently needed to help agents explore the environment more stably and efficiently, achieving [positive results].

[0025] To achieve better performance and improve the versatility of the algorithm. Summary of the Invention

[0026] This invention aims to overcome the shortcomings of existing actor-critic-based online reinforcement learning methods in terms of environment exploration capabilities, and proposes a reinforcement learning system and method for agent exploration in highly dynamic decision-making scenarios.

[0027] The methods include positive actor-critic model, negative actor-critic model, actor-critic model, policy similarity model, and policy action selection model. These methods guide the learning of the primary policy using both positive and negative policies, while minimizing the proximity of the primary policy to the positive policy and maximizing the proximity of the primary policy to the negative policy.

[0028] By increasing the probability of the primary policy generating high-value actions with a high probability and low-value actions with a low probability, the overall performance of Method 75 is improved. Simultaneously, during training, the policy action selection model probabilistically selects the primary policy action to execute.

[0029] Strategic actions, positive strategic actions, and negative strategic actions increase the diversity of action choices, indirectly improving the diversity of the exploration environment and ultimately achieving performance improvement.

[0030] To achieve the above objectives, the technical solution adopted by this invention is: a reinforcement learning system for intelligent agent exploration in highly dynamic decision-making scenarios, comprising at least a positive actor-critic model, a negative actor-critic model, an actor-critic model, a policy similarity model, and a policy-action selection model.

[0031] The positive actor-critic model includes a positive actor network and a positive critic network. The positive actor network acts as a positive strategy, generating positive actions by maximizing the Q-value. The positive critic network aims to minimize the temporal difference error between the generated Q-value and the target Q-value, and is used to give the value of the actions generated by the positive actor network in the current state.

[0032] The negative actor-critic model described in 85 includes a negative actor network and a negative critic network. The negative actor network acts as a negative strategy, generating negative actions by minimizing the Q-value. The negative critic network aims to minimize the temporal difference error between the generated Q-value and the target Q-value, and is used to give the value of the actions generated by the negative actor network in the current state.

[0033] The actor-critic model comprises an actor network and a critic network. The actor network serves as the primary strategy for generating actions that are close to positive strategies and far from negative strategies. The critic network is used to provide the value of the actions generated by the actor network in the current state.

[0034] The policy similarity model is implemented by an autoencoder, which takes two policies as input and outputs the similarity between the two actions.

[0035] The strategy action selection model: Based on the value of the action corresponding to the main strategy, positive strategy, and negative strategy, the model probabilistically selects the action to be executed and interact with the environment.

[0036] As an improvement of the present invention, the actor-critic model, the positive actor-critic model, and the negative actor-critic model are all deep neural networks. The deep neural network processes vector signals through 3 fully connected layers and processes image signals through 3 convolutional layers plus 2 fully connected layers.

[0037] To achieve the above objectives, the present invention also adopts the following technical solution: a positive and negative network contrastive reinforcement learning method for intelligent 100-body exploration in highly dynamic decision-making scenarios, which includes at least two stages: training and execution.

[0038] For continuous action scenarios: During the training phase, the agent's actions are selected by the policy action selection model from the actions obtained from the main policy, positive policy, and negative policy. The main policy, positive policy, and negative policy are all determined by the Gaussian distribution generated by the corresponding actor network, and the actions are obtained by resampling the Gaussian distribution.

[0039] During the execution phase, the mean of the Gaussian distribution corresponding to the main policy is used as the agent's action;

[0040] For discrete action scenarios: During the training phase, the agent's actions are selected by the policy action selection model from the actions obtained from the main policy, positive policy, and negative policy. The main policy, positive policy, and negative policy are all directly generated by the corresponding actor network to determine the probability of all actions being selected. The actions are obtained by sampling through the corresponding category distribution.

[0041] During the execution phase, the action with the highest probability of the corresponding category of the main policy is used as the agent's action.

[0042] As an improvement of the present invention, the training objective of the positive actor network is to maximize the expected Q value:

[0043]

[0044] in For the parameters of the positive actor network, s t Let D be the environmental state information at time t, and D be the experience replay pool. The actions obtained based on the positive actor network strategy, Let j represent the network of positive commentators. This represents the parameters of the j-th positive commentator network;

[0045] The training objective of the positive commentator network is to minimize the gap between the value estimate and the target value.

[0046]

[0047] in Represents the network of all positive commentators, φ + s represents the parameters of all positive commentator networks. t , r t ,d t Let represent the environmental state information at time t, the action taken by the agent, the reward given by the environment, and the completion status of the environment, respectively. t+1y represents the environmental state information at time t+1. + This represents the target value, which is a combination of immediate rewards and the target positive commentator network's estimate of future value.

[0048] The training objective of the negative actor network is to minimize the expected Q value:

[0049]

[0050] in For the parameters of the negative actor network, s t Let D be the environmental state information at time t, and D be the experience replay pool. To obtain actions based on the negative actor network strategy, Let j represent the network of negative commentators. This represents the parameters of the j-th negative critic network;

[0051] The training objective of the negative critic network is to minimize the gap between the value estimate and the target value.

[0052]

[0053] in Represents the network of all negative critics, φ - s represents the parameters of the entire negative critic network. t , r t ,d t Let represent the environmental state information at time t, the action taken by the agent, the reward given by the environment, and the completion status of the environment, respectively. t+1 y represents the environmental state information at time t+1. - This represents the target value, which is a combination of immediate rewards and the target negative commentator network's estimate of future value.

[0054] The training objective of the actor network is to minimize the following formula:

[0055]

[0056] in For the j-th critic network, φ j Let a be the parameter of the negative commentator network for the j-th objective. t For actions obtained based on the actor's network strategy, where:

[0057]

[0058] Where sim is the strategy similarity model, τ is the temperature balance coefficient, and N is the total number of negative action samples generated by the negative actor network strategy. According to s tThe strategy corresponding to the positive actor network Get, all According to s t Strategy corresponding to negative actor network get;

[0059] The training objective of the critic network is to minimize the gap between the value estimate and the target value.

[0060]

[0061] Q φ Let φ represent all critic networks, and s represent the parameters of all critic networks. t , r t ,d t Let represent the environmental state information at time t, the action taken by the agent, the reward given by the environment, and the completion status of the environment, respectively. t+1 y represents the environmental state information at time t+1, and y represents the target value, which is a combination of immediate reward and the target critic network's estimate of future value.

[0062] As another improvement of the present invention, the target value y + The specific calculation method is as follows:

[0063]

[0064] Where γ is the discount factor, which discounts future rewards. For the j-th target positive commentator network, Let j be the parameters of the positive commentator network for the j-th target. According to s t+1 This is obtained from the strategy corresponding to the positive actor network;

[0065] The target value y - The calculation method is as follows:

[0066]

[0067] in For the j-th objective, negative commentator network, Let j be the parameters of the negative critic network for the j-th target. According to s t+1 This is obtained from the strategy corresponding to the negative actor network;

[0068] The target value y is calculated as follows:

[0069]

[0070] in For the j-th target commentator network, Let a be the parameter of the network for the j-th target critic. t+1 According to s t+1 It is obtained from the strategy corresponding to the actor network.

[0071] As another improvement of the present invention, the training objective of the autoencoder of the policy similarity model is to minimize the gap between the predicted next-step environment state and the actual next-step environment:

[0072]

[0073] Where AE is the autoencoder model, β is the parameter of the autoencoder model, and s t Given the current time step environment state, action a t Derived from the main policy, MSE is the mean squared error function, s t+1 The environmental state for the next time step. This is the prediction of the environmental state for the next time step obtained by the autoencoder;

[0074] The strategy similarity calculation method is as follows:

[0075]

[0076] in Indicates the state of environment s t Below, actions from different strategies The similarity between them is given by d, where d is the Euclidean distance function and cos is the cosine similarity function. The calculation method is as follows:

[0077]

[0078] Where Encoder is the encoder model, s t This represents the current environmental state at the current time step. A strategy yields an action.

[0079] As a further improvement of the present invention, in the strategy action selection model, the probability of an action being selected is obtained in a Softmax normalized form, wherein the probability of the action corresponding to the main strategy being selected is calculated as follows:

[0080]

[0081] Where s t Given the current environment state, action a t For actions obtained through the main strategy, For actions obtained through a positive strategy, The action obtained through a negative strategy;

[0082] The probability of an action being selected for a positive strategy is calculated as follows:

[0083]

[0084] The probability of an action being selected for a negative strategy is calculated as follows:

[0085]

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] (1) This invention uses positive and negative strategies and randomizes and probabilistically selects actions using an action selection model, which increases the diversity of action selection for unmanned intelligent agents and improves their ability to explore complex environments, especially highly dynamic decision-making environments.

[0088] (2) This invention uses the similarity between the positive strategy, the negative strategy and the main strategy as a guide for the learning of the main strategy, optimizes the learning process of the main strategy, helps the main strategy to escape the local optimum and reach the global optimum, and improves the actual performance of the main strategy.

[0089] (3) The present invention uses positive and negative strategies to guide the learning of the main strategy, which enhances the directionality of strategy learning, helps the main strategy to converge to the global optimal solution more quickly, and improves the convergence speed of the method in the face of complex environments and diverse action spaces. Attached Figure Description

[0090] Figure 1 This is a schematic diagram of the architecture of positive and negative network contrastive learning in the method of this invention;

[0091] Figure 2 This is a schematic diagram of the autoencoder model in the system strategy similarity model of this invention;

[0092] Figure 3 This is a schematic diagram of the architecture for simulating the strategy similarity model calculation in the method of this invention;

[0093] Figure 4 This is a schematic diagram of the action selection model in the system of the present invention;

[0094] Figure 5 This is a flowchart of the overall operation and training process of this invention;

[0095] Figure 6 This is a schematic diagram of the main principle of the method of the present invention. Detailed Implementation

[0096] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0097] Example 1

[0098] A reinforcement learning system for exploring intelligent agents in highly dynamic decision-making scenarios includes at least a positive actor-critic model, a negative actor-critic model, an actor-critic model, a policy similarity model, and a policy-action selection model. The positive actor-critic model includes a positive actor network and a positive critic network. The positive actor network acts as a positive policy, generating positive actions by maximizing the Q-value. The positive critic network aims to minimize the temporal difference error between the generated Q-value and the target Q-value, providing the value of the actions generated by the positive actor network in the current state. The negative actor-critic model includes a negative actor network and a negative critic network. The negative actor network acts as a negative policy, generating negative actions by minimizing the Q-value. The negative critic network aims to minimize the temporal difference error between the generated Q-value and the target Q-value, providing the value of the actions generated by the negative actor network in the current state. The actor-critic model includes an actor network and a critic network. The actor network acts as the master policy, generating actions that are close to the positive policy and far from the negative policy. The critic network provides the value of the actions generated by the actor network in the current state. The policy similarity model is implemented by an autoencoder, taking the actions generated by two policies as input and outputting the similarity between the two actions. The policy-action selection model probabilistically selects the action execution and interaction with the environment based on the values ​​corresponding to the actions obtained from the master policy, positive policy, and negative policy.

[0099] In the Positive Actor-Critic model, the Positive Actor network aims to maximize the Q-value to obtain higher-value actions. This optimization objective is particularly important in highly dynamic decision-making scenarios because it helps the agent select the optimal action strategy in complex and ever-changing environments, thereby efficiently completing various tasks. Through learning and optimization, the Positive Actor network enables the agent to consistently choose actions that maximize long-term cumulative rewards when performing tasks. These actions are not only optimal in the current situation but also retain high value in the face of various changes and challenges that may arise in the future. For example, in strike missions, the Positive Actor network guides the agent to choose the best attack route and timing, thereby inflicting maximum damage on the target in the shortest time. In reconnaissance missions, the Positive Actor network can guide the agent to avoid enemy defense systems and safely and efficiently collect critical intelligence. In support missions, the Positive Actor network can help the agent quickly respond to friendly forces' distress signals, providing necessary fire support or resource replenishment to ensure the friendly forces' advantageous position. In patrol missions, the Positive Actor network enables the agent to rationally plan patrol routes, covering a larger area, while promptly detecting and responding to potential threats. In escort missions, the positive actor network guides the agent to select the optimal escort strategy, protecting important targets from enemy attacks. By maximizing the Q-value, the positive actor network not only improves the agent's performance in single tasks but also enhances its comprehensive capabilities in multi-task environments. In highly dynamic decision-making scenarios, agents need to face various emergencies and complex environmental changes. The optimization objective of the positive actor network enables the agent to quickly adapt and efficiently respond to these challenges. This not only improves the efficiency and accuracy of task completion but also significantly increases the agent's survivability and overall combat value on the battlefield. Therefore, in highly dynamic decision-making scenarios, the positive actor network, by maximizing the Q-value, enables the agent to take appropriate actions and efficiently complete tasks such as strikes, reconnaissance, support, patrols, and escorts, thereby achieving higher simulation value and combat effectiveness. Through continuous optimization and learning, the positive actor network enables agents to perform well in various complex tasks, providing solid technical support for achieving autonomous control and efficient combat of unmanned intelligent agents. The positive critic network aims to minimize the temporal difference error between the generated Q-value and the target Q-value to obtain better value estimation.

[0100] The overall goal of the negative actor-critic model is to generate low-quality actions. The negative actor network aims to minimize the Q-value, specifically by acquiring actions with lower value. In highly dynamic decision-making scenarios, the optimization objective of this network manifests as action policies that are completely opposite to the positive policy, providing negative examples. These negative examples, through negative guidance, help the main policy learn and optimize better. Specifically, the negative actor network creates a set of suboptimal or inferior policies by learning and selecting actions that may lead to lower rewards. These policies perform poorly in practical applications, but their existence has important reference value for the optimization of the main policy. For example, in executing a strike mission, the negative actor network may select actions that delay the attack, expose the location, or waste resources. These actions may seem harmless in the short term, but may lead to mission failure or significant losses in the long term. By observing these negative policies, the main policy can identify and avoid these potential errors. In reconnaissance missions, the negative actor network may guide the agent to take paths and methods that are easily detected by the enemy and fail to gather effective intelligence. By comparing itself with these inefficient strategies, the main strategy can better optimize its reconnaissance paths and concealment methods, improving the efficiency and security of intelligence gathering. In support missions, negative actor networks may choose actions that fail to respond to friendly forces' needs in a timely manner or have unreasonable resource allocation; the main strategy can learn from this how to more effectively allocate resources and provide support responses. In patrol missions, negative actor networks may plan schemes with insufficient coverage areas and unreasonable patrol routes; the main strategy can compare these schemes to optimize its own patrol paths and strategies, ensuring broader area coverage and more timely threat response. In escort missions, negative actor networks may adopt strategies with weak defenses and exposed targets; the main strategy can learn from this inverse process to formulate more robust and effective escort strategies, protecting important targets from enemy attacks. By minimizing the Q-value, negative actor networks not only provide examples of negative strategies but also help the main strategy identify and understand what constitutes suboptimal or poor decisions to a certain extent. This inverse guidance mechanism has unique value for the optimization of the main strategy, helping it converge to the optimal solution more quickly, thereby improving the agent's overall performance in highly dynamic decision-making scenarios.

[0101] In summary, in highly dynamic decision-making scenarios, the negative actor network aims to minimize the Q-value by providing negative examples of low-value actions, thus guiding the learning and optimization of the main policy in an alternative way. Through this reverse guidance mechanism, the main policy can better identify and avoid inefficient or erroneous decisions, thereby achieving more efficient and accurate autonomous control. The negative critic network aims to minimize the temporal difference error between the generated Q-value and the target Q-value to obtain better value estimation.

[0102] The actor-critic model, while considering traditional optimization objectives, is guided by positive and negative actor-critic models to obtain more accurate high-value actions. The actor model's objective is divided into two parts: the first part is to maximize the Q-value of the generated action, and the second part is to minimize the contrast loss between the generated action and positive and negative policy actions based on the policy similarity model. The critic model aims to minimize the temporal difference error between the generated Q-value and the target Q-value to obtain better value estimation.

[0103] Traditional methods only use the actor-critic model, which is inadequate in highly dynamic decision-making scenarios where situations change rapidly.

[0104] Traditional actor-critic models struggle to adapt quickly to frequent changes and complex disturbances, leading to lag in policy updates and impacting the timeliness and effectiveness of decision-making. Since updates rely on iterative policy processes—policy evaluation and policy improvement—errors in either process accumulate, causing actor networks to become trapped in local optima in complex decision spaces, hindering the exploration of globally optimal policies. Critics networks may also exhibit inaccurate value estimations, affecting overall policy optimization. Furthermore, traditional actor-critic models often rely on randomness or simple heuristics when exploring new policies, resulting in inefficient exploration and difficulty in efficiently discovering high-value policies in complex environments. Finally, the performance of actor-critic models can significantly degrade in uncertain and noisy environments, exhibiting poor robustness. This is particularly evident in high-risk and adversarial environments.

[0105] The actor-critic model based on contrastive learning using positive and negative networks enables the agent to identify and avoid low-value actions during exploration, thereby discovering high-value strategies more efficiently and adapting to environmental changes more quickly. The negative examples provided by the negative actor network help the critic network more accurately evaluate the value of actions, allowing the actor network to converge to a better strategy more rapidly. The introduction of the negative actor network enables the agent to identify and avoid potential erroneous decisions, improving the robustness of the overall policy. The agent can perform tasks more robustly in the face of uncertainty and noise. Furthermore, through contrastive learning, the positive and negative actor networks work together to help the agent escape local optima. The positive actor network focuses on maximizing the Q-value, while the negative actor network minimizes it; their mutual comparison makes policy optimization more comprehensive and balanced.

[0106] A policy similarity model is set up, implemented by an autoencoder. The encoder of the autoencoder takes the environment state and policy actions as input and outputs a latent representation; the decoder takes the latent representation output by the encoder as input and outputs a prediction of the environment state at the next time step. The training objective of the autoencoder is to minimize the gap between the actual environment state at the next time step and the environment state prediction generated by the autoencoder. Policy similarity is obtained from the environment state and two policy actions. Specifically, the environment state and policy actions are first processed by the encoder to obtain latent representations, and then the final policy similarity is obtained based on the Euclidean distance and cosine similarity distance between the two latent representations.

[0107] A policy-action selection model is established. During training, given an environment, the agent probabilistically selects actions to interact with the environment based on the values ​​corresponding to actions obtained from the primary policy, positive policy, and negative policy. The action selection probability is obtained by normalizing the Q-value of the corresponding policy using Softmax. This action selection model adds randomness to the agent's exploration of highly dynamic environments, significantly improving the agent's environment exploration capabilities.

[0108] Example 2

[0109] A reinforcement learning method for agent exploration in highly dynamic decision-making scenarios is proposed, aiming to address the shortcomings of traditional actor-critic architecture-based online deep reinforcement learning methods, such as insufficient adaptability to highly dynamic environments, insufficient environment exploration capabilities, insufficient utilization capabilities, and slow convergence speed. Using the system described in Example 1, the actor-critic architecture-based online deep reinforcement learning employs additional positive and negative policies for policy learning guidance. The training of the positive policy is consistent with the basic actor-critic algorithm, aiming to generate high-value actions. The training objective of the negative policy is opposite to that of the basic actor-critic algorithm, aiming to generate low-value actions. The positive, negative, and main policy actions are calculated using a policy similarity model to obtain a contrastive loss, which serves as an additional training objective for the main policy. During the overall training process, the actions of the agent's interaction with the environment are obtained probabilistically by an action selection model based on the value of different policies.

[0110] This method follows a Markov decision process in its overall training flow. The agent accumulates experience data through interaction with the environment, and a certain batch of past experience data is randomly sampled for training during the training process. The agent obtains corresponding actions based on the policy network and action selection model, and executes them in the environment. The environment simulates state transitions based on the current state and the agent's actions, obtaining the reward for this step and the next state, repeating this process. Through interaction with the environment, the agent accumulates sufficiently diverse and high-quality data, assisting the reinforcement learning algorithm in completing an efficient learning process and obtaining a high-performance agent.

[0111] Reinforcement learning methods for agent exploration in highly dynamic decision-making scenarios, such as Figure 1 As shown, this method designs positive and negative policies to guide policy learning, with the right side of each network representing its learning objective. The positive actor-critic model comprises a positive actor network and a positive critic network, with the training objective of the positive actor network maximizing:

[0112]

[0113] in For the parameters of the positive actor network, s t Let D be the environmental state information at time t, and D be the experience replay pool. The actions obtained based on the positive actor network strategy, Let j represent the network of positive commentators. This represents the parameters of the j-th positive commentator network;

[0114] The training objective of the positive commentator network is to minimize:

[0115]

[0116] in Represents the network of all positive commentators, φ + s represents the parameters of all positive commentator networks. t , r t ,d t Let represent the environmental state information at time t, the action taken by the agent, the reward given by the environment, and the completion status of the environment, respectively. t+1 This represents the environmental state information at time t+1. All five data points above come from the experience replay pool D, where y... + The calculation method is as follows:

[0117]

[0118] Where γ is the discount factor, which discounts future rewards. For the j-th target positive commentator network, Let be the parameters of the positive commentator network for the j-th target, where According to s t+1 This is obtained from the strategy corresponding to the positive actor network;

[0119] The training objective of the negative actor network is to minimize:

[0120]

[0121] in For the parameters of the negative actor network, s tLet D be the environmental state information at time t, and D be the experience replay pool. To obtain actions based on the negative actor network strategy, Let j represent the network of negative commentators. This represents the parameters of the j-th negative critic network;

[0122] The training objective of the negative critic network is to minimize:

[0123]

[0124] in Represents the network of all negative critics, φ - s represents the parameters of the entire negative critic network. t , r t ,d t Let represent the environmental state information at time t, the action taken by the agent, the reward given by the environment, and the completion status of the environment, respectively. t+1 This represents the environmental state information at time t+1. All five data points above come from the experience replay pool D, where y... - The calculation method is as follows:

[0125]

[0126] Where γ is the discount factor, which discounts future rewards. For the j-th objective, negative commentator network, Let be the parameters of the negative critic network for the j-th target, where According to s t+1 This is obtained from the strategy corresponding to the negative actor network;

[0127] The training objective of the actor network is to minimize:

[0128]

[0129] in For the j-th critic network, φ j Let a be the parameter of the negative commentator network for the j-th objective. t For actions obtained based on the actor's network strategy, where:

[0130]

[0131] Where sim is the strategy similarity model, τ is the temperature balance coefficient, and N is the total number of negative action samples generated by the negative actor network strategy. According to s t The strategy corresponding to the positive actor network Get, all According to st Strategy corresponding to negative actor network get;

[0132] The training objective of the critic network is to minimize:

[0133]

[0134] Q φ Let φ represent all critic networks, and s represent the parameters of all critic networks. t , r t ,d t Let represent the environmental state information at time t, the action taken by the agent, the reward given by the environment, and the completion status of the environment, respectively. t+1 This represents the environmental state information at time t+1. The above five data points all come from the experience replay pool D, where y is calculated as follows:

[0135]

[0136] Where γ is the discount factor, which discounts future rewards. For the j-th target commentator network, Let a be the parameter of the network for the j-th target critic, where a t+1 According to s t+1 It is obtained from the strategy corresponding to the actor network.

[0137] The actor-critic model, positive actor-critic model, and negative actor-critic model are all deep neural networks, and different architectures are used to process the environmental state representations of different application scenarios. For vector signal representation, a network architecture of 3 fully connected layers is used; for image signal representation, a network architecture of 3 convolutional layers plus 2 fully connected layers is used.

[0138] The target critic network parameters are not updated during algorithm training. Instead, the target critic network uses a soft update method for parameter updates. That is, each time the parameters of its corresponding critic network are updated, the critic network parameters are copied to the target critic network in a certain proportion. The soft update formula is as follows:

[0139]

[0140] Where σ is the soft update parameter, used to control the parameter update ratio and prevent parameter updates from becoming unstable.

[0141] The discount factor γ is set between 0 and 1 to measure the impact of future rewards on current decisions, preventing the cumulative rewards from growing indefinitely, while making the strategy more suitable for long-term planning.

[0142] The temperature balance coefficient τ controls the smoothness of the softmax function, affects the steepness of the similarity distribution, balances the contributions of positive and negative policy actions to the contrastive loss, effectively distinguishes positive and negative samples, and thus improves the contrastive learning effect of the algorithm.

[0143] In the high-dynamic decision-making scenario of the UAV, the environmental state includes two parts. The first part is the UAV's partial observation information of the environment. The environmental state is sampled as an 84x84 grayscale image, and the pixel values ​​are normalized to [0,1]. Four consecutive frames of images are stacked, and the final state is represented as an 84x84x4 stacked grayscale image. The second part is the agent state information, which includes agent coordinates, heading, radar on / off status, radar frequency, jamming radar on / off, jamming radar frequency, list of enemy units detected by the radar, and number of remaining missiles. All agent state information is composed of a high-dimensional vector information signal.

[0144] like Figure 2 , 3 As shown, the method of this invention designs a policy similarity model, which accepts the environment state and actions of different policies as input, and outputs the similarity between two policy actions in the current state. The policy similarity model consists of an encoder and a decoder of an autoencoder. The encoder's input is the current environment state and the action obtained by a certain policy, and it outputs a latent representation. The decoder's input is the latent representation output by the encoder, and it outputs a prediction of the environment state in the next time step. The training objective of the autoencoder is to minimize:

[0145]

[0146] Where AE is the autoencoder model, β is the parameter of the autoencoder model, and s t Given the current time step environment state, action a t Derived from the main policy, MSE is the mean squared error function, s t+1 The environmental state for the next time step. This is the prediction of the environmental state for the next time step obtained by the autoencoder;

[0147] Strategy similarity is calculated as follows:

[0148]

[0149] in Indicates the state of environment s t Below, actions from different strategies The similarity between them is given by d, where d is the Euclidean distance function and cos is the cosine similarity function. The Euclidean distance is calculated as follows:

[0150]

[0151] Where n is the potential representation Dimensions They represent In the i-th dimension component, the cosine similarity is calculated as follows:

[0152]

[0153] in For vectors Euclidean norm,

[0154] in The calculation method is as follows:

[0155]

[0156] Where Encoder is the encoder model, s t This represents the current environmental state at the current time step. A strategy yields an action.

[0157] like Figure 4 As shown, the method of this invention designs an action selection model. Under the current environmental state, based on the value of the actions obtained from the main policy, positive policy, and negative policy, the action corresponding to the policy is randomly selected in the form of probability. The probability of the action being selected is obtained in the form of Softmax normalization. The probability of the action corresponding to the main policy being selected is calculated as follows:

[0158]

[0159] The probability of an action being selected for a positive strategy is calculated as follows:

[0160]

[0161] The probability of an action being selected for a negative strategy is calculated as follows:

[0162]

[0163] During the training phase, the agent's actions are selected by the policy action selection model from the actions obtained from the main policy, positive policy, and negative policy. For continuous actions, the main policy, positive policy, and negative policy are all determined by the Gaussian distribution generated by the corresponding actor network, and the actions are obtained by resampling the Gaussian distribution. During the execution phase, the mean of the Gaussian distribution corresponding to the main policy is directly used as the agent's action. For discrete actions, the main policy, positive policy, and negative policy are all directly generated by the corresponding actor network to determine the probability of all actions being selected. The actions are obtained by sampling the corresponding category distribution, and during the execution phase, the action with the highest probability in the category distribution corresponding to the main policy is directly used as the agent's action.

[0164] The intelligent agent actions in the high-dynamic decision-making scenario of the UAV include two parts. The first part is the heading, which is a continuous value and takes the value [0, 359]. The second part is the radar switch and frequency, the jamming device switch and frequency, whether to launch missiles and attack targets, all of which are discrete integer values.

[0165] like Figure 5 As shown, the entire algorithm operation process is divided into an environment interaction phase and a training phase. In the environment interaction phase, the environment provides the current state. The agent observes the environment state and selects an action to interact with the environment based on the action selection model. The environment obtains the reward for this step and the state for the next step based on the agent's action. The agent stores the current state, the next step state, the action, and the reward information in the experience replay pool and determines whether to enter the training phase based on whether the update interval has been reached. In the training phase, the agent extracts a certain batch of data from the experience replay pool and uses this data to update the positive actor-critic network and the negative actor-critic network simultaneously. At the same time, the actor-critic network is updated according to the contrastive loss.

[0166] In summary, the method of this invention designs a positive and negative actor-critic model structure, uses positive and negative policies to guide the learning and training process of the main policy; uses a policy similarity model to measure the similarity between policies, making the main policy closer to the positive policy and further away from the negative policy; and uses an action selection model to obtain diverse actions to interact with the environment, thereby improving the algorithm's ability to explore highly dynamic environments.

[0167] 445 It should be noted that the above content merely illustrates the technical concept of the present invention and cannot be used to limit the scope of protection of the present invention. For those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A reinforcement learning system for agent exploration in highly dynamic decision-making scenarios, characterized by: It includes at least the positive actor-critic model, the negative actor-critic model, the actor-critic model, the strategy similarity model, and the strategy-action selection model. The positive actor-critic model includes a positive actor network and a positive critic network. The positive actor network acts as a positive strategy, generating positive actions by maximizing the Q-value. The positive critic network aims to minimize the temporal difference error between the generated Q-value and the target Q-value, and is used to give the value of the actions generated by the positive actor network in the current state. The negative actor-critic model includes a negative actor network and a negative critic network. The negative actor network acts as a negative strategy, generating negative actions by minimizing the Q-value. The negative critic network aims to minimize the temporal difference error between the generated Q value and the target Q value, and is used to give the value of the action generated by the negative actor network in the current state. The actor-critic model comprises an actor network and a critic network. The actor network serves as the primary policy for generating actions that are close to positive policies and far from negative policies. The critic network is used to provide the value of the actions generated by the actor network in the current state. The policy similarity model is implemented by an autoencoder, which takes two policies as input and outputs the similarity between the two actions. The strategy action selection model probabilistically selects the action to be executed and interact with the environment based on the value corresponding to the action obtained from the main strategy, positive strategy, and negative strategy.

2. The reinforcement learning system for agent exploration in highly dynamic decision-making scenarios as described in claim 1, characterized in that: The actor-critic model, positive actor-critic model, and negative actor-critic model are all deep neural networks. The deep neural networks process vector signals through 3 fully connected layers and image signals through 3 convolutional layers and 2 fully connected layers.

3. A reinforcement learning method for agent exploration in highly dynamic decision-making scenarios using the system described in claim 1, characterized in that: It includes at least two phases: training and execution. For continuous action scenarios: During the training phase, the agent's actions are selected by the policy action selection model from the actions obtained from the main policy, positive policy, and negative policy. The main policy, positive policy, and negative policy are all determined by the Gaussian distribution generated by the corresponding actor network, and the actions are obtained by resampling the Gaussian distribution. During the execution phase, the mean of the Gaussian distribution corresponding to the main policy is used as the agent's action; For discrete action scenarios: During the training phase, the agent's actions are selected by the policy action selection model from the actions obtained from the main policy, positive policy, and negative policy. The main policy, positive policy, and negative policy are all directly generated by the corresponding actor network to determine the probability of all actions being selected. The actions are obtained by sampling through the corresponding category distribution. During the execution phase, the action with the highest probability of the corresponding category of the main policy is used as the agent's action.

4. The reinforcement learning method for agent exploration in highly dynamic decision-making scenarios as described in claim 3, characterized in that: The training objective of the positive actor network is to maximize the expected Q value: ； in To provide parameters for the positive actor network, for Environmental state information at any given time For experience replay pool, The actions obtained based on the positive actor network strategy, Indicates the first A network of positive commentators, Indicates the first Parameters of a positive commentator network; The training objective of the positive commentator network is to minimize the gap between the value estimate and the target value. ； in This represents the network of all positive commentators. The parameters represent all positive commentator networks. They represent The information includes the environmental state at any given moment, the actions taken by the agent, the rewards given by the environment, and the completion status of the environment. express Environmental state information at any given time This represents the target value, which is a combination of immediate rewards and the target positive commentator network's estimate of future value. The training objective of the negative actor network is to minimize the expected Q value: ； in For the parameters of the negative actor network, for Environmental state information at any given time For experience replay pool, To obtain actions based on the negative actor network strategy, Indicates the first A network of negative critics, Indicates the first Parameters of a negative critic network; The training objective of the negative critic network is to minimize the gap between the value estimate and the target value. ； in This refers to the network of all negative critics. The parameters represent the network of all negative critics. They represent The information includes the environmental state at any given moment, the actions taken by the agent, the rewards given by the environment, and the completion status of the environment. express Environmental state information at any given time This represents the target value, which is a combination of immediate rewards and the target negative commentator network's estimate of future value. The training objective of the actor network is to minimize the following formula: ； in For the first A network of critics, For the first Parameters of a network of critics, For actions obtained based on the actor's network strategy, where: ； in For the strategy similarity model, The temperature balance coefficient, The total number of negative action samples generated by the negative actor network strategy. according to The strategy corresponding to the positive actor network Get, all according to Strategy corresponding to negative actor network get; The training objective of the critic network is to minimize the gap between the value estimate and the target value. ； in This represents the entire network of critics. The parameters represent the network of all critics. They represent The information includes the environmental state at any given moment, the actions taken by the agent, the rewards given by the environment, and the completion status of the environment. express Environmental state information at any given time This represents the target value, which is a combination of immediate rewards and the target commentator network's estimate of future value.

5. The reinforcement learning method for agent exploration in highly dynamic decision-making scenarios as described in claim 4, characterized in that: The target value The specific calculation method is as follows: ； in This is a discount factor that applies a discount to future rewards. For the first A network of commentators targeting specific goals. For the first The parameters of the target positive commentator network, according to This is obtained from the strategy corresponding to the positive actor network; The target value The calculation method is as follows: ； in For the first A network of negative commentators targeting specific goals. For the first Parameters of a target negative critic network according to This is obtained from the strategy corresponding to the negative actor network; The target value y is calculated as follows: ； in For the first A network of target critics, For the first Parameters of a target critic network, according to It is obtained from the strategy corresponding to the actor network.

6. The reinforcement learning method for agent exploration in highly dynamic decision-making scenarios as described in claim 4, characterized in that: The training objective of the autoencoder in the policy similarity model is to minimize the gap between the predicted next-step environment state and the actual next-step environment. ； in, It is an autoencoder model. For the parameters of the autoencoder model, Given the current time step environment state, the action From the main strategy, Let be the mean square error function. The environmental state for the next time step. This is the prediction of the environmental state for the next time step obtained by the autoencoder; The strategy similarity calculation method is as follows: ； in Indicates the environmental state Below, actions from different strategies Similarity between them Let be the Euclidean distance function. The cosine similarity function is used. The calculation method is as follows: ； in For the encoder model, This represents the current environmental state at the current time step. A strategy yields an action.

7. The reinforcement learning method for agent exploration in highly dynamic decision-making scenarios as described in claim 6, characterized in that: In the policy action selection model, the probability of an action being selected is obtained in the form of Softmax normalization, wherein the probability of the action corresponding to the main policy being selected is calculated as follows: ； in Given the current environment state, the action For actions obtained through the main strategy, For actions obtained through a positive strategy, The action obtained through a negative strategy; The probability of an action being selected for a positive strategy is calculated as follows: ； The probability of an action being selected for a negative strategy is calculated as follows: 。

Images