Waveform parameter intelligent decision-making method based on virtual game Nash equilibrium solution

Abstract

The invention provides a waveform parameter intelligent decision-making method based on virtual game Nash equilibrium solution. According to the technical scheme, the method comprises the four stages of processes of virtual game bilateral agent framework construction, Markov decision process modeling, Nash equilibrium solution algorithm design and network training optimization. The method comprises the following steps: establishing a double-agent system with a competitive relationship, defining a joint observation space and a strategy space, designing a state space, an action space and a return function, realizing dynamic description of environment interaction, adopting an NFSP network architecture and a mixed reward function, ensuring that a strategy is converged to Nash equilibrium, and implementing network training and optimization in combination with a four-stage training process. And the convergence speed and adaptability of the algorithm are improved. According to the method, reinforcement learning is innovatively used for waveform parameter solving, intelligent optimization of waveform parameters in a complex environment is realized by constructing a virtual game framework and a Nash equilibrium solving algorithm, and the robustness and efficiency of decision making are improved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent waveform design, specifically involving an intelligent decision-making method for waveform parameters based on virtual game Nash equilibrium solution. Background Technology

[0002] With the rapid development of wireless communication and radar, waveform parameter optimization has become one of the core challenges in improving system performance. Traditional waveform parameter decision-making methods are usually based on fixed rules or static optimization algorithms, which are difficult to adapt to real-time requirements in dynamic environments. Especially in complex electromagnetic environments and multi-agent scenarios, the communicating parties need to make adversarial decisions under limited spectrum resources. Traditional methods suffer from three major bottlenecks: poor environmental adaptability, single waveform strategy, and high computational complexity. Current waveform parameter optimization mainly relies on rule-based methods and traditional optimization-based methods. The former selects parameters through predefined expert experience or threshold rules, but its disadvantage is that it cannot cope with dynamic changes. The latter, such as genetic algorithms and particle swarm optimization, can achieve local optima, but they face problems of computational time consumption and failure to consider environmental feedback, resulting in insufficient robustness of waveform strategies.

[0003] In radar, communications, and other applications, parameter decision-making among multiple systems is essentially a non-cooperative game problem. Game theory methods, such as Nash equilibrium, can describe the stable policy states of each party under rational decision-making, but existing research suffers from low efficiency in equilibrium solution and a lack of virtual interaction. Classical Nash equilibrium solutions rely on matrix games or numerical iterations in continuous policy spaces, and the computational cost increases exponentially when the parameter dimension is high. Furthermore, existing methods often assume that the opponent's policy is known or fixed, while practical applications require learning mechanisms such as virtual games to predict the opponent's behavior. Traditional algorithms lack the ability to evolve policies online.

[0004] In recent years, the combination of deep reinforcement learning and game theory has provided new ideas for dynamic decision-making. However, directly applying deep reinforcement learning to waveform decision-making faces challenges such as state space explosion and uncertainty in equilibrium convergence. Waveform parameters constitute a high-dimensional discrete-continuous hybrid space, requiring the design of efficient parameter encoding and action mapping mechanisms. Multi-agent reinforcement learning is prone to getting trapped in local optima and it is difficult to guarantee the effectiveness of the policy. The industry urgently needs an intelligent decision-making framework that integrates virtual game learning, Nash equilibrium solving, and waveform parameter optimization. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent decision-making method for waveform parameters based on virtual game Nash equilibrium solution, which can realize intelligent optimization of waveform parameter generation strategy and solve the problems of insufficient dynamic adaptability of waveforms and low efficiency of parameter solution in complex electromagnetic environments.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: A waveform parameter intelligent decision-making method based on virtual game Nash equilibrium solution includes: Step 1: Construct a virtual game two-sided agent framework. The two players are modeled as agent A and agent B, respectively. Define the observation information and policy space, which includes the environmental information perceived during the observation period and a series of agent parameters to be decided. Step 2: Construct an interaction framework between the agent and the environment, design a Markov decision process, and study the competitive relationship and extend the modeling under the condition that both agents have the same learning level, so as to realize the synchronous adversarial execution and real-time feedback of the strategies of both parties. Step 3: Design a Nash equilibrium solution algorithm based on deep reinforcement learning, adopt the NFSP network architecture, alternately use the best response policy and the average policy, learn the agent's optimal response policy, and construct a hybrid reward function; Step 4: Implement network training and optimization, initialize the method network weight parameters, and design a four-stage training process: Stage 1 is opponent policy modeling, Stage 2 is optimal response training, Stage 3 is average policy update, and Stage 4 is convergence verification.

[0007] Optionally, step 1 specifically includes: Step 1.1: Define the two communication devices as agent A and agent B respectively. Agent A's action is to select a set of waveform parameters from multiple combinations, while agent B's action is to select one of multiple adversarial methods. Step 1.2: In the two agents, the perception result of one agent is regarded as the observation result of the other agent. The observation information in each observation period is qualitatively described and expressed by array.

[0008] Optionally, in step 2, the Markov decision process includes four key elements: state space, action space, state transition probability, and reward.

[0009] Optionally, in step 2, the interaction process between the agent and the environment can be described as follows: at a certain moment, the agent is in a certain state, selects an action according to a policy, the policy is a mapping function from the state to the probability distribution of the action, and transitions from the current state to the next state according to the transition probability, while receiving a reward. The goal of the agent is to maximize the long-term expected reward.

[0010] Optionally, step 3 specifically includes: Step 3.1: Use the best response and Nash equilibrium to measure the policy performance of the agents. When the policies chosen by agents A and B are both the best responses to the current situation, a Nash equilibrium is formed. Step 3.2: Solve the Nash equilibrium using NFSP, an intelligent method that combines reinforcement learning and game theory. The core idea is based on virtual self-game, where agents need to consider the policies of other agents, and the goal is to make the policies of all agents form a Nash equilibrium. Step 3.3: Alternately use the best response policy and the average policy. The former learns the optimal policy under the current policy through a deep Q-network, while the latter imitates the average behavior of historical policies through supervised learning, and together construct a hybrid reward function.

[0011] Optionally, in step 4, the opponent's strategy modeling in stage one specifically includes: the agent learns the behavior pattern of another agent through historical game data to predict the strategy distribution of the other agent.

[0012] Optionally, in step 4, the optimal response training in stage two specifically includes: using deep reinforcement learning to train the current optimal policy, the agent adjusts its own actions based on the policy prediction of another agent, and optimizes the policy network through a reward mechanism.

[0013] Optionally, in step 4, the average policy update in stage three specifically includes: averaging the historical best policy to form a hybrid policy.

[0014] Optionally, in step 4, the convergence verification of stage four specifically includes: when the strategies of agents A and B no longer change due to the opponent's adjustment, a Nash equilibrium is reached.

[0015] Compared with the prior art, the present invention has the following technical effects: (1) The intelligent waveform parameter decision-making method has a high degree of automation and intelligence, solving the problem of rigidity in traditional waveform parameter decision-making methods. This method can autonomously adapt to dynamic environments, learn online and dynamically adjust waveform strategies, significantly improving the robustness of the system; (2) The intelligent decision-making method for waveform parameters has high computational efficiency, overcoming the problem of high computational complexity of traditional Nash equilibrium solution methods when facing high-dimensional waveform parameter spaces. This method alternately uses the best response strategy and the average strategy to construct a hybrid reward function and learn the agent's optimal response strategy; (3) The waveform parameter intelligent decision-making method can achieve real-time stable decision-making in dynamic environments in terms of game equilibrium. This method ensures that the strategy satisfies the Nash equilibrium characteristic of mutual optimal response through the virtual game mechanism and convergence verification module, avoiding the problem of single agent reinforcement learning being prone to getting trapped in local optima. Attached Figure Description

[0016] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the drawings described below are one embodiment of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is a flowchart of the waveform parameter intelligent decision-making method based on virtual game Nash equilibrium in this invention; Figure 2 This is a schematic diagram of the multi-agent waveform decision-making process based on Markov game in this invention. Figure 3 This is a graphical representation of the agent waveform strategy after the model of this invention has converged. Detailed Implementation

[0017] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, further illustrates the solution proposed by the present invention. The advantages and features of the present invention will become clearer from the following description. It should be noted that the drawings are in a very simplified form and use non-precise proportions, used only to facilitate and clearly illustrate the embodiments of the present invention. Please refer to the drawings to make the objectives, features, and advantages of the present invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only for illustrative purposes to aid those skilled in the art and are not intended to limit the implementation conditions of the present invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to the size, without affecting the effects and objectives achieved by the present invention, should still fall within the scope of the technical content disclosed in the present invention.

[0018] Based on the problems existing in the prior art, this invention proposes an intelligent decision-making method for waveform parameters based on virtual game Nash equilibrium, aiming to solve the problems of real-time performance and adaptability in complex electromagnetic environments.

[0019] like Figure 1 As shown, the intelligent decision-making method for waveform parameters based on virtual game Nash equilibrium provided by the present invention specifically includes the following steps 1 to 4.

[0020] Step 1: Construct a virtual game two-sided agent framework. The two players are modeled as agent A and agent B, respectively. Define the observation information and policy space, which includes the environmental information perceived during the observation period and a series of agent parameters to be decided.

[0021] Step 1 specifically includes: Step 1.1: Define the two communication devices as agent A and agent B respectively. Agent A's action is to select a set of waveform parameters from a variety of combinations, while agent B's action is to select one of a variety of adversarial methods.

[0022] The set of actions for agent A is That is, agent A has One action; the action set of agent B is . That is, agent B has One action.

[0023] Step 1.2: In the two agents, the perception result of one agent is regarded as the observation result of the other agent. The observation information in each observation period is qualitatively described and expressed by array.

[0024] The perception result of agent A can be regarded as the observation result of agent B, and similarly, the perception result of agent B can be regarded as the observation result of agent A.

[0025] The observation information of the agent in each observation period t can be defined as: in, Represents the frequency at time t. For transmission power, For signal bandwidth, Represents position, Represents the signal type.

[0026] Step 2: Construct an interaction framework between the agent and the environment, design a Markov decision process, and study the competitive relationship and extend the modeling under the condition that both agents have the same learning level, so as to realize the synchronous adversarial execution and real-time feedback of the strategies of both parties.

[0027] The interaction framework between intelligent agents and the environment can be modeled as a Markov decision process. A Markov decision process comprises four key elements: state space, action space, state transition probabilities, and reward. Specifically: state space : Set of states initial state Belongs to a given distribution ; Action Space Action Set ; State transition probability Given the current environment state and action In this case, the environment transitions to the next state. The probability is expressed as ; Return Given the current state ,action and the next state At that time, the reward that the environment gives to the intelligent agent is .

[0028] The interaction between an agent and its environment can be described as follows: at a certain moment, the agent is in a certain state and chooses an action according to a policy, where the policy is a mapping function from the state to the probability distribution of the action. For example... Figure 2 As shown, in the initial state Next, agents A and B select and execute actions, at which point the system transitions to the next state according to the state transition function. Meanwhile, agents A and B receive rewards from the environment, and this process is repeated cyclically. Each matrix in the diagram represents the expected cumulative reward generated by each possible combination of actions of agents A and B in the current state. Taking the path corresponding to the solid line in the diagram as an example, when agents A and B are in state... Take action Agent A receives a reward of (The jammer is) The game has entered a state of flux. Next, in the state Agent A and Agent B perform actions, and Agent A receives a reward. The game then moves to the next state, and so on, until a complete game trajectory is obtained.

[0029] Based on the transition probability, the agent transitions from the current state to the next state while receiving a reward. The goal of the agent is to maximize the long-term expected reward, as shown below: in In order to expect a return, For long-term returns on discounts, This represents the reward the agent receives in each round of interaction. This represents the discount factor, indicating the degree of importance placed on future returns.

[0030] In Markov games, no single agent's strategy is optimal because its reward depends on the other agent. By extending the modeling of the relationship between agents A and B, with both agents possessing equal learning levels, the agents can learn optimal action strategies from a large amount of action reward data.

[0031] Step 3: Design a Nash equilibrium solution algorithm based on deep reinforcement learning. The algorithm adopts the NFSP network architecture, alternately uses the best response policy and the average policy, learns the agent's optimal response policy, and constructs a hybrid reward function.

[0032] Step 3 specifically includes: Step 3.1: Use optimal response and Nash equilibrium to measure the policy performance of the agents. When both agents A and B choose the best response to the current situation, a Nash equilibrium is formed, specifically: Given the policy of agent A If agent B's policy Satisfy the following formula: This means there are no other strategies better than strategy 1. Okay, then the strategy Called strategy The best response.

[0033] Nash equilibrium corresponds to a strategy pair The two strategies in this strategy pair are each other's best responses to the opponent's strategy, specifically: Step 3.2: Solve the Nash equilibrium using NFSP (Neural Virtual Self-Play), an intelligent method combining reinforcement learning and game theory. The core idea is based on virtual self-play, where agents need to consider the policies of other agents, with the goal of achieving a Nash equilibrium among all agents' policies. Specifically: The NFSP algorithm combines game theory and reinforcement learning. It uses the minimax method in game theory to correct the Q-value update process of traditional Q-learning and uses the temporal difference method to iteratively learn the state-value function or state-action-value function.

[0034] In multi-agent scenarios, state value refers to the state of all agents. According to their respective strategies The expected cumulative return obtained by taking subsequent actions is denoted as . , is represented as: Nash equilibrium strategy Represented as: Step 3.3: Alternately use the best response policy and the average policy. The former learns the optimal policy under the current policy through a deep Q-network, while the latter imitates the average behavior of historical policies through supervised learning, and together construct a hybrid reward function.

[0035] Step 4: Implement network training and optimization, initialize the method network weight parameters, and design a four-stage training process: Stage 1 is opponent policy modeling, Stage 2 is optimal response training, Stage 3 is average policy update, and Stage 4 is convergence verification.

[0036] Specifically, the opponent strategy modeling in Phase 1 involves the agent learning the behavior patterns of another agent through historical game data, constructing a behavior pattern model of the opponent, mining its potential strategy preferences, predicting the possible action space and probability weights of the other agent in the current state, and providing strategy priors for subsequent optimal response training. The optimal response training in Phase 2 is specifically as follows: Deep reinforcement learning is used to train the current optimal policy. The agent will take the prediction of the other agent's policy as part of the environmental dynamics, construct a Markov decision process model, design a composite reward function that takes into account both immediate gains and long-term policy advantages, and use NFSP reinforcement learning to adjust its own actions in the virtual environment. The average policy update in Phase 3 is as follows: The agent maintains a policy buffer to store the phase-optimal policy generated during the iteration process, averages the historical optimal policy, and generates a hybrid policy as the current agent's output policy. The convergence verification in stage four specifically involves reaching a Nash equilibrium when the policies of agents A and B no longer change due to the opponent's adjustments. When the algorithm converges, its effectiveness can be further verified by examining the Q-table at the root node of the game and the policies of agents A and B at the root node. Figure 3 As shown. Figure 3 This diagram presents the strategy effectiveness evaluation matrix for agents A and B at the root node of the game. The matrix uses 11 types of interference from agent A as the x-axis and 24 anti-interference waveforms from agent B as the y-axis. Each cell value represents the effectiveness score of the agent's strategy against the opponent's strategy after the reinforcement learning model has been trained; a higher score indicates better anti-interference performance. Analyzing this matrix allows for the intuitive identification of the Nash equilibrium point, which represents the optimal strategy combination for both agents A and B. Specifically, the Nash equilibrium point is represented as a "saddle point" in the matrix, where the score is the maximum value in both its row and column. The clear distribution of equilibrium points in the diagram not only proves that the algorithm has successfully converged (i.e., the strategies of both parties no longer change due to the opponent's adjustments), but also reveals the matching relationship and performance boundary between agent A's and agent B's strategies under optimal adversarial conditions. This provides crucial data support for evaluating algorithm effectiveness and guiding practical strategy deployment.

[0037] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0038] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A waveform parameter intelligent decision-making method based on virtual game Nash equilibrium solution, characterized in that, include: Step 1: Construct a virtual game two-sided agent framework. The two players are modeled as agent A and agent B, respectively. Define the observation information and policy space, which includes the environmental information perceived during the observation period and a series of agent parameters to be decided. Step 2: Construct an interaction framework between the agent and the environment, design a Markov decision process, and study the competitive relationship and extend the modeling under the condition that both agents have the same learning level, so as to realize the synchronous adversarial execution and real-time feedback of the strategies of both parties. Step 3: Design a Nash equilibrium solution algorithm based on deep reinforcement learning, adopt the NFSP network architecture, alternately use the best response policy and the average policy, learn the agent's optimal response policy, and construct a hybrid reward function; Step 4: Implement network training and optimization, initialize the method network weight parameters, and design a four-stage training process: Stage 1 is opponent policy modeling, Stage 2 is optimal response training, Stage 3 is average policy update, and Stage 4 is convergence verification.

2. The intelligent decision-making method for waveform parameters based on virtual game Nash equilibrium solution as described in claim 1, characterized in that, Step 1 specifically includes: Step 1.1: Define the two communication devices as agent A and agent B respectively. Agent A's action is to select a set of waveform parameters from multiple combinations, while agent B's action is to select one of multiple adversarial methods. Step 1.2: In the two agents, the perception result of one agent is regarded as the observation result of the other agent. The observation information in each observation period is qualitatively described and expressed by array.

3. The intelligent decision-making method for waveform parameters based on virtual game Nash equilibrium solution as described in claim 1, characterized in that, In step 2, the Markov decision process includes four key elements: state space, action space, state transition probability, and reward.

4. The intelligent decision-making method for waveform parameters based on virtual game Nash equilibrium solution as described in claim 1, characterized in that, In step 2, the interaction process between the agent and the environment can be described as follows: at a certain moment, the agent is in a certain state and chooses an action according to a policy. The policy is a mapping function from the state to the probability distribution of the action. According to the transition probability, the agent transitions from the current state to the next state and receives a reward. The goal of the agent is to maximize the long-term expected reward.

5. The intelligent decision-making method for waveform parameters based on virtual game Nash equilibrium solution as described in claim 1, characterized in that, Step 3 specifically includes: Step 3.1: Use the best response and Nash equilibrium to measure the policy performance of the agents. When the policies chosen by agents A and B are both the best responses to the current situation, a Nash equilibrium is formed. Step 3.2: Solve the Nash equilibrium using the NFSP intelligent method. The core idea is based on virtual self-game, where agents need to consider the policies of other agents, and the goal is to make the policies of all agents form a Nash equilibrium. Step 3.3: Alternately use the best response policy and the average policy. The former learns the optimal policy under the current policy through a deep Q-network, while the latter imitates the average behavior of historical policies through supervised learning, and together construct a hybrid reward function.

6. The intelligent decision-making method for waveform parameters based on virtual game Nash equilibrium solution as described in claim 1, characterized in that, In step 4, the opponent strategy modeling in stage one specifically includes: the agent learns the behavior pattern of another agent through historical game data to predict the strategy distribution of the other agent.

7. The intelligent decision-making method for waveform parameters based on virtual game Nash equilibrium solution as described in claim 1, characterized in that, In step 4, the optimal response training in stage two specifically includes: using deep reinforcement learning to train the current optimal policy, the agent adjusts its own actions based on the policy prediction of another agent, and optimizes the policy network through a reward mechanism.

8. The intelligent decision-making method for waveform parameters based on virtual game Nash equilibrium solution as described in claim 1, characterized in that, In step 4, the average policy update in stage three specifically includes: averaging the historical best policy to form a hybrid policy.

9. The intelligent decision-making method for waveform parameters based on virtual game Nash equilibrium solution as described in claim 1, characterized in that, In step 4, the convergence verification of stage four specifically includes: when the strategies of agents A and B no longer change due to the opponent's adjustment, a Nash equilibrium is reached.

Images