A PPO-based asymptotically optimal multi-agent deep reinforcement learning anti-interference method

Abstract

The application relates to the technical field of communication, and particularly discloses an anti-interference method based on a PPO (Proximal Policy Optimization) asymptotically optimal multi-agent deep reinforcement learning, which comprises the following steps: S1, constructing and training an anti-interference decision model; S2, acquiring a global state of an anti-interference communication system through an intelligent decision device at the beginning of each communication time slot; S3, calling the anti-interference decision model trained based on the global state through the intelligent decision device to generate a joint anti-interference strategy for each communication user; the strategy comprises channels and transmission powers allocated to the communication users; and S4, issuing the strategy to each communication user, so that each communication user communicates with a receiver using the specified channel and transmission power in the communication time slot according to the received strategy.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and specifically to a PPO-based asymptotically optimal multi-agent deep reinforcement learning anti-interference method. Background Technology

[0002] With the development of 5G and even 6G technologies, wireless data traffic and wireless communication devices are growing exponentially, leading to a scarcity of wireless channel resources. Furthermore, due to the open and shared nature of wireless channels, wireless communication systems are vulnerable to malicious interference. As communication systems become increasingly large-scale, relying solely on a single domain to combat complex interference is insufficient. Moreover, joint domain interference mitigation for multi-user communication systems faces challenges such as state space non-stationarity, large decision space, and difficulty in algorithm convergence.

[0003] Existing technologies and their shortcomings include: traditional single-domain anti-interference methods are difficult to cope with multi-dimensional interference and have limited anti-interference effects; existing deep reinforcement learning-based methods converge slowly due to the large decision space; existing multi-domain anti-interference methods are mostly limited to single-user scenarios, and the proposed anti-interference methods are less effective for multi-user systems and do not meet real-world needs. Summary of the Invention

[0004] In complex electromagnetic environments, wireless communication systems are susceptible to malicious interference, leading to decreased spectrum utilization and reduced communication quality. The purpose of this invention is to provide a PPO-based asymptotically optimal multi-agent deep reinforcement learning anti-interference method, suitable for joint domain anti-interference scenarios in multi-user communication systems, effectively improving the convergence speed, convergence performance, and robustness of joint domain anti-interference algorithms for multi-user systems.

[0005] This invention provides a PPO-based asymptotically optimal multi-agent deep reinforcement learning anti-jamming method, applied to an anti-jamming communication system including a jammer, a receiver, and N communication users, wherein the receiver is equipped with an intelligent decision-making device. The method includes: Step S1: Construct and train an anti-interference decision model; Each actor network contains a channel selection subnetwork and a power allocation subnetwork configured in parallel. The anti-interference decision model is a multi-agent deep reinforcement learning model based on near-end policy optimization of PPO; the multi-agent deep reinforcement learning model adopts a centralized training and distributed execution architecture, including one centrally trained global critic network and N decentralized actor networks; each actor network corresponds to one communication user as an agent, including a channel selection sub-network for channel selection and a power allocation sub-network for transmit power allocation set in parallel; N is a positive integer; Step S2: At the beginning of each communication time slot, the global state of the anti-interference communication system is obtained through the intelligent decision-making device; Step S3: The intelligent decision-making device invokes the anti-interference decision model trained based on the global state to generate a joint anti-interference strategy for each communication user; the strategy includes: the channel and transmission power allocated to each communication user; Step S4: The policy is distributed to each communication user, so that each communication user can communicate with the receiver using the specified channel and transmit power in the communication time slot according to the received policy.

[0006] In one possible implementation, the global state includes at least: the availability of all channels, the channel gain between each communication user and the receiver and jammer, historical policies, and immediate rewards.

[0007] In one possible implementation, step S1 includes: Construct an interference-resistant decision-making model comprising one centrally trained global critic network and N decentralized actor networks; Based on the anti-interference decision model, in each communication time slot, the communication user selects and executes a joint action based on its current strategy and local observations; the joint action includes the selected communication channel and transmission power. Store the global state, the joint actions of all communication users, the immediate rewards obtained, and the global state of the next communication slot as empirical data; The value assessment result is obtained by sampling from the stored empirical data and calculating the value estimate by a global commentator network based on the global state and joint actions of all communicating users; Based on the near-end policy optimization algorithm, the advantage function is calculated using the value assessment results, and the parameters of the actor network of all communication users are updated according to the advantage function to obtain the trained anti-interference decision model.

[0008] In one possible implementation, the joint action selected and executed by each communication user based on its current policy and local observations includes: Each communication user adopts a greedy strategy ε-greedy to select joint actions.

[0009] In one possible implementation, step S3 includes: The intelligent decision-making device extracts local observations of each communication user from the global state. Input the local observations of each communication user into its corresponding actor network; The local observations are processed by the channel selection subnetwork of the actor network, and the selection probability distribution on M selectable channels is output. The local observations are processed by the power allocation subnetwork of the actor network, and the selection probability distributions on N discrete power levels are output. The strategy is generated based on the selection probability distributions on the M selectable channels and the selection probability distributions on the N discrete power levels; M is a positive integer.

[0010] In one possible implementation, the local observation includes at least: the communication user's own location information, available channel status, and related historical observation information.

[0011] In one possible implementation, the instant reward includes a self-reward and a neighbor reward; it also includes: The instant reward is expressed by the following formula. : in, Representative of individuals The reward; Represents the reward for neighboring nodes; It is the utility coefficient, representing the proportion of neighbor rewards in the immediate rewards; , , These represent the reward coefficient, failure penalty coefficient, and power loss coefficient, respectively. User throughput; It is the transmitter's transmission power.

[0012] In one possible implementation, the reward is represented by the following formula: in, , , These represent the reward coefficient, failure penalty coefficient, and power loss coefficient, respectively. It is the neighbor reward coefficient; User throughput; It is the transmitter's transmission power.

[0013] In one possible implementation, step S4 includes: After the anti-interference decision model is trained, each actor network is independently deployed to the corresponding communication user terminal.

[0014] In one possible implementation, step S4 further includes: Each communication user independently generates decisions based on its own local observations using a locally deployed actor network, without needing to communicate or coordinate with the commentator network or other communication users.

[0015] The PPO-based asymptotically optimal multi-agent deep reinforcement learning anti-interference method provided by this invention adopts a centralized training and distributed execution architecture, which effectively improves the convergence speed, convergence effect and robustness of the joint domain anti-interference algorithm for multi-user systems, and is suitable for joint domain anti-interference scenarios in multi-user communication systems. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the PPO-based asymptotically optimal multi-agent deep reinforcement learning anti-interference method provided for embodiments of the present invention; Figure 2 A schematic diagram of an anti-interference wireless communication system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the multi-agent PPO architecture provided in an embodiment of the present invention. Detailed Implementation

[0017] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following detailed description of the embodiments and the accompanying drawings are used to illustrate the principles of the present invention by way of example, but should not be used to limit the scope of the present invention. That is, the present invention is not limited to the described preferred embodiments, and the scope of the present invention is defined by the claims.

[0018] In the description of this invention, it should be noted that, unless otherwise stated, "a plurality of" means two or more; the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance; those skilled in the art can understand the specific meaning of the above terms in this invention as appropriate.

[0019] Figure 1 A flowchart illustrating the PPO-based asymptotically optimal multi-agent deep reinforcement learning anti-interference method provided for embodiments of the present invention is shown below. Figure 1 As shown, this invention provides an asymptotically optimal multi-agent deep reinforcement learning anti-jamming method based on PPO, applied to an anti-jamming communication system including a jammer, a receiver, and N communication users. The receiver is equipped with an intelligent decision-making device. The method includes: Step S1: Construct and train an anti-interference decision model; Each actor network contains a channel selection subnetwork and a power allocation subnetwork configured in parallel. The anti-interference decision model is a multi-agent deep reinforcement learning model based on proximal policy optimization of PPO. The multi-agent deep reinforcement learning model adopts a centralized training and distributed execution architecture, including one centralized training global critic network and N decentralized actor networks. Each actor network corresponds to one communication user as an agent, including a channel selection sub-network for channel selection and a power allocation sub-network for transmit power allocation set in parallel. N is a positive integer.

[0020] Step S1 is the foundation and prerequisite for the implementation of the method, and its output is a fully trained and usable intelligent decision-making model. Model training is an offline process that may be time-consuming, but once trained, it can be used efficiently online.

[0021] Step S2: At the beginning of each communication time slot, the global state of the anti-interference communication system is obtained through the intelligent decision-making device; Step S3: The intelligent decision-making device invokes the anti-interference decision model trained based on global state to generate a joint anti-interference strategy for each communication user; the strategy includes: the channel and transmission power allocated to each communication user; Step S4: The policy is distributed to each communication user so that each communication user can communicate with the receiver using the specified channel and transmit power within the communication time slot according to the received policy.

[0022] Steps S2 to S4 constitute a closed loop in the online execution phase. This process is repeated for each communication time slot, enabling real-time, adaptive anti-interference in the face of dynamic interference environments.

[0023] The following is a detailed explanation of step S1.

[0024] In one possible implementation, an anti-interference decision model is constructed, comprising one centrally trained global critic network and N decentralized actor networks. Based on the anti-interference decision model, in each communication slot, communication users select and execute joint actions based on their current policies and local observations. The joint actions include the selected communication channel and transmission power. The global state, the joint actions of all communication users, the immediate rewards obtained, and the global state of the next communication slot are stored as empirical data. Samples are taken from the stored empirical data, and the global critic network calculates the value estimate based on the global state and joint actions of all communication users to obtain the value assessment result. Based on the near-end policy optimization algorithm, the advantage function is calculated using the value assessment result, and the parameters of the actor networks of all communication users are updated according to the advantage function to obtain the trained anti-interference decision model.

[0025] In this context, "centralized training" refers to a central "critic" network that collects experiential information from all agents (communication users) during the model learning phase. This provides a global perspective, enabling more accurate evaluations to guide policy improvements in individual "actor" networks. This stored "experience data" serves as the model's learning "textbook," recording what actions were taken in what states, what results (rewards) were obtained, and what new states were transitioned to. The critic network acts like a "referee," learning to evaluate the long-term value of a set of joint actions given a global state. The actor network is the final "decision-maker." The PPO algorithm uses the "advantage function" (judging how much better an action is than the average level) provided by the critics to update the actors' policies. Its core mechanism is to limit the magnitude of each policy update by pruning probability ratios, ensuring a smooth and efficient training process and preventing policy collapse—a key to achieving "asymptotically optimal" performance.

[0026] In one possible implementation, each communication user selects and executes a joint action based on local observations according to its current policy. This involves each user employing a greedy policy ε to select the joint action. This balances the relationship between policy exploration (trying new actions) and utilization (selecting the currently optimal action), thereby learning the optimal policy more effectively. In the early stages of training, a higher exploration probability ε helps the agent to try various channel and power combinations extensively, avoiding getting trapped in local optima. As training progresses, the exploration probability is gradually reduced, making more use of the currently learned optimal policy, leading to policy convergence.

[0027] like Figure 2 As shown, consider an anti-jamming wireless communication scenario involving one jammer, one receiver, and n drones. The set of all drones is denoted as N = {1, 2, ..., n}. The communication link relies on a set of M available channels, defined as M = {1, 2, ..., M}. The system uses time-slot communication, and all users are synchronized. Each user occupies one channel per time slot. The receiver uses an omnidirectional antenna, capable of simultaneously receiving signals from all drones. The jammer aims to... To disrupt communication, the system continuously transmits jamming signals on multiple channels. To effectively counter malicious interference during communication, an intelligent decision-making device is deployed at the receiver. This device continuously monitors the spectrum environment and runs an intelligent anti-jamming algorithm at the beginning of each time slot to select the optimal communication strategy for the drone. An intelligent decision-making device is also deployed at the jammer end. This device operates different jamming modes and can release jamming signals into the channel at each time slot. The maximum transmit power of each drone is recorded. The maximum transmit power of the jammer is denoted as . The available state vector of the channel is denoted as... ,in (0 indicates the channel is unavailable, 1 indicates the channel is available). Considering path loss, the channel gain is defined as... ,in It is the transmission distance. It is the path loss factor.

[0028] This is a typical dynamic spectrum sharing and countermeasures scenario. The agent (drone) needs to intelligently select channels (frequency hopping) and adjust power (power control) in the presence of interference to maintain reliable communication with the receiver. The presence of jammers makes the environment complex and non-static.

[0029] The intelligent decision-making equipment at the receiver end guides the drone's transmission based on the quality of the received signal. This equipment continuously monitors the entire communication frequency band and determines the communication channel for each drone at the start of each time slot. The decision is then fed back to each user via a reverse link. Because the reverse link is a low-capacity control channel, reliable transmission is guaranteed. Each drone attempts to transmit data to the receiver on its given channel. The receiver ultimately demodulates the received signal and determines whether the transmission was successful.

[0030] Decision time, feedback time, receipt time, and reward calculation time are respectively denoted as: Similarly, the feedback reception time and transmission time are denoted as follows: This invention assumes that the decision-making time, feedback time, and reward calculation time are much shorter than the reception time, i.e. Furthermore, the feedback reception time is much shorter than the transmission time. Therefore, these durations can be disregarded.

[0031] In communication time slot t, the channel gain between the UAV transmitter and receiver is denoted as . The channel gain between the jammer and the receiver is denoted as . The power spectral density functions of the UAV signal, interference signal, and noise are denoted as U(f), J(f), and N(t,f), respectively. Considering the presence of all signals, the PSD of the received signal at the receiver is defined as: in, and These represent the center frequencies of the drone signal and the jamming signal, respectively. and These represent the antenna transmission gain of the drone and the jammer, respectively. This indicates the antenna receiving gain of the receiver.

[0032] The following formula represents the drone. The signal transmitted to the receiver has an interference-to-noise ratio of : The signal-to-interference-plus-noise ratio (SINR) is a core indicator for measuring the quality of communication links.

[0033] The following formula represents the drone. The normalized communication throughput obtained, i.e., the number of successful communications: Where δ(x) is an indicator function. δ(x) = 1 when x is true, otherwise δ(x) = 0; The signal-to-interference-plus-noise ratio (SINORR) threshold required for the receiver to correctly demodulate the signal is denoted as .

[0034] This invention considers the joint interference immunity problem in the frequency domain and power domain. Therefore, taking into account the power loss of each drone, each drone... The revenue is defined as: Where β represents the drone The normalized throughput obtained , , These represent the weighting factors for successful communication, communication failure, and power consumption, respectively. This reward function provides the foundation for reward design in subsequent reinforcement learning.

[0035] This payoff function embodies the idea of ​​multi-objective optimization: it aims to achieve successful communication, avoid unnecessary transmissions under strong interference, and also consider energy conservation. Weighting factors can be used to adjust the relative importance of these three objectives.

[0036] The following is a detailed explanation of step S2.

[0037] In one possible implementation, the global state includes at least: the availability of all channels, the channel gain between each communication user and the receiver and jammer, historical policies, and immediate rewards. The global state serves as the information basis for intelligent decision-making devices to make decisions.

[0038] The following is a detailed explanation of step S3.

[0039] In one possible implementation, a smart decision-making device extracts local observations of each communication user from the global state; the local observations of each communication user are input into their corresponding actor network; the local observations are processed by the channel selection subnetwork of the actor network, outputting a selection probability distribution on M selectable channels; the local observations are processed by the power allocation subnetwork of the actor network, outputting a selection probability distribution on N discrete power levels; and a strategy is generated based on the selection probability distributions on the M selectable channels and the selection probability distributions on the N discrete power levels; where M is a positive integer.

[0040] In one possible implementation, local observations include at least: the communication user's own location information, available channel states, and relevant historical observation information. Local observations serve as inputs for each agent (communication user) to make independent decisions.

[0041] In one possible implementation, the instant reward includes both self-reward and neighbor-reward; the instant reward is expressed by the following formula. : in, Representative of individuals The reward; Represents the reward for neighboring nodes; It is the utility coefficient, representing the proportion of neighbor rewards in the immediate rewards; , , These represent the reward coefficient, failure penalty coefficient, and power loss coefficient, respectively. User throughput; It is the transmitter's transmission power.

[0042] This design encourages agents to consider the impact on their neighbors when making decisions, promoting cooperation among multiple agents and avoiding overall system performance degradation caused by selfish behavior. If only self-rewards are considered, agents might learn "selfish" strategies, such as preempting the optimal channel regardless of interference with neighbors. By introducing a neighbor reward term, agents, while pursuing high returns for themselves, also "hope" for their neighbors to perform well. The parameter β controls the degree of this "altruism" or cooperation. When all agents learn according to this rule, they will spontaneously tend towards an equilibrium state beneficial to the entire network.

[0043] In one possible implementation, the reward is represented by the following formula: in, , , These represent the reward coefficient, failure penalty coefficient, and power loss coefficient, respectively. It is the neighbor reward coefficient; User throughput; It is the transmitter's transmission power.

[0044] This self-reward function simultaneously incentivizes successful communication, penalizes failed communication, and constrains energy consumption, guiding the agent to learn efficient and energy-saving anti-interference strategies. The self-reward provides positive feedback for successful communication and negative feedback for failure, driving the agent to learn how to avoid interference or enhance the signal. The energy consumption penalty encourages the agent to conserve power as much as possible while meeting communication needs, which is particularly important for energy-constrained devices such as drones.

[0045] A locally interactive Markov game is defined as a seven-tuple. ,in: S represents the state set. .

[0046] N represents the player set.

[0047] A represents a set of joint actions. ,in Let n be the set of actions for player n. The actions of a single player are: .

[0048] M represents the set of neighbors for each player. ,in It is the neighborhood set of player n, defined as: ,in and These represent the positions of player m and player n, respectively.

[0049] F represents the state transition matrix, which describes the dynamic characteristics of the state transition.

[0050] R represents the reward function. By considering system loss and local interactions between neighbors, the reward function of this invention is defined as: Here, β represents the weighted neighbor welfare. γ represents the time discount factor, which determines the importance of immediate and future rewards in the cumulative time return. The value of γ is greater than 0 and less than 1. The smaller the value of γ, the more emphasis is placed on immediate rewards, thus giving them a higher weight in the total return.

[0051] The reward mechanism is designed to consider both individual rewards and neighbor rewards. The reward value for player n in time slot t is defined as follows: The definition of future cumulative time return is as follows: To eliminate the randomness of cumulative time returns, consider its expected value: The utility function of player n with time-series decision accumulation is defined as follows: in, This utility function combines the weighted anti-interference effect of the current and historical time slots, ensuring that all players maximize the anti-interference utility of the current time slot while also maximizing the cumulative anti-interference effect over the entire system cycle.

[0052] Figure 3 This diagram illustrates the multi-agent PPO architecture provided in an embodiment of the present invention. The present invention employs a multi-agent PPO architecture, achieving collaborative training through a centralized critic and distributed actors. Furthermore, by leveraging the advantages of PPO, learning efficiency and stability in a multi-agent environment are enhanced. During the training phase, the critic network can access global state information and the actions of other agents, thereby learning a more accurate value function. Each agent utilizes its own local observations to optimize its policy based on the critic's guidance. During the execution phase, each agent makes decisions solely based on its own observed local state and policy, ensuring distributed control of the system.

[0053] The variance of the value function is reduced by learning a global critic, enabling individual PPO agents to coordinate through this global value function. Each agent i selects an action based on its local observation oi and its individual policy π, aiming to maximize the cumulative discounted reward. During action selection, a [missing information - likely a specific algorithm or method] is employed. A greedy strategy is used to balance exploration and exploitation. Specifically, the agent uses a probabilistic approach. Choose a random action based on the current strategy, with a probability of 1- The agent selects the action with the highest probability from the policy. During the training phase, all agents transmit their state observation data to the central critic network. After obtaining the global state S, the critic network trains the model and then transmits the optimal decentralized control policy back to each agent. After training is complete, the agents no longer need to communicate with the critic network; instead, they can generate optimal actions based on local observations through their own policy functions.

[0054] Each agent's policy network employs a dual-branch parallel architecture, comprising two independent subsystems: channel selection (channel selection subnetwork) and power allocation (power allocation subnetwork). These two subnetworks update their parameters through a parallel learning mechanism while sharing the same reward function as the optimization objective. The core advantage of this design lies in significantly reducing the dimensionality of the decision space. Assuming the system has M selectable channels and N power levels, a traditional single-policy network requires outputting M×N joint decisions, corresponding to M×N output neurons. However, with the parallel subnetwork architecture, the total number of output neurons is reduced to M+N, effectively reducing the complexity of the exploration space and accelerating the policy convergence process.

[0055] This is a significant innovation of this invention. Traditional joint decision-making treats the channel and power as a combined action, resulting in an action space that grows with the product of parameters, making learning extremely difficult and slow. The dual-branch parallel architecture decomposes the joint decision into two parallel sub-decisions, each with an independent action space (M and N), making the total complexity additive (M+N). This not only greatly reduces the size of the neural network's output layer and accelerates training, but also makes the policy easier to learn and generalize. The two sub-networks share the reward, ensuring they optimize collaboratively under a common goal.

[0056] The following is a detailed explanation of step S4.

[0057] In one possible implementation, after the robust decision-making model is trained, each actor network is independently deployed to its corresponding communication user terminal. Each communication user independently generates decisions based on its own local observations using its locally deployed actor network, without needing to communicate or coordinate with the commentator network or other communication users.

[0058] This achieves a transition from "centralized training" to "distributed execution." After training, the intelligent decision-making device only needs to distribute the trained, lightweight actor network model parameters to each user. In actual operation, each user runs their own actor network independently, requiring only locally perceived information to make decisions within milliseconds. This completely eliminates the dependence on real-time computation by the intelligent decision-making device and eliminates communication overhead between users, resulting in extremely strong system robustness, real-time performance, and scalability.

[0059] In summary, this invention provides an asymptotically optimal multi-agent deep reinforcement learning anti-jamming method based on PPO. This method is applied to a wireless communication system comprising one jammer, one receiver, and multiple communication users (such as drones). The core of this method is to deploy an intelligent decision-making device at the receiver end, running a multi-agent deep reinforcement learning model called PSPMMA.

[0060] The model employs a "centralized training, distributed execution" architecture. During the training phase, a centralized "commentator" network acquires global state information from all users to evaluate value, guiding multiple distributed "actor" networks (one for each user) to optimize their policies. Each actor network uses an innovative dual-branch parallel sub-network structure to independently handle channel selection and power allocation decisions, thereby reducing the dimensionality of the joint decision from a product level (M×N) to a summation level (M+N), significantly improving learning efficiency and convergence speed.

[0061] To promote collaboration among agents, a reward function designed for each agent integrates its own communication utility (based on communication success and energy consumption) and the rewards from its neighbors. The model is trained using the Proximity Policy Optimization (PPO) algorithm to maximize long-term cumulative rewards. After training, each user can make real-time, distributed, and robust decisions based solely on its own agent network and local observations, without needing continuous communication with a central node. This achieves a comprehensive improvement in robustness, decision-making speed, and overall system robustness.

[0062] Simulations show that, compared with the Independent Decision Training (IIIDT) algorithm and the Actor-Critic algorithm, the proposed PSPMMA method achieves significantly faster convergence speed, higher convergence efficiency, and better stability. Furthermore, this method exhibits strong robustness under various disturbance environments, maintaining high communication efficiency even in large-scale UAV swarm scenarios.

[0063] Normalized Total Throughput (NAT) during the training process of different algorithms in the face of DQN interference. The proposed PSPMMA algorithm converges in approximately 80 rounds, exhibiting higher and relatively stable NAT. The first 300 rounds represent the intensive training phase of the proposed method, during which parameters are continuously optimized and gradually converge. The subsequent 200 rounds constitute the execution phase. During this phase, the algorithm's communication success rate consistently remains at the highest level, indicating that the trained decision model has effectively learned the optimal anti-interference strategy, enabling it to avoid or suppress interference signals.

[0064] The PSPMMA method demonstrates communication success rates under comb interference, frequency sweep interference at varying power levels, and DQN-based intelligent interference. The success rate represents the ratio of the cumulative number of successful communications up to the current round to the total number of communications. Therefore, when the curve flattens, it indicates that the algorithm has converged. The success rate will eventually stabilize at a consistent value. PSPMMA exhibits satisfactory performance under various interference types. As the number of rounds increases, the curve gradually flattens, and the communication success rate steadily increases, eventually converging to a value exceeding 90%.

[0065] As the number of users increases, the proposed algorithm maintains a high SNAT, consistently above 460. In contrast, the SNAT of the Actor-Critic method gradually decreases with the number of users, while the IIIDT algorithm, due to its lack of information interaction, shows a sharp decline when the number of users increases. This demonstrates that the proposed algorithm remains effective even when handling a large number of users.

[0066] This invention models the anti-interference problem in multi-user systems as a locally interactive Markov game. By constructing an efficient potential function, it rigorously proves that this game is an exact potential game, thus ensuring the existence of at least one pure policy Nash equilibrium.

[0067] This invention designs a parallel sub-network PPO method to solve the multi-domain anti-interference problem. This method adopts parallel policy sub-networks, which focus on channel selection and power allocation respectively, and ensures stable and efficient learning by pruning the probability ratio.

[0068] This invention proposes a PSPMMA method for solving the Nash equilibrium of the aforementioned game model. This method employs a "centralized training, distributed execution" architecture, and its convergence and asymptotic optimality are theoretically proven, ensuring its stability and reliability in practical applications.

[0069] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A PPO-based asymptotically optimal multi-agent deep reinforcement learning anti-jamming method applied to an anti-jamming communication system comprising a jammer, a receiver and N communication users, characterized in that, The receiver is equipped with an intelligent decision-making device, and the method includes: Step S1: Construct and train an anti-interference decision model; Each actor network contains a channel selection subnetwork and a power allocation subnetwork configured in parallel. The anti-interference decision model is a multi-agent deep reinforcement learning model based on near-end policy optimization of PPO; the multi-agent deep reinforcement learning model adopts a centralized training and distributed execution architecture, including one centrally trained global critic network and N decentralized actor networks; each actor network corresponds to one communication user as an agent, including a channel selection sub-network for channel selection and a power allocation sub-network for transmit power allocation set in parallel; N is a positive integer; Step S2: At the beginning of each communication time slot, the global state of the anti-interference communication system is obtained through the intelligent decision-making device; Step S3: The intelligent decision-making device invokes the anti-interference decision model trained based on the global state to generate a joint anti-interference strategy for each communication user; the strategy includes: the channel and transmission power allocated to each communication user; Step S4: The policy is distributed to each communication user, so that each communication user can communicate with the receiver using the specified channel and transmit power in the communication time slot according to the received policy.

2. The PPO-based asymptotically optimal multi-agent deep reinforcement learning jamming-resistant method according to claim 1, characterized in that, The global state includes at least: the availability of all channels, the channel gain between each communication user and the receiver and jammer, historical policies, and immediate rewards.

3. The PPO-based asymptotically optimal multi-agent deep reinforcement learning jamming-resistant method according to claim 1, wherein, Step S1 includes: Construct an interference-resistant decision-making model comprising one centrally trained global critic network and N decentralized actor networks; Based on the anti-interference decision model, in each communication time slot, the communication user selects and executes a joint action based on its current strategy and local observations; the joint action includes the selected communication channel and transmission power. Store the global state, the joint actions of all communication users, the immediate rewards obtained, and the global state of the next communication slot as empirical data; The value assessment result is obtained by sampling from the stored empirical data and calculating the value estimate by a global commentator network based on the global state and joint actions of all communicating users; Based on the near-end policy optimization algorithm, the advantage function is calculated using the value assessment results, and the parameters of the actor network of all communication users are updated according to the advantage function to obtain the trained anti-interference decision model.

4. The PPO-based asymptotically optimal multi-agent deep reinforcement learning jamming-resistant method according to claim 3, characterized in that, Each communication user selects and executes a joint action based on its current strategy and local observations, including: Each communication user adopts a greedy strategy ε-greedy to select joint actions.

5. The PPO-based asymptotically optimal multi-agent deep reinforcement learning jamming-resistant method according to claim 1, wherein, Step S3 includes: The intelligent decision-making device extracts local observations of each communication user from the global state. Input the local observations of each communication user into its corresponding actor network; The local observations are processed by the channel selection subnetwork of the actor network, and the selection probability distribution on M selectable channels is output. The local observations are processed by the power allocation subnetwork of the actor network, and the selection probability distributions on N discrete power levels are output. The strategy is generated based on the selection probability distributions on the M selectable channels and the selection probability distributions on the N discrete power levels; M is a positive integer.

6. The PPO-based asymptotically optimal multi-agent deep reinforcement learning jamming-resistant method according to claim 5, characterized in that, The local observations include at least: the location information of the communication user, the available channel status, and related historical observation information.

7. The PPO-based asymptotically optimal multi-agent deep reinforcement learning jamming-resistant method according to claim 3, characterized in that, The instant rewards include self-rewards and neighbor rewards; and also include: The instant reward is expressed by the following formula. : in, Representative of individuals The reward; Represents the reward for neighboring nodes; It is the utility coefficient, representing the proportion of neighbor rewards in the immediate rewards; , , These represent the reward coefficient, failure penalty coefficient, and power loss coefficient, respectively. User throughput; It is the transmitter's transmission power.

8. The PPO-based asymptotically optimal multi-agent deep reinforcement learning anti-interference method according to claim 7, characterized in that, Also includes: The self-reward can be expressed using the following formula: in, , , These represent the reward coefficient, failure penalty coefficient, and power loss coefficient, respectively. It is the neighbor reward coefficient; User throughput; It is the transmitter's transmission power.

9. The PPO-based asymptotically optimal multi-agent deep reinforcement learning anti-interference method according to claim 1, characterized in that, Step S4 includes: After the anti-interference decision model is trained, each actor network is independently deployed to the corresponding communication user terminal.

10. The PPO-based asymptotically optimal multi-agent deep reinforcement learning anti-interference method according to claim 9, characterized in that, Step S4 further includes: Each communication user independently generates decisions based on its own local observations using a locally deployed actor network, without needing to communicate or coordinate with the commentator network or other communication users.

Images