A communication interference resource allocation method based on multi-agent reinforcement learning

Abstract

A communication interference resource allocation method based on multi-agent reinforcement learning, which relates to a communication interference resource allocation method. The present invention aims to solve the problem that the existing interference decision algorithm does not take into account that with the increase in the number of multi-agents, the action space and the state space will grow rapidly, and the algorithm may be difficult to converge. On the basis of the multi-agent proximal strategy optimization algorithm, the present invention proposes a multi-agent proximal strategy optimization algorithm with a selective experience storage strategy, selectively stores the experience entering the experience pool, improves the convergence ability of the algorithm, and adds an attention mechanism to the multi-agent proximal strategy optimization algorithm. The present invention belongs to the field of communication technology.

Description

Technical Field

[0001] The invention relates to a communication interference resource allocation method, belonging to the technical field of communications. Background Art

[0002] The electromagnetic spectrum is the main carrier of information transmission. In modern warfare, how to seize the right to use and control the electromagnetic spectrum has become the focus of both parties. The combat process of the electromagnetic spectrum includes four processes: perception, decision-making, action, and evaluation. Intelligent decision-making is a key link and has become an important research direction of cognitive electronic warfare. How to allocate limited jamming resources to maximize the interference of the enemy's communication system and optimize the allocation of jamming resources is an urgent problem that needs to be solved, which brings huge challenges to the command system. In order to achieve intelligent jamming decision-making, researchers have successively applied genetic algorithms, game theory, distributed optimization and other theories to the task of jamming resource allocation.

[0003] With the development of artificial intelligence, researchers have gradually applied artificial intelligence to the task of interference resource allocation. Since reinforcement learning does not require prior information, it can optimize strategies by interacting with the environment, and has been initially applied in the field of communication interference decision-making. In 2019, ZHAO N et al. adopted a competitive dual Q network structure in a distributed deep Q network, and the device obtained global state information through information transmission for random game. In 2020, GUO DL et al. used a centralized policy gradient method to solve the collaborative multi-agent tasks of frequency switching and power allocation. In 2021, Rao et al. proposed an interference decision algorithm based on the Actor-Critic reinforcement learning framework, which can combine information entropy to avoid the optimization process of interference decision-making from falling into local optimality. However, the above method does not take into account the problem that as the number of multi-agents increases, the action space and state space will grow rapidly, and the algorithm may be difficult to converge. Summary of the invention

[0004] In order to solve the problem that the existing interference decision algorithm does not take into account the rapid growth of action space and state space with the increase of the number of multi-agents, and the algorithm may be difficult to converge, the present invention proposes a communication interference resource allocation method based on multi-agent reinforcement learning.

[0005] The technical solution adopted by the present invention to solve the above problems is: the steps of the present invention include:

[0006] Step 1: Establish a communication confrontation scenario;

[0007] Step 2: Establish a communication node model;

[0008] Step 3: Establish an interference point model;

[0009] Step 4: Obtain the observed state of the jammer;

[0010] Step 5: Get the global status of the jammer;

[0011] Step 6. Get the jammer reward;

[0012] Step 7: Store the experience tuple;

[0013] Step 8: Use MAPPO algorithm to iteratively optimize network model parameters;

[0014] Step 9: Allocate interference resources for multi-jammer collaborative communication.

[0015] Furthermore, the communication confrontation scenario built in step 1 includes two parts: the communication party and the interference party. The communication party includes three communication links and uses the TCP / IP protocol for communication. The frequencies of the three communication links are set to f1, f2 and f3, respectively, and are represented by a vector f = [f1, f2, f3]. The modulation modes that can be selected by each communication node are BPSK, QPSK and EPSK. After quantization, they are represented by a vector c = [0, 1, 2]. The selectable transmission power is divided into five cases, which are 20%, 40%, 60%, 80% and 100% of the maximum transmission power, respectively, and are represented by a vector p = [0.2, 0.4, 0.6, 0.8, 1]. The interference party is composed of three jammers, which detect the communication parameters of the communication party through signal detection and identification methods to interfere.

[0016] Furthermore, in step 2, the communication node adopts the DQN algorithm to perform anti-interference, including:

[0017] Step 201, initializing an experience pool with a capacity of D;

[0018] Step 202, initializing the parameters θ of the main network and the parameters θ′ of the target network;

[0019] Step 203: In the initial state, each communication node randomly selects an initial state and selects an action a with the maximum Q value through the Q-Network with an ε probability according to the ε-greedy strategy. t =[c t ,p t ], randomly select an action a in the action space with probability 1-ε t =[c t ,p t ], set ε = 0.1;

[0020] Step 204: At time t, each communication node executes the action a obtained by the DQN algorithm. t , set the modulation mode and transmit power;

[0021] Step 205: Get the communication node status:

[0022] The state space of the communication node consists of the interference power, After each communication node completes the action, it returns to the next state s t+1 and reward r t , the experience tuple [s t ,a t ,r t ,s t+1 ] is stored in the experience pool, where the reward function is the bit error rate of the communication link;

[0023] Step 206: Update the current state s, let s t =s t+1 , update ε, set ε = 0.1 + 0.001 × episode, repeat steps 203 to 206 until the experience pool is full;

[0024] Step 207: Update the network in DQN, randomly sample from the experience pool, and store the sampled data [s j ,a j ,r j ,s j+1 ] is sent to the target network to calculate y j value, and y j The value is sent to Q-Network to calculate the loss value and start updating Q-Network;

[0025] y j =r j +γmax a′ Q(s j+1 ,a′;θ),

[0026] loss=(y j -Q(s j ,a j ;θ)) 2 ,

[0027] Step 208: Various communication nodes repeat steps 203 to 207 until the Q-Network completes convergence;

[0028] Step 209: After the parameters of the main network Q-Network are updated, all updated parameters of the main network are copied to the target network Target Q-Network;

[0029] Step 210: Use the trained DQN algorithm to output a corresponding anti-interference strategy according to the current state of the communication node.

[0030] Furthermore, the steps of establishing the interference node model in step 3 are:

[0031] Step 301, set the parameters of the jammer Actor network to θ, and the parameters of the jammer Critic network to ω;

[0032] Step 302: Set the initial value of the parameter θ of the jammer Actor network to θ(0), and the initial value of the parameter ω of the jammer Critic network to ω(0).

[0033] Furthermore, the steps of obtaining the jammer observation state in step 4 are:

[0034] Step 401: At time t, jammer j obtains the jamming effect at time t-1, i.e., the modulation pattern and power composition of the communication party at time t constitute the local observation value at the current time.

[0035] Step 402: The observed state of the jth jammer at the tth moment In the Actor network of the jammer with the input initial value θ(0), the Actor network of the jammer outputs the jamming target and jamming power decision result of the jth jammer, that is, the selected action is

[0036] Step 403: Each jammer outputs its own decision result according to its own local observation state to obtain a joint action

[0037] Furthermore, the global state at the tth moment in step 5 is the set of local observation states of each agent, expressed as

[0038] Furthermore, in step 6, each jammer selects the jamming frequency and jamming power according to its own decision result, and after jamming the communication link, calculates the error rate of the communication party to obtain R SEi (t), calculate the total error rate R of the three jammers SEtotal (t) and the total interference power used The reward value at time t is calculated using the reward function. Since the collaborative resource allocation of the three jammers is modeled as a fully collaborative multi-agent task, all jammers share a reward value r t .

[0039] Furthermore, the steps for storing the experience tuple in step 7 are:

[0040] Step 701: After each agent completes the action, it obtains the next state s t+1 , using a computer to convert [s t ,a t ,s t+1 ,r t ] as the experience tuple at the tth moment, judging rt Is it greater than the recycling standard W? t >W, store the data into the experience tuple; when r t When <W, the system randomly generates a random number from 0 to 1. When the random number is less than the selection factor β, the data is stored in the experience tuple, and the remaining data that does not meet the conditions are discarded. The detailed parameters of the experience tuple are as follows:

[0041]

[0042] Step 702, repeat steps 4 to 701, obtain the experience tuple of the next moment, and store it in the buffer until the buffer is full.

[0043] Furthermore, the steps of iteratively optimizing the network model parameters using the MAPPO algorithm in step 8 are:

[0044] Step 801: Randomly sample small batches from the experience pool and input them into the algorithm to train the model. Perform gradient ascent optimization on the parameters θ of the jammer Actor network to obtain the optimized values ​​of the parameters θ of the jammer Actor network. At the same time, perform gradient descent optimization on the parameters ω of the jammer Critic network to obtain the optimized values ​​of the parameters ω of the jammer Critic network.

[0045] Step 802, repeating step 801, each time taking the values ​​after the previous round of optimization as the initial parameters of the network, performing a new round of optimization on the parameters θ of the Actor network of the jammer and the parameters ω of the Critic network of the jammer;

[0046] Step 803: Clear the experience pool;

[0047] Step 804: Obtain the next round of training data according to the method described in steps 4 to 702;

[0048] Step 805, repeating steps 4 to 804 until the maximum number of training rounds max_episode is reached, and obtaining the max_episode-th round optimization value of the parameter θ of the jammer Actor network and the max_episode-th round optimization value of the parameter ω of the jammer Critic network;

[0049] Step 806: Return the network parameters of each jammer.

[0050] Further, the step of allocating multi-jammer cooperative communication interference resources in step 9 is:

[0051] Step 901, obtaining an optimized network model according to the max_episode-th round optimization value of the parameter θ of the jammer Actor network and the max_episode-th round optimization value of the parameter ω of the jammer Critic network;

[0052] Step 902: Each jammer uses the trained model to make jamming target and jamming power decisions based on the local observation state at the current moment, and outputs a collaborative optimization action strategy.

[0053] The beneficial effects of the present invention are:

[0054] 1. The present invention can provide the optimal interference resource allocation scheme while ensuring the interference effect, greatly saving interference resources and reducing resource waste;

[0055] 2. The communication node of the present invention is provided with an anti-interference method based on DQN. This communication party has good anti-interference ability. Through TCP / IP protocol communication, the present invention can effectively interfere with this communication scenario;

[0056] 3. The present invention introduces an attention mechanism into MAPPO, so that the intelligent agent can select complex information, avoiding the problem that the algorithm is difficult to converge due to the increase in the number of intelligent agents, thereby avoiding the waste of resources;

[0057] 4. The present invention uses a selective experience storage strategy to selectively store the experience entering the experience pool, thereby improving the convergence ability of the algorithm;

[0058] 5. The present invention introduces a multi-agent reinforcement learning model with centralized training and distributed execution to solve the problem of non-stationarity in the multi-agent environment, achieves good training convergence performance, and provides a low-complexity resource allocation algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a schematic diagram of the three-to-three communication anti-interference scenario;

[0060] Figure 2 It is a flowchart of the present invention;

[0061] Figure 3 It is the implementation flow chart of the communication node DQN algorithm;

[0062] Figure 4 This is a schematic diagram of MAPPO with an attention mechanism added; Figure 4 (a) is the Actor network structure diagram. Figure 4 (b) is a schematic diagram of the Critic network structure;

[0063] Figure 5 It is the implementation flow chart of the multi-agent reinforcement learning algorithm. DETAILED DESCRIPTION

[0064] Specific implementation method 1: Combination Figures 1 to 5 This embodiment is described. The communication interference resource allocation method based on multi-agent reinforcement learning described in this embodiment is implemented through the following steps:

[0065] Step 1: Establish a communication confrontation scenario;

[0066] Step 2: Establish a communication node model;

[0067] Step 3: Establish an interference point model;

[0068] Step 4: Obtain the observed state of the jammer;

[0069] Step 5: Get the global status of the jammer;

[0070] Step 6. Get the jammer reward;

[0071] Step 7: Store the experience tuple;

[0072] Step 8: Use MAPPO algorithm to iteratively optimize network model parameters;

[0073] Step 9: Allocate interference resources for multi-jammer collaborative communication.

[0074] Specific implementation method 2: Combination Figures 1 to 5 To explain this embodiment, in step 1 of a communication interference resource allocation method based on multi-agent reinforcement learning described in this embodiment, a communication confrontation scenario is built, which includes two parts: a communication party and an interference party. The communication party includes three communication links, and uses the TCP / IP protocol for communication. The frequencies of the three communication links are set to f1, f2 and f3, respectively, and are represented by a vector f=[f1, f2, f3]. Each communication node has three selectable modulation modes: BPSK, QPSK and EPSK, which are quantized and represented by a vector c=[0, 1, 2]. The selectable transmission power is divided into five cases, which are 20%, 40%, 60%, 80% and 100% of the maximum transmission power, respectively, and are represented by a vector p=[0.2, 0.4, 0.6, 0.8, 1]. The interference party is composed of three jammers, and the communication parameters of the communication party are detected and interfered by signal detection and identification methods.

[0075] Specific implementation method three: Combination Figures 1 to 5 To illustrate this embodiment, in step 2 of a communication interference resource allocation method based on multi-agent reinforcement learning described in this embodiment, the step of using the DQN algorithm by the communication party node to perform anti-interference includes:

[0076] Step 201, initializing an experience pool with a capacity of D;

[0077] Step 202, initializing the parameters θ of the main network and the parameters θ′ of the target network;

[0078] Step 203: In the initial state, each communication node randomly selects an initial state and selects an action a with the maximum Q value through the Q-Network with an ε probability according to the ε-greedy strategy. t =[c t ,p t ], randomly select an action a in the action space with probability 1-ε t =[c t ,p t ], set ε = 0.1;

[0079] Step 204: At time t, each communication node executes the action a obtained by the DQN algorithm. t , set the modulation mode and transmit power;

[0080] Step 205: Get the communication node status:

[0081] The state space of the communication node consists of the interference power, After each communication node completes the action, it returns to the next state s t+1 and reward r t , the experience tuple [s t ,a t ,r t ,s t+1 ] is stored in the experience pool, where the reward function is the bit error rate of the communication link;

[0082] Step 206: Update the current state s, let s t =s t+1 , update ε, set ε = 0.1 + 0.001 × episode, repeat steps 203 to 206 until the experience pool is full;

[0083] Step 207: Update the network in DQN, randomly sample from the experience pool, and store the sampled data [s j ,a j ,r j ,s j+1 ] is sent to the target network to calculate y j value, and y j The value is sent to Q-Network to calculate the loss value and start updating Q-Network;

[0084] y j =r j +γmax a′ Q(s j+1 ,a′;θ),

[0085] loss=(y j -Q(s j ,a j ;θ)) 2 ,

[0086] Step 208: Various communication nodes repeat steps 203 to 207 until the Q-Network completes convergence;

[0087] Step 209: After the parameters of the main network Q-Network are updated, all updated parameters of the main network are copied to the target network Target Q-Network;

[0088] Step 210: Use the trained DQN algorithm to output a corresponding anti-interference strategy according to the current state of the communication node.

[0089] Specific implementation method four: Combination Figures 1 to 5 To illustrate this implementation mode, the step of establishing an interference node model in step 3 of a communication interference resource allocation method based on multi-agent reinforcement learning described in this implementation mode is:

[0090] Step 301, set the parameters of the jammer Actor network to θ, and the parameters of the jammer Critic network to ω;

[0091] Step 302: Set the initial value of the parameter θ of the jammer Actor network to θ(0), and the initial value of the parameter ω of the jammer Critic network to ω(0).

[0092] Specific implementation method five: Combination Figures 1 to 5 To illustrate this implementation mode, the step of obtaining the observation state of the jammer in step 4 of the communication jamming resource allocation method based on multi-agent reinforcement learning described in this implementation mode is:

[0093] Step 401: At time t, jammer j obtains the jamming effect at time t-1, i.e., the modulation pattern and power composition of the communication party at time t constitute the local observation value at the current time.

[0094] Step 402: The observed state of the jth jammer at the tth moment In the Actor network of the jammer with the input initial value θ(0), the Actor network of the jammer outputs the jamming target and jamming power decision result of the jth jammer, that is, the selected action is

[0095] Step 403: Each jammer outputs its own decision result according to its own local observation state to obtain a joint action

[0096] Specific implementation method six: Combination Figures 1 to 5 To illustrate this embodiment, the global state at the tth moment in step 5 of the communication interference resource allocation method based on multi-agent reinforcement learning described in this embodiment is the set of local observation states of each agent, expressed as

[0097] Specific implementation method seven: Combination Figures 1 to 5 This embodiment is described. In step 6 of the communication interference resource allocation method based on multi-agent reinforcement learning described in this embodiment, each jammer selects the interference frequency and interference power according to its own decision result, and after interfering with the communication link, calculates the error rate of the communication party to obtain R SEi (t), calculate the total error rate R of the three jammers SEtotal (t) and the total interference power used The reward value at time t is calculated using the reward function. Since the collaborative resource allocation of the three jammers is modeled as a fully collaborative multi-agent task, all jammers share a reward value r t .

[0098] Specific implementation method eight: Combination Figures 1 to 5 To illustrate this embodiment, the step of storing the experience tuple in step 7 of the communication interference resource allocation method based on multi-agent reinforcement learning described in this embodiment is:

[0099] Step 701: After each agent completes the action, it obtains the next state s t+1 , using a computer to convert [s t ,a t ,s t+1 ,r t ] as the experience tuple at the tth moment, judging r t Is it greater than the recycling standard W? t >W, store the data into the experience tuple; when r t When <W, the system randomly generates a random number from 0 to 1. When the random number is less than the selection factor β, the data is stored in the experience tuple, and the remaining data that does not meet the conditions are discarded. The detailed parameters of the experience tuple are as follows:

[0100]

[0101] Step 702, repeat steps 4 to 701, obtain the experience tuple of the next moment, and store it in the buffer until the buffer is full.

[0102] Specific implementation method nine: Combination Figures 1 to 5 To illustrate this implementation, the steps of iteratively optimizing network model parameters using the MAPPO algorithm in step 8 of a communication interference resource allocation method based on multi-agent reinforcement learning described in this implementation are:

[0103] Step 801: Randomly sample small batches from the experience pool and input them into the algorithm to train the model. Perform gradient ascent optimization on the parameters θ of the jammer Actor network to obtain the optimized values ​​of the parameters θ of the jammer Actor network. At the same time, perform gradient descent optimization on the parameters ω of the jammer Critic network to obtain the optimized values ​​of the parameters ω of the jammer Critic network.

[0104] Step 802, repeating step 801, each time taking the values ​​after the previous round of optimization as the initial parameters of the network, performing a new round of optimization on the parameters θ of the Actor network of the jammer and the parameters ω of the Critic network of the jammer;

[0105] Step 803: Clear the experience pool;

[0106] Step 804: Obtain the next round of training data according to the method described in steps 4 to 702;

[0107] Step 805, repeating steps 4 to 804 until the maximum number of training rounds max_episode is reached, and obtaining the max_episode-th round optimization value of the parameter θ of the jammer Actor network and the max_episode-th round optimization value of the parameter ω of the jammer Critic network;

[0108] Step 806: Return the network parameters of each jammer.

[0109] Specific implementation method ten: Combination Figures 1 to 5 To illustrate this implementation mode, the step of allocating multi-jammer collaborative communication interference resources in step 9 of a communication interference resource allocation method based on multi-agent reinforcement learning described in this implementation mode is:

[0110] Step 901, obtaining an optimized network model according to the max_episode-th round optimization value of the parameter θ of the jammer Actor network and the max_episode-th round optimization value of the parameter ω of the jammer Critic network;

[0111] Step 902: Each jammer uses the trained model to make jamming target and jamming power decisions based on the local observation state at the current moment, and outputs a collaborative optimization action strategy.

[0112] How it works

[0113] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A communication interference resource allocation method based on multi-agent reinforcement learning, characterized in that: The communication interference resource allocation method based on multi-agent reinforcement learning is implemented by the following steps: Step 1: Establish a communication confrontation scenario; Step 2: Establish a communication node model; the steps of using the DQN algorithm to resist interference by the communication node include: Step 201, initializing an experience pool with a capacity of D; Step 202, initializing the parameters θ of the main network and the parameters θ′ of the target network; Step 203: In the initial state, each communication node randomly selects an initial state and selects an action a corresponding to the maximum Q value through the Q-Network with an ε probability according to the ε-greedy strategy. t =[c t ,p t ], randomly select an action a in the action space with probability 1-ε t =[c t ,p t ], set ε = 0.1; Step 204: At time t, each communication node executes the action a obtained by the DQN algorithm. t , set the modulation mode and transmit power; Step 205: Get the communication node status: The state space of the communication node consists of the interference power, After each communication node completes the action, it returns to the next state s t+1 and reward r t , the experience tuple [s t ,a t ,r t ,s t+1 ] is stored in the experience pool, where the reward function is the bit error rate of the communication link; Step 206: Update the current state s, let s t =s t+1 , update ε, set ε = 0.1 + 0.001 × episode, repeat steps 203 to 206 until the experience pool is full; Step 207: Update the network in DQN, randomly sample from the experience pool, and store the sampled data [s j ,a j ,r j ,s j+1 ] is sent to the target network to calculate y j value, and y j The value is sent to Q-Network to calculate the loss value and start updating Q-Network; y j =r j +γmax a′ Q(s j+1 ,a′;θ), loss=(y j -Q(s j ,a j ;θ)) 2 , Step 208: Various communication nodes repeat steps 203 to 207 until the Q-Network completes convergence; Step 209: After the parameters of the main network Q-Network are updated, all updated parameters of the main network are copied to the target network TargetQ-Network; Step 210: using the trained DQN algorithm, output a corresponding anti-interference strategy according to the current state of the communication node; Step 3: Establish an interference point model. The steps to establish an interference node model are: Step 301, set the parameters of the jammer Actor network to θ, and the parameters of the jammer Critic network to ω; Step 302: Set the initial value of the parameter θ of the jammer Actor network to θ(0), and the initial value of the parameter ω of the jammer Critic network to ω(0); Step 4: Obtain the observation state of the jammer. The steps for obtaining the observation state of the jammer are: Step 401: At time t, jammer j obtains the jamming effect at time t-1, i.e., the modulation pattern and power composition of the communication party at time t constitute the local observation value at the current time. Step 402: The observed state of the jth jammer at the tth moment In the Actor network of the jammer with the input initial value θ(0), the Actor network of the jammer outputs the jamming target and jamming power decision result of the jth jammer, that is, the selected action is Step 403: Each jammer outputs its own decision result according to its own local observation state to obtain a joint action Step 5: Get the global status of the jammer; Step 6: Obtain the reward of the jammer; each jammer selects the jamming frequency and jamming power according to its own decision results, and after jamming the communication link, calculates the error rate of the communication party to obtain R SEi (t), calculate the total error rate R of the three jammers SEtotal (t) and the total interference power used The reward value at time t is calculated using the reward function. The collaborative resource allocation of the three jammers is modeled as a fully collaborative multi-agent task, and all jammers share a reward value r t ; Step 7: Store the experience tuple. The steps for storing the experience tuple are: Step 701: After each agent completes the action, it obtains the next state s t+1 , using a computer to convert [s t ,a t ,s t+1 ,r t ] as the experience tuple at the tth moment, judging r t Is it greater than the recycling standard W? t >W, store the data into the experience tuple; when r t When <W, the system randomly generates a random number from 0 to 1. When the random number is less than the selection factor β, the data is stored in the experience tuple, and the remaining data that does not meet the conditions are discarded. The detailed parameters of the experience tuple are as follows: Step 702: repeat step 4 to step 701 to obtain the experience tuple at the next moment and store it in the buffer until the buffer is full. Step 8: Use the MAPPO algorithm to iteratively optimize the network model parameters. The steps of using the MAPPO algorithm to iteratively optimize the network model parameters are as follows: Step 801: Randomly sample small batches from the experience pool and input them into the algorithm to train the model. Perform gradient ascent optimization on the parameters θ of the jammer Actor network to obtain the optimized values ​​of the parameters θ of the jammer Actor network. At the same time, perform gradient descent optimization on the parameters ω of the jammer Critic network to obtain the optimized values ​​of the parameters ω of the jammer Critic network. Step 802, repeating step 801, each time taking the values ​​after the previous round of optimization as the initial parameters of the network, performing a new round of optimization on the parameters θ of the Actor network of the jammer and the parameters ω of the Critic network of the jammer; Step 803: Clear the experience pool; Step 804: Obtain the next round of training data according to the method described in steps 4 to 702; Step 805, repeating steps 4 to 804 until the maximum number of training rounds max_episode is reached, and obtaining the max_episode-th round optimization value of the parameter θ of the jammer Actor network and the max_episode-th round optimization value of the parameter ω of the jammer Critic network; Step 806: Return the network parameters of each jammer; Step 9: Allocate multi-jammer cooperative communication interference resources; the steps of allocating multi-jammer cooperative communication interference resources are: Step 901, obtaining an optimized network model according to the max_episode-th round optimization value of the parameter θ of the jammer Actor network and the max_episode-th round optimization value of the parameter ω of the jammer Critic network; Step 902: Each jammer uses the trained model to make jamming target and jamming power decisions based on the local observation state at the current moment, and outputs a collaborative optimization action strategy.

2. According to claim 1, a communication interference resource allocation method based on multi-agent reinforcement learning is characterized in that: The communication confrontation scenario established in step 1 includes two parts: the communication party and the interference party. The communication party contains three communication links and uses the TCP / IP protocol for communication. The frequencies of the three communication links are set to f1, f2 and f3, respectively, and are represented by a vector f = [f1, f2, f3]. The modulation modes that can be selected by each communication node are BPSK, QPSK and EPSK. After quantization, they are represented by a vector c = [0, 1, 2]. The selectable transmission power is divided into five cases, which are 20%, 40%, 60%, 80% and 100% of the maximum transmission power, respectively, and are represented by a vector p = [0.2, 0.4, 0.6, 0.8, 1]. The interference party is composed of three jammers, which detect the communication parameters of the communication party through signal detection and identification methods to interfere.

3. According to claim 1, a communication interference resource allocation method based on multi-agent reinforcement learning is characterized in that: The global state at the tth moment is the set of local observation states of each agent, expressed as

Images