Interference power distribution method based on multi-agent deep reinforcement learning

Abstract

The invention relates to an interference power distribution method based on multi-agent deep reinforcement learning, and belongs to the technical field of wireless mobile communication. The method comprises the following steps: S1, in a many-to-many communication confrontation scene, establishing a cooperative interference model of multiple interference devices confronting multiple unmanned aerial vehicle targets; s2, constructing a distributed local observable Markov decision process (De-POMDP), and modeling an interference power distribution problem into a multi-agent complete cooperation model; s3, designing a'decentralized training, distributed decision '(DTDE) multi-agent framework, and reducing the collaborative decision dimension of interference power resources; s4, an extractor is designed, intelligent agent input state information is preprocessed, and the training efficiency is improved; and S5, realizing efficient exploration of a power distribution strategy based on a discrete flexible actor-commentator (DSAC) algorithm. According to the method, the interference power distribution of the multi-interference equipment during confrontation of multiple unmanned aerial vehicle targets can be realized based on multi-agent deep reinforcement learning, the interference resource utilization rate is effectively improved, and the method has good expansibility in a complex communication confrontation scene.

Description

Technical Field

[0001] The present invention belongs to the field of wireless mobile communication technology and relates to an interference power allocation method based on multi-agent deep reinforcement learning. Background Art

[0002] With the advancement of artificial intelligence (AI) technology, especially machine learning (ML), electronic warfare (EW) is undergoing a major transformation from "human-led" to "machine-led". In modern warfare, the threat of intelligent electronic equipment is increasing, and the competition for control of the electromagnetic spectrum on the battlefield has become more intense. Limited resources have become a key factor restricting the effectiveness of both sides in electronic warfare confrontation. Therefore, how to reasonably and effectively allocate these limited interference resources has become a core issue in improving the effectiveness of electronic warfare.

[0003] Deep reinforcement learning (DRL) combines deep learning (DL) with reinforcement learning (RL), giving intelligent systems powerful autonomous decision-making and environmental understanding capabilities. It has the advantages of fast solution speed, multi-dimensional decision-making and strong generalization ability. With the continuous evolution of technology, the environment of electronic warfare is becoming increasingly complex, and the "many-to-many" clustered combat mode has gradually become the mainstream confrontation method of electronic warfare. Therefore, traditional multi-objective optimization theory faces problems such as slow convergence and poor scalability when solving high-dimensional interference resource allocation problems.

[0004] At present, some related works have proposed to use multi-agent deep deterministic policy gradient algorithm to realize joint optimization of UAV trajectory and jamming power (reference: ZHANG Y, MOU ZY, GAO FF, et al. UAV-enabled secure communications by multi-agent deep reinforcement learning [J]. IEEE Transactions on Vehicular Technology, 2020, 69 (10): 11599-11611.); some research works have proposed to use hierarchical multi-agent reinforcement learning to realize frequency domain collaborative decision-making (reference: ZHANG WX, ZHAO T, ZHAO ZK, et al. An intelligent strategy decision method for collaborative jamming based on hierarchical multi-agent reinforcement learning [J]. IEEE Transactions on Cognitive Communications and Networking, 2024.). However, most existing studies use the "centralized training and decentralized execution" (CTDE) multi-agent framework. The centralized training method leads to high training cost and poor scalability in the scenario of changing the number of communication adversarial members. Summary of the invention

[0005] The purpose of the present invention is to provide an interference power allocation method based on multi-agent deep reinforcement learning. By designing a "decentralized training and distributed decision-making" (Decentralized Training and Decentralized Execution, DTDE) multi-agent framework, the agent training cost is reduced and good expansion is achieved in the scenario where the number of communication confrontation members changes.

[0006] The technical solution of the present invention is as follows:

[0007] S1: In the "many-to-many" communication confrontation scenario, a collaborative interference model of multiple interference devices against multiple UAV targets is established. Multiple interference devices are distributed and deployed on the ground. Each UAV target forms a flight formation and is commanded by an early warning aircraft.

[0008] S2: Construct a distributed partially observable Markov decision process (Dec-POMDP) ​​to model the interference power allocation problem as a multi-agent fully cooperative model;

[0009] S3: Design a DTDE (Decentralized Training and Decentralized Execution) multi-agent framework to reduce the collaborative decision-making dimension of interference power resources;

[0010] S4: Design an extractor to preprocess the agent’s input state information to improve training efficiency;

[0011] S5: To achieve the exploration and utilization of power allocation strategies, the agent network is trained based on the Discrete Soft Actor-Critic (DSAC) algorithm.

[0012] Furthermore, in S1, a collaborative interference model of multiple interference devices against UAV targets is constructed, and the interference party includes a command center and M interference devices, which implement suppressive interference on a UAV formation composed of N UAVs commanded by an enemy early warning aircraft; let N e ={1,2,…,N} and M e = {1, 2, ..., M} represents the set of N UAVs and jammers in the UAV formation. Each jammer can generate multiple power-controllable jammer beams, and the jammer m∈M is defined. e The beam pointing vector b m =[b m,1 ,b m,2 ,...,b m,N ] T , where b m,n ∈{0,1} represents the beam indicator variable. When the jammer m allocates a beam to the drone target n∈N e When interference is implemented, b m,n =1, otherwise b m,n =0, [·] T represents the transposition operation, and the subscripts m and n represent the index numbers of the jammer and the drone target, respectively. Based on this, the jammer beam allocation matrix is ​​defined as

[0013] B=[b1,b2,...,b M ] T

[0014] The command center issues a unified beam allocation command and defines the power allocation vector p of the interference device m according to the interference beam allocation result. m =[p m,1 ,p m,2 ,...,p m,N ] T , where p m,n Indicates that m,n To efficiently utilize the interference power, the interference power is non-zero only when the interference beam irradiates the target, that is,

[0015]

[0016] Where P max represents the maximum radiated power of the interference device. Therefore, for the interference device m,

[0017]

[0018] According to the power allocation vectors of M interfering devices, the interference power allocation matrix is ​​defined as

[0019] P=[p1,p2,...,p M ] T

[0020] When multiple jamming devices cooperate to jam a UAV formation, B and P can be used to characterize the allocation of jamming resources.

[0021] Through interference coordination, the interference signals radiated by multiple interference devices are superimposed at the target, causing the target communication receiver to lose its normal working ability. The interference effect is usually measured by the Jamming Signal Ratio (JSR). Assuming that the drone target receiver uses an omnidirectional antenna and ignoring the effects of polarization loss and bandwidth mismatch loss, at a certain moment, the JSR at the drone target n receiver is

[0022]

[0023] where p n is the signal power transmitted by the early warning aircraft to the UAV target n through the communication link, G nt G represents the transmitting antenna gain of the AWACS to the UAV target n, nr is the receiving antenna gain of the UAV target n, L nr =32.44+20lg(f n )+20lg(R nr ) represents the path propagation loss from the AWACS to the UAV target n, f n is the UAV communication frequency, R nr G represents the distance from the AWACS to the UAV target n;m,n represents the transmitting antenna gain of the jammer m on the UAV target n, L m,n =32.44+20lg(f n )+20lg(R m,n ) represents the path propagation loss from the jammer m to the drone target n, where R m,n represents the distance from the jammer m to the drone target n, σ 2 Represents the ambient noise power.

[0024] When the JSR at the receiver of the UAV target n exceeds the interference suppression coefficient K n When , the interference of the jammer on the UAV target n is considered effective, that is,

[0025] JSR n ≥K n

[0026] Furthermore, in S2, a distributed locally observable Markov decision process (Dec-POMDP) ​​is constructed to model the interference power allocation problem as a multi-agent fully cooperative model, where each interfering device is regarded as an agent; Dec-POMDP can be used as a seven-tuple <M e ,S p ,{A m},T,{O m},Z,r> means, where M e is the set of intelligent agents; S p is the state space; A m is the action space of agent m; T is the state transition function; O m is the observation space of agent m; Z represents the observation function; r represents the global reward function;

[0027] In the collaborative jamming task of multiple jammers, after the jamming beam is determined, the jammers need to explore the joint power allocation strategy π=<π1,...,π M >To maximize the expected value of the cumulative discounted reward Solve

[0028]

[0029] Among them, π * is the optimal joint power allocation strategy, argmax represents the independent variable value corresponding to the maximum value of the function, It means to solve the expected value, s p,t Indicates the environmental state, a p,t represents the joint action of all interference devices, ρ π Represents a state-action pair (s p,t ,a p,t) corresponding to the distribution function, r(s p,t ,a p,t ) means in state s p,t Perform joint action a p,t The instant reward obtained after , γ is the discount factor, the subscript p represents power, and t represents the time step.

[0030] Furthermore, in S3, a DTDE multi-agent framework is designed, and the agent does not need to obtain the state and action information of other agents during training, and does not need to know the decision actions of other interference devices when deciding the interference power, and only relies on its own observation to decide the interference power. Each interference device has the same physical and functional properties, and they can be regarded as isomorphic agents. To this end, these devices can share the same training parameters, that is, the single-agent DRL method can be used to train the multi-agent network parameters, thereby significantly improving the training efficiency and decision-making effect.

[0031] Furthermore, in S4, an extractor is designed for the agent to pre-process the input state information before making a decision on the interference power. The command center issues a beam allocation instruction to each interference device, and the agent extracts information related to the assigned target from the observation information according to the beam allocation result, removes information related to the unassigned UAV target, and makes an interference power decision for the assigned UAV target based on the pre-processed observation information. The extractor reduces the input dimension of the neural network while reducing the interference of irrelevant information on the decision-making process, thereby improving the efficiency and accuracy of the system decision-making.

[0032] Furthermore, in S5, the interference power resource allocation problem is solved based on DSAC. The DSAC model is based on the actor-critic (AC) architecture and maximum entropy reinforcement learning theory, aiming to maximize the expected return and the entropy of the strategy; at the same time, DSAC can reduce the sensitivity of the algorithm to model and estimation errors, making the algorithm more robust.

[0033] In order to take into account both the interference benefit and the randomness of the joint power allocation strategy, the optimal joint power allocation strategy is transformed into

[0034]

[0035] in For the joint power allocation strategy π in state s p,t The entropy of , log represents the logarithm, and α represents the entropy temperature coefficient.

[0036] A target evaluation network with the same structure as the evaluation (Critic) network is used to alleviate the problem of Q-value over-estimation. The target Q-value is the sum of the reward value obtained and the flexible Q-value output by the target Critic network. At the same time, to ensure the stability of training, a dual neural network structure is used in both the Critic network and the target Critic network, and a single neural network structure is used in the policy (Actor) network.

[0037] Further, in S5, implementing interference power resource allocation based on DSAC includes the following steps:

[0038] S71: Initialize the number of interference devices M, the number of drone targets N, the total number of training rounds max_episode, the total interaction time step per round max_t, the parameters of the Actor, Critic network and target Critic, the experience replay pool, and the threshold Φ p

[0039] S72: If the number of training rounds exceeds max_episode, the training is terminated and S76 is executed; otherwise, max_t is initialized and S73 is executed;

[0040] S73: If the number of interaction time steps exceeds max_t, execute S72, otherwise execute S74;

[0041] S74: The agent receives the observation information, the extractor performs preprocessing, and then performs power allocation for the assigned drone targets, generates experience samples and stores them in the experience replay pool;

[0042] S75: If the number of samples in the experience replay pool is greater than the threshold Φ p , randomly extract batch samples and train the Actor and Critic networks, otherwise execute S73;

[0043] S76: Save the DSAC model;

[0044] S77: Each interference device uses the DSAC model to make distributed decisions on interference power

[0045] The beneficial effects of the present invention are as follows: the present invention proposes an interference power allocation method based on multi-agent deep reinforcement learning for the high-dimensional interference power allocation problem in the "many-to-many" communication confrontation scenario. By constructing a multi-agent-based collaborative interference model, the interference power allocation problem is modeled as Dec-POMDP, which effectively improves the efficiency of collaborative interference between multiple interference devices. The designed DTDE multi-agent framework significantly reduces the collaborative decision-making dimension of interference power resource allocation, thereby reducing the computational complexity and improving the real-time response capability. In addition, the present invention introduces a feature extractor to pre-process the agent input state information, combines the DSAC algorithm to achieve efficient strategy exploration, and significantly improves the training efficiency and the optimality of the strategy. Adaptive collaboration can be achieved between multiple agents, interference power can be reasonably allocated, resources can be maximized, and energy consumption and training costs can be reduced. At the same time, the system can still maintain high robustness and decision-making efficiency in the dynamic and complex "many-to-many" confrontation scenario, thus having a strong application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0047] Figure 1 Coordinated jamming model for multiple jammers against drone formations Figure 2 DTDE multi-agent framework

[0048] Figure 3 Schematic diagram of the extractor

[0049] Figure 4 Flowchart for interference power resource allocation DETAILED DESCRIPTION

[0050] The embodiment of the present invention proposes an interference power allocation method based on multi-agent deep reinforcement learning, which can effectively realize the reasonable allocation of high-dimensional interference power resources.

[0051] The specific steps of this method are as follows:

[0052] S1: Construct a coordinated jamming model of multiple jamming devices against UAV targets. The jammers include a command center and M jamming devices, which implement suppressive jamming on a UAV formation composed of N UAVs commanded by an enemy early warning aircraft. e ={1,2,…,N} and M e={1,2,…,M} respectively represent the set of N drones and jammers in the drone formation. The jammer command center collects intelligence information such as the drone target communication frequency and bandwidth through the information exchange link, and issues jammer task instructions to the jammer; during the flight of the drone formation toward the jammer, the jammer continuously suppresses and jams all communication links between the enemy early warning aircraft and the drone formation, in order to destroy the connection between the enemy drone formation and the early warning aircraft. Figure 1 This is a collaborative jamming model for multiple jammers to counter UAV formations. Each jammer can generate multiple power-controllable jamming beams, and the jammer m∈M is defined. e The beam pointing vector b m =[b m,1 ,b m,2 ,...,b m,N ] T , where b m,n ∈{0,1} represents the beam indicator variable. When the jammer m allocates a beam to the drone target n∈N e When interference is implemented, b m,n =1, otherwise b m,n =0, [·] T represents the transposition operation, and the subscripts m and n represent the index numbers of the jammer and the drone target, respectively. Based on this, the jammer beam allocation matrix is ​​defined as

[0053] B=[b1,b2,...,b M ] T

[0054] The command center issues a unified beam allocation command and defines the power allocation vector p of the interference device m according to the interference beam allocation result. m =[p m,1 ,p m,2 ,....,p m,N ] T , where p m,n Indicates that m,n To efficiently utilize the interference power, the interference power is non-zero only when the interference beam irradiates the target, that is,

[0055]

[0056] Where P max represents the maximum radiated power of the interference device. Therefore, for the interference device m,

[0057]

[0058] According to the power allocation vectors of M interfering devices, the interference power allocation matrix is ​​defined as

[0059] P=[p1,p2,...,p M ] T

[0060] When multiple jamming devices cooperate to jam a UAV formation, B and P can be used to characterize the allocation of jamming resources.

[0061] Through interference coordination, the interference signals radiated by multiple interference devices are superimposed at the target, causing the target communication receiver to lose its normal working ability. The interference effect is usually measured by the Jamming Signal Ratio (JSR). Assuming that the drone target receiver uses an omnidirectional antenna and ignoring the effects of polarization loss and bandwidth mismatch loss, at a certain moment, the JSR at the drone target n receiver is

[0062]

[0063] where p n is the signal power transmitted by the early warning aircraft to the UAV target n through the communication link, G nt G represents the transmitting antenna gain of the AWACS to the UAV target n, nr is the receiving antenna gain of the UAV target n, L nr =32.44+20lg(f n )+20lg(R nr ) represents the path propagation loss from the AWACS to the UAV target n, f n is the UAV communication frequency, R nr G represents the distance from the AWACS to the UAV target n; m,n represents the transmitting antenna gain of the jammer m on the UAV target n, L m,n =32.44+20lg(f n )+20lg(R m,n ) represents the path propagation loss from the jammer m to the drone target n, where R m,n represents the distance from the jammer m to the drone target n, σ 2 Represents the ambient noise power.

[0064] When the JSR at the receiver of the UAV target n exceeds the interference suppression coefficient K n When , the interference of the jammer on the UAV target n is considered effective, that is,

[0065] JSR n ≥K n

[0066] To characterize the effectiveness of interference power allocation, we define sign(JSR n -K n) is the benefit brought by implementing interference power allocation for UAV target n, where sign(·) represents the sign function.

[0067] S2: Construct a distributed locally observable Markov decision process Dec-POMDP, model the interference power allocation problem as a multi-agent fully cooperative model, and regard each interfering device as an agent; Dec-POMDP can be used as a seven-tuple <M e ,S p ,{A m},T,{O m},Z,r> means, where M e is the set of intelligent agents; S p is the state space; A m is the action space of agent m; T is the state transition function; O m is the observation space of agent m; Z represents the observation function; r represents the global reward function;

[0068] In the collaborative jamming task of multiple jammers, once the jamming beam is determined, the jammer needs to find the optimal joint power allocation strategy π=<π1,...,π M >To maximize the expected value of the cumulative discounted reward Solve

[0069]

[0070] Among them, π * is the optimal joint power allocation strategy, argmax represents the independent variable value corresponding to the maximum value of the function, It means to solve the expected value, s p,t Indicates the environmental state, a p,t represents the joint action of all interference devices, ρ π Represents a state-action pair (s p,t ,a p,t ) corresponding to the distribution function, r(s p,t ,a p,t ) means in state s p,t Perform joint action a p,t The instant reward obtained after , γ is the discount factor, the subscript p represents power, and t represents the time step.

[0071] At time step t, the local observation information of the interfering device m is o m,p,t =[o m,1,p,t ,o m,2,p,t ,...,o m,N,p,t ]∈O p , which contains observation information for N drone targets; o m,n,p,t=[m,n,R m,n,t ,R nr,t ,f n,t ,p m,n,t-1 ,JSR n,t ] represents the observation information about the UAV target n obtained by the jammer m, R m,n,t and R nr,t represents the jammer m and the distance between the AWACS and the UAV target n, respectively, n,t is the communication frequency, p m,n,t-1 is the interference power of the interference device m on the interference drone target n at the previous moment, JSR n,t is the interference-to-signal ratio at the UAV target n at time t; p represents the observation space of the interfering device. The global state s is defined by the local observation information of all interfering devices p,t =[o 1,p,t ,o 2,p,t ,...,o M,p,t ]∈S p To ensure that each jammer can independently implement jamming under parameter sharing conditions, the model not only integrates local observation information, but also introduces the index of each jammer and the index of the target drone. This design allows the model to perform personalized processing for each jammer and its assigned target to improve the accuracy of jamming decisions.

[0072] P max As a benchmark, the interference power is divided into L p Level, P L ={0,1,2,3,…,L p} represents the set of jamming power levels, 0 represents no jamming power allocation, let I be the maximum number of UAV targets that can be jammed simultaneously by a single jamming device, b m is the beam allocation vector of the mth interfering device, and the action space A p Defined as

[0073]

[0074] Where × represents the Cartesian product. Correspondingly, the joint action is a p,t =[a 1,p,t ,a 2,p,t ,...,a M,p,t ], where a m,p,t ∈A p Indicates the action of the interference device m

[0075] A distributed architecture is used to assign power jamming tasks to various jamming devices, and the global reward of power jamming tasks is decomposed. At the same time, considering resource efficiency, the jamming power consumption is introduced as a penalty term into the reward function, and the definition is Represents the normalized power consumption of the jammer m. To ensure that each jammer can complete its own jamming subtask and thus effectively suppress all UAV targets, the following reward function is defined.

[0076]

[0077] Among them, ρ1, ρ2 and ρ3 are all positive numbers, and ρ3 is used to control the emphasis on power consumption during training. The above formula shows that the less interference resources are used to suppress the UAV target, the higher the overall interference benefit of the system will be.

[0078] S3: The command center uniformly issues beam allocation commands to decompose the overall interference task. The interference equipment does not need to know the decision actions of other interference equipment when deciding the interference power, which avoids the impact of dynamic changes in complex environments on power decisions to a certain extent. The DTDE multi-agent framework is designed. The agent does not need to obtain the status and action information of other agents during training, and does not need to know the decision actions of other interference devices when deciding the interference power. It only relies on its own observations to decide the interference power. Each interference device has the same physical and functional properties, and they can be regarded as isomorphic agents. To this end, these devices can share the same training parameters, that is, the single-agent DRL method can be used to train the multi-agent network parameters, thereby significantly improving the training efficiency and decision-making effect. Figure 2 It is a DTDE multi-agent framework.

[0079] S4: Design an extractor for the agent to pre-process the input state information before deciding the jamming power. The command center issues beam allocation instructions to each jamming device. Jamming device m UAV target allocation result b m and the corresponding local observation information o m,p,t After that, the extractor is from o m,p,t The information related to the unassigned UAV targets is removed to obtain the processed observation data based on The agent makes interference power decisions for the assigned drone targets. The extractor not only reduces the dimensionality of the neural network input, but also reduces the interference of irrelevant information on the decision-making process, thereby improving the decision-making efficiency and accuracy of the system. Figure 3 The schematic diagram of the extractor.

[0080] S5: Solve the interference power resource allocation problem based on DSAC. The DSAC model is based on the AC architecture and maximum entropy reinforcement learning theory, aiming to maximize the expected return and the entropy of the strategy; at the same time, DSAC can reduce the algorithm's sensitivity to model and estimation errors, making the algorithm more robust.

[0081] In order to take into account both the interference benefit and the randomness of the joint power allocation strategy, the optimal joint power allocation strategy is transformed into

[0082]

[0083] in For the joint power allocation strategy π in state s p,t The entropy of , log represents the logarithm, and the entropy temperature coefficient α determines the relative importance of the reward value and the entropy regularization term.

[0084] In order to solve the optimal joint power allocation strategy, an alternating iterative method is used to evaluate and improve the strategy based on the flexible strategy within the maximum entropy framework. In the strategy evaluation stage, the flexible state value function of the interference device m is defined as

[0085]

[0086] Among them, Q m (o m,p,t ,a m,p,t ) is the observation o m,p,t Execute action a m,p,t The corresponding action value function, -log(π m (a m,p,t |o m,p,t )) is action a m,p,t The corresponding policy entropy. The extractor's local observation information of the interfering device o m,p,t Preprocessing is performed to extract the content related to the assigned drone target and obtain Therefore, the above formula can be further expressed as

[0087]

[0088] Define the flexible action value function (flexible Q function) based on the Bellman equation

[0089]

[0090] Then the strategy is improved by solving the following optimization problem

[0091]

[0092] Where D KL represents the Kullback-Leibler divergence, represents the strategy to be optimized, Π represents the candidate strategy set, argmin represents the independent variable value when the function reaches the minimum value, represents the flexible Q function under the current strategy, is the partition function of the current strategy, and exp represents the exponential function.

[0093] Using Critic network and Actor network to implement flexible Q function and the policy function Parameterization is performed, and each agent uses the same network model. To alleviate the problem of overestimation of Q value, a target critic network with the same structure as the critic network is used, and the target Q value is the sum of the reward value obtained and the flexible Q value output by the target critic network; at the same time, to ensure the stability of training, a dual neural network structure is used in both the critic network and the target critic network, and a single neural network structure is used in the actor network.

[0094] Therefore, in the process of strategy evaluation, the loss function of the Critic network can be expressed as

[0095]

[0096] Among them, θ i are the parameters of the Critic network, is the experience replay pool, y m,p,t For the status Take action a m,p,t The flexible target Q value can be expressed as

[0097]

[0098] Among them, min means to find the minimum value, are the parameters of the target Critic network, and φ are the parameters of the Actor network.

[0099] To minimize J Q (θ i ) as the target, the parameters of the Critic network are updated based on the Stochastic Gradient Descent (SGD), that is,

[0100]

[0101] where λ Q is the learning rate of the Critic network, Express J Q (θ i ) in the parameter θ i Perform derivation.

[0102] In the strategy improvement phase, the following loss function is defined, and the Actor network parameters are updated based on SGD.

[0103]

[0104] In the above formula, φ is the parameter of the Actor network. For observation The policy vector under φ is the learning rate of the Actor network, Express J π The parameter φ in (φ) is differentiated.

[0105] Define the loss function of the following entropy temperature coefficient α to achieve adaptive update of α

[0106]

[0107] in, represents the target entropy.

[0108] Use SGD to update α

[0109]

[0110] In the above formula, λ α is the learning rate of α, It means to derive the parameter α in J(α).

[0111] Finally, the parameters of the target Critic network are updated in a flexible way.

[0112]

[0113] Where τ is the flexible update coefficient.

[0114] The implementation of interference power resource allocation based on DSAC specifically includes the following steps:

[0115] S71: Initialize the number of interference devices M, the number of drone targets N, the total number of training rounds max_episode, the total interaction time step per round max_t, the parameters of the Actor, Critic network and target Critic, the experience replay pool, and the threshold Φ p ;

[0116] S72: If the number of training rounds exceeds max_episode, the training is terminated and S76 is executed; otherwise, max_t is initialized and S73 is executed;

[0117] S73: If the number of interaction time steps exceeds max_t, execute S72, otherwise execute S74;

[0118] S74: The agent receives the observation information, the extractor performs preprocessing, and then performs power allocation for the assigned drone targets, generates experience samples and stores them in the experience replay pool;

[0119] S75: If the number of samples in the experience replay pool is greater than the threshold Φp , randomly extract batch samples and train the Actor and Critic networks, otherwise execute S73;

[0120] S76: Save the DSAC model;

[0121] S77: Each interference device uses the DSAC model to make a distributed decision on interference power.

Claims

1. Interference power allocation method based on multi-agent deep reinforcement learning, characterized by: The following steps are involved: S1: In the "many-to-many" communication confrontation scenario, a collaborative interference model of multiple interference devices against multiple UAV targets is established. Multiple interference devices are distributed and deployed on the ground. Each UAV target forms a flight formation and is commanded by an early warning aircraft. S2: Construct a distributed partially observable Markov decision process (Dec-POMDP) ​​to model the interference power allocation problem as a multi-agent fully cooperative model; S3: Design a "decentralized training and distributed execution" (DTDE) multi-agent framework to reduce the collaborative decision-making dimension of interference power resources; S4: Design an extractor to preprocess the agent’s input state information to improve training efficiency; S5: To achieve the exploration and utilization of power allocation strategies, the agent network is trained based on the Discrete Soft Actor-Critic (DSAC) algorithm.

2. The interference power allocation method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In S1, a collaborative interference model of multiple interference devices against UAV targets is constructed. The interference party includes a command center and M interference devices, which implement suppressive interference on a UAV formation composed of N UAVs commanded by an enemy early warning aircraft. e ={1,2,…,N} and M e = {1, 2, ..., M} represents the set of N UAVs and jammers in the UAV formation. Each jammer can generate multiple power-controllable jammer beams, and the jammer m∈M is defined. e The beam pointing vector b m =[b m,1 ,b m,2 ,...,b m,N ] T , where b m,n ∈{0,1} represents the beam indicator variable. When the jammer m allocates a beam to the drone target n∈N e When interference is implemented, b m,n =1, otherwise b m,n =0, [·] T represents the transposition operation, and the subscripts m and n represent the index numbers of the jammer and the drone target, respectively. Based on this, the jammer beam allocation matrix is ​​defined as B=[b1,b2,...,b M ] T The command center issues a unified beam allocation command and defines the power allocation vector p of the interference device m according to the interference beam allocation result. m =[p m,1 ,p m,2 ,...,p m,N ] T , where p m,n Indicates that m,n To efficiently utilize the interference power, the interference power is non-zero only when the interference beam irradiates the target, that is, Where P max represents the maximum radiated power of the interference device. Therefore, for the interference device m, According to the power allocation vectors of M interfering devices, the interference power allocation matrix is ​​defined as P=[p1,p2,...,p M ] T When multiple jamming devices cooperate to jam a UAV formation, B and P can be used to characterize the allocation of jamming resources. Through interference coordination, the interference signals radiated by multiple interference devices are superimposed at the target, causing the target communication receiver to lose its normal working ability. The interference effect is usually measured by the Jamming Signal Ratio (JSR). Assuming that the drone target receiver uses an omnidirectional antenna and ignoring the effects of polarization loss and bandwidth mismatch loss, at a certain moment, the JSR at the drone target n receiver is where p n is the signal power transmitted by the early warning aircraft to the UAV target n through the communication link, G nt G represents the transmitting antenna gain of the AWACS to the UAV target n. nr is the receiving antenna gain of the UAV target n, L nr =32.44+20lg(f n )+20lg(R nr ) represents the path propagation loss from the AWACS to the UAV target n, f n is the UAV communication frequency, R nr G represents the distance from the AWACS to the UAV target n; m,n represents the transmitting antenna gain of the jammer m on the drone target n, L m,n =32.44+20lg(f n )+20lg(R m,n ) represents the path propagation loss from the jammer m to the drone target n, where R m,n represents the distance from the jammer m to the drone target n, σ 2 Represents the ambient noise power. When the JSR at the receiver of the UAV target n exceeds the interference suppression coefficient K n When , the interference of the jammer on the UAV target n is considered to be effective, that is, JSR n ≥K n。 3. The interference power allocation method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In S2, a distributed locally observable Markov decision process Dec-POMDP is constructed to model the interference power allocation problem as a multi-agent fully cooperative model, where each interfering device is regarded as an agent; Dec-POMDP can be used with seven tuples <M e ,S p ,{A m },T,{O m },Z,r> means, where M e is the set of intelligent agents; S p is the state space; A m is the action space of agent m; T is the state transition function; O m is the observation space of agent m; Z represents the observation function; r represents the global reward function; In the collaborative jamming task of multiple jammers, once the jamming beam is determined, the M jammers need to explore the joint power allocation strategy π=<π1,...,π M >To maximize the expected value of the cumulative discounted reward Solve Among them, π * is the optimal joint power allocation strategy, argmax represents the independent variable value corresponding to the maximum value of the function, It means to solve the expected value, s p,t Indicates the environmental state, a p,t represents the joint action of all interference devices, ρ π Represents a state-action pair (s p,t ,a p,t ) corresponding to the distribution function, r(s p,t ,a p,t ) means in state s p,t Perform joint action a p,t The instant reward obtained after , γ is the discount factor, the subscript p represents power, and t represents the time step.

4. The interference power allocation method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In S3, a DTDE multi-agent framework is designed, and the agent does not need to obtain the state and action information of other agents during training, and does not need to know the decision actions of other interference devices when deciding the interference power, and only relies on its own observation to decide the interference power. Each interference device has the same physical and functional properties, and they can be regarded as isomorphic agents. To this end, these devices can share the same training parameters, that is, the single-agent deep reinforcement method can be used to train the multi-agent network parameters, thereby significantly improving the training efficiency and decision-making effect.

5. The interference power allocation method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In S4, an extractor is designed for the agent to pre-process the input state information before making a decision on the interference power. The command center issues a beam allocation instruction to each interference device. The agent extracts information related to the assigned target from the observation information according to the beam allocation result, removes information related to the unassigned UAV target, and makes an interference power decision for the assigned UAV target based on the pre-processed observation information. The extractor reduces the input dimension of the neural network while reducing the interference of irrelevant information on the decision-making process, thereby improving the efficiency and accuracy of the system decision-making.

6. The interference power allocation method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In S5, the interference power resource allocation problem is solved based on DSAC. The DSAC model is based on the actor-critic (AC) architecture and maximum entropy reinforcement learning theory, aiming to maximize the expected return and the entropy of the strategy; at the same time, DSAC can reduce the algorithm's sensitivity to model and estimation errors, making the algorithm more robust. In order to take into account both the interference benefit and the randomness of the joint power allocation strategy, the optimal joint power allocation strategy is transformed into in For the joint power allocation strategy π in state s p,t The entropy of , log represents the logarithm, and α represents the entropy temperature coefficient. A target evaluation network with the same structure as the evaluation (Critic) network is used to alleviate the problem of Q-value over-estimation. The target Q-value is the sum of the reward value obtained and the flexible Q-value output by the target Critic network. At the same time, to ensure the stability of training, a dual neural network structure is used in both the Critic network and the target Critic network, and a single neural network structure is used in the policy (Actor) network.

7. The interference power allocation method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In S5, implementing interference power resource allocation based on DSAC includes the following steps: S71: Initialize the number of interference devices M, the number of drone targets N, the total number of training rounds max_episode, the total interaction time step per round max_t, the parameters of the Actor, Critic network and target Critic, the experience replay pool, and the threshold Φ p S72: If the number of training rounds exceeds max_episode, the training is terminated and S76 is executed; otherwise, max_t is initialized and S73 is executed; S73: If the number of interaction time steps exceeds max_t, execute S72, otherwise execute S74; S74: The agent receives the observation information, the extractor performs preprocessing, and then performs power allocation for the assigned drone targets, generates experience samples and stores them in the experience replay pool; S75: If the number of samples in the experience replay pool is greater than the threshold Φ p , randomly extract batch samples and train the Actor and Critic networks, otherwise execute S73; S76: Save the DSAC model; S77: Each interference device uses the DSAC model to make a distributed decision on interference power.

Images