Deep reinforcement learning assisted target threat degree evaluation model and interference beam allocation method

Abstract

The invention relates to a deep reinforcement learning assisted target threat degree evaluation model and an interference beam allocation method, and belongs to the technical field of wireless mobile communication. The method comprises the following steps: S1, in a wartime communication confrontation scene, constructing a cooperative interference model of a multi-interference device confronting enemy unmanned aerial vehicle formation; s2, analyzing constraint conditions of an interference beam distribution process; s3, introducing a target threat degree evaluation link, and constructing an interference beam resource allocation optimization problem; s4, constructing a target threat degree evaluation model based on a tactical importance plotting (TSM) function, and generating a target threat degree evaluation matrix; s5, improving the TSM function by using a discrete flexible actor-commentator (DSAC) algorithm, and reconstructing a target threat degree evaluation matrix; and S6, according to the target threat degree evaluation matrix, executing interference beam allocation. According to the invention, reasonable allocation of interference beams in a many-to-many wartime communication confrontation scene can be realized, so that the utilization rate of interference resources is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of wireless mobile communication technology and relates to a target threat assessment model and interference beam allocation method assisted by deep reinforcement learning. Background Art

[0002] The deep integration of artificial intelligence (AI) technology, especially machine learning (ML), and electronic warfare (EW) has promoted the leapfrog development of EW from "artificial cognition" to "machine cognition". In the wartime environment where the threat of intelligent electronic equipment is becoming increasingly severe, the competition for control of the electromagnetic spectrum between the two sides of cognitive EW has become more intense, and resource constraints have become the main constraint on the effectiveness of the confrontation between the two sides. Therefore, whether limited jamming resources can be allocated efficiently and reasonably has become the key to maximizing the effectiveness of EW jamming.

[0003] Deep reinforcement learning (DRL) combines deep learning (DL) with reinforcement learning (RL), giving the intelligent agent powerful autonomous decision-making capabilities and abstract representation capabilities of the environment. It has the advantages of fast solution speed, support for multi-dimensional decision-making, and strong generalization ability. With the continuous advancement of technology, the electronic warfare environment is becoming increasingly complex, among which "many-to-many" clustered operations have gradually become the mainstream confrontation method of wartime electronic warfare. Jamming equipment is required to have the ability to jam multiple targets at the same time.

[0004] Some scholars have studied the resource allocation problem for multi-target interference in multi-beam surface-to-air radar jamming systems by comprehensively considering the individual and overall interference effects (Reference: Cui Zheming, Peng Shirui, Ren Mingqiu, et al. Research on multi-target interference decision-making in multi-beam jamming systems [J]. Firepower and Command Control, 2021, 46(12): 149-155.); some research work has explored the joint scheduling of jamming beams and power (Reference: ZHANG D, SUN J, YI W, et al. Joint jamming beamand power scheduling for suppressing netted radar system [C] / / 2021IEEE RadarConference (RadarConf21). IEEE, 2021: 1-6.). However, these studies mainly rely on conventional multi-objective optimization theory in the design of multi-objective jamming resource allocation schemes; however, due to the complexity of high-dimensional jamming resource allocation, these schemes often face challenges such as slow convergence and poor scalability, and it is difficult to meet the needs of cognitive electronic countermeasure systems for intelligent allocation of large-scale jamming resources. In addition, related work simplifies or lacks the target threat assessment link when solving the jamming resource allocation problem. Target threat assessment is a core component of command, control, decision-making and other links in the complex electronic countermeasure process during wartime, and is a prerequisite and important basis for realizing jamming resource allocation. Summary of the invention

[0005] The present invention aims to provide a target threat assessment model and interference beam allocation method assisted by deep reinforcement learning, which can achieve reasonable allocation of high-dimensional interference beam resources by introducing a target threat assessment link. The technical solution of the present invention is as follows:

[0006] S1: In the wartime communication confrontation scenario, a collaborative interference model of multiple interference devices against enemy drone formations is constructed;

[0007] S2: Analyze the resource constraints of the beam resource allocation process;

[0008] S3: Introduce the target threat assessment link, construct the interference beam resource allocation optimization problem, and guide the reasonable allocation of interference beam resources;

[0009] S4: Build a target threat assessment model based on the Tactical Significance Map (TSM) function and generate a target threat assessment matrix;

[0010] S5: Use the Discrete Soft Actor-Critic (DSAC) algorithm to improve the TSM function and reconstruct the target threat assessment matrix;

[0011] S6: Based on the target threat assessment matrix, execute the interference beam resource allocation strategy and generate the interference beam resource allocation matrix.

[0012] Furthermore, in S1, a "many-to-many" wartime communication confrontation scenario is considered, in which the jammer M jammers implement suppressive jamming on a drone formation composed of N drones commanded by an enemy early warning aircraft; e ={1,2,…,N} and M e ={1,2,…,M} respectively represent the set consisting of N drones and M jammers in the drone formation. In order to avoid jamming, the speed and direction of the drone formation are in dynamic change while flying towards the jammer’s position; the jammer can generate multiple beams to jam multiple drone targets at the same time.

[0013] Furthermore, in S2, the beam pointing vector of the interference device m is b m =[b m,1 ,b m,2 ,...,b m,N ] T , where b m,n ∈{0,1} is the beam pointing variable, [·] T represents the transposition operation, m, n represent the index numbers of the jammer and the drone target. e Assign beam to UAV target n∈N e When interference is implemented, b m,n =1, otherwise b m,n = 0; define the beam resource allocation matrix B = [b1, b2, ..., b M ] T , considering that the UAV targets in the UAV formation are relatively dispersed in space, a single beam cannot cover multiple targets at the same time. It is assumed that a beam of each jammer can only interfere with a single UAV target; at the same time, restricted by the performance of the jammer, a single jammer can interfere with at most I UAV targets at the same time:

[0014]

[0015] To ensure the reasonable use of interference beam resources, each UAV is limited to be interfered by a maximum of U beams:

[0016]

[0017] Furthermore, in S3, when allocating interference resources, the differences in the threat levels of different UAV targets need to be considered. According to the evaluation result of the interference device m, the target threat level evaluation vector w is defined m =[w m,1 ,wm,2,... ,w m,N ] T , where 0≤w m,n ≤1; Comprehensively evaluate the results of all interference devices and define the target threat evaluation matrix as W = [w1,w2,...,w M ] T ; Define the average threat level of drone target n as The jamming beam resource allocation problem of multiple jammers for UAV formations can be expressed as:

[0018]

[0019] In the above formula, P1 is the interference beam allocation problem, max means maximization, and P1 means maximizing the sum of the average threat levels of all UAV targets by solving the beam resource allocation matrix B under the constraints C1 and C2.

[0020] Furthermore, in S4, the target threat level assessment is performed based on the TSM function, and the TSM function can be expressed as:

[0021]

[0022] in, is the state vector of the drone target n at the current moment, [x n ,y n ] T and They represent the position vector and velocity vector of the drone target n, respectively, and the reference point coordinate is h c =[x n ,y n ] T , exp is the exponential function, represents the distance between the drone target n and the reference point, k0 and m0 are preset constants, is the absolute speed of the drone target n, θ c,n is the heading angle of the UAV target n relative to the reference point, and its expression is arccos represents the inverse cosine function; all interference devices are taken as reference points, and target threat assessment is performed on the UAV target to generate the target threat assessment matrix W.

[0023] Furthermore, in S5, when the drone target is far away from the jammer, the target threat assessment result based on the TSM function will degenerate into the assessment result based on a single jammer. To address this problem, the target threat assessment matrix reconstruction problem is modeled as a Markov decision process (MDP), including state space, action space and reward function. The motion state information of the drone target n at time t for M jammers is s n,b,t =[o 1,b,t ,o 2,b,t ,...,o M,b,t ], where the subscript b represents the beam, t represents the time, and o m,b.t =[R m,n,t ,v n,t ,θ m,n,t ], including the distance R from the drone target n to the jammer m at time t m,n,t , speed v n,t and heading angle θ m,n,t Based on s n,b,t , define the state of MDP as s b,t =[s 1,b,t ,s 2,b,t ,...,s N,b,t ]∈S b , where S b is the state space; the action space is M e ; The reward function is based on the TSM function. When a jammer is selected to jam the drone target n, if the threat evaluation value w of the selected jammer m is m,n In all evaluation results w n =[w m,1 ,w m,2 ,...,w m,N ] T The higher the value, the higher the reward value.

[0024] Further, in said S5, the target threat level assessment matrix is ​​reconstructed using the DSAC algorithm. The target threat level assessment matrix reconstruction process includes the following steps:

[0025] S71: Initialize the number of interference devices M, the number of drone targets N, the total number of training rounds max_episode, the total interaction time step per round max_t, the parameters of the policy (Actor) network, the evaluation (Critic) network and the target evaluation (Critic) network, the experience replay pool, and the threshold Φ b , let B = 0, W = 0;

[0026] S72: If the number of training rounds exceeds max_episode, the training is terminated and S76 is executed; otherwise, max_t is initialized and S73 is executed;

[0027] S73: If the number of interaction time steps exceeds max_t, execute S72, otherwise execute S74;

[0028] S74: For drone target n, the agent Actor network input state s n,b,t , output action a n,b,t And get reward n,b,t , the state is transferred to s n,b,t+1 , the experience sample (s n,b,t ,a n,b,t ,r n,b,t ,s n,b,t+1 ) is stored in the experience replay pool;

[0029] S75: If the number of samples in the experience replay pool is greater than the threshold Φ b , randomly extract batch samples and train the Actor and Critic networks, otherwise execute S73;

[0030] S76: Save the DSAC model;

[0031] Further, in S6, based on the target threat assessment matrix, the interference beam resource allocation strategy is executed to generate an interference beam resource allocation matrix, including the following steps:

[0032] S81: For each drone target n, the agent Actor network input s n,b,t , output probability vector ξ n And save, reconstruct the target threat assessment matrix W = [ξ1,ξ2,...,ξ N ];

[0033] S82: Loop through W, select the position with the largest threat assessment value in W in descending order and set it to 0, and map the value of the corresponding position in the interference beam allocation matrix B to 1. During the mapping process, the C1 and C2 constraints must be satisfied at the same time until each UAV target is illuminated by an interference beam;

[0034] S83: Based on S82, the mapping process is repeatedly cycled through, and when the number of drones assigned to each jammer reaches the maximum number of drone targets I that can be jammed simultaneously, the beam resource allocation process ends;

[0035] S84: Output interference beam resource allocation matrix B

[0036] The beneficial effect of the present invention is that the present invention aims at the problem of high-dimensional interference beam resource allocation difficulties caused by the dynamic characteristics of the environment and the increase in the number of communication confrontation members in the "many-to-many" communication confrontation scenario, and proposes a deep reinforcement learning-assisted communication confrontation interference resource allocation method. First, a collaborative interference model of multiple interference devices against enemy drone formations is established; then, considering the constraints such as limited interference resources and interference device performance, the target threat assessment link is introduced, and the interference beam resource allocation optimization problem is established; then, the target threat assessment matrix is ​​constructed based on the TSM function, and the target threat assessment matrix is ​​reconstructed using DSAC to address the shortcomings of the TSM function; finally, the DSAC network model is saved, and the interference beam allocation matrix is ​​decided using the Actor network, effectively realizing the reasonable allocation of high-dimensional interference beam resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0038] Figure 1 Coordinated jamming model for multiple jammers to counter enemy drone formations Figure 2 DSAC algorithm network structure diagram

[0039] Figure 3 Overall flow chart for interference beam allocation DETAILED DESCRIPTION

[0040] The embodiment of the present invention proposes a deep reinforcement learning-assisted target threat assessment model and interference beam allocation method, which can effectively realize the reasonable allocation of high-dimensional interference beam resources.

[0041] The specific steps of this method are as follows:

[0042] S1: In the wartime communication confrontation scenario, a coordinated interference model of multiple interference devices against the enemy UAV formation is constructed. In this model, the interference party includes M interference devices, which implement suppressive interference on the UAV formation composed of N UAVs commanded by the enemy early warning aircraft; e ={1,2,…,N} and M e ={1,2,…,M} respectively represent the set consisting of N drones and M jammers in the drone formation. In order to avoid interference, the speed and direction of the drone formation change dynamically during the flight towards the jammer’s position. The jammer can generate multiple beams to interfere with multiple drone targets at the same time.

[0043] S2: The beam pointing vector of the interference device m is b m =[b m,1 ,b m,2 ,…,b m,N ]T , where b m,n ∈{0,1} is the beam pointing variable, [·] T represents the transposition operation, and the subscripts m and n represent the index numbers of the jammer and the drone target. e Assign beam to UAV target n∈N e When interference is implemented, b m,n =1, otherwise b m,n = 0; define the beam resource allocation matrix B = [b1, b2, ..., b M ] T , considering that the UAV targets in the UAV formation are relatively dispersed in space, a single beam cannot cover multiple targets at the same time. It is assumed that a beam of each jammer can only interfere with a single UAV target; at the same time, restricted by the performance of the jammer, a single jammer can interfere with at most I UAV targets at the same time:

[0044]

[0045] To ensure the reasonable use of interference beam resources, each UAV is limited to be interfered by a maximum of U beams:

[0046]

[0047] S3: In communication confrontation, the jamming effect is closely related to the way jamming resources are allocated. In actual communication confrontation, the greater the threat level of the target, the more attention it should receive. Therefore, when allocating jamming resources, it is necessary to consider the differences in the threat levels of different drone targets. To this end, based on the evaluation results of the jamming device m, the target threat evaluation vector w is defined m =[w m,1 ,w m,2 ,...,w m,N ] T , where 0≤w m,n ≤1; then, the evaluation results of all interference devices are integrated to define the target threat evaluation matrix W = [w1,w2,...,w M ] T ;

[0048] In wartime scenarios, in order to ensure the survivability of each jammer, each jammer is usually deployed in a distributed manner. Therefore, different jammers have different threat assessments for different targets. Therefore, in order to give full play to the geographical advantages of the jammer, any UAV target should be jammed by a jammer with a higher threat assessment as much as possible. Therefore, the jammer beam allocation can be determined based on the target threat assessment matrix. To this end, the average threat level of the UAV target n is defined as The jamming beam allocation problem of multiple jammers for UAV formations can be expressed as:

[0049]

[0050] In the above formula, P1 is the interference beam allocation problem, max means maximization, and P1 means maximizing the sum of the average threat levels of all UAV targets by solving the beam resource allocation matrix B under the constraints C1 and C2.

[0051] S4: Considering the differences in the motion states of different UAV targets, a dynamic evaluation model of target threat degree is established based on TSM. The TSM function can be expressed as:

[0052]

[0053] in, is the state vector of the drone target n at the current moment, [x n ,y n ] T and They represent the coordinates and speed of the drone target n, respectively, and the reference point coordinate is h c =[x c ,y c ] T , exp is the exponential function, represents the distance between the drone target n and the reference point, k0 and m0 represent preset constants, is the absolute speed of the drone target n, θ c,n is the heading angle of the UAV target n relative to the reference point, and its expression is arccos represents the inverse cosine function; all interference devices are used as reference points, and target threat assessment is performed on the UAV target to generate the target threat assessment matrix W

[0054] S5: The dynamic target threat assessment model based on the TSM function uses different jamming devices as references to assess the target threat based on the motion state of the UAV target. When the UAV target is far away from the jamming device, the motion state information of the same UAV target for different jamming devices is similar. Therefore, in the target threat assessment matrix W determined by the TSM function, the difference between the threat values ​​of the same UAV target between different jamming devices is very small. At this time, the assessment result in W will degenerate into a threat assessment based on a single jamming device.

[0055] In view of the above problems, DSAC is used to reconstruct the target threat assessment matrix, and the motion states of all jammers targeting the same UAV target are used as the input of the policy (Actor) network. The target threat assessment results based on TSM are used to design the reward function, and finally the output probability vector of the Actor network output layer (this layer is a fully connected layer, outputting a probability vector with a sum of 1) is used as the new threat assessment vector, that is, each probability value is used as the new threat assessment value of each jammer to the UAV target. In this way, the difference in threat assessment between different jammers can be magnified, and then a new target threat assessment matrix W can be constructed. The target threat assessment problem is modeled as a Markov decision process (MDP), including state space, action space and reward function.

[0056] The motion state information of the drone target n against the M jammers at time t is s n,b,t =[o 1,b,t ,o 2,b,t ,...,o M,b,t ], where b represents beam, t represents time, and o m,b.t =[R m,n,t ,v n,t ,θ m,n,t ], including the distance R from the drone target n to the jammer m at time t m,n,t , speed v n,t and heading angle θ m,n,t Based on s n,b,t , define the state of MDP as s b,t =[s 1,b,t ,s 2,b,t ,...,s N,b,t ]∈S b , where S b is the state space.

[0057] The action space is M e , action a n,b,t ∈M e It means that a pair of UAV targets n are selected from M jammers to perform jamming; from the target threat assessment matrix W determined based on TSM, the nth column, i.e., w, is selected for the jamming target n. n =[w 1,n ,w 2,n ,...,w M,n ] T , design reward function

[0058]

[0059] where w n [a n,b,t ] indicates that from w nGet action a n,b,t The target threat assessment value w of the corresponding jammer m against the drone target n m,n ,sorted(w n ) represents the vector w n The elements in are sorted from small to large, idx(w n [a n,b,t ],sorted(w n )) is used to obtain w m,n After sorting, the vector w n The position index in (assuming the position order is represented by (1, 2, ..., M)). This formula shows that when a jammer is selected to jam the drone target n, if the threat assessment value w of the selected jammer is m,n In all evaluation results w n The higher the value, the higher the reward value; define the joint reward r b,t =[r 1,b,t ,r 2,b,t ,...,r N,b,t ], which contains the rewards for N drone targets.

[0060] The DSAC algorithm can not only maximize expected returns and policy entropy at the same time, but also reduce sensitivity to model and estimation errors, and has the advantages of fast, comprehensive and stable training. The optimal joint strategy solution is constructed as shown in the following formula:

[0061]

[0062] Among them, π * is the optimal joint strategy, argmax represents the independent variable value corresponding to the maximum value of the function, represents the expected value, ρ π Represents a state-action pair (s b,t ,a b,t ) corresponds to the distribution function, γ is the discount factor, That is, the strategy π in state s b,t The entropy of , log represents the logarithm, and the entropy temperature coefficient α determines the relative importance of the reward value and the entropy term, thereby controlling the randomness of the optimal strategy.

[0063] In order to improve the stability of training and reduce the estimation deviation of the target Q value, this paper adopts a target evaluation (Critic) network with the same structure as the evaluation (Critic) network, and regularly updates the parameters of the Critic network to the target Critic network; in addition, in order to further alleviate the problem of over-estimation of the Q value, this paper adopts a dual neural network structure in both the Critic network and the target Critic network, that is, each network contains two neural networks with the same structure. During the training process, the Critic network and the target Critic network each select the smaller of the Q values ​​in the two neural networks to update the network parameters to optimize the performance and ensure the accuracy of the estimation.

[0064] To evaluate the state s of the drone target n at time t n,b,t The value of , defines the soft value function

[0065]

[0066] where θ i is the Critic network parameter, for a n,b,t The corresponding action value function, -log(π(a n,b, t|s n,b,t )) represents the policy entropy, and φ is the Actor network parameter.

[0067] In status n,b,t Execute action a n,b,t After that, the update process of the target Q value can be expressed according to the soft value function as follows:

[0068]

[0069] Among them, y(a n,b,t ,s n,b,t+1 ) is the target Q value, For the target Critic network in state s n,b,t+1 Take action a n,b,t+1 The corresponding action value, is the parameter of the target Critic network, and min means taking the minimum value.

[0070] In order to effectively evaluate the quality of the actions performed, the loss function of the Critic network is defined

[0071]

[0072] in, For the experience replay pool.

[0073] In the strategy evaluation phase, the Critic network parameters are updated by stochastic gradient descent.

[0074]

[0075] Among them, λ Q is the learning rate of the loss function of the Critic network, Express J Q (θ i ) parameter θ in the function i Derivation;

[0076] In the strategy improvement phase, the strategy is updated by minimizing the Kullback-Leibler (KL) divergence.

[0077]

[0078] Where Π represents the candidate strategy set, π(·|s n,b,t ) represents the strategy to be optimized, represents the Q function under the current strategy, is the distribution function of the current strategy. After simplification, the loss function of the Actor network is obtained as follows

[0079]

[0080] Update Actor network parameters using stochastic gradient descent

[0081]

[0082] where λ φ is the learning rate of the loss function of the Actor network, Express J π Derivative of the parameter φ in (φ);

[0083] By setting the target entropy, using the gradient descent method to adaptively update α, the loss function of the entropy temperature coefficient is defined

[0084]

[0085] in, is the target entropy, which is related to the agent action dimension;

[0086] Update the entropy temperature coefficient α by training J(α)

[0087]

[0088] where λ α is the learning rate of the loss function of the entropy temperature coefficient, It means to derive the parameter α in J(α);

[0089] Finally, the parameters of the target Critic network are updated in a flexible way.

[0090]

[0091] Among them, τ is the flexible update coefficient.

[0092] The reconstruction process of the target threat assessment matrix includes the following steps:

[0093] S71: Initialize the number of interference devices M, the number of drone targets N, the total number of training rounds max_episode, the total interaction time step per round max_t, the parameters of the Actor, Critic network and target Critic, the experience replay pool, and the threshold Φ b , let B = 0, W = 0;

[0094] S72: If the number of training rounds exceeds max_episode, the training is terminated and S76 is executed; otherwise, max_t is initialized and S73 is executed;

[0095] S73: If the number of interaction time steps exceeds max_t, execute S72, otherwise execute S74;

[0096] S74: For drone target n, agent Actor network input s n,b,t , output action a n,b,t And get rewards rn,b,t , the state is transferred to s n,b,t+1 , the experience sample (s n,b,t ,a n,b,t ,r n,b,t ,s n,b,t+1 ) is stored in the experience replay pool;

[0097] S75: If the number of samples in the experience replay pool is greater than the threshold Φ b , randomly extract batch experience samples to train the Actor and Critic networks, otherwise continue to execute S73;

[0098] S76: Save DSAC model

[0099] S6: Based on the target threat assessment matrix, the interference beam resource allocation strategy is executed to generate an interference beam resource allocation matrix, which specifically includes the following steps:

[0100] S81: For each drone target, the agent Actor network inputs n and outputs a probability vector ξ n And save, reconstruct the target threat assessment matrix W = [ξ1,ξ2,...,ξ N ]

[0101] S82: Loop through W, select the position with the largest threat assessment value in W in descending order and set it to 0, and map the value of the corresponding position in the interference beam allocation matrix B to 1. During the mapping process, the C1 and C2 constraints must be satisfied at the same time until each UAV target is illuminated by an interference beam;

[0102] S83: Based on S82, the mapping process is repeated in a loop. When the number of drones assigned to each interference device reaches the maximum number of drone targets I that can be interfered with simultaneously, the interference beam allocation process ends.

[0103] S84: Output interference beam allocation matrix B.

Claims

1. A target threat assessment model and interference beam resource allocation method assisted by deep reinforcement learning, characterized by: The following steps are involved: S1: In the wartime communication confrontation scenario, a collaborative interference model of multiple interference devices against enemy drone formations is constructed; S2: Analyze the resource constraints of the beam resource allocation process; S3: Introduce the target threat assessment link, construct the interference beam resource allocation optimization problem, and guide the reasonable allocation of interference beam resources; S4: Build a target threat assessment model based on the Tactical Significance Map (TSM) and generate a target threat assessment matrix; S5: Use the Discrete Soft Actor-Critic (DSAC) algorithm to improve the TSM function and reconstruct the target threat assessment matrix; S6: Based on the target threat assessment matrix, execute the interference beam resource allocation strategy and generate an interference beam resource allocation matrix.

2. The target threat assessment model and interference beam allocation method assisted by deep reinforcement learning according to claim 1, characterized in that: In S1: consider a "many-to-many" wartime communication confrontation scenario, in which the jammer M jammers implement suppressive jamming on a drone formation composed of N drones commanded by an enemy early warning aircraft; N e ={1,2,...,N} and M e ={1,2,…,M} respectively represent the set of N drones and M jammers in the drone formation. In order to avoid interference, the speed and direction of the drone formation are in dynamic change during the flight towards the jammer’s position; the jammer can generate multiple beams to interfere with multiple drone targets at the same time.

3. The target threat assessment model and interference beam allocation method assisted by deep reinforcement learning according to claim 1, characterized in that: In S2: the beam pointing vector of the interference device m is b m =[b m,1 ,b m,2 ,...,b m,N ] T , where b m,n ∈{0,1} is the beam pointing variable, [·] T represents the transposition operation, m, n represent the index numbers of the jammer and the drone target. e Assign beam to UAV target n∈N e When interference is implemented, b m ,n=1, otherwise b m ,n=0; define beam resource allocation matrix B=[b1,b2,...,b M ] T , considering that the UAV targets in the UAV formation are relatively dispersed in space, a single beam cannot cover multiple targets at the same time. It is assumed that a beam of each jammer can only interfere with a single UAV target; at the same time, restricted by the performance of the jammer, a single jammer can interfere with at most I UAV targets at the same time: To ensure the reasonable use of interference beam resources, each UAV is limited to be interfered by a maximum of U beams:

4. The target threat assessment model and interference beam allocation method assisted by deep reinforcement learning according to claim 1, characterized in that: In S3: When allocating jamming resources, it is necessary to consider the differences in the threat levels of different drone targets. Based on the evaluation results of the jamming device m, the target threat level evaluation vector w is defined m =[w m,1 ,w m,2 ,…,w m,N ] T , where 0≤w m,n ≤1; Comprehensively evaluate the results of all interference devices and define the target threat evaluation matrix as W = [w1,w2,...,w M ] T ; Define the average threat level of drone target n as The jamming beam allocation problem of multiple jammers for UAV formations can be expressed as: In the above formula, P1 is the interference beam allocation problem, max means maximization, and P1 means maximizing the sum of the average threat levels of all UAV targets by solving the beam resource allocation matrix B under the constraints C1 and C2.

5. The target threat assessment model and interference beam allocation method assisted by deep reinforcement learning according to claim 1, characterized in that: In S4, assuming that the communication confrontation scenario is an XY two-dimensional space, the target threat assessment is performed based on the TSM function, and the TSM function can be expressed as: in, is the state vector of the drone target n at the current moment, [x n ,y n ] T and They represent the position vector and velocity vector of the drone target n, respectively, and the reference point coordinate is h c =[x c ,y c ] T , exp is the exponential function, represents the distance between the drone target n and the reference point, k0 and m0 are preset constants, is the absolute speed of the drone target n, θ c,n is the heading angle of the UAV target n relative to the reference point, and its expression is arccos represents the inverse cosine function; all interference devices are taken as reference points, and target threat assessment is performed on the UAV target to generate the target threat assessment matrix W.

6. The target threat assessment model and interference beam allocation method assisted by deep reinforcement learning according to claim 1, characterized in that: In S5, when the drone target is far away from the jammer, the target threat assessment result based on the TSM function will degenerate into the assessment result based on a single jammer. To address this problem, the target threat assessment matrix reconstruction problem is modeled as a Markov decision process (MDP), including state space, action space and reward function. The motion state information of the drone target n at time t for M jammers is s n,b,t =[o 1,b,t ,o 2,b,t ,...,o M,b,t ], where the subscript b represents the beam, the subscript t represents the time, and o m,b.t =[R m,n,t ,v n,t ,θ m,n,t ], including the distance R from the drone target n to the jammer m at time t m,n,t , speed v n,t and heading angle θ m,n,t Based on s n,b,t , define the state of MDP as s b,t =[s 1,b,t ,s 2,b,t ,...,s N,b,t ]∈S b , where S b is the state space; the action space is M e ; The reward function is based on the TSM function. When a jammer is selected to jam the drone target n, if the threat evaluation value w of the selected jammer m is m,n In all evaluation results w n =[w m,1 ,w m,2 ,...,w m,N ] T The higher the value, the higher the reward value.

7. The target threat assessment model and interference beam allocation method assisted by deep reinforcement learning according to claim 1, characterized in that: In S5, the DSAC algorithm is used to improve the TSM function, and the reconstruction process of the target threat assessment matrix includes the following steps: S71: Initialize the number of interference devices M, the number of drone targets N, the total number of training rounds max_episode, the total interaction time step per round max_t, the parameters of the policy (Actor) network, the evaluation (Critic) network and the target evaluation (Critic) network, the experience replay pool, and the threshold Φ b , let B = 0, W = 0; S72: If the number of training rounds exceeds max_episode, the training is terminated and S76 is executed; otherwise, max_t is initialized and S73 is executed; S73: If the number of interaction time steps exceeds max_t, execute S72, otherwise execute S74; S74: For drone target n, agent Actor network input s n,b,t , output action a n,b,t And get reward n,b,t , the state is transferred to s n,b,t+1 , the experience sample (s n,b,t ,a n,b,t ,r n,b,t ,s n,b,t+1 ) is stored in the experience replay pool; S75: If the number of samples in the experience replay pool is greater than the threshold Φ b , randomly extract batch samples and train the Actor and Critic networks, otherwise execute S73; S76: Save the DSAC model.

8. The target threat assessment model and interference beam allocation method assisted by deep reinforcement learning according to claim 1, characterized in that: In S6, based on the target threat assessment matrix, the interference beam allocation strategy is executed to generate an interference beam resource allocation matrix, which specifically includes the following steps: S81: For each drone target n, the agent Actor network input s n,b,t , output probability vector ξ n And save, reconstruct the target threat assessment matrix W = [ξ1,ξ2,...,ξ N ]; S82: loop through W, select the position with the largest threat assessment value in W in descending order and set it to 0, and map the value of the corresponding position in the interference beam allocation matrix B to 1. During the mapping process, the C1 and C2 constraints must be satisfied at the same time; S83: Based on S82, the mapping process is repeatedly cycled through, and when the number of drones assigned to each jammer reaches the maximum number of drone targets I that can be jammed simultaneously, the beam resource allocation process ends; S84: Output interference beam resource allocation matrix B.

Images