A radar jamming resource allocation method based on MADDPG and auction algorithm

Abstract

The present invention discloses a radar jamming resource allocation method based on MADDPG and auction algorithm. The method first establishes an interference confrontation model and a radar detection model, and based on the interference confrontation model, realizes target allocation through an auction algorithm. Secondly, a state space of a jammer is established, including its own information, information of other jammers and target radar information. Then, a multi-agent decomposition method is adopted to decompose each jammer into three decision-making agents, and an action space of each decision-making agent is established. Finally, based on the state space and the action space, combined with the allocation matrix of the target allocation in the auction algorithm, the interference resource allocation is realized through the MADDPG algorithm. The present invention dynamically and real-timely allocates the interference resources of the jammer, and realizes the real-time and efficient allocation of the interference resources of the jammer in the high-dimensional action space.

Description

Technical Field

[0001] The present invention belongs to the technical field of radar jamming decision making, and in particular relates to a radar jamming resource allocation method based on multi-agent deep deterministic policy gradient (MADDPG) and an auction algorithm. Background Art

[0002] At present, with the widespread application of adaptive radars, the demand for jamming resource allocation strategies is increasing. The jammer needs to be able to adjust its own strategy in real time according to the parameters of the surrounding radars to autonomously select the optimal jamming strategy. Due to the large variety and quantity of jamming resources, the discrete action space dimension of the jammer as an intelligent agent is high, which makes it difficult for the model training to converge. With the complexity of the modern electronic countermeasure environment, radar systems are characterized by large number, wide distribution and strong performance. The traditional single jammer to multi-radar jamming mode faces problems such as insufficient jamming resources, reduced jamming efficiency and increased decision-making pressure. In order to solve the above problems, the multi-jammer collaborative mode is introduced. However, the multi-jammer system has problems such as low efficiency of collaborative decision-making and resource allocation conflicts. Summary of the invention

[0003] The purpose of the present invention is to provide a radar jamming resource allocation decision method based on MADDPG and auction algorithm in response to the deficiencies of the prior art. A two-layer decision architecture is adopted, and the upper layer allocates N radars to M jammers through an auction algorithm, so that when the number of jamming radars is limited, each jammer can allocate high-threat-level radars to jammers with higher priorities as much as possible; secondly, in view of the problem that high-dimensional discrete action space is difficult to converge, the lower layer uses the MADDPG algorithm to split the jammer penetration task into three subtasks: jamming target selection, jamming style selection, and jamming power selection, and dynamically and in real time allocates the jammer's jamming resources. In order to achieve the above purpose, the technical solution adopted by the present invention includes the following steps:

[0004] Step 1: Establish the jammer countermeasure model and radar detection model. The jammer countermeasure model includes the jammer set , radar collection , the penetration path and location information of the jammer. The radar detection model is the maximum detection distance of the radar in a noise and interference environment .

[0005] Step 2: Based on the interference adversarial model, the auction algorithm achieves target allocation. Initialize the price matrix and the allocation matrix , establish the value assessment function , calculate each jammer For each radar The profit value is used as the basis for bidding decisions; the maximum net profit is selected through iterative bidding radar targets and price matrix and the allocation matrix to update.

[0006] Step 3: Establish the jammer's state space , the state space S includes its own information, other jammers’ information and target radar information. Target radar information is the current working mode of all radars. In a multi-function radar system, the radar has three working modes: search, track and lock. The radar’s ability to detect a target mainly depends on the distance between the radar and the target: when the target is The maximum range of the radar When within the range (i.e. ), the radar can effectively detect and find the target. On the contrary, if the target distance exceeds the maximum range of the radar (i.e. ), the target will not be detected by the radar.

[0007] Step 4: High-dimensional discrete action space processing, using the multi-agent decomposition method to decompose each jammer into three specialized decision-making agents, and establish the action space of each decision-making agent The interference target selection agent is responsible for determining the specific target set that needs to be interfered from all possible targets. The interference style selection agent allocates the most suitable interference mode to each selected target. The interference power selection agent reasonably allocates interference power to the selected target according to the specified interference style based on the decision results of the first two layers.

[0008] Step 5: MADDPG algorithm implements interference resource allocation. Each agent consists of a dual network structure: a policy network (Actor) responsible for action decision-making and a value network (Critic) responsible for state evaluation. The jammer selects the target from the auction allocation scheme. The allocation matrix obtains the target allocation result and builds status information , input into the strategy network of each agent to obtain the strategy set of all agents , execute the strategy to update the data and return the reward and the next moment state .Will Store to experience replay pool In, from Random sampling The batch samples update the Critic network and the Actor network, and soft update the target network. For each radar, in this round of detection, when the radar's range Greater than the radial distance from the jammer to the radar The target can be detected by the radar when the search mode is the initial state and the lock mode is the terminal state. After entering the lock mode, the iteration ends. If the penetration distance of the jammer is greater than the set maximum penetration distance, the training ends. The radar jamming resource allocation is performed using the neural network model that meets the penetration mission requirements. Jammer status Input into the neural network to obtain the strategy set adopted by the jammer group , that is, the real-time allocation result of radar jamming resources of the drone swarm.

[0009] The beneficial effects of the present invention are:

[0010] The present invention introduces MADDPG and auction algorithm into the multi-to-multi radar jamming resource allocation. The jammer improves the collaborative performance among multiple agents through the auction algorithm, obtains the allocated target radar at the same time, and models the jamming resource allocation as a Markov decision process to obtain the state space, reward function, and action space of the problem, wherein the high-dimensional discrete action space is processed, and the single jammer agent is divided into three heterogeneous agents for collaborative decision-making to reduce the dimension of the action space. Then, based on the MADDPG algorithm, the jamming resource allocation strategy is designed to dynamically and real-time allocate the jammer's jamming resources, thereby realizing the real-time and efficient allocation of the jammer's jamming resources in the high-dimensional action space. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0012] Figure 1 This is the schematic diagram of the MADDPG algorithm;

[0013] Figure 2 It is a flow chart of the scheme of the present invention;

[0014] Figure 3 It is a schematic diagram of electromagnetic confrontation scenario;

[0015] Figure 4 It is the working mode conversion diagram of the multi-function radar;

[0016] Figure 5 It is a curve diagram of the maximum penetration steps of the algorithm of the present invention and the DQN algorithm during the training process in a one-to-many scenario;

[0017] Figure 6 It is a curve chart of the maximum number of penetration steps of the algorithm of the present invention during the training process in a many-to-many scenario. DETAILED DESCRIPTION

[0018] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0019] like Figure 1 As shown in the figure, the present invention applies MADDPG and auction algorithm to the radar jamming resource allocation method. In a complex electromagnetic confrontation environment, it ensures that the jamming resources of multiple jammers can be dynamically adjusted in the high-dimensional action space according to the actual situation. The solution process is as follows Figure 2 As shown, the specific steps include:

[0020] Step 1: Establish a jamming countermeasure model, which includes the location and number of radars, the penetration path and distance of jammers. Specifically, there are M jammers and N radars, and their position relationship is as follows: Figure 3 As shown, the total penetration distance is , the radial distance between each jammer and the radar is , the distance between each jammer is , the speed is , the distance between the target radars is .

[0021] The radar detection model is established as:

[0022] Where: is the Boltzmann constant; is standard room temperature, usually ; is the internal noise bandwidth of the receiver; is the noise factor; is the received sea clutter power; is the signal-to-noise ratio measured at the receiver output, i.e. Pulse signal-to-noise ratio; is the jammer antenna gain; To deviate from the radar Receiving gain at angle; is the radial distance from the jammer to the radar; is the polarization factor; is the wavelength of the transmitted signal; is the target scattering cross-sectional area; is the pulse compression ratio; is the radar receiver loss; is the interference power; is the radar transmitting power; is the radar antenna gain; is the radar receiving antenna gain. Here, the radar receiving signal direction is considered to be the main lobe direction of the radar antenna, so In this area, the jammer carries interference resources such as noise amplitude modulation jammer, noise frequency modulation jammer, smart noise jammer and dense false target jammer to jointly suppress and interfere with the radar system, reducing the radar detection performance. The jammer's penetration route is a straight line. It is considered a successful penetration if it passes through the radar protection area and safely reaches the designated position. When the radar enters the locking mode, it is considered a penetration failure.

[0023] Step 2 uses an auction algorithm to achieve target allocation.

[0024] Initialize the price matrix , let all elements ; Initialize the allocation matrix , let all elements ; To quantify the effectiveness of the jammer on the radar target, a value evaluation function is constructed:

[0025]

[0026] In the formula Indicates the distance factor, reflecting the positional relationship between the jammer and the target radar. Indicates the target threat level, reflecting the threat level of the radar's current working mode to the jammer. are the distance factor and the weight coefficient of the target's threat degree respectively.

[0027] For each jammer , calculate its response to each target radar Net income:

[0028]

[0029] in For the goal The current price of .

[0030] Each jammer finds the target that provides the greatest net benefit and suboptimal goals :

[0031]

[0032]

[0033]

[0034] Bidding increment:

[0035]

[0036] in The minimum positive number representing the bid increment.

[0037] Update target price:

[0038]

[0039] Update the allocation relationship:

[0040]

[0041] in For the jammer in the allocation matrix Jamming radar elements.

[0042] The algorithm is implemented when all jammers have completed their assignment or are continuous. The round allocation scheme remains unchanged.

[0043] Step 3: Establish the state space of each agent in the jammer , where each agent includes its own information, other agents’ information, and target radar information, so they have the same state set ,in , Contains target radar, interference pattern, and interference power information. Indicates the distance between the current jammer and the radar position. Indicates the current working mode of all radars. For the jammer, the multi-function radar mainly has search mode, tracking mode and lock mode. For each radar step, in this round of detection, when the radar's working distance Greater than the distance between the target and the radar When , the target can be found by the radar. Search mode is the initial state, and lock mode is the termination state. After entering the lock mode, this iteration ends.

[0044] Step 4 processes the high-dimensional discrete action space and uses the multi-agent decomposition method to decompose each jammer into three specialized decision-making agents: jamming target selection agent, jamming style selection agent, and jamming power selection agent. And establish the action space of each agent , these three agents form a hierarchical decision-making structure, making decisions in the order of target selection, interference style selection, and finally interference power allocation. Department radar, There are four types of jammers: noise amplitude modulation, noise frequency modulation, smart noise and dense false targets, and the corresponding jamming power is . Establish the action space of each agent in the jammer ,in is the target radar selection action space, is the interference style selection action space, is the interference power selection action space.

[0045] The target selection agent is responsible for Choose no more than The action space is dimensional binary vector space, that is , for The elements.

[0046] The style selection agent assigns the selected target based on the target selection result. One of the available interference styles, the dimension of the action space depends on the number of selected targets, that is, .

[0047] The power selection agent allocates interference power to the selected target based on the decisions of the first two agents, which must satisfy both the single target power upper limit and the total power constraint, that is, .

[0048] Step 5: MADDPG algorithm realizes radar jammer resource allocation

[0049] Initialize the Critic network of all agents, namely the target radar selection agent, the interference style selection agent, and the interference power selection agent, i.e. the value network and target value network , and Actor Networks as Policy Networks and target strategy network The value network requires global information, while the policy network requires local information. Initialize the experience replay buffer , used to store the agent's experience data (state, action, reward, next state) in the environment interaction according to the auction algorithm's sub-target allocation results and strategy network ,get The strategy set of each agent at the moment ,award , the next moment state , storing it in the experience replay buffer .

[0050] The three agents achieve collaborative decision-making through a reward distribution mechanism. The reward obtained by each agent after interacting with the environment.

[0051] Successful Interference Reward :

[0052]

[0053] If the radar keeps the working mode unchanged, the working mode If the radar working mode changes from search mode to tracking mode or from tracking mode to locking mode, then Increase; if the radar working mode changes from tracking mode to search mode, then Radar operating mode conversion is as follows Figure 4 shown.

[0054] Interference Power Bonus :

[0055]

[0056] in The value range is .

[0057] Disruption Style Reward

[0058]

[0059]

[0060] Among them, during the breakthrough process The four distance nodes correspond to different reward functions according to the interference patterns.

[0061] From the buffer The random sampling batch is of samples, one of which is , calculate the target Q value for each agent , and update the Critic network by minimizing the loss. The minimization loss function is:

[0062]

[0063] Combined with the Q value function of the Critic training network, the Actor network is updated according to the policy gradient:

[0064]

[0065] The target strategy network and target value network are updated through the soft update mechanism, specifically:

[0066]

[0067]

[0068] In the formula is the learning rate.

[0069] Table 1 shows the specific parameter settings for simulation training using the MADDPG algorithm.

[0070]

[0071] Figure 5 This is a comparison of the maximum penetration step curves of the MADDPG algorithm and the reinforcement learning algorithm DQN in the training process in a one-to-many scenario. The parameter settings are shown in Table 1. It can be seen from the figure that after about 600 trainings, the maximum penetration distance of the MADDPG algorithm of the present invention has been basically stable, while the DQN algorithm needs more than 800 trainings to achieve stability, and the penetration effect is far lower than that of the MADDPG algorithm. This is because there are 3*4*10 action combinations in the action space of DQN, which makes the action space dimension too large to train and converge. Figure 6 In order to maximize the number of penetration steps in the training process in a many-to-many scenario based on MADDPG and auction algorithm, the jammers were first assigned to target radars, and then the dimension of the action space was greatly reduced by processing the high-dimensional discrete action space. Finally, the MADDPG algorithm was used to improve the speed and effect of training.

Claims

1. A radar jammer resource allocation method based on MADDPG and auction algorithm, characterized in that: The following steps are involved: Step 1: Establish interference countermeasure model and radar detection model; Step 2: Based on the interference adversarial model, target allocation is achieved through an auction algorithm; Step 3: Establish the jammer’s state space S, including its own information, other jammers’ information, and target radar information; Step 4: Use the multi-agent decomposition method to decompose each jammer into three decision-making agents, and establish the action space A of each decision-making agent; Step 5: Based on the state space S and action space A, combined with the allocation matrix X of the target allocation in the auction algorithm, the interference resource allocation is realized through the MADDPG algorithm; The specific process of implementing interference resource allocation by the MADDPG algorithm is as follows: In the MADDPG algorithm, each agent consists of a dual network structure: a policy network Actor responsible for action decision-making and a value network Critic responsible for state evaluation. The jammer obtains the target allocation result from the allocation matrix X in the auction allocation scheme, constructs the state information s∈S, inputs it into the policy network of each agent to obtain the policy set a∈A of all agents, executes the policy update data and returns the reward r and the state s′ at the next moment; stores (s, a, r, s′) in the experience replay pool d, randomly samples H batches of samples from D to update the Critic network and the Actor network, and soft-updates the target network; For each radar, in this round of detection, when the radar's range R max Greater than the radial distance R from the jammer to the radar d When , the target is discovered by the radar; the search mode is the initial state, and the locking mode is the termination state. After entering the locking mode, this iteration ends. If the penetration distance of the jammer is greater than the set maximum penetration distance, the training ends; the radar jamming resource allocation is performed using the neural network model that meets the penetration task, and the state S of the M jammers at a certain moment is input into the neural network to obtain the strategy set a adopted by the jammer group, that is, the real-time allocation result of the radar jamming resources of the drone group.

2. The radar jammer resource allocation method based on MADDPG and auction algorithm according to claim 1, characterized in that: The interference countermeasure model includes a jammer set J={J1, J2, ..., J M }、Radar set R={R1,R2,…,R N }, the penetration path and location information of the jammer; The radar detection model is the maximum detection distance R of the radar in a noise and interference environment. max : Where: k is the Boltzmann constant; T0 is the standard room temperature; B n is the internal noise bandwidth of the receiver; F n is the noise factor; P c is the received sea clutter power; D s (n) is the signal-to-noise ratio measured at the receiver output, i.e., the signal-to-noise ratio of n pulses; G j is the jammer antenna gain; G t (θ) is the receiving gain when the radar deviates from the angle θ; R d is the radial distance from the jammer to the radar; γ j is the polarization factor; λ is the wavelength of the transmitted signal; σ is the target scattering cross-sectional area; D is the pulse compression ratio; L t is the radar receiver loss; P j is the interference power; P t is the radar transmitting power; G t is the radar antenna gain; G r is the radar receiving antenna gain; here, the radar receiving signal direction is considered to be the main lobe direction of the radar antenna, so G t =G r =G.

3. The radar jammer resource allocation method based on MADDPG and auction algorithm according to claim 2, characterized in that: The implementation process of step 2 is as follows: Initialize the price matrix P and the allocation matrix X, and let all elements p ij =0,x ij =0; Construct value evaluation function: V(J i ,R j )=w1·D(J i ,R j )+w2·T(R j ); Where D(J i ,R j ) represents the distance factor, which reflects the positional relationship between the jammer and the target radar. j ) represents the target threat degree, reflecting the threat degree of the radar's current working mode to the jammer, w1 and w2 are the weight coefficients of the distance factor and the target threat degree respectively; For each jammer J i , calculate its response to each target radar R j Net income: U(J i ,R j )=V(J i ,R j )-P(R j ); Where P(R j ) is the target R j The current price of Each jammer finds the radar target u1 and the second best target u2 that provides the maximum net benefit: Bidding increment: γ=u1-u2+ε; Where ε represents the minimum positive number of the bidding increment; Update target price: Update the allocation relationship: X(J i ,R j )=1; Where X(J i ,R j ) is the allocation matrix for jammer J i Jamming Radar R j Elements of The process terminates when all jammers have completed their allocation or the allocation scheme remains unchanged for K consecutive rounds.

4. The radar jammer resource allocation method based on MADDPG and auction algorithm according to claim 3, characterized in that: The target radar information is the current working mode of all radars. In the multi-function radar system, the radar has three working modes: search, track and lock. The radar's ability to detect the target depends on the distance between the radar and the target: when the target is at a distance R d The maximum range of the radar is R max When the target is within the range, the radar can effectively detect and discover the target; otherwise, if the target distance exceeds the maximum effective range of the radar, the target will not be detected by the radar.

5. The radar jammer resource allocation method based on MADDPG and auction algorithm according to claim 4, characterized in that: In step 4, for the three decision agents: a decision agent for interference target selection is responsible for determining the specific target set for interference from all targets; a decision agent for interference style selection allocates interference mode to each selected target; a decision agent for interference power selection allocates interference power to the selected target according to the interference style based on the decision results of the first two layers; the details are as follows: In this penetration scenario, there are N radars and M jammers with four jamming styles: noise amplitude modulation, noise frequency modulation, smart noise and dense false targets. The corresponding jamming power is P min ~P max ; Establish the action space A of each agent in the jammer = [a r ,a t ,a p ], where a r is the target radar selection action space, a t is the interference style selection action space, a p It is the interference power selection action space; The decision agent for interference target selection is responsible for selecting no more than n targets to interfere with among N targets. Its action space is an N-dimensional binary vector space, i.e., a r ={a1∈{0,1} N |∑ i a 1i ≤n}, a 1i is the i-th element in a1; The decision agent for interference pattern selection assigns one of K = 4 interference patterns to the selected target based on the target selection result. The dimension of its action space depends on the number of selected targets, i.e., a t (a1)={a2∈{1,...,K} N |a 2i =0 if a 1i =0}; The decision agent for interference power selection allocates interference power to the selected target based on the decisions of the first two agents, which must satisfy both the single target power upper limit and the total power constraint, i.e., a p (a1,a2)={a3∈[0,P max ] N |a 3i =0 ifa 1i =0,∑ i a 3i ≤P max }.

Images