A joint optimization method of jamming patterns and operating parameters for multi-function radar

Abstract

The present invention proposes a joint optimization method for interference strategies of interference patterns and working parameters for multi-function radars. Compared with the game confrontation between strategy-level radar and interference, a joint optimization model of strategy and parameter levels is constructed. First, the confrontation problem of the multi-function radar is modeled as a Markov decision process model, and then the state space, action space and reward function of the model are clarified. According to different action spaces, corresponding deep reinforcement learning neural networks are designed for optimization and solution to obtain the maximum cumulative reward, thereby obtaining the optimal solution for the interference strategy of the confrontation process. The present invention uses the continuous interaction between the intelligent jammer of the deep reinforcement learning algorithm and the electromagnetic environment, and optimizes its own strategy through environmental feedback, thereby improving the redeployment speed of the interference strategy, the scenario robustness and the accuracy of the strategy.

Description

Technical Field

[0001] The present invention belongs to the technical field of the intersection of radar electronic countermeasures and artificial intelligence, and specifically relates to a joint optimization method for interference patterns and operating parameters for multi-function radars. Background Art

[0002] Multifunction radars play a vital role in electronic countermeasures due to their rapid response, flexible beams, and excellent anti-interference capabilities. Modern multifunction radars can continuously switch between candidate operating modes based on reconnaissance of the external environment. Traditional jamming decision-making systems typically target a specific radar operating mode, so developing intelligent algorithms to counter multifunction radars is an important research direction in current radar electronic countermeasures. Game theory, a mathematical theory that studies cooperative or competitive phenomena, is widely used in electronic countermeasures research. However, the premise of using game theory analysis is to ensure that all participants in the game are rational, which makes it difficult to adapt to the highly subjective nature of electronic countermeasures. Secondly, the goal of game theory is to find the Nash equilibrium or Stackelberg equilibrium of the problem, and proving that this equilibrium is the optimal solution to a given problem is extremely difficult.

[0003] The existing intelligent jamming decision-making of multi-function radars mainly studies strategic-level confrontation. The jammer optimizes its own jamming strategy based on the relationship between the advantages and disadvantages of the radar working mode and the alternative jamming styles according to the mode set by expert knowledge. However, this strategic-level confrontation strategy without parameter optimization is difficult to apply to the actual complex electromagnetic environment. Summary of the Invention

[0004] In view of this, the present invention provides a joint optimization method for interference patterns and operating parameters for multi-function radars, which can solve the optimization problem of the interference patterns and operating parameters of the interference party for different operating modes of the multi-function radar in a complex and dynamic radar confrontation environment.

[0005] A method for joint optimization of jamming patterns and operating parameters for a multi-function radar comprises the following steps:

[0006] Step 1: Set the electronic countermeasure scenario of multi-function radar and intelligent jammer, and set the multi-function radar to have N mode Working mode, with radiation power P R Send reconnaissance signals to the target. The intelligent jammer can choose two jamming styles: suppression jamming and deception jamming. Jmax The interference signal is transmitted to the radar, and the background noise power is P noiseSet the working mode state transition table of the multi-function radar and the radar state transition report table under different interference patterns. The multi-function radar switches states according to the interference effect. The jammer can select an interference pattern at each moment and select an operating parameter within the maximum radiation power range to interfere with the radar. Set the position of each radar (x r ,y r , z r ) and the initial position of the target (x0, y0, z0), and the target's speed at a fixed height (v x , v y );

[0007] Step 2: Model the Markov decision model and determine the five-tuple of the decision process established by the game confrontation process. Where N is the number of jammers, which is 1. The Markov decision process is used to model the confrontation scenario, and the state space is constructed according to the characteristic set of radar working state, jamming pattern and working parameters. Action Space According to the optimization goal determined by the jammer, that is, the optimization goal of minimizing the consumption of power resources as much as possible while successfully jamming the multi-function radar, the reward function R, the state transfer matrix P, and the discount factor γ are determined;

[0008] In this step, the state space of the Markov decision process at step t is defined as [W t , W t-1 , JSR t , a t-1 ], where JSR t Represents the interference effect of the current t-th step, W t and W t-1 Respectively represent the radar working mode of this step and the previous step, a t-1 Indicates the interference action performed in the previous step;

[0009] In this step, the jamming action of the Markov decision process at time t is defined as (T, p), where T∈{0,1} represents the jamming pattern indicator of deceptive jamming and suppression jamming, and p represents the jamming power selected by the jammer under the current jamming pattern. The present invention constructs the action space in the following two ways:

[0010] 1) Discrete action space, discretize the interference power range into p∈{0, 1, ..., P Jmax}, the power values ​​of deceptive interference and suppression interference are selected from m discrete values;

[0011] 2) Hybrid action space: The interference action is constructed as a hybrid action space, where the interference pattern is a discrete action. Under each discrete interference action, there are continuously selectable working parameters, that is, arbitrary interference power.

[0012] Step 3: Based on the constructed action space and state space Establish a deep reinforcement learning neural network and set the number of network training times N episode , the number of training steps in each round of training is N step , initialize the neural network parameters and the jammer state;

[0013] Among them, different deep reinforcement learning neural networks are used according to the different action spaces constructed in step 2:

[0014] 1) For discrete action spaces, the D3QN deep reinforcement learning neural network is used;

[0015] 2) For the mixed action space, HPPO deep reinforcement learning neural network is used;

[0016] Step 4: Train the D3QN or HPPO network selected in step 3 until the network converges or reaches the termination condition. The training is completed.

[0017] During network training, the neural network outputs the next action a0 based on the initial state s0, obtains the next state s1 through interaction in the electromagnetic environment, and obtains the immediate reward r1 corresponding to the action. These data are stored in cache B (s0, a0, s1, r1), and the state s1 is set as the starting state of the neural network for the next step. The above process is repeated to continuously obtain the next action, the next state, and the immediate reward corresponding to the action.

[0018] Step 5: Within the same number of training rounds as step 4, if the actual number of training steps in step 3 reaches the set N step , or the early termination condition is reached, then stop the training of this round and enter the next round of training. Otherwise, repeat step 4 to continue training, and repeat step 5 until the number of training rounds reaches N episode After completing the training;

[0019] Step 6: Based on the jamming strategy obtained using deep reinforcement learning neural network, the intelligent jammer obtains a jamming solution for the multi-function radar during the confrontation process in the confrontation environment.

[0020] Preferably, the D3QN network consists of an input layer, two fully connected layers, and an output layer. The input layer is the state input of 4 neurons, and the output layer neurons correspond to the discrete action space.

[0021] The HPPO network consists of a performer network and a critic network. The performer network consists of an input layer, two fully connected layers, and an output layer. The input layer is the state input of four neurons. Each layer of the fully connected layer is connected to the Tanh activation function. The output layer is divided into two branches. Branch 1 uses the Softmax layer to output discrete actions, and branch 2 uses the Beta distribution as a continuous power value, using the Softplus layer and adding 1 to output continuous power.

[0022] The outputs of branch 1 and branch 2 are combined to obtain the interference joint action, and the critic network evaluates the joint action selected by the performer network.

[0023] Preferably, the interference effect described in step 1 is expressed by the interference-to-signal ratio, and the current interference effect JSR at time t is t The mathematical expression is:

[0024]

[0025] Among them, P J , P R , P N are the jammer power, radar power and the ambient N~(0,σ) Gaussian noise power respectively. and They represent the channel gains of the jammer and radar signal transmission channels, p j and p r is the transmitting power of the transmitter. According to the radar formula, it can be expressed as:

[0026]

[0027] Among them, G t and G r Represents the gain of the transmitting and receiving antennas of the multi-function radar, G I represents the gain of the interfering transmitting antenna, σ RCS represents the radar interception area, λ is the signal wavelength, and d is the radial distance from the multi-function radar to the interference.

[0028] Preferably, in step 1, when the jammer selects to suppress the jamming, JSR t ≥γ, it is considered that the suppression interference is effective. When the jammer chooses deception interference, ρ2≥JSR t ≥ρ1, the deception interference is considered effective, and in other cases the interference is considered unsuccessful, where γ represents the suppression interference coefficient, ρ2 and ρ1 represent the effective upper and lower bounds of the deception interference coefficient, respectively.

[0029] Preferably, in step 1 and step 2, the reward function R of the Markov model at time t can be expressed by the following mathematical formula:

[0030]

[0031] in Indicates the power return when the interference is successful, k2(W t , W t+1 ) and k3(W t , W t+1 ) represent the adjacent working modes W of the multi-function radar under different interference patterns. t and W t+1 The mode return after switching, k4 represents the return of whether the interference is successful or not, and the Sign function is the sign function.

[0032] Preferably, for discrete action spaces, a D3QN deep reinforcement learning neural network is designed. The D3QN network combines the Double DQN and Dueling DQN networks and calculates the objective function using the following formula:

[0033]

[0034] where R t Indicates the reward value of the action, s t+1 Indicates execution of action a t+1 The states after θ and θ - They represent the network parameters of the network and the parameters of the target network respectively. The agent adopts the ε-greedy strategy when selecting the strategy;

[0035] The mathematical expression of the value function is as follows:

[0036]

[0037] Among them, V represents the state value function, Q represents the action state value function, A represents the advantage function of the action state, and α and β represent the network parameters of the two fully connected layer branches of the network.

[0038] Preferably, for the mixed action space, a HPPO deep reinforcement learning neural network is designed. Based on the PPO algorithm, performer networks are constructed for discrete and continuous actions respectively. Specifically, the agent's observations are first fed into the fully connected layer. Then the output is divided into two parts: the performer network of the interference pattern and the interference parameters. After merging the outputs, the interference joint action is obtained. The critic network evaluates the joint action selected by the performer network. The optimization objectives for discrete and continuous actions are respectively:

[0039]

[0040]

[0041] in and Represent the probability ratio of the new and old strategies for discrete actions and continuous actions respectively, and The policy with fixed old parameters for discrete and continuous action performer networks, and are the strategies to be updated for the discrete and continuous action performer networks respectively; clip represents the clipping function, which is used to and The range of is controlled within the range of 1-∈ to 1+∈, where ∈ is a clipping hyperparameter used to control the difference between the new and old strategies; is the target advantage function at time t, and the generalized advantage estimation technique is used to estimate the advantage function

[0042]

[0043] Among them, when 1=0, is the GAE parameter, State s t+1 , s t The objective value function under represents the parameters of the target critic network.

[0044] Preferably, the discrete action performer network uses the Softmax activation function to output the discrete action strategy with the highest probability, and the continuous action performer network samples the mean and variance from the Beta distribution to output the continuous action strategy.

[0045] Preferably, in step 4, the intelligent jammer and the environment form a trajectory during the interaction process [s t , a t , s t+1 , r t , a t+1 , s t+2 , r t+1 ,...], the goal of the agent is to maximize the cumulative reward obtained from multiple interaction trajectory sequences, which can be expressed as follows:

[0046]

[0047] Among them, T is the total time of the interaction between the agent and the environment, that is, the number of training steps.

[0048] Preferably, in step 4, the trajectory sequence is stored in an experience cache, and a batch of data is randomly selected from the cache each time during training to update the network.

[0049] The present invention has the following beneficial effects:

[0050] Aiming at the multi-function radar confrontation problem, the present invention proposes a joint optimization method for interference strategies of interference patterns and working parameters of multi-function radars. Compared with the game confrontation between strategy-level radar and interference, a joint optimization model of strategy and parameter levels is constructed. First, the confrontation problem of multi-function radar is modeled as a Markov decision process model, and then the state space, action space and reward function of the model are clarified. According to different action spaces, corresponding deep reinforcement learning neural networks are designed for optimization and solution to obtain the maximum cumulative reward, thereby obtaining the optimal solution for the interference strategy of the confrontation process. The present invention uses the continuous interaction between the intelligent jammer of the deep reinforcement learning algorithm and the electromagnetic environment, and optimizes its own strategy through environmental feedback, thereby improving the redeployment speed of the interference strategy, the scenario robustness and the accuracy of the strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is the experimental flow chart of the present invention.

[0052] Figure 2 Schematic diagram of the D3QN algorithm in the present invention.

[0053] Figure 3 Schematic diagram of the HPPO algorithm in the present invention.

[0054] Figure 4 This is the training result diagram of the D3QN algorithm.

[0055] Figure 5 This is the training result diagram of the HPPO algorithm.

[0056] Figure 6 This is a graph showing the change in the interference-to-signal ratio of the intelligent jammer during the countermeasure process.

[0057] Figure 7 This is a graph showing the change in interference power of the intelligent jammer during the countermeasure process.

[0058] Figure 8 This is a sequence diagram of the multi-function radar working mode in the countermeasure process.

[0059] Figure 9 The following are the training results of D3QN and HPPO networks under different interference coefficients. DETAILED DESCRIPTION

[0060] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. A method for jointly optimizing jamming patterns and operating parameters for a multi-function radar comprises the following steps:

[0061] Step 1: Set the electronic countermeasure scenario of multi-function radar and intelligent jammer, and set the multi-function radar to have Nmode Working mode, with radiation power P R Send reconnaissance signals to the target. The intelligent jammer can choose two jamming styles: suppression jamming and deception jamming. The interference signal is transmitted to the radar, and the background noise power is P noise Set the working mode state transition table of the multi-function radar and the radar state transition report table under different interference patterns. The multi-function radar switches states according to the interference effect. The jammer can select an interference pattern at each moment and select an operating parameter within the maximum radiation power range to interfere with the radar. Set the position of each radar (x r ,y r , z r ) and the initial position of the target (x0, y0, z0), and the target's speed at a fixed height (v x , v y );

[0062] Step 2: Model the Markov decision model and determine the five-tuple of the decision process established by the game confrontation process. Where N is the number of jammers, which is 1. The Markov decision process is used to model the confrontation scenario, and the state space is constructed according to the characteristic set of radar working state, jamming pattern and working parameters. Action Space According to the optimization goal determined by the jammer, that is, the optimization goal of minimizing the consumption of power resources as much as possible while successfully jamming the multi-function radar, the reward function R, the state transfer matrix P, and the discount factor γ are determined;

[0063] In this step, the state space of the Markov decision process at step t is defined as [W t , W t-1 , JSR t , a t-1 ], where JSR t Represents the interference effect of the current t-th step, W t and W t-1 Respectively represent the radar working mode of this step and the previous step, a t-1 Indicates the interference action performed in the previous step.

[0064] In this step, the jamming action of the Markov decision process at time t is defined as (T, p), where T∈{0,1} represents the jamming pattern indicator of deceptive jamming and suppression jamming, and p represents the jamming power selected by the jammer under the current jamming pattern. The present invention constructs the action space in the following two ways:

[0065] 1) Discrete action space, discretize the interference power range into The power values ​​of deceptive interference and suppression interference are selected from m discrete values;

[0066] 2) Hybrid action space: The interference action is constructed as a hybrid action space, where the interference pattern is a discrete action. Under each discrete interference action, there are continuously selectable working parameters, that is, arbitrary interference power.

[0067] Step 3: Based on the constructed action space and state space Establish a deep reinforcement learning neural network and set the number of network training times N episode , the number of training steps in each round of training is N step , initialize the neural network parameters and the jammer state;

[0068] Among them, different deep reinforcement learning neural networks are used according to the different action spaces constructed in step 2:

[0069] 1) For discrete action spaces, the D3QN deep reinforcement learning neural network is used;

[0070] 2) For the mixed action space, HPPO deep reinforcement learning neural network is used;

[0071] The D3QN network consists of an input layer, two fully connected layers, and an output layer. The input layer is the state input of 4 neurons. The number of neurons in the two fully connected layers is 128 and 64 respectively. Each layer is connected to the ReLU activation function. The output layer includes 200 neurons corresponding to the discrete action space.

[0072] The HPPO network consists of a performer network and a critic network. The performer network consists of an input layer, two fully connected layers, and an output layer. The input layer is the state input of 4 neurons. The number of neurons in the fully connected layer is 64. Each layer is connected to the Tanh activation function. The output layer is divided into two branches. Branch 1 uses the Softmax layer to output discrete actions. Branch 2 uses the Beta distribution as a continuous power value, uses the Softplus layer and adds 1 to output continuous power.

[0073] The outputs of branch 1 and branch 2 are combined to obtain the interference joint action, and the critic network evaluates the joint action selected by the performer network;

[0074] Step 4: Train the D3QN or HPPO network selected in step 3 until the network converges or reaches the termination condition. The training is completed.

[0075] During network training, the neural network outputs the next action a0 based on the initial state s0, obtains the next state s1 through interaction in the electromagnetic environment, and obtains the immediate reward r1 corresponding to the action. These data are stored in cache B (s0, a0, s1, r1), and the state s1 is set as the starting state of the neural network for the next step. The above process is repeated to continuously obtain the next action, the next state, and the immediate reward corresponding to the action.

[0076] Step 5: Within the same number of training rounds as step 4, if the actual number of training steps in step 3 reaches the set N step , or the early termination condition is reached, then stop the training of this round and enter the next round of training. Otherwise, repeat step 4 to continue training, and repeat step 5 until the number of training rounds reaches N episode After completing the training;

[0077] Step 6. Based on the jamming strategy obtained by using deep reinforcement learning neural network, the intelligent jammer can quickly obtain the optimized jamming solution for the multi-function radar during the confrontation process in the confrontation environment.

[0078] The jamming effect described in step 1 is expressed by the jamming-to-signal-ratio (JSR). The current jamming effect JSR at time t is t The mathematical expression is:

[0079]

[0080] Among them, P J , P R , P N are the jammer power, radar power and the ambient N~(0,σ) Gaussian noise power respectively. and They represent the channel gains of the jammer and radar signal transmission channels, p j and p r is the transmitting power of the transmitter. According to the radar formula, it can be expressed as:

[0081]

[0082] Among them G t and G r Represents the gain of the transmitting and receiving antennas of the multi-function radar, G I represents the gain of the interfering transmitting antenna, σ RCS represents the radar interception area, λ is the signal wavelength, and d is the radial distance from the multi-function radar to the interference.

[0083] Furthermore, different interference signal ratio thresholds are set for different interference patterns. Specifically, when the jammer chooses to suppress interference, the JSR t ≥γ, it is considered that the suppression interference is effective. When the jammer chooses deception interference, ρ2≥JSR t ≥ρ1, the deception interference is considered effective, and in other cases the interference is considered unsuccessful, where γ represents the suppression interference coefficient, ρ2 and ρ1 represent the effective upper and lower bounds of the deception interference coefficient, respectively.

[0084] Furthermore, in step 2, the reward function R of the Markov model at time t can be expressed by the following mathematical formula:

[0085]

[0086] in represents the power return when the jamming is successful, k2 and k3 represent the adjacent working modes W of the multi-function radar under different jamming patterns. t and W t+1 The mode return after switching, k4 represents the return of whether the interference is successful or not, and the Sign function is the sign function.

[0087] Furthermore, in step three,

[0088] 1) For discrete action spaces, we design the D3QN deep reinforcement learning neural network. The D3QN network combines the DoubleDQN and Dueling DQN networks and calculates the objective function using the following formula:

[0089]

[0090] where R t Indicates the reward value of the action, s t+1 Indicates execution of action a t+1 The states after θ and θ - They represent the network parameters of the network and the parameters of the target network respectively. The agent adopts the ε-greedy strategy when selecting the strategy.

[0091] The mathematical expression of the value function is as follows:

[0092]

[0093] Where V represents the state value function, Q represents the action state value function, A represents the advantage function of the action state, and α and β represent the network parameters of the two fully connected layer branches of the network.

[0094] 2) For the mixed action space, a HPPO deep reinforcement learning neural network is designed. Based on the PPO algorithm, performer networks are constructed for discrete and continuous actions respectively. Specifically, the agent's observations are first fed into a fully connected layer. The output is then divided into two parts: the perturbation pattern and the perturbation parameter performer network. The outputs are merged to obtain the perturbation joint action. The critic network evaluates the joint action selected by the performer network. The optimization objectives for discrete and continuous actions are:

[0095]

[0096]

[0097] in and Represent the probability ratio of the new and old strategies for discrete actions and continuous actions respectively, and The policy with fixed old parameters for discrete and continuous action performer networks, and are the strategies to be updated for the discrete and continuous action performer networks respectively. clip represents the clipping function, which is used to and The range of is controlled within the range of 1-∈ to 1+∈, where ∈ is a clipping hyperparameter used to control the difference between the new and old strategies. is the target advantage function at time t, and the generalized advantage estimation technique is used to estimate the advantage function

[0098]

[0099] Among them, when 1=0, is the GAE parameter, State s t+1 , s t The objective value function under Represents the parameters of the target critic network, whose objective function and parameter update method are consistent with the traditional PPO algorithm.

[0100] Furthermore, in step 3, the discrete action performer network uses the Softmax activation function to output the discrete action strategy with the highest probability, and the continuous action performer network samples the mean and variance from the Beta distribution to output the continuous action strategy.

[0101] In step 4, the intelligent jammer and the environment will form a trajectory during the interaction process [s t , a t , s t+1 , Rt , a t+1 , s t+2 , R t+1 ,...], the goal of the agent is to maximize the cumulative reward obtained from multiple interaction trajectory sequences, which can be expressed as follows:

[0102]

[0103] Where T is the total time of interaction between the agent and the environment, that is, the number of training steps; according to Total Reward t Determine whether the training has converged.

[0104] Furthermore, in step 4, the trajectory sequence is stored in the experience cache, and a batch of data is randomly taken from the cache to update the network each time during training.

[0105] Furthermore, the termination condition described in step five is when the multi-function radar working sequence appears in the "identification" mode three times consecutively.

[0106] Example:

[0107] (1) Experimental scene setting

[0108] The scenario of this experiment is a confrontation between an intelligent jammer and a multi-function radar. Assume that the multi-function radar has N mode = 4 working modes, namely search, confirmation, tracking and identification, which are represented by 0, 1, 2 and 3 respectively. The multi-function radar is based on the radiation power P R = 200MW transmits reconnaissance signals to the target, and considers the multi-function radar as the environment of the constructed reinforcement learning algorithm, that is, the radar's working mode is the observation of the jammer, and the jammer has N jam = 2 types of interference, namely suppression interference and deception interference, with maximum radiation power The interference signal is transmitted to the radar, and the background noise power is P noise =10 -12 W. Set Table 1 as the radar state transition table under various interference conditions, Tables 2 and 3 as the state transition report tables, set the spatial range of the scenario, the radar position is fixed at (0, 0, 0), the initial position of the target is (40, 40, 20) km, and the target departs from the initial position at a speed of (-50, 0) m / s. Table 4 shows other scenario parameters.

[0109] The parameters of the deep reinforcement learning neural network in the embodiment are set. The D3QN network includes an input layer, two fully connected layers, and an output layer. The input layer is the state input of 4 neurons. The number of neurons in the two fully connected layers is 128 and 64 respectively. Each layer is connected to the ReLU activation function. The output layer includes 200 neurons corresponding to the discrete action space. The network diagram of D3QN is shown below. Figure 2 As shown in Figure 5, the HPPO network includes a performer network and a critic network. The performer network includes an input layer, two fully connected layers, and an output layer. The input layer is the state input of 4 neurons. The number of neurons in the fully connected layer is 64. Each layer is connected to the Tanh activation function. The output layer is divided into two branches. Branch 1 uses the Softmax layer to output discrete actions. Branch 2 uses the Beta distribution as the source of continuous action parameters, uses the Softplus layer and adds 1 to output continuous actions. Table 5 shows the detailed parameters of the neural network. The network diagram of HPPO is shown in Figure 5. Figure 3 shown.

[0110] Table 1 Radar state transition table under various interference conditions

[0111]

[0112]

[0113] Table 2 State transition report table under interference suppression

[0114]

[0115] Table 3. State transition report table under deception interference

[0116]

[0117] Table 4. Parameter settings for adversarial scenarios

[0118] variable symbols Value size Operating frequency f of radar and jammer 3000MHz Wavelength λ 0.1m Suppression interference coefficient δ 2 <![CDATA[Upper and lower limits of deception jamming coefficient ρ2 and ρ1]]> 1.3，0.7 <![CDATA[Radar cross section σ RCS > <![CDATA[1m 2 ]]> <![CDATA[Gain G of the radar transmitting antenna t > 40dB <![CDATA[Radar receiving antenna gain G r > 40dB <![CDATA[Interference emission antenna gain G I > 10dB

[0119] Table 5. D3QN and HPPO network parameter settings

[0120]

[0121] (2) Experimental process

[0122] The method flow chart of the present invention is as follows Figure 1 shown.

[0123] First, a one-on-one game environment between a multi-function radar and an intelligent jammer is constructed. The environment is initialized according to the parameters in the scenario settings. Similarly, a deep reinforcement learning neural network is constructed according to the set parameters and the parameters of the neural network are initialized. episodeThe interaction trajectories with the environment are stored in the experience buffer. These are then retrieved as batches and used to train the D3QN and HPPO networks separately until the networks converge or a termination condition is reached. Finally, the trained policies are tested in an adversarial game environment. The details and calculation examples are described below.

[0124] Figure 4 The training process of the D3QN network is shown. It can be seen that the average return of the network first decreases and then increases. The training return curve fluctuates greatly. The return value is close to convergence around 6000 rounds, and the average return value is around 2000. It is considered that the interference strategy training is completed after 6000 rounds.

[0125] Figure 5 The training process of the HPPO network is shown. It can be seen that compared with the D3QN network, the HPPO training curve converges faster and the curve is smoother. It converges after about 1500 training rounds, and the average round return is about 2500.

[0126] Figure 6 The curve of the interference-to-signal ratio change of the intelligent jammer in the confrontation process is shown. It can be seen that the HPPO algorithm produces two strategies. The first strategy shows that the intelligent jammer continuously chooses deception jamming, and the second strategy shows that it continuously chooses suppression jamming. When the interference-to-signal ratio at the target leaves the effective range of jamming, the jammer will reselect the jamming strategy; the D3QN algorithm shows that the intelligent jammer switches between deception jamming and suppression jamming. Figure 6 As can be seen from the figure, D3QN has more invalid strategies than the HPPO algorithm, which is consistent with the fluctuations in the training process.

[0127] Figure 7 The interference power change curve of the intelligent jammer in the confrontation process is shown. The interference power under each algorithm corresponds to the above-mentioned interference-to-signal ratio curve. It can be seen that the continuous selection of suppression interference has the highest interference reliability, but the high transmission power brings the risk of target exposure. Deception interference uses less interference power than suppression interference, but there will be some moments when the interference fails.

[0128] Figure 8 The sequence of multi-function radar working modes in the countermeasure process is shown. It can be seen that in strategy 2 obtained by the HPPO algorithm, the number of multi-function radar identifications is the least. Although higher power comes at the expense of the size of the reward function, this strategy reduces the possibility of interference failure.

[0129] Figure 9The training curves of D3QN and HPPO networks under varying interference coefficients are shown. Specifically, the suppression interference coefficient γ is set to 3 and 4, and the deception interference coefficients ρ1 and ρ2 are set to 0.75, 1.25 and 0.8, 1.2, respectively. Figure 9 It can be seen that as the suppression interference coefficient increases and the deception interference coefficient decreases, the convergence time required for network training increases and the average round return decreases. This is because the interference-to-signal ratio range within which the jamming effect is effective decreases, requiring longer time to explore and utilize an effective strategy. Although the training effect decreases, both networks still converge, demonstrating the robustness of the present invention to the optimization of jamming patterns and operating parameters of multi-function radars.

[0130] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A joint optimization method for jamming patterns and operating parameters of a multi-function radar, characterized in that: The following steps are involved: Step 1: Set the electronic countermeasure scenario of multi-function radar and intelligent jammer, and set the multi-function radar to have N mode Working mode, with radiation power P R Send reconnaissance signals to the target. The intelligent jammer can choose two jamming styles: suppression jamming and deception jamming. The interference signal is transmitted to the radar, and the background noise power is P noise ; Set the working mode state transition table of the multi-function radar and the radar state transition report table under different interference patterns. The multi-function radar switches states according to the interference effect. The jammer can select an interference pattern at each moment and select an operating parameter within the maximum radiation power range to interfere with the radar. Set the position of each radar (x r ,y r ,z r ) and the initial position of the target (x0, y0, z0), and the target's speed at a fixed height (v x ,v y ); Step 2: Model the Markov decision model and determine the five-tuple of the decision process established by the game confrontation process.<N,S,A,P,R,γ> , where N is the number of jammers and is set to 1. The Markov decision process is used to model the confrontation scenario. The state space and action space are constructed based on the characteristic set of radar working state, jamming pattern and working parameters. According to the optimization goal determined by the jammer, that is, the optimization goal of minimizing power resource consumption as much as possible while successfully jamming the multi-function radar, the reward function R, the state transition matrix P, and the discount factor γ are determined. In this step, the state space of the Markov decision process at step t is defined as [W t ,W t-1 ,JSR t ,a t-1 ], where JSR t Indicates the interference effect of the current t-th step, W t and W t-1 Respectively represent the radar working mode of this step and the previous step, a t-1 Indicates the interference action performed in the previous step; In this step, the jamming action of the Markov decision process at time t is defined as (T, p), where T∈{0,1} represents the jamming pattern indicator of deceptive jamming and suppression jamming, and p represents the jamming power selected by the jammer under the current jamming pattern. The present invention constructs the action space in the following two ways: 1) Discrete action space, discretize the interference power range into The power values ​​of deceptive interference and suppression interference are selected from m discrete values; 2) Hybrid action space: The interference action is constructed as a hybrid action space, where the interference pattern is a discrete action. Under each discrete interference action, there are continuously selectable working parameters, that is, arbitrary interference power; Step 3: Based on the constructed action space and state space, establish a deep reinforcement learning neural network and set the number of network training times N episode , the number of training steps in each round of training is N step , initialize the neural network parameters and the jammer state; Among them, different deep reinforcement learning neural networks are used according to the different action spaces constructed in step 2: 1) For discrete action spaces, the D3QN deep reinforcement learning neural network is used; 2) For the mixed action space, HPPO deep reinforcement learning neural network is used; Step 4: Train the D3QN or HPPO network selected in step 3 until the network converges or reaches the termination condition. The training is completed. During network training, the neural network outputs the next action a0 based on the initial state s0, obtains the next state s1 through interaction in the electromagnetic environment, and obtains the immediate reward r1 corresponding to the action. These data are stored in cache B (s0, a0, s1, r1), and the state s1 is set as the starting state of the neural network for the next step. The above process is repeated to continuously obtain the next action, the next state, and the immediate reward corresponding to the action. Step 5: Within the same number of training rounds as step 4, if the actual number of training steps in step 3 reaches the set N step , or the early termination condition is reached, then stop the training of this round and enter the next round of training. Otherwise, repeat step 4 to continue training, and repeat step 5 until the number of training rounds reaches N episode After completing the training; Step 6: Based on the jamming strategy obtained using deep reinforcement learning neural network, the intelligent jammer obtains a jamming solution for the multi-function radar during the confrontation process in the confrontation environment.

2. The method for joint optimization of jamming patterns and operating parameters for a multi-function radar according to claim 1, wherein: in, The D3QN network consists of an input layer, two fully connected layers, and an output layer. The input layer is the state input of 4 neurons, and the output layer neurons correspond to the discrete action space. The HPPO network consists of a performer network and a critic network. The performer network consists of an input layer, two fully connected layers, and an output layer. The input layer is the state input of four neurons. Each layer of the fully connected layer is connected to the Tanh activation function. The output layer is divided into two branches. Branch 1 uses the Softmax layer to output discrete actions, and branch 2 uses the Beta distribution as a continuous power value, using the Softplus layer and adding 1 to output continuous power. The outputs of branch 1 and branch 2 are combined to obtain the interference joint action, and the critic network evaluates the joint action selected by the performer network.

3. The method for joint optimization of jamming patterns and operating parameters for a multifunctional radar according to claim 1, wherein: The interference effect described in step 1 is expressed by the interference-to-signal ratio. The current interference effect JSR at time t is t The mathematical expression is: Among them, P J , P R , P N are the jammer power, radar power and the ambient N~(0,σ) Gaussian noise power respectively; and They represent the channel gains of the jammer and radar signal transmission channels, p j and p r is the transmitting power of the transmitting end; according to the radar formula, it can be expressed as: Among them, G t and G r Represents the gain of the transmitting and receiving antennas of the multi-function radar, G I represents the gain of the interfering transmitting antenna, σ RCS represents the radar interception area, λ is the signal wavelength, and d is the radial distance from the multi-function radar to the interference.

4. The method for joint optimization of jamming patterns and operating parameters for a multi-function radar according to claim 3, wherein: In step 1, when the jammer chooses to suppress the jamming, JSR t ≥γ, it is considered that the suppression interference is effective. When the jammer chooses deception interference, ρ2≥JSR t ≥ρ1, the deception interference is considered effective, and in other cases the interference is considered unsuccessful, where γ represents the suppression interference coefficient, ρ2 and ρ1 represent the effective upper and lower bounds of the deception interference coefficient, respectively.

5. The method for joint optimization of jamming patterns and operating parameters for a multifunctional radar according to claim 1, wherein: In step 1 and step 2, the reward function R of the Markov decision model at time t can be expressed by the following mathematical formula: in Indicates the power return when the interference is successful, k2(W t ,W t+1 ) and k3(W t ,W t+1 ) represent the adjacent working modes W of the multi-function radar under different interference patterns. t and W t+1 The mode return after switching, k4 represents the return of whether the interference is successful or not, and the Sign function is the sign function.

6. The method for joint optimization of jamming patterns and operating parameters for a multifunctional radar according to claim 1, wherein: For discrete action spaces, we design the D3QN deep reinforcement learning neural network. The D3QN network combines the DoubleDQN and Dueling DQN networks and calculates the objective function using the following formula: where R t Indicates the reward value of the action, s t+1 Indicates execution of action a t+1 The states after θ and θ - They represent the network parameters of the network and the parameters of the target network respectively. The agent adopts the ε-greedy strategy when selecting the strategy; The mathematical expression of the value function is as follows: Among them, V represents the state value function, Q represents the action state value function, A represents the advantage function of the action state, and α and β represent the network parameters of the two fully connected layer branches of the network.

7. The method for joint optimization of jamming patterns and operating parameters for a multifunctional radar according to claim 1, wherein: For the mixed action space, a HPPO deep reinforcement learning neural network is designed. Based on the PPO algorithm, performer networks are constructed for discrete and continuous actions respectively. Specifically, the agent's observations are first fed into a fully connected layer. The output is then divided into two parts: the perturbation pattern and the perturbation parameter performer network. The outputs are merged to obtain the perturbation joint action. The critic network evaluates the joint action selected by the performer network. The optimization objectives for discrete and continuous actions are: in and Represent the probability ratio of the new and old strategies for discrete actions and continuous actions respectively, and The policy with fixed old parameters for discrete and continuous action performer networks, and are the strategies to be updated for the discrete and continuous action performer networks respectively; clip represents the clipping function, which is used to and The range of is controlled within the range of 1-∈ to 1+∈, where ∈ is a clipping hyperparameter used to control the difference between the new and old strategies; is the target advantage function at time t, and the generalized advantage estimation technique is used to estimate the advantage function Among them, when l = 0, is the GAE parameter, State s t+1 , s t The objective value function under represents the parameters of the target critic network.

8. The method for joint optimization of jamming patterns and operating parameters for a multifunctional radar according to claim 1, wherein: The discrete action performer network uses the Softmax activation function to output the discrete action strategy with the highest probability, and the continuous action performer network samples the mean and variance from the Beta distribution to output the continuous action strategy.

9. The method for joint optimization of jamming patterns and operating parameters for a multifunctional radar according to claim 1, wherein: In step 4, the intelligent jammer and the environment form a trajectory during the interaction process [s t ,a t ,s t+1 ,r t ,a t+1 ,s t+2 ,r t+1 ,…], the goal of the agent is to maximize the cumulative reward obtained from multiple interaction trajectory sequences, which can be expressed as follows: Among them, T is the total time of the interaction between the agent and the environment, that is, the number of training steps.

10. The method for joint optimization of jamming patterns and operating parameters for a multifunctional radar according to claim 1, wherein: In step 4, the trajectory sequence is stored in the experience cache, and a batch of data is randomly selected from the cache to update the network during training.

Images