Multifunctional radar interference decision network coding method

Abstract

The invention provides a multifunctional radar interference decision network coding method, which is based on the thought of mapping single-valued linear data codes to a high-dimensional nonlinear space to enhance the feature representation capability of data and improve the network fitting capability, and comprises the following steps of: designing a variational auto-encoder to carry out state coding to unify the input dimension; compared with the prior art, the learning and convergence speed is improved. In addition, a multifunctional radar interference decision generally lacks prior information in practical application, an unknown distribution function, an unknown transition probability and an unknown state space enable a reward function and a discount factor of reinforcement learning to be difficult to design, and some actions which are harmful for a short time but beneficial for a long time are difficult to learn. In order to solve the problem, a forward experience feedback mechanism is designed, and an experience accumulation reward return function is optimized by modifying an experience time sequence. Compared with the existing method, the method provided by the invention improves the network robustness under the condition of not reducing the learning speed.

Description

Technical Field

[0001] The present invention relates to radar communication technology and deep learning technology, and in particular to multifunctional radar interference decision network coding technology. Background Art

[0002] The diversity of modern radars makes the electronic countermeasure environment more complex and changeable. The widespread application of multi-system multi-function radars poses a greater challenge to the interference party of electronic countermeasures, and requires an interference decision-making method with strong decision-making ability and high learning speed. With the increase in the number of radar states, the decision-making ability of expert experience decisions with long learning cycles, strong experience dependence and slow decision-making speed gradually decreases. Although intelligent decision-making methods based on machine learning such as DDQN can improve the network decision-making ability by increasing network complexity or optimizing network structure and improving parameter optimization, these methods will also increase memory overhead and reduce learning efficiency and convergence speed. Since the interference evaluation part of the multi-function radar represents the radar state as a state level, the state corresponding to each state level is usually mapped to a linear serial number code, which makes the learning ability and efficiency of the neural network low. Therefore, state coding technology can be used to encode single-valued states into high-dimensional nonlinear vectors. However, with the increase in the number of radar states, traditional coding methods such as One-Hot have the problem of code length expansion, which not only makes the network input dimension inconsistent, but also slows down the learning and convergence speed. In addition, the long-term reward discount for state transitions and the negative rewards for rising state levels in the radar jamming scenario reduce the network's ability to learn actions that are harmful in the short term but beneficial in the long term, and its ability to find the best decision is lower.

[0003] The overall interactive process of multi-function radar jamming decision-making is as follows: Figure 1 As shown. The radar that needs to be interfered is called the target radar. The time t is quantified as the time period for capturing, analyzing the target radar signal, formulating and executing the interference strategy for the tth time. After our jammer captures the radiation signal of the target radar, it extracts the parameters such as carrier frequency, pulse width, bandwidth, repetition rate and power to form a pulse description word PDW (Pulse Description Word), and uses PDW to analyze the opponent's working status s t , such as search state, tracking state, etc. Then our jammer selects jamming action a according to the jamming strategy t Jam the target radar, and then according to the state of the target radar at the next time s t+1 Evaluate the interference effect to obtain the reward R for this interference t , accumulate experience(s t ,a t ,R t ,s t+1 ) is used for network learning to establish a state-to-action strategy mapping.

[0004] As the radar state space and jammer action space increase, the Q-value table maintenance cost and learning time of Q learning increase accordingly. However, DQN uses the same network for Q value estimation and maximum estimate update, which may lead to over-estimation problems during the learning process. To solve this problem, Double DQN introduces two estimation networks Q with the same structure but different parameters. eval and the target network Q target The two processes of action selection and evaluation are separated. Action selection relies on the estimation network as shown in formula (1):

[0005]

[0006] Among them, a ′ is the expected action for the next step, s t+1 is the next time state, w is the estimated network Q eval The network parameters of A are the action space, including deception interference, FM interference, etc. The evaluation of the action depends on the target network as (2):

[0007] y D2 =R(s t ,a t )+γ Q target (s t+1 ,a ′ ;w ′ ) (2)

[0008] Among them, y D2 is the target Q value of DDQN, γ is the reward discount factor, and w ′ The target network Q target The network parameters are as follows. As in (3), the mean square error function is used as the loss function L for updating the neural network. MSE :

[0009] L MSE (w) = E[Q eval (s t ,a t ;w)-y D2 ] 2 (3)

[0010] The overall process of radar jamming decision making is as follows: Figure 2 As shown in the figure, the jammer intercepts the target radar signal and analyzes its working status. t , and use its state number as the input of the decision network, where the state space S has a size of |S|=N. The jammer decides the jamming action a according to the jamming strategy. t And execute, then intercept the radar signal analysis working status s at the next time t+1 , according to the state transition, obtain the reward value R of this interference t , accumulate experience(st ,a t ,R t ,s t+1 ) is used for network learning. Since intelligent control networks usually adopt distributed models in actual application scenarios, radar individual identification, interference decision-making and effect evaluation are usually divided into different sub-networks. In order to evaluate the interference effect, the radar state is encoded from high to low according to the state level: {N-1, N-2, ..., 1, 0}. The higher the state level, the greater the threat of our jammer being discovered and tracked. However, such a state input with a strong linear relationship makes the learning and fitting ability of the neural network weaker, and the output of the network input N2 may show a strong correlation with the output of the input N1 synchronously.

[0011] In addition, it is difficult to design an appropriate reward function when prior conditions are missing, and long-term reward discount may make the expected reward too small, so the network cannot learn certain actions that are harmful in the short term but beneficial in the long term, and thus cannot obtain the best decision route. Existing methods of interfering with decision-making are as follows:

[0012] Existing method 1

[0013] DDQN uses a Q-value neural network Q eval (s t ,a t ; w) estimates the action value, but has a weak learning ability for single-valued states. Method 1 uses the dueling double Q network D3QN (Dueling Double DQN), as shown in formula (4), through two neural networks V (s t ,a t ;w V ) and A(s t ,a t ;w A ) introduces state value function and action advantage function to optimize the neural network structure, which makes the estimation of decision value more precise and the utilization rate of state information and action value higher, but it will also reduce the network learning speed to a certain extent.

[0014]

[0015] Among them, V(s t ,a t ;w V ) is the network parameter w V The state value neural network, A(s t ,a t ;w A ) is the network parameter w A The action advantage neural network is , |A| is the size of the action space.

[0016] Existing method 2

[0017] like Figure 3 ,exist Figure 2 Based on the basic process of radar interference decision-making shown in the figure, state encoding is performed after state recognition to map the low-dimensional linear state number to a high-dimensional nonlinear space. Since the neural network requires a unified dimension, equal-length encoding is used for state encoding. Method 2 uses traditional machine learning encoding, such as One-Hot encoding, and uses formula (5) to convert single-valued linear data into an N-bit code sequence E One-Hot :

[0018]

[0019] Among them, i∈[0,N-1], E One-Hot [i] represents the i-th bit of One-Hot encoding, and k∈[0,N-1] is the input sequence number.

[0020] Gray code can be compressed to the minimum code length L of binary equal length code Gray , such as (6):

[0021]

[0022] in, Indicates rounding up.

[0023] However, as the state space increases, the code length of equal-length coding is still increasing, which is not conducive to the unification of input dimensions, network migration and incremental learning, and the coding space will also be wasted.

[0024] Existing method 3

[0025] Each decision-making step evaluates the state transition results of the next two or more steps, so as to obtain the confidence of the advantage of the transition from the current state to the next state, and use the long-term state transition results to feedback the value of the current action. To a certain extent, it can learn actions that are harmful in the short term but beneficial in the long term. Summary of the invention

[0026] The technical problem to be solved by the present invention is to provide an interference decision method with a unified input dimension of a neural network in combination with the distributed interference assessment and decision-making scenarios of a multi-function radar, aiming at the problem of poor performance of the decision network when the number of radar states and the number of actions are large.

[0027] The technical solution adopted by the present invention to solve the above technical problems is a multifunctional radar interference decision network coding method, comprising the steps of:

[0028] Step 1: The jammer captures the target radar radiation signal, analyzes the pulse description word PDW of the target radar and maps it to the target radar status level, and uses the target radar status level as the radar working state; the higher the target radar status level, the greater the threat of the jammer being discovered and tracked;

[0029] Step 2: Input the radar working state into the variational autoencoder VAE coding network. The VAE coding network samples and outputs VAE radar state coding extraction latent variables with uniform dimension and characterization capability after code length compression, and inputs them into the Q-value neural network as state coding data. The Q-value neural network is used to receive the input state coding data and output the current best interference action. The Q-value neural network includes an estimation network and a target network. The estimation network is used for interference action selection. The target network is used for interference action evaluation.

[0030] Step 3: The estimation network in the Q-value neural network obtains the next expected action based on the input state encoding data and outputs it to the target network. The target network obtains the action value evaluation of the next expected action, calculates the loss based on the action value evaluation, updates the estimation network, and outputs the current optimal interference action. During the training process of the Q-value neural network, the estimation network parameters are copied to the target network after the set number of steps.

[0031] Step 4: The jammer executes the jamming action output by the Q-value neural network and determines whether the target radar state level is reduced to the lowest. If not, it obtains rewards based on the state transfer results, and stores an experience formed by the state transfer results, jamming actions and rewards into the experience replay pool. The experience replay pool accumulates experience for Q-value neural network learning, improves the Q-value neural network to establish a strategy mapping from the radar working state to the jamming action, and the radar working state is transferred to the next state, and then go to step 2; if yes, then go to step 1.

[0032] The present invention is based on the idea of ​​mapping single-valued linear data encoding to high-dimensional nonlinear space to enhance the feature representation ability of data and improve the network fitting ability, and proposes a solution to map linear state encoding to discrete high-dimensional space using encoding technology. However, the mainstream encoding technology of traditional machine learning such as One-Hot has the problem of code length expansion as the state space increases, and variable-length coding makes the input dimension of the neural network non-uniform. Therefore, the present invention unifies the input dimension by designing a variational autoencoder for state encoding, which improves the learning and convergence speed compared with the existing method.

[0033] In addition, multi-function radar jamming decisions usually lack prior information in real applications. The unknown distribution function, transition probability and state space make it difficult to design the reward function and discount factor of reinforcement learning, and it is difficult to learn certain actions that are harmful in the short term but beneficial in the long term. To address this problem, the present invention designs a forward experience feedback mechanism that optimizes the reward return function of experience accumulation by modifying the experience sequence. Compared with existing methods, the proposed method improves the robustness of the network without reducing the learning speed, and enhances the adaptability to the reward function and reward discount. Preferably, in step 4, the Q-value neural network accumulates one experience each time it interacts, using the transfer s of the current target radar state level. t-1 ,s t , the next target radar status level transition s t ,s t+1 To correct the reward R of the previous step in the sequence t-1 , that is, the experience is (s t-1 ,a t-1 ,R t-1 ,s t ,s t+1 ), where t is the interaction times variable of the current round.

[0034] The beneficial effect of the present invention is that the method of using variational autoencoder VAE (Variational Auto Encoder) for state encoding solves the problem of poor performance of the decision network when the number of radar states is large, and unifies the interference decision method of the input dimension of the neural network. In addition, in view of the decline in decision-making ability caused by negative rewards and reward depreciation, the present invention also introduces a forward experience feedback mechanism in the interference decision method to improve the robustness of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of the multi-function radar jamming decision interaction;

[0036] Figure 2 Provides the basic process for radar jammer decision making;

[0037] Figure 3 This is a schematic diagram of a coded radar jammer decision;

[0038] Figure 4 is the VAE structure;

[0039] Figure 5 This is a flowchart of the decision-making process of radar interference with coding;

[0040] Figure 6 This is the standard experience playback sequence;

[0041] Figure 7 It is the timing of forward experience feedback;

[0042] Figure 8 The optimal transfer route is shown in FIG.

[0043] Fig. 9 is the decision accuracy-learning time curve. DETAILED DESCRIPTION

[0044] The variational autoencoder VAE is used to implement a confrontation between reconstruction loss and KL divergence. Its structure is as follows Figure 4 , the input data is passed through the encoding network to obtain the normal distribution mean μ and variance σ, and then the latent variable z is obtained through sampling. The decoding network attempts to reconstruct the original data and uses the mean square error function to establish the reconstruction loss function. The KL divergence is defined as (7):

[0045]

[0046] Among them, q(z|x) and p(x|z) represent the encoding probability distribution and decoding probability distribution estimated by the network, respectively. VAE can transform discrete value encoding into probability distribution mapping, which can ensure the uniformity of output dimension and compress code length while ensuring representation capability.

[0047] like Figure 5 The multifunctional radar jamming decision process for network coding is shown in Figure 1, where the radar state coding is the radar state coding using VAE, and the steps are as follows:

[0048] Step 1: The jammer captures the target radar radiation signal, analyzes the pulse description word PDW of the target radar and maps it to the radar status level s t As radar working status;

[0049] Step 2: Input the radar working state into the VAE encoding network, sample and obtain the latent variables and input them into the Q value neural network as state encoding data;

[0050] Step 3: The Q-value neural network selects the current best interference action a t And output: Q value neural network internal estimation network Q eval and the target network Q target Output the next expected action a respectively ′ and action value evaluation y D2 , where the action selection of the estimation network is as shown in formula (1), and the action value evaluation of the target network is as shown in formula (2); using the loss function L MSE Update the estimated network, as shown in formula (3). After a certain number of steps, copy the estimated network parameters to the target network;

[0051] Step 4: Execute the interference action output by the Q-value neural network and go to step 1: If the target radar state level has not been reduced to the minimum, then according to the state s t to t+1The transfer result obtains the reward R of this action t , will (s t ,a t ,R t ,s t+1 ) is stored in the experience replay pool as an experience; otherwise, end this round of interaction, enter the next round of interaction, and go to step 1.

[0052] The timing of the standard experience replay method is as follows Figure 6 , starting from time T0, each interaction accumulates one experience (s t ,a t ,R t ,s t+1 ), until s t+1 = 0, that is, the radar state level drops to the lowest. Among them, t = 0, 1, 2, ..., P-1 is the number of interactions in the current round, and P is the total number of interactions in the current round. When the state level of the radar working state is reduced, a positive reward is given, indicating that the action interference is effective, otherwise a negative reward is given. The feedback of reward R is immediate and only related to the current state, decision action and next state, as shown in formula (8):

[0053]

[0054] In order to improve the adaptability of the network to the reward function and reward loss, it is necessary to modify the feedback method of the reward function. Although the backward call feedback method of the existing method 3 is easier to implement, this method needs to consider the subsequent multi-step transfer results when calculating the reward at each step. Therefore, one interaction requires multiple calls to the network for transmission and prediction, which reduces the learning efficiency and convergence speed. Therefore, preferably, the present invention proposes a forward experience feedback mechanism based on time series.

[0055] Different from standard experience feedback (s t ,a t ,R t ,s t+1 ) mode, the forward experience feedback uses the transfer of the current state and the next state to correct the experience reward of the previous step in the sequence, that is, (s t-1 ,a t-1 ,R t-1 ,s t ,s t+1 ) mode. This mode does not require multiple predictions of subsequent state transitions in one interaction, saving time for model training and calling. Considering that short-term harmful and long-term beneficial actions usually prohibit repeated switching between adjacent states, when s is satisfied t-1 -s t+1 >3, the value of performing this action to quickly reduce the status level must be higher than gradually reducing it. t-1After being corrected to a positive reward, it is stored in the experience replay area, as shown in formula (9):

[0056]

[0057] In addition, the forward experience feedback aims to use multi-step state transitions to correct the action rewards of the previous time step, but the samples used for network training only need to accumulate the corrected state transitions and rewards as experience to evaluate the value of using a certain action in the current state. Therefore, in order to reduce the memory usage of experience, the sample input of experience playback is (s t-1 ,a t-1 ,R t-1 ,s t ). The forward experience feedback sequence is as follows Figure 7 As shown, no experience is accumulated at time T0. From T1 onwards, each interaction obtains the reward R of the previous interaction based on the current state and the state before and after. t-1 , accumulate experience t-1 ,a t-1 ,R t-1 ,s t ), until s t+1 = 0, that is, the radar status level drops to the lowest. Where t = 0, 1, 2, ..., P-1 is the number of interactions in the current round, and P is the total number of interactions in the current round. The last time step T P-1 Accumulate two experiences (s P-2 ,a P-2 ,R P-2 ,s P-1 ) and (s P-1 ,a P-1 ,R P-1 ,s P =0).

[0058] Experimental comparison:

[0059] Design a system with 17 states S = {s0, s1, ..., s 16} state transition spectrum, the state identification radar corresponds to different working modes of different state levels, such as Figure 8 , where the node represents the identified radar working state, each directed edge represents a possible state transition path, and the direction is the state transition direction. For example, in the figure, state 2 can be transferred to state 0, and state 12 and state 5 can be converted in both directions. Define s 16 is the highest status level, and s0 is the lowest status level.

[0060] The optimal decision route is the route with the fewest steps from the initial state to the lowest state level. The optimal route includes a state level upgrade process s8→s 11 , used to mislead network learning. The optimal decision path is as follows Figure 8 The red route, the decision network finally converges to 6 steps to complete the interference task as the best decision route.

[0061] Fig. 9 (a) shows the curve of the decision route accuracy changing over time under different encoding methods, where the accuracy refers to the proportion of the best route selected to the total number of decisions, the horizontal axis is the learning time, and the vertical axis is the accuracy. Fig. 9 (a) shows that the VAE encoding method has a faster learning speed and can learn and converge to the best decision in a shorter time. Since the early exploration process and learning process are random, the final accuracy is affected to a certain extent by the randomness of the early stage, resulting in high or low. In addition, using quaternary Gray code encoding can compress the state encoding to the same dimension as VAE, as shown in formula (10):

[0062]

[0063] However, due to the weakening of its nonlinear representation ability, the convergence speed decreases rapidly, and it is more difficult for the agent to learn the potential relationship and value of the data, such as Fig. 9 As shown in (b), VAE compresses the encoding state to a lower dimension to speed up network learning, making the network lightweight while ensuring relatively strong data representation capabilities compared to One-Hot encoding and Gray code.

[0064] Table 1. Comparison of reward-value redundancy

[0065]

[0066] Fig. 9 (c) shows the decision accuracy curve under different interactive feedback optimization methods, indicating that the selected forward feedback experience method can not damage the network performance and increase the learning time. As shown in Table 1, the s8 state reward-value redundancy under different encodings and different experience accumulation mechanisms. Among them, VAE_M is the VAE encoding under multi-step predictive backward call feedback, and VAE_Ours is the VAE encoding under the forward experience feedback designed by the present invention. The results show that the forward experience feedback mechanism used in the present invention can make the network's value redundancy and redundancy rate for short-term harmful and long-term beneficial action evaluation 0.114 and 22.28% respectively under the same VAE encoding, which is 61.93% and 8.57% higher than the 0.0704 and 13.71% of the standard experience playback, respectively. It shows that the forward experience feedback mechanism can make the network have a greater tolerance for inappropriate reward function settings and excessive reward discounts in the absence of prior conditions without changing the training speed, thereby improving the robustness of the network.

[0067] Table 2. Comparison of results

[0068]

[0069] As shown in Table 2, the average network interaction time under VAE coding is 0.00701 seconds, compared with the average interaction time of Gray code and One-Hot coding of 0.00808 seconds and 0.00868 seconds, the interaction speed is increased by 15.3% and 23.8%, respectively; the total number of Q network parameters under VAE coding is only 1673, compared with the network parameters of Gray code and One-Hot coding of 1929 and 3465, the network parameters are reduced by 13.27% and 51.72%, respectively; in addition, VAE convergence takes 4.20786 seconds, compared with Gray code and One-Hot coding of 5.46548 seconds and 6.73827 seconds, the convergence speed is increased by 29.89% and 60.14%, respectively, which shows the performance efficiency and stability of the method proposed in the present invention in radar jamming intelligent decision-making.

Claims

1. A multifunctional radar interference decision network coding method, characterized in that: Includes steps: Step 1: The jammer captures the target radar radiation signal, analyzes the pulse description word PDW of the target radar and maps it to the target radar state level, and uses the target radar state level as the radar working state; A higher target radar status level indicates a greater threat of jammer detection and tracking; Step 2: Input the radar working state into the variational autoencoder VAE coding network. The VAE coding network samples and outputs VAE radar state coding extraction latent variables with uniform dimension and characterization capability after code length compression, and inputs them into the Q-value neural network as state coding data. The Q-value neural network is used to receive the input state coding data and output the current best interference action. The Q-value neural network includes an estimation network and a target network. The estimation network is used for interference action selection. The target network is used for interference action evaluation. Step 3: The estimation network in the Q-value neural network obtains the next expected action based on the input state encoding data and outputs it to the target network. The target network obtains the action value evaluation of the next expected action, calculates the loss based on the action value evaluation, updates the estimation network, and outputs the current optimal interference action. During the training process of the Q-value neural network, the estimation network parameters are copied to the target network after the set number of steps. Step 4: The jammer executes the jamming action output by the Q-value neural network and determines whether the target radar state level is reduced to the minimum. If not, it obtains a reward based on the state transfer result, and stores an experience formed by the state transfer result, the jamming action and the reward into the experience replay pool. The experience replay pool accumulates experience for Q-value neural network learning, improves the Q-value neural network to establish a strategy mapping from the radar working state to the jamming action, and the radar working state is transferred to the next state, and then goes to step 2; If yes, go to step 1.

2. The method according to claim 1, characterized in that: The VAE encoding network uses variational autoencoder VAE to achieve the confrontation between reconstruction loss and KL divergence, maps the input radar working state to the mean and variance of the normal distribution, and then obtains the latent variable z through sampling. The VAE decoding network is used to reconstruct the data of the original radar working state using the latent variable z, and uses the mean square error function to establish the reconstruction loss function.

3. The method according to claim 2, characterized in that: KL divergence is expressed as: Among them, q(z|x) and p(x|z) represent the encoding probability distribution of the VAE encoding network and the decoding probability distribution of the VAE decoding network respectively, and x represents the working state of the radar.

4. The method according to claim 1, characterized in that: In step 4, each interaction accumulates one experience (s t ,a t ,R t ,s t+1 ), until s t+1 Reduce to the minimum; where t is the number of interactions in the current round, s t is the current target radar status level, s t+1 is the next target radar status level, a t is the current best interference action, R t is the current reward; when the status level of the radar working status is reduced, a positive reward is given, indicating that the action interference is effective, otherwise a negative reward is given.

5. The method according to claim 4, characterized in that: The reward feedback is immediate and only related to the current state, decision action and next state.

6. The method according to claim 4, characterized in that: Reward R t The specific way to obtain is:

7. The method according to claim 1, characterized in that: In step 4, the Q-value neural network accumulates one experience per interaction using the transfer s of the current target radar state level t-1 ,s t , the next target radar status level transition s t ,s t+1 To correct the reward R of the previous step in the sequence t-1 , that is, the experience is (s t-1 ,a t-1 ,R t-1 ,s t ,s t+1 ), where t is the interaction times variable of the current round.

8. The method according to claim 7, characterized in that: Correct the reward R of the previous step in the sequence t-1 The specific method is:

Images