A network coding method for multifunctional radar jamming decision making

Abstract

This invention provides a network coding method for multifunctional radar jamming decision-making. Based on the concept of mapping single-valued linear data codes into a high-dimensional nonlinear space to enhance the data's feature representation and improve network fitting capabilities, this method unifies the input dimensionality by designing a variational autoencoder for state encoding, thereby improving learning and convergence speeds compared to existing methods. Furthermore, multifunctional radar jamming decision-making often lacks prior information in real-world applications. The unknown distribution function, transition probability, and state space make it difficult to design the reward function and discount factor for reinforcement learning, making it difficult to learn certain actions that are detrimental in the short term but beneficial in the long term. To address this issue, the invention designs a forward experience feedback mechanism that optimizes the reward function for accumulated experience by modifying the experience time series. Compared to existing methods, the proposed method improves network robustness without reducing learning speed.

Description

Technical Field

[0001] The present invention relates to radar communication technology and deep learning technology, and in particular to multifunctional radar interference decision network coding technology. Background Art

[0002] The diversity of modern radars has made the electronic countermeasures (ECM) environment more complex and dynamic. The widespread use of multi-system, multi-function radars poses a greater challenge to ECM jammers, necessitating a jamming decision-making method with strong decision-making capabilities and rapid learning speed. As the number of radar states increases, the decision-making capabilities of expert-based decision-making, which has long learning cycles, strong reliance on experience, and slow decision-making speed, gradually decline. While machine learning-based intelligent decision-making methods such as DDQN can improve network decision-making capabilities by increasing network complexity, optimizing network structure, or improving parameter optimization, these methods also increase memory overhead, reducing learning efficiency and convergence speed. Because the interference assessment component of a multi-function radar represents radar states as state levels, the states corresponding to each state level are typically mapped to a linear ordinal code, resulting in reduced learning capabilities and efficiency of the neural network. Therefore, state encoding techniques can be used to encode single-valued states into high-dimensional nonlinear vectors. However, as the number of radar states increases, traditional encoding methods such as one-hot encoding suffer from code length expansion, resulting in non-uniform network input dimensions and slow learning and convergence speeds. In addition, the long-term reward discount for state transitions and the negative rewards for rising state levels in the radar jamming scenario reduce the network's ability to learn actions that are harmful in the short term but beneficial in the long term, and its ability to find the best decision is lower.

[0003] The overall interactive process of multi-function radar jamming decision-making is as follows: Figure 1 As shown. The radar that needs to be jammed is called the target radar. Time t is quantified as the time period for capturing, analyzing the target radar signal, formulating and executing the jamming strategy for the tth time. After our jammer captures the radiation signal of the target radar, it extracts parameters such as carrier frequency, pulse width, bandwidth, repetition rate and power to form a pulse description word PDW (Pulse Description Word), and uses PDW to analyze the opponent's working status s t , such as search state, tracking state, etc. Then our jammer selects jamming action a according to the jamming strategy t Jam the target radar, and then according to the state of the target radar at the next time s t+1 Evaluate the interference effect to obtain the reward R for this interference t , accumulate experience(s t ,a t ,R t ,s t+1 ) is used for network learning to establish a state-to-action strategy mapping.

[0004] As the radar state space and jammer action space increase, the Q-value table maintenance cost and learning time of Q learning increase accordingly. However, DQN uses the same network for Q-value estimation and maximum estimate update, which may lead to over-estimation problems during the learning process. To solve this problem, Double DQN introduces two estimation networks Q with the same structure but different parameters. eval and the target network Q target The two processes of action selection and evaluation are separated. Action selection relies on the estimation network as shown in formula (1):

[0005]

[0006] Among them, a ′ is the expected action for the next step, s t+1 is the next time state, w is the estimated network Q eval The network parameters are as follows: A is the action space, including deception interference, FM interference, etc. The evaluation of the action depends on the target network as (2):

[0007] y D2 =R(s t ,a t )+γ Q target (s t+1 ,a ′ ;w ′ ) (2)

[0008] Among them, y D2 is the target Q value of DDQN, γ is the reward discount factor, and w ′ The target network Q target The network parameters. As (3) uses the mean square error function as the loss function L for the neural network update MSE :

[0009] L MSE (w)=E[Q eval (s t ,a t ;w)-y D2 ] 2 (3)

[0010] The overall process of radar jamming decision making is as follows: Figure 2 As shown, the jammer intercepts the target radar signal and analyzes its working status s t , and use its state number as the input of the decision network, where the state space S size |S| = N. The jammer decides the jamming action a according to the jamming strategy t And execute, then intercept the radar signal analysis working status s at the next time t+1 , according to the state transition, the reward value R of this interference is obtained t , accumulate experience(st ,a t ,R t ,s t+1 ) is used for network learning. Since intelligent control networks in practical application scenarios typically adopt distributed models, radar individual identification, jamming decision-making, and effect evaluation are usually divided into different subnetworks. To evaluate the jamming effect, radar states are encoded in descending order of state level: {N-1, N-2, …, 1, 0}. The higher the state level, the greater the threat of detection and tracking by our jammer. However, such strongly linear state inputs weaken the learning and fitting capabilities of the neural network. The output of network input N2 may simultaneously show a strong correlation with the output of input N1.

[0011] In addition, it is difficult to design an appropriate reward function when there are no prior conditions, and long-term reward depreciation may make the expected reward too small. As a result, the network cannot learn certain actions that are harmful in the short term but beneficial in the long term, and thus cannot obtain the optimal decision path. Existing methods for interfering with decision-making are as follows:

[0012] Existing method 1

[0013] DDQN uses the Q-value neural network Q eval (s t ,a t ; w) estimates the action value, but has a weak learning ability for single-valued states. Method 1 uses the Dueling Double DQN (D3QN), as shown in formula (4), through two neural networks V(s t ,a t ;w V ) and A(s t ,a t ;w A ) introduces state value function and action advantage function to optimize the neural network structure, making the estimation of decision value more precise and the utilization of state information and action value higher, but it will also reduce the network learning speed to a certain extent.

[0014]

[0015] Among them, V(s t ,a t ;w V ) is the network parameter w V The state value neural network, A(s t ,a t ;w A ) is the network parameter w A The action advantage neural network is represented by |A|, which is the size of the action space.

[0016] Existing method 2

[0017] like Figure 3 ,exist Figure 2 Based on the basic process of radar interference decision-making shown in the figure, state encoding is performed after state recognition, mapping the low-dimensional linear state number to a high-dimensional nonlinear space. Since the neural network requires a unified dimension, equal-length coding is used for state encoding. Method 2 uses traditional machine learning coding, such as One-Hot coding, and uses formula (5) to convert single-valued linear data into an N-bit code sequence E One-Hot :

[0018]

[0019] Among them, i∈[0,N-1], E One-Hot [i] represents the i-th bit of the One-Hot encoding, and k∈[0,N-1] is the input sequence number.

[0020] Gray code can be compressed to the minimum code length L of binary equal length code Gray , such as (6):

[0021]

[0022] in, Indicates rounding up.

[0023] However, as the state space increases, the code length of equal-length coding continues to increase, which is not conducive to input dimension unification, network migration and incremental learning, and the coding space will also be wasted.

[0024] Existing method 3

[0025] Each decision-making step evaluates the state transition results of the next two or more steps, thereby obtaining the confidence level of the advantage of the transition from the current state to the next state. The long-term state transition results are used to feedback the value of the current action. To a certain extent, actions that are harmful in the short term but beneficial in the long term can be learned. Summary of the Invention

[0026] The technical problem to be solved by the present invention is to provide an interference decision method with a unified input dimension of the neural network, combining the distributed interference assessment and decision-making scenarios of the multi-function radar, to address the problem of poor performance of the decision network when the number of radar states and actions is large.

[0027] The technical solution adopted by the present invention to solve the above technical problems is a multifunctional radar interference decision network coding method, comprising the steps of:

[0028] Step 1: The jammer captures the target radar radiation signal, analyzes the pulse descriptor word PDW of the target radar and maps it to the target radar status level, which is used as the radar working status. The higher the target radar status level, the greater the threat of the jammer being discovered and tracked.

[0029] Step 2: The radar operating state is input into the variational autoencoder (VAE) encoding network. The VAE encoding network samples and outputs the VAE radar state encoding extracted latent variables with uniform dimension and code length compression, which have representation capabilities. This is input into the Q-value neural network as state encoding data. The Q-value neural network is used to receive the input state encoding data and output the current optimal interference action. The Q-value neural network includes an estimation network and a target network. The estimation network is used for interference action selection, and the target network is used for interference action evaluation.

[0030] Step 3: The estimation network in the Q-value neural network obtains the next expected action based on the input state encoding data and outputs it to the target network. The target network obtains the action value evaluation of the next expected action, calculates the loss based on the action value evaluation, updates the estimation network, and outputs the current optimal interference action. During the training process of the Q-value neural network, after a set number of steps, the estimation network parameters are copied to the target network.

[0031] Step 4: The jammer executes the jamming action output by the Q-value neural network and determines whether the target radar state level is reduced to the lowest. If not, it obtains a reward based on the state transition result and stores an experience formed by the state transition result, the jamming action, and the reward into the experience replay pool. The experience replay pool accumulates experience for Q-value neural network learning, improves the Q-value neural network to establish a strategy mapping from the radar working state to the jamming action, and the radar working state changes to the next state, and then goes to step 2; if so, then goes to step 1.

[0032] Based on the idea of ​​mapping single-valued linear data encoding to a high-dimensional nonlinear space to enhance the data's feature representation capabilities and improve network fitting capabilities, this paper proposes a solution that uses encoding technology to map linear state encoding to a discrete high-dimensional space. However, traditional mainstream machine learning encoding technologies such as One-Hot have code length expansion problems as the state space increases, and variable-length coding makes the neural network input dimension non-uniform. Therefore, this paper designs a variational autoencoder to perform state encoding to unify the input dimension, thereby improving learning and convergence speed compared to existing methods.

[0033] In addition, multi-function radar jamming decision-making usually lacks prior information in real-world applications. The unknown distribution function, transition probability and state space make it difficult to design the reward function and discount factor of reinforcement learning, and it is difficult to learn certain actions that are harmful in the short term but beneficial in the long term. To address this problem, the present invention designs a forward experience feedback mechanism that optimizes the reward return function of experience accumulation by modifying the experience sequence. Compared with existing methods, the proposed method improves the robustness of the network without reducing the learning speed, and enhances the adaptability to the reward function and reward discount. Preferably, in step 4, the Q-value neural network accumulates one experience each time it interacts using the transfer s of the current target radar state level. t-1 ,s t , the next target radar status level transition s t ,s t+1 To correct the reward R of the previous step in the sequence t-1 , that is, the experience is (s t-1 ,a t-1 ,R t-1 ,s t ,s t+1 ), where t is the number of interactions in the current round.

[0034] The present invention utilizes a variational autoencoder (VAE) state encoding method to address the issue of poor decision-making network performance when a large number of radar states are present, unifying the input dimensions of the neural network to achieve interference decision-making. Furthermore, to address the decline in decision-making performance caused by negative rewards and reward depreciation, the present invention introduces a forward experience feedback mechanism into the interference decision-making method to improve network robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of the multi-function radar jamming decision interaction;

[0036] Figure 2 Provide the basic process for radar jamming decision making;

[0037] Figure 3 This is a schematic diagram of a coded radar jammer decision;

[0038] Figure 4 It is a VAE structure;

[0039] Figure 5 The following is a flowchart of the decision-making process of radar jamming with coding;

[0040] Figure 6 This is the standard experience playback sequence;

[0041] Figure 7 It is the timing of forward experience feedback;

[0042] Figure 8 The figure shows the optimal transfer route.

[0043] Figure 9 is the decision accuracy-learning time curve. DETAILED DESCRIPTION

[0044] Variational autoencoder VAE is used to implement a confrontation between reconstruction loss and KL divergence. Its structure is as follows Figure 4 , the input data is passed through the encoding network to obtain the normal distribution mean μ and variance σ, and then the latent variable z is obtained through sampling. The decoding network attempts to reconstruct the original data and uses the mean square error function to establish the reconstruction loss function. The KL divergence is defined as (7):

[0045]

[0046] Where q(z|x) and p(x|z) represent the network-estimated encoding and decoding probability distributions, respectively. VAEs can transform discrete-value encodings into probability distribution mappings, ensuring uniform output dimensionality and compressing code length while maintaining representational power.

[0047] like Figure 5 The multi-function radar jamming decision process for network coding is shown in Figure 1, where the radar state coding is the radar state coding using VAE. The steps are as follows:

[0048] Step 1: The jammer captures the target radar radiation signal, analyzes the pulse description word PDW of the target radar and maps it to the radar status level s t As radar working status;

[0049] Step 2: Input the radar working state into the VAE encoding network, sample and obtain the latent variables and input them into the Q-value neural network as state encoding data;

[0050] Step 3: The Q-value neural network selects the current best interference action a t And output: Q value neural network internal estimation network Q eval and the target network Q target Output the next expected action a respectively ′ and action value evaluation y D2 , where the action selection of the estimation network is as shown in formula (1), and the action value evaluation of the target network is as shown in formula (2); using the loss function L MSE Update the estimated network, as shown in Equation (3). After a certain number of steps, copy the estimated network parameters to the target network;

[0051] Step 4: Execute the interference action output by the Q-value neural network and go to step 1: If the target radar status level has not been reduced to the lowest, then according to the status s t to s t+1The transfer result obtains the reward R of this action t , will (s t ,a t ,R t ,s t+1 ) is stored in the experience replay pool as an experience; otherwise, end this round of interaction, enter the next round of interaction, and go to step 1.

[0052] The timing of the standard experience replay method is as follows Figure 6 , starting from time T0, each interaction accumulates an experience (s t ,a t ,R t ,s t+1 ), until s t+1 = 0, that is, the radar state level drops to the lowest. Where t = 0, 1, 2, ..., P-1 is the number of interactions in the current round, and P is the total number of interactions in the current round. When the state level of the radar working state decreases, a positive reward is given, indicating that the action interference is effective. Otherwise, a negative reward is given. The feedback of reward R is immediate and only related to the current state, decision action, and next state, as shown in formula (8):

[0053]

[0054] To improve the network's adaptability to reward functions and reward depreciation, the reward function's feedback method needs to be modified. While the backward call feedback method of existing method 3 is relatively easy to implement, this method requires considering the subsequent multi-step transfer results when calculating the reward at each step. Therefore, a single interaction requires multiple calls to the network for transmission and prediction, which reduces learning efficiency and convergence speed. Therefore, the present invention preferably proposes a forward experience feedback mechanism based on time series.

[0055] Different from standard experience feedback (s t ,a t ,R t ,s t+1 ) mode, the forward experience feedback uses the transfer of the current state and the next state to correct the experience reward of the previous step in the sequence, that is, (s t-1 ,a t-1 ,R t-1 ,s t ,s t+1 ) mode. This mode does not require multiple predictions of subsequent state transitions in one interaction, saving time for model training and calling. Considering that short-term harmful and long-term beneficial actions usually prohibit repeated switching between adjacent states, when s is satisfied t-1 -s t+1 >3, the value of executing this action to quickly reduce the status level must be higher than gradually reducing it. t-1After being corrected to a positive reward, it is stored in the experience replay area, as shown in formula (9):

[0056]

[0057] In addition, the forward experience feedback aims to use multi-step state transitions to correct the action rewards of the previous time step, but the samples used for network training only need to accumulate the corrected state transitions and rewards as experience to evaluate the value of taking a certain action in the current state. Therefore, in order to reduce the memory usage of experience, the sample input of experience playback is (s t-1 ,a t-1 ,R t-1 ,s t ). The forward experience feedback sequence is as follows Figure 7 As shown, no experience is accumulated at time T0. Starting from T1, each interaction obtains the reward R of the previous interaction based on the current state and the state before and after time. t-1 , accumulate an experience t-1 ,a t-1 ,R t-1 ,s t ), until s t+1 = 0, that is, the radar status level drops to the lowest. Among them, t = 0, 1, 2, ..., P-1 is the number of interactions in the current round, and P is the total number of interactions in the current round. The last time step T P-1 Accumulate two experiences (s P-2 ,a P-2 ,R P-2 ,s P-1 ) and (s P-1 ,a P-1 ,R P-1 ,s P =0).

[0058] Experimental comparison:

[0059] Design a system with 17 states S={s0,s1,…,s 16} state transition spectrum, the state identification radar corresponds to different working modes of different state levels, such as Figure 8 , where the node represents the identified radar working state, each directed edge represents a possible state transition path, and the direction is the state transition direction. For example, state 2 in the figure can be transferred to state 0, and state 12 and state 5 can be transferred in both directions. Define s 16 is the highest status level, and s0 is the lowest status level.

[0060] The optimal decision route is the route with the fewest steps from the initial state to the lowest state level. The optimal route includes a step of state level improvement process s8→s 11 , used to mislead network learning. The optimal decision path is as follows Figure 8 The red route, where the decision network finally converges to 6 steps to complete the interference task, is the optimal decision route.

[0061] Figure 9 (a) shows the curve of the decision route accuracy changing over time under different encoding methods, where the accuracy refers to the proportion of the optimal route selected in the total number of decisions. The horizontal axis is the learning time and the vertical axis is the accuracy. Figure 9 (a) shows that the VAE encoding method has a faster learning speed and can learn and converge to the optimal decision in a shorter time. Due to the randomness of the early exploration and learning processes, the final accuracy is affected to a certain extent by the early randomness, resulting in high or low. In addition, using quaternary Gray-like code encoding can compress the state code to the same dimension as VAE, as shown in Equation (10):

[0062]

[0063] However, due to the weakening of its nonlinear representation ability, the convergence speed decreases rapidly, and it is more difficult for the intelligent agent to learn the potential relationship and value of the data, such as Figure 9 As shown in (b), VAE compresses the encoding state to a lower dimension compared to One-Hot encoding and Gray code to speed up network learning, make the network lightweight, and also ensure relatively strong data representation capabilities.

[0064] Table 1. Comparison of reward-value redundancy

[0065]

[0066] Figure 9 (c) shows the decision accuracy curves for different interactive feedback optimization methods, demonstrating that the selected forward feedback experience method can be used without compromising network performance or increasing learning time. Table 1 shows the reward-value redundancy of the s8 state under different encodings and different experience accumulation mechanisms. VAE_M is the VAE encoding for multi-step predictive backward call feedback, and VAE_Ours is the VAE encoding for the forward experience feedback designed by the present invention. The results show that the forward experience feedback mechanism used in the present invention, when using the same VAE encoding, can achieve a value redundancy and redundancy rate of 0.114 and 22.28% for the network's short-term harmful and long-term beneficial action assessments, respectively. These are improvements of 61.93% and 8.57% compared to the 0.0704 and 13.71% for standard experience replay, respectively. This demonstrates that the forward experience feedback mechanism can make the network more resilient to inappropriate reward function settings and excessive reward discounting in the absence of prior conditions, while maintaining training speed, thereby improving network robustness.

[0067] Table 2. Comparison of results

[0068]

[0069] As shown in Table 2, the average network interaction time under VAE coding is 0.00701 seconds, compared with the average interaction time of Gray code and One-Hot coding of 0.00808 seconds and 0.00868 seconds, the interaction speed is improved by 15.3% and 23.8%, respectively; the total number of Q network parameters under VAE coding is only 1673, compared with the network parameters of Gray code and One-Hot coding of 1929 and 3465, the network parameters are reduced by 13.27% and 51.72%, respectively; in addition, VAE convergence takes 4.20786 seconds, compared with Gray code and One-Hot coding of 5.46548 seconds and 6.73827 seconds, the convergence speed is improved by 29.89% and 60.14%, respectively, which shows the performance efficiency and stability of the proposed method in radar jamming intelligent decision-making.

Claims

1. A multifunctional radar interference decision network coding method, characterized in that: Including steps: Step 1: The jammer captures the target radar radiation signal, analyzes the pulse description word PDW of the target radar and maps it to the target radar status level, and uses the target radar status level as the radar working state; A higher target radar status level indicates a greater threat to the jammer being detected and tracked; Step 2: The radar operating state is input into the variational autoencoder (VAE) encoding network. The VAE encoding network samples and outputs the VAE radar state encoding extracted latent variables with uniform dimension and code length compression, which have representation capabilities. This is input into the Q-value neural network as state encoding data. The Q-value neural network is used to receive the input state encoding data and output the current optimal interference action. The Q-value neural network includes an estimation network and a target network. The estimation network is used for interference action selection, and the target network is used for interference action evaluation. Step 3: The estimation network in the Q-value neural network obtains the next expected action based on the input state encoding data and outputs it to the target network. The target network obtains the action value evaluation of the next expected action, calculates the loss based on the action value evaluation, updates the estimation network, and outputs the current optimal interference action. During the training process of the Q-value neural network, after a set number of steps, the estimation network parameters are copied to the target network. Step 4: The jammer executes the jamming action output by the Q-value neural network and determines whether the target radar state level has been reduced to the lowest. If not, it obtains a reward based on the state transition result and stores an experience formed by the state transition result, the jamming action, and the reward into the experience replay pool. The experience replay pool accumulates experience for Q-value neural network learning, improves the Q-value neural network to establish a strategy mapping from the radar working state to the jamming action, and the radar working state changes to the next state, and then goes to step 2. If yes, go to step 1.

2. The method according to claim 1, wherein: The VAE encoding network uses variational autoencoder (VAE) to achieve the confrontation between reconstruction loss and KL divergence, maps the input radar working state to the mean and variance of the normal distribution, and then obtains the latent variable z through sampling. The VAE decoding network is used to reconstruct the original radar working state data using the latent variable z, and uses the mean square error function to establish the reconstruction loss function.

3. The method according to claim 2, wherein: KL divergence is expressed as: Among them, q(z|x) and p(x|z) represent the encoding probability distribution of the VAE encoding network and the decoding probability distribution of the VAE decoding network, respectively, and x represents the radar working state.

4. The method according to claim 1, wherein: In step 4, each interaction accumulates an experience (s t ,a t ,R t ,s t+1 ), until s t+1 Reduce to the minimum; where t is the number of interactions in the current round, s t is the current target radar status level, s t+1 is the next target radar status level, a t is the current best interference action, R t is the current reward obtained; when the status level of the radar working status is reduced, a positive reward is given, indicating that the interference action is effective; otherwise, a negative reward is given.

5. The method according to claim 4, wherein: The reward feedback is immediate and only related to the current state, decision action and next state.

6. The method according to claim 4, wherein: Reward R t The specific way to obtain it is:

7. The method according to claim 1, wherein: In step 4, the Q-value neural network accumulates an experience per interaction using the transfer s of the current target radar state level. t-1 ,s t , the next target radar status level transition s t ,s t+1 To correct the reward R of the previous step in the sequence t-1 , that is, the experience is (s t-1 ,a t-1 ,R t-1 ,s t ,s t+1 ), where t is the number of interactions in the current round.

8. The method according to claim 7, wherein: Correct the reward R of the previous step in the time series t-1 The specific method is:

Images