A spatial main lobe interference suppression method based on reinforcement learning

Abstract

The application provides a spatial main lobe interference suppression method based on reinforcement learning, which comprises the following steps: sampling target echo signals, noise and interference signals to obtain a radar array antenna receiving signal matrix; selecting a radar as an agent for reinforcement learning to determine a state space and an action space of the agent; designing a reward signal required by reinforcement learning based on the agent, the state space and the action space thereof set in step S2; training the agent by using a DQN algorithm based on the reinforcement learning model constructed based on the radar array antenna receiving signal matrix and the content described in steps S2 and S3 to obtain a prediction of an interference angle; and performing blocking matrix processing based on the predicted interference angle to cancel the main lobe interference signals and complete main lobe anti-interference. The application can quickly lock to the interference angle under a fixed interference angle, and when the interference angle changes, the application can quickly follow the angle change and converge to the interference angle, so that the signal-to-interference-and-noise ratio of the system output meets the requirements of subsequent processing.

Description

Technical Field

[0001] This invention belongs to the field of radar anti-jamming, specifically a method for suppressing spatial main lobe interference based on reinforcement learning. Background Technology

[0002] The complex electromagnetic environment and rapidly developing radar detection technology have made the confrontation between radar and targets more intense and complex. Radar anti-jamming measures are mainly divided into three directions: active anti-jamming, passive anti-jamming, and intelligent anti-jamming. Commonly used anti-jamming methods mainly suppress interference from aspects such as radar system, operating mode, operating frequency, transmitted waveform, polarization characteristics, spatial angle, signal processing algorithms, and multi-sensor cooperation. The blocking matrix preprocessing method is a passive anti-jamming method from the spatial angle. First, the main lobe interference angle is estimated, then a blocking matrix is ​​designed, and the received data is preprocessed with the blocking matrix to suppress interference. Then, adaptive beamforming is performed on the preprocessed received data to suppress sidelobe interference. The interference zero depth formed by the blocking matrix preprocessing method has a small affected angle range and is an effective main lobe interference suppression method. Traditional blocking matrix preprocessing uses the ES-DOA method to estimate the interference angle and uses interference noise data analysis during transmission to perform interference, avoiding the influence of the signal. This is difficult to achieve in practice, as the received signal is usually a mixture of interference and target signals. Summary of the Invention

[0003] This invention proposes a spatial main lobe interference suppression method based on reinforcement learning.

[0004] The technical solution for implementing this invention is: a spatial main lobe interference suppression method based on reinforcement learning, the method comprising the following steps:

[0005] S1. Sample the target echo signal, noise, and interference signal to obtain the radar array antenna received signal matrix;

[0006] S2. Select the radar as the agent for reinforcement learning and determine the agent's state space S and action space A;

[0007] S3. Based on the agent and its state space S and action space A set in step S2, design the reward signal required for reinforcement learning;

[0008] S4. Based on the radar array antenna received signal matrix obtained in step S1 and the reinforcement learning model constructed in steps S2 and S3, the DQN algorithm is used to train the agent to obtain the prediction of the interference angle.

[0009] S5. Based on the angle obtained in S4, perform blocking matrix processing to cancel the main lobe interference signal and complete the main lobe anti-interference.

[0010] Preferably, the state space S of the agent is the set of all possible states s of the agent, and the state s at time t is denoted as st. t :

[0011] s t =[θ,SINR,NUM]

[0012] In the formula, SINR is the signal-to-interference-plus-noise ratio, NUM is the number of mismatches, and θ is the current blocking angle.

[0013] Preferably, the formula for estimating the signal-to-interference-plus-noise ratio is:

[0014]

[0015] In the formula, Y is the amplitude of the measured element, M is the number of reference elements, and Z is the total amplitude of all reference elements.

[0016] Preferably, the action space is the set of actions taken by the radar agent, including four selectable actions, each representing an angle adjustment in different directions and with different precision, specifically:

[0017]

[0018] In the formula, φ is the maximum change angle of the radar agent's output angle each time.

[0019] Preferably, the radar agent receives a reward signal after interacting with the environment. The reward signal can be divided into three categories: "success" reward, "penalty" reward, and regular reward. The specific reward signal structure is as follows:

[0020]

[0021] Where a>0, b<0, SINR' is the expected signal-to-interference-plus-noise ratio when the system successfully resists interference, SINR is the current BMP output signal-to-interference-plus-noise ratio, a is the "success" reward signal, b is the "penalty" signal, and α is the coefficient.

[0022] Preferably, the specific process of training the agent using the DQN algorithm based on the radar array antenna received signal matrix obtained in step S1 and the reinforcement learning model constructed in steps S2 and S3 is as follows:

[0023] S41. Initialize the system: radar array antenna shape, total capacity of the experience replay pool, minimum training data batch size, learning rate, reward discount factor, target network update frequency, random initialization of training network parameters, and initial state of the agent.

[0024] S42. The radar agent interacts and explores with the environment. The radar, as the agent, uses the current signal-to-interference ratio (SIR) as the observation value in the reinforcement learning model. At each time step t, the radar is in state s.t According to the current network Q w (s,a) selects action a according to the ∈-greedy strategy. t This causes the state to transition to s. t+1 and receive a reward r t The interaction information between the agent and the environment constitutes a Markov decision process quadruple (s t ,a t ,r t ,s t+1 Store this quadruple in the experience replay pool;

[0025] S43. When the amount of data in the experience replay pool is greater than the size of the training samples required each time, randomly sample N quadruplets from the experience pool. For the i-th quadruplet, use the target network to calculate the temporal difference target y. i =r i + Where r i Q is the reward for the i-th quadruple. w -(s i+1 a) is the current target network's fitted output of the action-value function, where γ is the reward discount coefficient. Then, the target loss L = ∑ i (y i -Q w (s i ,a i )) 2 / N, Q w (s i ,a i To train the network to fit the action value function, the obtained parameters are used to update the training network Q. w (s,a);

[0026] S44. Update the training network parameters to the target network according to the initial update frequency.

[0027] S45. Repeat steps S42 to S44 until the training reward value converges stably.

[0028] Compared with existing technologies, the significant advantages of this invention are: the average number of system iterations required to search for spatial interference angles using reinforcement learning is greatly reduced, the output signal-to-interference-plus-noise ratio (SIR) is more stable, angle convergence is faster, and the system is more stable. Under a fixed interference angle, the reinforcement learning algorithm can quickly lock onto the interference angle; when the interference angle changes, the reinforcement learning algorithm can quickly follow the angle change and converge to the interference angle, ensuring that the system's output SIR meets the requirements for subsequent processing.

[0029] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0030] Figure 1 To enhance the learning participation process of radar anti-jamming.

[0031] Figure 2 This method estimates the signal-to-interference-plus-noise ratio (SIR / NNR) by statistically analyzing the signal strength of the target cell and neighboring cells.

[0032] Figure 3 The results of reinforcement learning under ULA are shown when the angle changes by 0.1°, 0.2°, 0.5°, and 1° respectively.

[0033] Figure 4 This represents the results of reinforcement learning under ULA with randomly varying angles.

[0034] Figure 5 This represents the training result of the algorithm proposed in this invention when the elevation angle of the incoming interference signal remains constant but the azimuth angle changes.

[0035] Figure 6 This represents the training result of the algorithm proposed in this invention when the azimuth angle of the incoming interference signal remains constant while the elevation angle changes. Detailed Implementation

[0036] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] A reinforcement learning-based method for suppressing spatial main lobe interference, the method comprising the following steps:

[0038] S1. Sample the target echo signal, noise, and interference signal to obtain the radar array antenna received signal matrix.

[0039] S2. Select the radar as the agent for reinforcement learning and determine the agent's state space S and action space A;

[0040] S3. Based on the agent and its state space S and action space A set in step S2, design the reward signal required for reinforcement learning;

[0041] S4. Based on the radar array antenna received signal matrix obtained in step S1 and the reinforcement learning model constructed in steps S2 and S3, the agent is trained using the DQN algorithm to obtain the prediction of the interference angle.

[0042] S5. Based on the angle obtained in S4, perform blocking matrix processing to cancel the main lobe interference signal and complete the main lobe anti-interference.

[0043] In a further embodiment, in step S2, the radar acts as an agent in reinforcement learning, and its state space revolves around the radar, used to express relevant information about the current interference cancellation blocking matrix. State S is the set of all possible states s of the agent, and state s at time t is denoted as st. t :

[0044] s t =[θ,SINR,NUM]

[0045] The signal-to-interference-plus-noise ratio (SIR) is modeled after constant false alarm rate (CFAR) detection methods, such as... Figure 2 As shown, FFT processing is performed on the array antenna received signal matrix in step S1 to determine the amplitude Y of the unit under test (CUT). The two units closest to the CUT are removed. Then, the amplitudes of the M / 2 units on the left and the M / 2 units on the right are summed and averaged to obtain Z / M, thus obtaining the estimation formula for the signal-to-interference-plus-noise ratio:

[0046]

[0047] The mismatch count is used to calculate the number of times the blocking matrix and the interference direction mismatch when the interference angle changes; the initial value is 0. Within each time step, if SINR... t -SINR t-1 If the value is less than 15, then NUM = NUM ​​+ 1; otherwise, NUM remains unchanged. Initially, NUM = 0.

[0048] The action space is the set of actions taken by the radar agent. In this reinforcement learning environment model, four selectable actions are set, which represent angle adjustments of different directions and with different precision.

[0049] Table 3 Composition of ULA Action Space

[0050]

[0051] In a further embodiment, step S3 further provides the reward signal in the reinforcement learning model of the present invention. The radar agent receives a reward signal after interacting with the environment. In the method of the present invention, the reward signal can be divided into three categories: "success" reward, "penalty" reward, and regular reward. When interference cancellation is successful, the main task is completed, and the agent radar receives a positive "success" reward signal 'a'. The "penalty" reward and regular reward are used to regulate the agent's behavior: when the blocking angle reaches an angle outside the main lobe range, the agent radar receives a "penalty" signal 'b'. The regular reward -α(SINR'-SINR) / SINR' is used to evaluate behavior under non-boundary conditions, which should both encourage the agent to move closer to the target and prevent the agent from exploring indefinitely to obtain higher expected rewards.

[0052] The reward signal structure is as follows:

[0053]

[0054] Where a>0, b<0, SINR' is the expected signal-to-interference-plus-noise ratio (SIR) when the system successfully resists interference, and SINR is the current SIR output of the BMP. Multiplying by a coefficient α is to ensure that the reward value under normal exploration is greater than the reward value under boundary violation, so as to avoid the agent lingering at the boundary and stopping.

[0055] In a further embodiment, in step S4, the radar, acting as an agent, uses the current signal-to-interference ratio (SIR) as an observation in the reinforcement learning model. At each time step t, the radar is in state s. t Observe the current information ratio to determine the action. t This causes the state to transition to s. t+1 and receive a reward r t .

[0056] The DQN algorithm used employs a deep neural network to fit the action-value function Q in the reinforcement learning model. w (s,a), the DQN algorithm consists of two deep neural networks, one for training, called the training network, denoted as Q. w (s, a); a set of computations used for updates, called the target network, denoted as Q. w -(s, a). The reinforcement learning process is as follows:

[0057] S41. Initialize the system: radar array antenna shape, total capacity of the experience replay pool, minimum training data volume, learning rate, reward discount factor, target network update frequency, etc., randomly initialize training network parameters, and initial state of the agent.

[0058] S42. The radar agent interacts and explores with the environment. The radar, as the agent, uses the current signal-to-interference ratio (SIR) as the observation value in the reinforcement learning model. At each time step t, the radar is in state s. t According to the current network Q w (s, a) selects action a according to the ∈-greedy strategy. t This causes the state to transition to s. t+1 and receive a reward r t The interaction information between the agent and the environment constitutes a Markov decision process quadruple (s...). t a t r t s t+1 Store this quadruple in the experience replay pool.

[0059] S43. When the amount of data in the experience replay pool is greater than the size of the training samples required each time, randomly sample N quadruplets from the experience pool. For the i-th quadruplet, use the target network to calculate the temporal difference target. Where r i Q is the reward for the i-th quadruple. w -(s i+1 a) is the current target network's fitted output of the action-value function, and γ is the reward discount coefficient. Then, the target loss L = ∑ i (y i -Q w (s i a i )) 2 / N, Q w (s i a i To train the network to fit the action value function, the obtained parameters are used to update the training network Q. w (s, a);

[0060] S44. Update the training network parameters to the target network according to the initial update frequency.

[0061] S45. Repeat steps S42 to S44 until the training reward value converges stably.

[0062] In step S5, the radar agent adjusts the blocking angle according to the strategy trained in step S4, and performs blocking matrix processing on the radar array antenna received signal matrix in step S1 based on the adjusted angle.

[0063] This invention studies a fast angle search technique for spatial main lobe interference from the perspective of reinforcement learning. When the interference angle changes over time, the blocking matrix preprocessing can quickly track the interference angle, reduce the search time, and thus better suppress the interference.

[0064] The following examples provide a more detailed description.

[0065] Example 1: Uniform Linear Array (ULA)

[0066] For a 16-element linear uniformly distributed array antenna, assuming the target echo is f d The signal is a 1MHz sine wave with the target wave direction remaining constant at 0°, and consists of additive white Gaussian noise. The interference signal is a noise-modulated frequency signal with a sampling frequency f. s =10MHz, signal snapshot count 128; input signal-to-interference ratio -30dB, input signal-to-noise ratio 10dB.

[0067] The state-space composition of a uniform linear array antenna is shown in Table 1.

[0068] Table 1. Composition of the ULA state space

[0069] quantity Observations meaning Range of values 0 θ Current blocking angle <![CDATA[[-θ 0.5 ,i 0.5 ]]> 1 SINR BMP outputs SINR [-Inf,Inf] 2 NUM Number of mismatches [0,+∞]

[0070] In this environment, the action space is a finite space, meaning the total number of possible actions 'a' within action set A is finite. This RL environment model includes four selectable actions, representing angle adjustments of different directions and precisions. The composition of the ULA's action space is shown in Table 2.

[0071] Table 3 Composition of ULA Action Space

[0072] quantity meaning 0 Current angle shifted 1° to the left 1 The current angle shifts 1° to the right. 2 The current angle has shifted 0.1° to the left. 3 The current angle has shifted 0.1° to the right.

[0073] The reward signal in this model is specifically defined as follows:

[0074]

[0075] Where SINR' is the expected signal-to-interference-plus-noise ratio (SNR) when the system successfully resists interference, and SINR is the current SNR output by the BMP. Multiplying by a coefficient α is to ensure that the reward value under normal exploration is greater than the reward value under boundary violation, so as to avoid the agent lingering at the boundary and stopping.

[0076] The neural network used in this model has four layers (excluding the input layer), including two hidden layers consisting of 128 neurons and 64 neurons respectively; one activation layer activated by the ReLU function; and the other output layer.

[0077] Figure 3 and Figure 4 The results of training the algorithm proposed in this invention when the interference angle changes are compared with those of random strategy selection. It can be seen that the algorithm of this invention can converge the interference angle better and has a stable output signal-to-interference-plus-noise ratio.

[0078] Example 2: Uniform Circular Array

[0079] For a 21-element circular uniformly distributed array antenna, assuming the target echo is f d A 1MHz sine wave signal, with the target wave direction maintaining an azimuth angle of 45° and an elevation angle (angle with the z-axis) of 30°, is generated. Additive white Gaussian noise is also present, and the interfering signal is a noise-frequency modulated signal with a sampling frequency of f. s =10MHz, signal snapshot count 128, input signal-to-interference ratio -30dB, input signal-to-noise ratio 10dB. A Householder matrix is ​​used as the blocking matrix. The state space composition of the UCA is shown in Table 3.

[0080] Table 4. Composition of the UCA state space

[0081]

[0082] In this RL environment, the action space is still a finite space. There are four selectable actions in action set A, representing angle adjustments in different directions, as shown in Table 4.

[0083] Table 5. Composition of the UCA motion space

[0084]

[0085] The positive value of the reward signal is adjusted appropriately based on the size of the exploration space. The specific definition of the reward signal in this model is as follows:

[0086]

[0087] The neural network used in this model has four layers (excluding the input layer), including two hidden layers consisting of 256 and 128 neurons respectively; one activation layer activated by the ReLU function; and the other output layer.

[0088] Figure 5 This represents the training results of the algorithm proposed in this invention when the elevation angle of the incoming interference signal remains constant but the azimuth angle changes. Figure 6 The training results of the algorithm proposed in this invention are shown when the azimuth angle of the incoming interference signal remains unchanged and the elevation angle changes. Compared with the random strategy selection, it can be seen that the algorithm of this invention can converge the interference angle better and has a stable output signal-to-interference-plus-noise ratio.

Claims

1. A spatial main lobe interference suppression method based on reinforcement learning, characterized in that, The method includes the following steps: S1. Sample the target echo signal, noise, and interference signal to obtain the radar array antenna received signal matrix; S2. Select the radar as the agent for reinforcement learning, and determine the agent's state space S and action space A; the agent's state space S is the set of all possible states s of the agent, and the state s at time t is represented as... : In the formula, Here, S is the signal-to-interference-plus-noise ratio, and NUM is the number of mismatches. The current blocking angle; The formula for estimating the signal-to-interference-plus-noise ratio (SINR): In the formula, Y is the amplitude of the measured unit, M is the number of reference units, and Z is the total amplitude of all reference units; S3. Based on the agent and its state space S and action space A set in step S2, design the reward signal required for reinforcement learning; S4. Based on the radar array antenna received signal matrix obtained in step S1 and the reinforcement learning model constructed in steps S2 and S3, the agent is trained using the DQN algorithm to obtain the prediction of the interference angle. S5. Based on the angle obtained in S4, perform blocking matrix processing to cancel the main lobe interference signal and complete the main lobe anti-interference.

2. The spatial main lobe interference suppression method based on reinforcement learning according to claim 1, characterized in that, The action space is the set of actions taken by the radar agent, including four selectable actions, each representing an angle adjustment in different directions and with different precision: shift the current angle left by ϕ°, shift the current angle right by ϕ°, and shift the current angle left by ϕ°. The current angle shifts to the right. ϕ represents the maximum change in the output angle of the radar agent in each operation.

3. The spatial main lobe interference suppression method based on reinforcement learning according to claim 1, characterized in that, The radar agent receives a reward signal after interacting with the environment; the reward signal can be divided into three categories: "success" reward, "penalty" reward, and regular reward. The specific reward signal structure is as follows: in , The desired signal-to-interference-plus-noise ratio (SIR) when the system successfully resists interference. Let be the current BMP output signal-to-interference-plus-noise ratio, 'a' be the "success" reward signal, and 'b' be the "penalty" signal. is a coefficient.

Images