Adaptive anti-jamming beamforming method based on pre-processing deep reinforcement learning

Abstract

The application discloses an adaptive anti-interference beam forming method based on pre-processing deep reinforcement learning and belongs to the technical field of navigation, comprising the following steps: constructing a GPS terminal signal model, including a 2*2 dual-polarized antenna array, a control variable, a time-varying gain interference variable and Gaussian noise; constructing a deep learning convolutional neural network CNN, including a data feature extraction network layer, a convolution network layer, a pooling layer, an activation function layer and a full link layer; performing reinforcement learning processing and decision implementation on the deep learning convolutional neural network to obtain a deep reinforcement learning Q network; and training the deep reinforcement learning Q network to obtain a trained deep reinforcement learning convolutional neural network. The application controls array signals by beams, realizes beam main lobe positioning and simultaneously traps interference signals by side lobe, learns interference data features by the deep Q network to automatically determine the next execution action, and does not need human intervention, thereby greatly improving the autonomy of the intelligent antenna.

Description

Technical Field

[0001] This invention discloses an adaptive anti-interference beamforming method based on preprocessing deep reinforcement learning, which belongs to the field of navigation technology. Background Technology

[0002] Adaptive anti-interference beamforming, as a key research area in smart antenna anti-interference technology, is widely used in transportation, surveying, telecommunications, water conservancy, fisheries, natural disaster relief, and aerospace, possessing significant commercial and economic value. Currently, adaptive anti-interference beamforming technology adjusts the weighting factors of each array element signal based on a certain criterion algorithm, thereby adjusting the radiation pattern of the antenna array to enhance the desired signal and suppress interference signals. However, in practical wireless channel transmission, due to complex electromagnetic environments and dynamically changing interference, non-uniform imbalances, anisotropy, and uncertain oscillations occur during array beamforming. These adverse factors cause distortion of the antenna array's wavefront, resulting in angular spread. This causes beam components to converge and suspend at the transmission channel azimuth of the interference source, leading to variable virtual interference signals, i.e., spatial superposition interference. Research on array beamforming algorithms targeting wavefront distortion caused by dynamic interference is relatively limited. However, with the development of artificial intelligence, applying machine learning anti-interference strategies to the beamforming field allows for modeling and learning of dynamic interference environments and constraining the desired signal based on wavefront distortion.

[0003] Related published patents include CN202110568887.4: A fast adaptive anti-jamming method for large arrays based on convolutional neural networks. Convolutional neural networks are used to solve the problems of large computational load and poor beamform preservation in the adaptive beamforming of large phased arrays in the prior art, thereby controlling the beamforming to achieve the purpose of anti-jamming. Related literature has also studied the anti-jamming method implemented by machine learning, such as the recently published [1] Z. Xiao, B. Gao, S. Liu and L. Xiao. Learning Based Power Control for mmWave Massive MIMO against Jamming[C]. IEEE Global Communications Conference (GLOBECOM), pp. 1-6, 2018. Xiao et al. used the DQN learning method to improve the overall system power of anti-jamming in unknown low-complexity environments. [2] H.Yang, Z.Xiong, J.Zhao, D.Niyato, L.Xiao and Q.Wu. Deep Reinforcement Learning-Based Intelligent Reflecting Surface for Secure Wireless Communications[J].IEEE Transactions on Wireless Communications, vol.20, no.1, pp.375-388, Jan.2021. A DRL algorithm based on DQN is proposed, which can quickly obtain the optimal convex solution of the anti-jamming strategy while ensuring system security. X.Liu, Y.Xu, L.Jia, Q.Wu and A.Anpalagan. Anti-Jamming Communications Using Spectrum Waterfall: A Deep Reinforcement Learning Approach[J].IEEE Communications Letters, vol.22, no.5, pp.998-1001, May 2018. The anti-jamming beamforming strategy is learned through the DRL algorithm, which obtains the optimal convex solution of the anti-jamming beam while maximizing the overall system power. There are two limitations to learning decisions regarding anti-interference: i) Some information in the direction of the interference beam may be lost due to the unknown environment; ii) Anti-interference strategies can intelligently switch according to the dynamic environment, but it is difficult to track the interference signal in real time.

[0004] Reinforcement learning, also known as reinforcement learning, mainly includes two types of methods: value-based and probability-based methods. Value-based methods learn from existing experience to optimize the estimation function of the value of actions in different states, thereby obtaining the optimal action control strategy. Reinforcement learning has surpassed human performance in most Atari games. Deep neural networks have achieved remarkable results in the field of computer science, especially in computer vision. Convolutional neural networks can effectively extract convolutional features of images and have achieved excellent results in nonlinear fitting, object localization, object recognition, and image semantic segmentation. However, when faced with phase disturbances of the antenna array, system random errors, or the influence of channel transmission clutter, the interference source can still destroy the array beam and cause distortion, resulting in enhanced interference spatial overlap. The beam steering vector of the beam formation will produce phase diffusion, causing rank skewness in the signal covariance matrix. This phenomenon is equivalent to the interference signal undergoing fission in the channel environment, thereby significantly consuming the array degrees of freedom, resulting in the loss of some information in the direction of the interference beam, and severely degrading the system transmission performance. Therefore, how to eliminate wavefront distortion and establish a stable anti-interference strategy in practical engineering applications is of great significance for studying the maximum optimization of the total system transmission rate under the wavefront spatial superposition interference model. Summary of the Invention

[0005] This invention proposes an adaptive anti-interference beamforming method based on preprocessing deep reinforcement learning, which solves the problem in the prior art where interference signals undergo fission in the channel environment, resulting in the loss of some information in the direction of the interference beam.

[0006] An adaptive anti-jamming beamforming method based on preprocessing deep reinforcement learning includes:

[0007] S1. Construct a GPS terminal signal model, including a 2×2 dual-polarized antenna array, manipulation variables, time-varying gain interference variables, and Gaussian noise;

[0008] S2. Construct a deep learning convolutional neural network (CNN), including a data feature extraction network layer, a convolutional network layer, a pooling layer, an activation function layer, and a fully connected layer;

[0009] S3. Perform reinforcement learning processing and decision-making on the deep learning convolutional neural network using Q-network to obtain a deep reinforcement learning Q-network;

[0010] S4. Train the deep reinforcement learning Q-network to obtain the trained deep reinforcement learning convolutional neural network.

[0011] Preferably, in S1, the 2×2 dual-polarized antenna array obtains the gain of 8 polarization ports, manipulates variables to achieve beam patterns at different azimuth angles, the time-varying gain interference variable satisfies time-varying Rayleigh attenuation, and Gaussian noise is used as an interference auxiliary quantity.

[0012] The 2×2 dual-polarized antenna array is composed of a double-layer "Rogers RO3010" material substrate with a relative permittivity of 10.2 and a dielectric loss tangent of 0.0035.

[0013] Preferably, in S1, the control variables are specifically the i-th pitch and azimuth of the downlink navigation transmission link #1.

[0014]

[0015] In the formula, θ and These are the azimuth and initial elevation angles, respectively. Here, is the beam control vector, diag() represents the diagonal matrix symbol, exp() represents the exponential function, j is the imaginary number symbol, ω is the angular frequency, sin()cos() represents the trigonometric product, and R=d / λ is the ratio of the element spacing d to the resonant wavelength λ.

[0016] Preferably, in S1, the time-varying gain interference variable and Gaussian noise are the interference quantities of the total received signal model of GPS in downlink navigation transmission links 3# and 4#, and the structural formula of the time-varying gain interference variable is: In the formula E is the amount of interference that is similar to the desired signal structure. j H t Let be the time-varying Rayleigh attenuation interference t. Based on the downlink navigation transmission link, the overall GPS signal model is as follows:

[0017]

[0018] In the formula, In order to receive the total time-varying GPS signal, For the desired GPS signal in link #1, and These are the manipulation variables for the desired GPS signal in links #1 and #2, respectively. Here, n(t) is the manipulation variable for interference signals in links #3 and #4, and n(t) is the Gaussian noise signal. Where i = 1x or 2x or j, Let j be an N×1 phase keying matrix for the IRS sensor, where j is an imaginary number. The amplitude and phase of the phase keying unit of the IRS sensor are η∈0,1,φ∈[0,2π].

[0019] Preferably, in S2, the data feature extraction network layer is connected to a convolutional network layer, the convolutional network layer is connected to a pooling layer, and the pooling layer and the activation function layer are respectively connected to fully connected layers;

[0020] The data feature extraction network layer is a network layer that performs data dimensionality reduction and standardization after the input layer. It extracts the feature values ​​of the input data and inputs them into the data feature extraction network. The output of the data feature extraction network is the convolutional feature value corresponding to the standardization process at time t.

[0021] The convolutional network layer gradually recognizes higher-level perceptual regions from the initial region perception, and the convolutional output is... In the formula, For the Cartesian inner product, X t ,,W c b and b are the input, weight, and bias variables in the convolutional layer, respectively;

[0022] The pooling layer reduces dimensionality and compresses data to avoid overfitting, and selects max pooling to filter out excessive noise, s = s x δ(s x ≥n th In the formula, s x The raw data input to the network, δ(s) x ≥n th ) represents the original data s x ≥n th The impulse function, n th The minimum noise threshold;

[0023] The activation function layer uses the Rectified Lu() function as the activation function to achieve the non-linear transformation from convolutional pooling to fully connected layers. Its expression is f. ReLu The maximum value of the function f(x) = max(0,x) ReLu (x) represents the mathematical symbol for the ReLU(x) activation function, and max(0,x) represents the maximum function value between 0 and x;

[0024] The fully linked layer combines all features of the previous layer node at time t, Y t =W f ·X t Y t+1 =W f ·X t+1 =Y t +W f ·ΔX t+1 , where Y t and Y t+1 The output data at times t and t+1 are respectively, W f Let ΔX be the weight vector of the fully connected layer. t+1=X t+1 -X t Let ΔX represent the error variable between the input data X at times t and t+1. According to the recursive nature of the Markov decision process, ΔX... t+1 ={X t+1 -X t ,X t -X t-1 ,…,X t-T+2 -X t-T+1 It only affects the changes in input information at times t and t+1, but remains unchanged at time t-1.

[0025] Preferably, S3 includes:

[0026] S3.1.π Strategy Based on State s at Time t t and available action a t Low variance estimation of the expected Q function Implementing deep Q-learning with expectation operator s represents the initial state of Q-learning, a represents the initial learning activity of Q, and the π policy is based on the system state at time t. Available actions The mapping probability distribution, π(s) t ,a t ):

[0027] S3.2.t+1 time state s t+1 and available action a t+1 Get optimization Value, based on the π strategy, the i-th weight w i To obtain the expected target Q value, In the formula, r t Let μ be the cost function at time t, μ be the controllable factor, and max() be the maximum solver.

[0028] S3.3. Nonlinear CNNs preprocess elements by randomly selecting uniformly distributed elements. Make it approach the target expected value: In the formula, r(a t )=α(a t )-γδ(a t ≠a t-1 ), where the reward factor α(a t )=δ(ξ k (a t )≥ξ th In the formula, ξ k (a t Action a is available at time t. tThe reward amount; in the reward function, γ represents the cost coefficient of the loss of transmitted signal power, ξ th The threshold set for data transmission at the GNSS transmitter;

[0029] S3.4. Input Status Cross-entropy error is used to prevent gradient vanishing during each training iteration of the loss function, which is: In the formula, log() is the logarithmic function;

[0030] S3.5. Optimization of the gradient descent process for the loss function in S3.4 under a β learning rate, to obtain the algorithm for updating the Q function:

[0031] In the formula, ψ t Let be the gradient value at time t;

[0032] S3.6. To avoid local convergence, an ∈-greedy strategy is adopted to select the desired target action. The maximum Q-table value is selected based on the random probability ε≤p≤1-ε, and the current action is selected using p≤ε. The ∈-greedy strategy for the proxy action is expressed as follows:

[0033]

[0034] Preferably, S4 includes:

[0035] S4.1. In each learning time t, the learning agent uses a CNN to process the state information S t Preprocessing is performed to observe the system state at each event step in order to execute actions, which include anti-interference weights and phase shifts;

[0036] S4.2. Use an ∈-greedy strategy to select the optimal Q-function to balance exploration and exploitation;

[0037] S4.3. Within each time slot, obtain the maximum Q-function through the probability range ε≤p≤1-ε, and execute (s t ,a t After that, the reward r(s) is obtained. t ,a t ), and set the next state S t+1 Stored in set D to test the sample at the next time slot t+1;

[0038] S4.4. Feed the updated Q-value back to the CNN to take the next action, until the loop reaches the maximum number of iterations.

[0039] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention utilizes deep reinforcement learning to perform beam control on array signals, achieving main lobe localization of the beam while simultaneously nulling sidelobe interference signals by controlling manipulating variables. During the anti-interference process, the deep Q-network learns the characteristics of the interference data and autonomously determines the next action, eliminating the need for human intervention and greatly improving the autonomy of the smart antenna. The representation learning capability of the deep convolutional network performs translation-invariant classification of the input information (beamforming), max-pools the input data, and through the sharing of convolutional kernel parameters within the hidden layers and the sparsity of inter-layer connections, the convolutional neural network can process the beamfront covariance matrix with a small computational load, making it approximate the target value of the Q function. The reinforcement learning method strengthens the target Q value, weakens the spatial overlap of the interference signal, and thus eliminates beamfront distortion data, seeking to maximize the total transmission rate of the target anti-interference system, thereby reducing the consumption of the array antenna's degrees of freedom. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the technical process of the present invention.

[0041] Figure 2 This is a schematic diagram of the deep learning convolutional neural network (CNN) operation of the method of the present invention;

[0042] Figure 3 This is a schematic diagram of the deep reinforcement learning Q-network training stage of the method of the present invention;

[0043] Figure 4 This is a schematic diagram showing the comparison of results from an embodiment of the present invention. Detailed Implementation

[0044] The specific embodiments of the present invention will be further described below with reference to specific examples:

[0045] An adaptive anti-jamming beamforming method based on preprocessing deep reinforcement learning, such as Figure 1 ,include:

[0046] S1. Construct a GPS terminal signal model, including a 2×2 dual-polarized antenna array, manipulation variables, time-varying gain interference variables, and Gaussian noise;

[0047] S2. Construct a deep learning convolutional neural network (CNN), such as Figure 2 It includes a data feature extraction network layer, a convolutional network layer, a pooling layer, an activation function layer, and a fully connected layer;

[0048] S3. Perform reinforcement learning processing and decision-making on the deep learning convolutional neural network using Q-network to obtain a deep reinforcement learning Q-network;

[0049] S4. Train the deep reinforcement learning Q-network, such as... Figure 3The trained deep reinforcement learning convolutional neural network is obtained.

[0050] In S1, the 2×2 dual-polarized antenna array obtains the gain of 8 polarization ports, manipulates variables to achieve beam patterns at different azimuth angles, the time-varying gain interference variable satisfies time-varying Rayleigh attenuation, and Gaussian noise is used as an interference auxiliary quantity.

[0051] The 2×2 dual-polarized antenna array is composed of a double-layer "Rogers RO3010" material substrate with a relative permittivity of 10.2 and a dielectric loss tangent of 0.0035.

[0052] In S1, the control variables are specifically the i-th pitch and azimuth of the downlink navigation transmission link #1.

[0053] In the formula, θ and These are the azimuth and initial elevation angles, respectively. Here, is the beam control vector, diag() represents the diagonal matrix symbol, exp() represents the exponential function, j is the imaginary number symbol, ω is the angular frequency, sin()cos() represents the trigonometric product, and R=d / λ is the ratio of the element spacing d to the resonant wavelength λ.

[0054] In S1, the time-varying gain interference variable and Gaussian noise represent the interference amount in the GPS total received signal model of downlink navigation transmission links 3# and 4#. The structural formula of the time-varying gain interference variable is: In the formula E is the amount of interference that is similar to the desired signal structure. j H t Let be the time-varying Rayleigh attenuation interference t. Based on the downlink navigation transmission link, the overall GPS signal model is as follows:

[0055]

[0056] In the formula, In order to receive the total time-varying GPS signal, For the desired GPS signal in link #1, and These are the manipulation variables for the desired GPS signal in links #1 and #2, respectively. Here, n(t) is the manipulation variable for interference signals in links #3 and #4, and n(t) is the Gaussian noise signal. Where i = 1x or 2x or j, Let j be an N×1 phase keying matrix for the IRS sensor, where j is an imaginary number. The amplitude and phase of the phase keying unit of the IRS sensor are η∈0,1,φ∈[0,2π].

[0057] In S2, the data feature extraction network layer is connected to the convolutional network layer, the convolutional network layer is connected to the pooling layer, and the pooling layer and the activation function layer are respectively connected to the fully connected layer.

[0058] The data feature extraction network layer is a network layer that performs data dimensionality reduction and standardization after the input layer. It extracts the feature values ​​of the input data and inputs them into the data feature extraction network. The output of the data feature extraction network is the convolutional feature value corresponding to the standardization process at time t.

[0059] The convolutional network layer gradually recognizes higher-level perceptual regions from the initial region perception, and the convolutional output is... In the formula, For the Cartesian inner product, X t ,,W c b and b are the input, weight, and bias variables in the convolutional layer, respectively;

[0060] The pooling layer reduces dimensionality and compresses data to avoid overfitting, and selects max pooling to filter out excessive noise, s = s x δ(s x ≥n th In the formula, s x The raw data input to the network, δ(s) x ≥n th ) represents the original data s x ≥n th The impulse function, n th The minimum noise threshold;

[0061] The activation function layer uses the Rectified Lu() function as the activation function to achieve the non-linear transformation from convolutional pooling to fully connected layers. Its expression is f. ReLu The maximum value of the function f(x) = max(0,x) ReLu (x) represents the mathematical symbol for the ReLU(x) activation function, and max(0,x) represents the maximum function value between 0 and x;

[0062] The fully linked layer combines all features of the previous layer node at time t, Y t =W f ·X t Y t+1 =W f ·X t+1 =Y t +W f ·ΔX t+1 , where Y t and Y t+1 The output data at times t and t+1 are respectively, W f Let ΔX be the weight vector of the fully connected layer. t+1 =Xt+1 -X t Let ΔX represent the error variable between the input data X at times t and t+1. According to the recursive nature of the Markov decision process, ΔX... t+1 ={X t+1 -X t ,X t -X t-1 ,…,X t-T+2 -X t-T+1 It only affects the changes in input information at times t and t+1, but remains unchanged at time t-1.

[0063] S3 includes:

[0064] S3.1.π strategy based on state s at time t t and available action a t Low variance estimation of the expected Q function Implementing deep Q-learning with expectation operator s represents the initial state of Q-learning, a represents the initial learning activity of Q, and the π policy is based on the system state at time t. Available actions The mapping probability distribution, π(s) t ,a t ):

[0065] S3.2.t+1 time state s t+1 and available action a t+1 Get optimization Value, based on the π strategy, the i-th weight w i To obtain the expected target Q value, In the formula, r t Let μ be the cost function at time t, μ be the controllable factor, and max() be the maximum solver.

[0066] S3.3. Nonlinear CNNs preprocess elements by randomly selecting uniformly distributed elements. Make it approach the target expected value: In the formula, r(a t )=α(a t )-γδ(a t ≠a t-1 ), where the reward factor α(a t )=δ(ξ k (a t )≥ξ th In the formula, ξ k (a t Action a is available at time t. tThe reward amount; in the reward function, γ represents the cost coefficient of the loss of transmitted signal power, ξ th The threshold value set when transmitting data at the GNSS transmitter;

[0067] S3.4. Input Status Cross-entropy error is used to prevent gradient vanishing during each training iteration of the loss function, which is: In the formula, log() is the logarithmic function;

[0068] S3.5. Optimization of the gradient descent process for the loss function in S3.4 under a β learning rate, to obtain the algorithm for updating the Q function:

[0069] In the formula, ψ t Let be the gradient value at time t;

[0070] S3.6. To avoid local convergence, an ∈-greedy strategy is adopted to select the desired target action. The maximum Q-table value is selected based on the random probability ε≤p≤1-ε, and the current action is selected using p≤ε. The ∈-greedy strategy for the proxy action is expressed as follows:

[0071]

[0072] S4 includes:

[0073] S4.1. In each learning time t, the learning agent uses a CNN to process the state information S t Preprocessing is performed to observe the system state at each event step in order to execute actions, which include anti-interference weights and phase shifts;

[0074] S4.2. Use an ∈-greedy strategy to select the optimal Q-function to balance exploration and exploitation;

[0075] S4.3. Within each time slot, obtain the maximum Q-function through the probability range ε≤p≤1-ε, and execute (s t ,a t After that, the reward r(s) is obtained. t ,a t ), and set the next state S t+1 Stored in set D to test the sample at the next time slot t+1;

[0076] S4.4. Feed the updated Q-value back to the CNN to take the next action, until the loop reaches the maximum number of iterations.

[0077] The data calculation results are as follows Figure 4The simulation was performed using Python v3.6.6 on a Win64 system with TensorFlow v1 and the Keras learning library, on a computer with four Intel(R) i5-6500 CPU cores(TM), a 1258 GPU, and 8GB of RAM. Ansoft HFSS 15.0 software was used to simulate a 2×2 dual-polarized GNSS smart antenna array. All experimental programs were run on PyCharm Version 2018 to evaluate the anti-jamming performance of the proposed DRL.

[0078] The convolutional neural network uses the Adam method to update network parameters; the convolution depth is 32, and the bias is 8 bits; the max pooling is set to: 1 pooling layer, 1 tensor, and padding = 'VALID'; the input data is converted to shape, with 1024 samples and an 8×1 matrix; the initial learning rate is 0.01, and the learning rate is divided by 2 whenever the loss stops decreasing during training; the number of training iterations is 200; the mini-batch size is 32; and the number of hidden layer nodes is set to 1024.

[0079] Simulation parameters: Based on the reference, the learning rate λ = 0.1 for the Q-function, the maximum greedy ε = 0.9, the incremental coefficient Δε = 0.1, the discount coefficient μ = 0.7 in the reward function, the cost coefficient γ = 0.2, and the threshold ξ... th =15dB; the Rayleigh fading gain parameter is set to λ=1, f c =1.5GHz, c=3×10 8 m / s, α=2.8, d l =2m; the channel loss L-dB is set to PL0=30dB, d0=1m, γ=0.8. Furthermore, the CNN convolutional kernel layer count is set to 5 layers, with CL having 8 kernels of size 1×3 and FCL having 32 neurons, with a maximum PL. Additionally, to compare the performance of the proposed method, the following three methods are used:

[0080] i) The state space of the GNSS transmission system is optimized using a neural network-based RL method (Q-learning), in which the neural network uses 8 neural kernels, 1 ReLU layer and 1 output layer (named reinforcement learning);

[0081] ii) A greedy learning method (referred to as greedy learning, Q-learning not required) is proposed, which uses CNN and ∈-greedy strategy to optimize the 1# transmit power sent by the GNSS dual-polarized antenna array to the user, without Q-learning (named greedy learning);

[0082] iii) The minimum distortion-free adaptive anti-interference beamforming method (spatial filtering) is adopted. This method adaptively changes the weight vector of the received data by means of minimum variance distortion-free response, so as to minimize the average power of the array output (named minimum distortion-free response).

[0083] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. An adaptive anti-interference beamforming method based on preprocessing deep reinforcement learning, characterized in that, include: S1. Construct a GPS terminal signal model, including a 2×2 dual-polarized antenna array, manipulation variables, time-varying gain interference variables, and Gaussian noise; S2. Construct a deep learning convolutional neural network (CNN), including a data feature extraction network layer, a convolutional network layer, a pooling layer, an activation function layer, and a fully connected layer; S3. Perform reinforcement learning processing and decision-making on the deep learning convolutional neural network using Q-network to obtain a deep reinforcement learning Q-network; S4. Train the deep reinforcement learning Q-network to obtain the trained deep reinforcement learning convolutional neural network. ; S3 includes: S3.1.π strategy based on Moment State and available actions Low variance estimation expectation function, For the expectation operator, implement depth study , express The initial state of learning, express In the initial learning activity, the π strategy is from System status at all times Available actions The mapping probability distribution, ; S3.

2. Moment State and available actions Get optimization Value, based on the π strategy, the th Weight Achieve the expected goals value, In the formula, for Time-cost function, The controllable factor is max(), which is the maximum solver. S3.

3. Nonlinear CNNs preprocess elements by randomly selecting uniformly distributed elements. This brings it closer to the target expected value. In the formula, , Among them, reward factors In the formula, for Actions available at any time The amount of reward; in the reward function The cost factor representing the loss of transmitted signal power. The threshold set for data transmission at the GNSS transmitter; S3.

4. Input Status Cross-entropy error is used to prevent gradient vanishing during each training loss function iteration. The loss function is: In the formula, log() is the logarithmic function; S3.

5. Algorithm for optimizing the gradient descent processing of the loss function in S3.4 under different learning rates to obtain the updated Q-function: In the formula, Let be the gradient value at time t; S3.

6. To avoid local convergence, an ε-greedy strategy is used to select the desired target action based on random probability. Select the largest Q-table value, and use... The ε-greedy strategy for selecting the current action is as follows: 。 2. The adaptive anti-interference beamforming method based on preprocessing deep reinforcement learning according to claim 1, characterized in that, In S1, the 2×2 dual-polarized antenna array obtains the gain of 8 polarization ports, manipulates variables to achieve beam patterns at different azimuth angles, the time-varying gain interference variable satisfies time-varying Rayleigh attenuation, and Gaussian noise is used as an interference auxiliary quantity. The 2×2 dual-polarized antenna array is composed of a double-layer "Rogers RO3010" material substrate with a relative permittivity of 10.2 and a dielectric loss tangent of 0.0035.

3. The adaptive anti-interference beamforming method based on preprocessing deep reinforcement learning according to claim 2, characterized in that, In S1, the control variables are specifically the i-th pitch and azimuth of the downlink navigation transmission link #1. : In the formula, and These are the azimuth and initial elevation angles, respectively. Here, is the beam control vector, diag() represents the diagonal matrix symbol, exp() represents the exponential function, j is the imaginary number symbol, ω is the angular frequency, sin()cos() represents the trigonometric product, and R=d / λ is the ratio of the element spacing d to the resonant wavelength λ.

4. The adaptive anti-interference beamforming method based on preprocessing deep reinforcement learning according to claim 3, characterized in that, In S1, the time-varying gain interference variable and Gaussian noise represent the interference amount in the GPS total received signal model of downlink navigation transmission links 3# and 4#. The structural formula of the time-varying gain interference variable is: In the formula The amount of interference is similar to the desired signal structure. Let be the time-varying Rayleigh attenuation interference t. Based on the downlink navigation transmission link, the overall GPS signal model is as follows: In the formula, In order to receive the total time-varying GPS signal, for GPS signal is expected in the link. and They are respectively and Manipulated variables for the desired GPS signal in the link. for Manipulated variables of interference signals in the link. For Gaussian noise signals, set Where i = 1x or 2x or j, Let j be an N×1 phase keying matrix for the IRS sensor, where j is an imaginary number. The amplitude and phase of the phase keying unit of the IRS sensor are respectively... .

5. The adaptive anti-interference beamforming method based on preprocessing deep reinforcement learning according to claim 4, characterized in that, In S2, the data feature extraction network layer is connected to the convolutional network layer, the convolutional network layer is connected to the pooling layer, and the pooling layer and the activation function layer are respectively connected to the fully connected layer. The data feature extraction network layer is a network layer that performs data dimensionality reduction and standardization after the input layer. It extracts the feature values ​​of the input data and inputs them into the data feature extraction network. The output of the data feature extraction network is the convolutional feature value corresponding to the standardization process at time t. The convolutional network layer gradually recognizes higher-level perceptual regions from the initial region perception, and the convolutional output is... In the formula, ⊗ is the Cartesian inner product, X t W c b and b are the input, weight, and bias variables in the convolutional layer, respectively; The pooling layer reduces dimensionality and compresses data to avoid overfitting, and selects max pooling mode to filter out excessive noise. In the formula, The raw data input to the network, Original data The impulse function, The minimum noise threshold; The activation function layer uses the Rectified Lu() function as the activation function to achieve the non-linear transformation from convolutional pooling to fully connected layers. Its expression is: The maximum function value, ReLU ( The mathematical notation of the activation function. This represents the maximum function value between 0 and x; The fully linked layer combines all features of the nodes in the previous layer at time t. , ,in and The output data at times t and t+1 are respectively. The weight vector of the fully connected layer. This represents the input data at times t and t+1. The error variable between them, according to the recursive property of the Markov decision process, It only affects the changes in input information at times t and t+1, but remains unchanged at time t-1.

6. The adaptive anti-interference beamforming method based on preprocessing deep reinforcement learning according to claim 5, characterized in that, S4 include: S4.

1. In each learning time t, the learning agent uses a CNN to process the state information S t Preprocessing is performed to observe the system state at each event step in order to execute actions, which include anti-interference weights and phase shifts; S4.

2. Use an ε-greedy strategy to select the optimal Q-function to balance exploration and exploitation; S4.

3. Passage probability range within each time slot Obtain the maximum Q function and execute. Afterwards, you will receive a reward. And set the next state S t+1 Stored in set D to test the sample at the next time slot t+1; S4.

4. Feed the updated Q-value back to the CNN to take the next action, until the loop reaches the maximum number of iterations.

Images