Radar anti-interference decision generation method and system based on experience playback and strategy gradient, medium and product

Abstract

The invention discloses a radar anti-interference decision generation method and system based on experience playback and strategy gradient, a medium and a product. The method comprises the following steps: constructing an experience playback pool; a reinforcement learning GPPO model is constructed; training the reinforcement learning GPPO model by using the experience playback pool to obtain a trained reinforcement learning GPPO model; and the trained reinforcement learning GPPO model is used to generate an optimal anti-interference strategy for an interference signal to be suppressed. According to the method, the current state of the linear frequency modulation radar can be better estimated by using the current and historical radar interference data, so that the online generation of the optimal anti-interference decision is realized, and the anti-interference decision generation performance in the dynamic electromagnetic countermeasure environment is also improved.

Description

Technical Field

[0001] The present invention relates to the field of radar anti-interference technology, and in particular to a radar anti-interference decision-making generation method, system, medium and product based on experience replay and policy gradient. Background Art

[0002] Radar's all-weather sensing capabilities are crucial to modern warfare, but their performance is threatened by advanced jamming technologies, such as digital radio frequency memory (DRM). Consequently, there is a need to develop better anti-jamming decision-making methods in complex electromagnetic environments. There are many different anti-jamming methods for specific jamming signals, and the most effective method must be selected from the available options. Decision-making for anti-jamming methods primarily relies on manual intervention, which is time-consuming and inefficient. Summary of the Invention

[0003] In order to solve the technical problems existing in the background technology, the present invention proposes a radar anti-interference decision generation method, system, medium and product based on experience replay and policy gradient.

[0004] The present invention proposes a radar anti-interference decision generation method based on experience replay and policy gradient, comprising:

[0005] Build an experience replay pool;

[0006] Build a reinforcement learning GPPO model;

[0007] Use the experience replay pool to train the reinforcement learning GPPO model to obtain a trained reinforcement learning GPPO model;

[0008] The trained reinforcement learning GPPO model is used to generate the optimal anti-interference strategy for the interference signal to be suppressed.

[0009] Preferably, building an experience replay pool includes:

[0010] The radar system is considered as an intelligent agent in deep reinforcement learning, the interference signal is used as the state, the anti-interference strategy is used as the action, and the effectiveness evaluation of the anti-interference strategy is used as the reward. The radar anti-interference environment is mapped to the deep reinforcement learning environment to obtain an online anti-interference environment.

[0011] The online anti-interference environment uses the radar system as an intelligent agent to interact with the online anti-interference environment to obtain an experience revisit pool.

[0012] Preferably, the interference signals include distance deception interference DDJ, coherent suppression interference CSJ, spectrum dispersion interference SMSPJ, intermittent sampling repetitive cycle interference ISCJ and dense false target interference DFTJ, and the anti-interference strategy is used to represent the blind source separation method, singular value ratio spectrum method, energy detection method, time-frequency domain filtering method and fractional Fourier domain filtering method in the anti-interference method library.

[0013] Preferably, the online anti-interference environment is modeled as a Markov decision process, which is represented by a five-tuple (S, A, P, R, γ), where S is the state set, which is used to represent all types of interference signals encountered by the radar; A is the action set, which is used to represent all anti-interference methods that the radar can take; P is the state transition probability, which is used to represent the probability of transitioning from the current state s to the next state s' after taking a certain action a, that is, R is the reward function, which is used to evaluate the effect of taking an action, that is, the effectiveness of the anti-interference strategy; γ is the discount factor, which is used to calculate the present value of future rewards;

[0014] At time t, the state of the online anti-interference environment is s t ∈S, action is a t ∈A, reward is r t ∈R, then the state transition probability Indicates the possibility of transferring from the current state s to the next state s' after taking an action a at the current moment; expected reward Given s t =s and a t =a, reward r at time t+1 t+1 The expected value of where E[·] represents the expected value operation.

[0015] Preferably, the radar system is used as an intelligent agent to interact with the online anti-interference environment to obtain an experience revisit pool, specifically including:

[0016] At each moment, the interference signal in the interference signal library of the radar system is obtained, and the interference signal is identified using a preset interference identification model to obtain the interference identification result at each moment;

[0017] The interference recognition result at each moment is regarded as the interference state. According to the interference state and the observation equation, anti-interference result equation and reward function of the Markov decision process, the observation result, anti-interference completion mark and reward at each moment are obtained.

[0018] Based on the observation results and completion flag of the interference signal at each moment and the observation results, actions, rewards and experience tuples of the interference signal at the previous moment, the experience tuple is expressed as<o,a,r,o',done> , o represents the observation result of the interference signal in the previous round, a represents the action, r represents the reward, o' represents the observation result of the interference signal in the current round, and done represents the completion flag;

[0019] Based on the experience tuple, an experience replay pool is obtained.

[0020] Preferably, the interference identification model is a pre-trained two-dimensional convolutional neural network for identifying interference category labels of interference signals; wherein the two-dimensional convolutional neural network includes four layers of convolution.

[0021] Preferably, the reinforcement learning GPPO model includes a policy network and a value network;

[0022] The value network is defined as a neural network whose basic structure is a gated recurrent unit; in the value network, Where V π (s) represents the maximization value function, γ represents the discount factor, E[·] represents the expected value operation, π represents the optimization goal of the value network, that is, the optimal anti-interference strategy; T represents the total time step, k represents the time step, which is used to traverse each time step after the current time t, t represents the time, r k represents the reward value obtained at time step k, s represents the interference state, s t represents the interference state at time t;

[0023] Among them, the policy network is defined as a neural network whose basic structure is the gated recurrent unit (GRU); in the policy network, Where Q π (s,a) represents the action value function, γ represents the discount factor, E[·] represents the expected value operation, T represents the total time step, t represents the moment, k represents the time step, r k represents the reward value obtained at time step k, a represents the action, a t represents the action at time t;

[0024] Among them, the advantage function of the reinforcement learning GPPO model is A π (s,a)=Q π (s,a)-V π (s,a); where A π (s,a) represents the advantage function.

[0025] Preferably, the reinforcement learning GPPO model is constructed, including:

[0026] The gated recurrent unit is integrated into the policy network and value network of the proximal policy optimization algorithm to obtain the reinforcement learning GPPO model.

[0027] Preferably, the experience replay pool is used to train the reinforcement learning GPPO model to obtain a trained reinforcement learning GPPO model, specifically including:

[0028] Randomly extract a batch of experience tuples from the experience replay pool D;

[0029] Input the experience tuple into the reinforcement learning GPPO model, calculate the gradient and update the parameters of the policy network and the value network in the reinforcement learning GPPO model;

[0030] Based on the updated policy network, the optimal anti-interference strategy a' is selected from the pre-built anti-interference method library; based on the updated value network and the optimal anti-interference strategy a', a new interference state is generated;

[0031] Based on the new interference state, determine whether the reinforcement learning GPPO model meets the preset convergence conditions;

[0032] If so, output the trained reinforcement learning GPPO model;

[0033] If not, based on the new interference state, the observation equation, anti-interference result equation and reward function of the Markov decision process, new observation results, anti-interference completion flag and reward are obtained;

[0034] Based on the new observation results, anti-interference completion flag and reward, a new batch of experience tuples is called and the reinforcement learning GPPO model is iteratively trained until the reinforcement learning GPPO model reaches the preset convergence condition;

[0035] Preferably, the parameters of the policy network and the value network are updated as follows:

[0036] Update the parameters of the policy network by optimizing the objective function of the policy network;

[0037] The parameters of the value network are updated by minimizing the loss function of the value network.

[0038] Preferably, the objective function J(φ k ) can be defined as

[0039]

[0040] Where, J(φ k ) represents the objective function;

[0041] Among them, the function h(e,A) is defined as

[0042] The loss function of the value network L(ψ k ) can be defined as

[0043]

[0044] Preferably, the update formula of the policy network parameters is Where η is the learning rate and φ represents the parameter;

[0045] The update formula of the value network parameters is Where η is the learning rate and ψ represents the parameter.

[0046] In a second aspect, the present invention further proposes a radar anti-interference decision generation system based on experience replay and policy gradient, comprising:

[0047] Experience replay pool construction module, used to build the experience replay pool;

[0048] Model building module, used to build reinforcement learning GPPO model;

[0049] The training model is used to train the reinforcement learning GPPO model using the experience replay pool to obtain a trained reinforcement learning GPPO model;

[0050] The strategy generation module is used to generate the optimal anti-interference strategy for the interference signal to be suppressed using the trained reinforcement learning GPPO model.

[0051] In a third aspect, the present invention further proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the radar anti-interference decision-making method based on experience replay and policy gradient as described in any one of the first aspects.

[0052] In a fourth aspect, the present invention further proposes a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the radar anti-interference decision-making method based on experience replay and policy gradient as described in any one of the third aspects.

[0053] The present invention proposes a radar anti-interference decision-making method, system, medium, and product based on experience replay and policy gradients. This method constructs an experience replay pool and a reinforcement learning (GPPO) model. This model is then trained using the experience replay pool to generate a trained GPPO model. This trained GPPO model is then used to generate an optimal anti-interference strategy for the interference signal to be suppressed. This method can better utilize current and historical radar interference data to estimate the current state of a linear frequency modulation (LFM) radar. This not only enables online generation of optimal anti-interference decisions, but also improves the performance of anti-interference decision-making in dynamic electromagnetic countermeasure environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of a radar anti-interference decision-making method based on experience replay and policy gradient in one embodiment of the present invention.

[0055] Figure 2 Figure 2 is a pulse compression diagram of a radar signal before and after suppression according to an embodiment of the present invention.

[0056] Figure 3 This is a schematic diagram of a reward value curve when the reinforcement learning GPPO model in one embodiment of the present invention is trained in an online anti-interference environment.

[0057] Figure 4 Schematic diagram of the reward value curve when the PPO model in the prior art is trained in an online anti-interference environment. DETAILED DESCRIPTION

[0058] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0059] First, as Figure 1 As shown, the present invention proposes a radar anti-interference decision generation method based on experience replay and policy gradient, including:

[0060] Build an experience replay pool;

[0061] Build a reinforcement learning GPPO model;

[0062] Use the experience replay pool to train the reinforcement learning GPPO model to obtain a trained reinforcement learning GPPO model;

[0063] The trained reinforcement learning GPPO model is used to generate the optimal anti-interference strategy for the interference signal to be suppressed.

[0064] This application constructs an experience replay pool and a reinforcement learning GPPO model, then uses the experience replay pool to train the reinforcement learning GPPO model to obtain a trained reinforcement learning GPPO model. This trained reinforcement learning GPPO model is then used to generate an optimal anti-interference strategy for the interference signal to be suppressed. This invention can better utilize current and historical radar interference data to estimate the current state of the linear frequency modulation radar. This not only enables the online generation of optimal anti-interference decisions, but also improves the performance of anti-interference decision generation in dynamic electromagnetic countermeasure environments.

[0065] In this embodiment, the experience replay pool is constructed, including:

[0066] Build an online anti-interference environment;

[0067] Use the radar system as an intelligent agent to interact with the online anti-interference environment to obtain an experience revisit pool.

[0068] This embodiment applies deep reinforcement learning to radar anti-interference decision-making. It can collect interaction data between the radar system and the electromagnetic environment, thereby enhancing the decision-making performance of the interfered radar, and enables the intelligent agent to fully explore environmental data, thereby improving the efficiency of the decision-making process.

[0069] In a further embodiment, building an online anti-interference environment includes:

[0070] The radar system is regarded as an intelligent agent in deep reinforcement learning, the interference signal is regarded as the state, the anti-interference strategy is regarded as the action, and the effectiveness evaluation of the anti-interference strategy is regarded as the reward. The radar anti-interference environment is mapped with the deep reinforcement learning environment to obtain an online anti-interference environment.

[0071] Among them, the online anti-interference environment is modeled as a Markov decision process. Markov decision process is a mathematical method for making sequential decisions under uncertain conditions, which is represented by a five-tuple (S, A, P, R, γ). In the online anti-interference environment, the state set S represents all the types of interference signals that the radar may encounter; the action set A represents all the anti-interference methods that the radar can take; P is the state transition probability, which represents the probability of transitioning from the current state s to the next state s' after taking a certain action a, that is, R is the reward function, which is used to evaluate the effect of taking an action and the effectiveness of the anti-interference strategy; γ is the discount factor, which is used to calculate the present value of future rewards and is usually between 0 and 1. At time t, the state of the online anti-interference environment is s t ∈S, action is a t ∈A, reward is r t ∈R, then the state transition probability Indicates the possibility of transferring from the current state s to the next state s' after taking an action a at the current moment; expected reward Given s t =s and a t =a, reward r at time t+1 t+1 The expected value of where E[·] represents the expected value operation.

[0072] The Markov decision process includes the observation equation O(s), the anti-interference result equation T(s) and the reward function R(s,a).

[0073] The reward function R(s,a) is defined as the reward value r obtained after implementing action a of a certain anti-interference method under interference state s. The reward value r equally comprehensively considers the impulse response width, peak sidelobe ratio, integrated sidelobe ratio, and signal-to-interference-noise ratio before and after the implementation of the anti-interference method, a total of four quantitative evaluation indicators.

[0074] It's important to understand that the anti-interference result equation, T(s), defines the effective anti-interference result under interference state s. This is determined by comparing the signal-to-noise ratio (SNR) of the interference signal before and after implementing the anti-interference method. If the SNR improves by more than 5dB after implementing the anti-interference method, the anti-interference completion flag (done) is output; otherwise, the anti-interference failure flag is output.

[0075] In a further embodiment, a radar system is used as an intelligent agent to interact with an online anti-interference environment to obtain an experience revisit pool, specifically including:

[0076] At each moment, the interference signal in the interference signal library of the radar system is obtained, and the interference signal is identified using a preset interference identification model to obtain the interference identification result at each moment;

[0077] The interference recognition result at each moment is regarded as the interference state s at each moment. According to the interference state s at each moment and the observation equation O(s), the anti-interference result equation T(s) and the reward function R(s,a) of the Markov decision process, the observation result o', the anti-interference completion flag done and the reward r at each moment are obtained;

[0078] Based on the observation result o' and completion flag done of the interference signal at each moment and the observation result o, action a and reward r of the interference signal at the previous moment, an experience tuple is constructed; wherein the experience tuple is represented as<o,a,r,o',done> ;

[0079] Based on the experience tuple, an experience replay pool is obtained.

[0080] It should be understood that the experience replay pool D is defined as a fixed-size buffer used to store the experience tuples generated by the radar system as an intelligent agent interacting with the online anti-interference environment.<o,a,r,o',done> ,If the number of experience tuples in the experience replay pool reaches the preset number, the earliest experience tuple is replaced by the new experience tuple.

[0081] The jamming signal library contains five types of jamming signals: distance deception jamming (DDJ), coherent suppression jamming (CSJ), spectrum dispersion jamming (SMSPJ), intermittent sampling repetitive cycle jamming (ISCJ), and dense false target jamming (DFTJ). Therefore, the jamming signal labels are DDJ, CSJ, SMSPJ, ISCJ, and DFTJ.

[0082] The jamming signals in this embodiment come from a jamming signal library, including distance deception jammer DDJ, coherent suppression jammer CSJ, spectrum dispersion jammer SMSPJ, intermittent sampling repetitive cycle jammer ISCJ, and dense false target jammer DFTJ.

[0083] In one specific embodiment, the interference recognition model is a two-dimensional convolutional neural network for identifying interference category labels of interference signals; wherein the two-dimensional convolutional neural network includes four layers of convolution.

[0084] The interference signal is identified using a preset interference identification model to obtain the interference identification results at each moment, specifically including:

[0085] Perform short-time Fourier transform on the interference signal to obtain the time-frequency image corresponding to the interference signal;

[0086] The time-frequency image is input into a two-dimensional convolutional neural network to obtain an image feature vector, which is used as the interference recognition result corresponding to the interference signal at each moment.

[0087] In order to improve the recognition accuracy of the two-dimensional convolutional neural network, in a further embodiment, before using the two-dimensional convolutional neural network to identify the interference signal and obtain the interference recognition result at each moment, it also includes:

[0088] Obtain radar active jamming signal dataset with jamming type labels;

[0089] The radar active jamming signal dataset is divided into recognition training set, recognition verification set and recognition test set;

[0090] The two-dimensional convolutional neural network is trained, verified and tested in sequence using the recognition training set, recognition verification set and recognition test set to obtain a trained two-dimensional convolutional neural network.

[0091] That is to say, the two-dimensional convolutional neural network in this embodiment is a pre-trained two-dimensional convolutional neural network.

[0092] Among them, the radar active jamming signal dataset is divided into recognition training set, recognition verification set and recognition test set, specifically including:

[0093] Divide the radar active jamming signal dataset into corresponding data subsets according to the jamming type labels;

[0094] Samples are randomly selected from each data subset according to the set ratio to form the training set, validation set and test set.

[0095] Specifically, the radar active jamming signal dataset includes five categories of radar active jamming signals, with 150 signals in each category, for a total of 750 radar active jamming signals. The ratio of the training set, validation set, and test set is 0.6:0.2:0.2.

[0096] During the training process, the interference signal of the identification training set is input into the two-dimensional convolutional neural network to obtain the corresponding time-frequency image; the video image is input into the pre-trained classification head to perform classification prediction of the interference signal to obtain the predicted classification result; based on the predicted classification result and the corresponding interference type label, the recognition loss is calculated, and based on the recognition loss, the parameters of the two-dimensional convolutional neural network are optimized until the two-dimensional convolutional neural network reaches the preset convergence condition; the parameter weight file that meets the convergence condition is saved for directly loading the weight to identify the interference signal when adjusting the deep reinforcement learning algorithm.

[0097] Among them, during the verification and testing process, the interference signal in the recognition verification set or the recognition test set is input into the two-dimensional convolutional neural network to obtain the corresponding time-frequency image; the video image is input into the pre-trained classification head to perform classification prediction of the interference signal to obtain the predicted classification result; based on the predicted classification result and the corresponding interference type label, the corresponding recognition accuracy is calculated; if the recognition accuracy reaches the preset verification index or test index, the verification is passed or the test is passed; if the recognition accuracy does not reach the preset verification index or test index, retraining is performed until the verification and test are passed; the parameter weight file when the test index is reached is saved for directly loading the weight for interference signal recognition when adjusting the deep reinforcement learning algorithm.

[0098] In a specific embodiment, the training process of the two-dimensional convolutional neural network uses a stochastic gradient descent optimizer for 100 rounds with a batch size of 32.

[0099] Specifically, the recognition loss of the two-dimensional convolutional neural network is

[0100] Where, represents the recognition loss, y i Represents the one-hot encoding of the label information corresponding to the time-frequency image, C is the index of the interference category, p i Represents the predicted probability of the image classification head, N represents the total number of recognition samples, i = 1, 2, 3, ..., N.

[0101] In this embodiment, a reinforcement learning GPPO model is constructed, including:

[0102] The Gated Recurrent Units (GRU) are integrated into the policy network and value network of the Proximal Policy Optimization (PPO) algorithm to obtain the reinforcement learning GPPO model.

[0103] This implementation integrates a gated recurrent unit (GRU) into the proximal policy optimization algorithm. The GRU serves as the backbone of the PPO algorithm's policy and value networks, effectively leveraging current and historical radar jamming data to estimate the current state of the chirp radar. This not only enables online generation of optimal anti-jamming decisions, but also improves the performance of the basic PPO algorithm in generating anti-jamming decisions in dynamic electromagnetic countermeasure environments.

[0104] In this embodiment, the reinforcement learning GPPO model includes a policy network and a value network.

[0105] The value network is defined as a neural network whose basic structure is a gated recurrent unit (GRU); the value function V in the value network π (s) is defined as

[0106]

[0107] Where γ is the discount factor, E[·] represents the expected value operation; the optimization goal of the value network is to find the optimal anti-interference strategy π and maximize the value function V π (s), T represents the total time step, t represents the moment, k represents the time step, r k represents the reward value obtained at time step k, s represents the interference state, s t represents the interference state at time t.

[0108] The policy network is defined as a neural network whose basic structure is a gated recurrent unit (GRU); the action value function Q in the policy network is π (s,a) is defined as

[0109]

[0110] Where γ is the discount factor, E[·] represents the expected value operation, T represents the total time step, t represents the moment, k represents the time step, r k represents the reward value obtained at time step k, a represents the action, a t Represents the action at time t.

[0111] Advantage function A in the reinforcement learning GPPO model π (s,a) is used to calculate the action value function Q π (s,a) and value function V π (s), that is,

[0112] A π (s,a)=Q π (s,a)-V π (s,a).

[0113] Among them, the anti-interference method library includes blind source separation, singular value ratio spectrum method, energy detection method, time-frequency domain filtering method and fractional Fourier domain filtering method.

[0114] The library of anti-interference methods includes five categories: blind source separation, singular value ratio spectrum, energy detection, time-frequency domain filtering, and fractional Fourier domain filtering. Anti-interference strategies refer to the anti-interference methods in the library.

[0115] It should be noted that the anti-interference strategy in this embodiment is defined as π, and π(a|s; θ) represents the probability of taking action a in state s, where θ is a parameter of the strategy network.

[0116] The training process of the reinforcement learning GPPO model in this embodiment includes:

[0117] S1. Randomly extract a batch of experience tuples from the experience replay pool D;

[0118] S2. Input the experience tuple into the reinforcement learning GPPO model, calculate the gradient and update the parameters of the policy network and the value network in the reinforcement learning GPPO model;

[0119] S3. Based on the updated policy network, select the optimal anti-interference strategy a' from the pre-built anti-interference method library; use the updated value network to execute the optimal anti-interference strategy a' to generate a new interference state;

[0120] S4. Based on the new interference state, determine whether the reinforcement learning GPPO model meets the preset convergence conditions; if so, enter S5; if not, enter S6;

[0121] S5. Output the trained reinforcement learning GPPO model;

[0122] S6. Based on the new interference state, the observation equation, anti-interference result equation and reward function of the Markov decision process, new observation results, anti-interference completion flag and reward are obtained; based on the new observation results, anti-interference completion flag and reward, a new batch of experience tuples is called, and S2-S4 are iteratively repeated until the reinforcement learning GPPO model reaches the preset convergence condition.

[0123] This embodiment is configured in this way, which can help the reinforcement learning GPPO model learn a wider range of data distribution and improve the generalization ability of the reinforcement learning GPPO model.

[0124] The library of anti-interference methods includes five categories: blind source separation, singular value ratio spectrum, energy detection, time-frequency domain filtering, and fractional Fourier domain filtering. Anti-interference strategies refer to the anti-interference methods in the library.

[0125] Among them, in the policy network, by optimizing the objective function J(φ k ) to update the parameters of the policy network; in the value network, by minimizing the loss function L(ψ k ) to update the parameters of the value network.

[0126] Specifically, during the training process, the experience replay pool D is generated through L iterations, and the tuple τ from the initial step to the final step T is compiled.<o,a,r,o',done> , the objective function of the policy network J(φ k ) can be defined as

[0127]

[0128] Among them, the function h(e,A) is defined as

[0129] Specifically, during the training process, the experience replay pool D is generated through L iterations, and the tuple τ from the initial step to the final step T is compiled.<o,a,r,o',done> , the loss function of the value network L(ψ k ) can be defined as

[0130]

[0131] Among them, during the training process, the update formula of the policy network parameter φ is defined as

[0132] Where η is the learning rate;

[0133] The update formula of the value network parameter ψ is defined as

[0134] Where η is the learning rate.

[0135] In this embodiment, the trained reinforcement learning GPPO model is used to generate the optimal anti-interference strategy for the interference signal to be suppressed, specifically including:

[0136] Identify the interference signal to be suppressed and obtain the interference signal identification result; regard the interference identification result as a state, and obtain the observation result, anti-interference completion flag and reward based on the interference identification result and the observation equation, anti-interference result equation and reward function of the Markov decision process; call the experience tuple in the experience replay pool based on the observation result, anti-interference completion flag and reward<o,a,r,o',done> , input into the policy network and value network of the reinforcement learning GPPO model to optimize the objective function J(φ k ) and the loss function L(ψ k), select the best anti-interference strategy from the anti-interference method library to suppress the current interference signal.

[0137] In summary, the radar anti-interference decision-making method based on experience replay and policy gradient proposed in this paper generates anti-interference decision-making through a deep reinforcement learning algorithm. This decision-making process does not require expert participation, and the decision results are automatically generated, saving time and efficiency. Furthermore, by using GRU to improve the decision network and value network of the PPO algorithm, it can estimate the current state of the linear frequency modulation radar by linking current and historical radar interference data, more closely matching the dynamic electromagnetic countermeasure condition assumptions. The improved reinforcement learning GPPO model further improves the decision-making speed and algorithm stability of the PPO algorithm, demonstrating its practical application value.

[0138] In a second aspect, the present invention further proposes a radar anti-interference decision generation system based on experience replay and policy gradient, comprising:

[0139] Experience replay pool construction module, used to build the experience replay pool;

[0140] Model building module, used to build reinforcement learning GPPO model;

[0141] The training model is used to train the reinforcement learning GPPO model using the experience replay pool to obtain a trained reinforcement learning GPPO model;

[0142] The strategy generation module is used to generate the optimal anti-interference strategy for the interference signal to be suppressed using the trained reinforcement learning GPPO model.

[0143] The construction process of the experience replay pool in this embodiment is the same as the construction method of the experience replay pool described in the first aspect. The structure of the reinforcement learning GPPO model and its training method are the same as those described in the first aspect. The optimal anti-interference strategy is also the same as that described in the first aspect, and will not be repeated here.

[0144] In a third aspect, the present invention further proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the radar anti-interference decision-making method based on experience replay and policy gradient as described in any one of the first aspects.

[0145] In a fourth aspect, the present invention further proposes a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the radar anti-interference decision-making method based on experience replay and policy gradient as described in any one of the third aspects.

[0146] The present invention will be described below with reference to specific embodiments.

[0147] Example 1

[0148] The interference signal is identified using the two-dimensional convolutional neural network in the radar anti-interference decision generation method based on experience replay and policy gradient proposed in the present invention. Based on the recognition results, a recognition confusion matrix is ​​constructed to evaluate the performance of the two-dimensional convolutional neural network in identifying interference signals. The recognition confusion matrix results are shown in Table 1.

[0149] The two-dimensional convolutional neural network in this embodiment is trained using a stochastic gradient descent optimizer for 100 rounds with a batch size of 32.

[0150] Table 1

[0151]

[0152] Example 2

[0153] The radar anti-interference decision generation method based on experience replay and policy gradient proposed in this invention is used to generate anti-interference strategies, and the generated anti-interference strategies are used to suppress interference on radar signals. The pulse compression diagrams of radar signals before and after suppression are shown in Figure 2. Figure 2 shown.

[0154] Example 3

[0155] The reinforcement learning GPPO model in the radar anti-interference decision generation method based on experience replay and policy gradient proposed in this invention was trained in an online anti-interference environment for 400 cycles, and each experiment was repeated 10 times. The reward value curve is shown in Figure 2. Figure 3 As shown. Among them, Figure 3 Center, line indicates mean reward value, and shaded area indicates variance across repeated trials.

[0156] Comparative Example 1

[0157] The PPO model in the prior art was trained for 400 cycles in an online anti-interference environment, and each experiment was repeated 10 times. The reward value curve is as follows Figure 4 As shown. Among them, Figure 4 Center, line indicates mean reward value, and shaded area indicates variance across repeated trials.

[0158] Table 1 illustrates the performance of the two-dimensional convolutional neural network using a confusion matrix, where the values ​​on the diagonal correspond to the recognition accuracy. As can be seen from Table 1, the two-dimensional convolutional neural network in this embodiment achieves recognition accuracy exceeding 96% for all five types of interference signals, providing an accurate interference state set S for online anti-interference environments.

[0159] from Figure 2 As you can see, Figure 2The signal pulse compression diagrams before and after the optimal anti-interference method is adopted are displayed. It can be clearly seen that after applying different anti-interference methods, the interference signals in the radar signal are successfully suppressed.

[0160] from Figure 2 It can be seen that the most effective anti-interference strategies for DDJ, CSJ, SMSPJ, DFTJ and ISCJ are energy detection, blind source separation, singular value ratio spectrum method, time-frequency domain filtering method and fractional Fourier domain filtering method respectively.

[0161] from Figure 3 and Figure 4 As can be seen from the figure, under the same experimental conditions, the reinforcement learning GPPO model shows a significant advantage in terms of stability. In contrast, the variance of the PPO model is large, while the variance of the reinforcement learning GPPO model is extremely small.

[0162] At the same time, the reward values ​​of the reinforcement learning GPPO model continue to increase, indicating that its accuracy in making decisions with robustness to interference is also gradually improving. This is attributed to the integration of gated recurrent units in the policy and value networks, which enables the algorithm to incorporate historical observations, thereby achieving more consistent performance estimates.

[0163] In other words, the reinforcement learning GPPO model in this paper achieved higher reward values ​​than the standard PPO model, demonstrating its superior decision-making ability in dynamic jamming scenarios. This advancement is crucial for the future of radar security and provides a powerful strategy for responding to complex jamming threats.

[0164] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A radar anti-interference decision generation method based on experience replay and policy gradient, characterized in that: include: Build an experience replay pool; Build a reinforcement learning GPPO model; Use the experience replay pool to train the reinforcement learning GPPO model to obtain a trained reinforcement learning GPPO model; The trained reinforcement learning GPPO model is used to generate the optimal anti-interference strategy for the interference signal to be suppressed.

2. The radar anti-interference decision generation method based on experience replay and policy gradient according to claim 1 is characterized in that: Build an experience replay pool, including: The radar system is considered as an intelligent agent in deep reinforcement learning, the interference signal is used as the state, the anti-interference strategy is used as the action, and the effectiveness evaluation of the anti-interference strategy is used as the reward. The radar anti-interference environment is mapped to the deep reinforcement learning environment to obtain an online anti-interference environment. Use the radar system as an intelligent agent to interact with the online anti-interference environment to obtain an experience revisit pool; Preferably, the interference signals include distance deception interference DDJ, coherent suppression interference CSJ, spectrum dispersion interference SMSPJ, intermittent sampling repetitive cycle interference ISCJ and dense false target interference DFTJ, and the anti-interference strategy is used to represent the blind source separation method, singular value ratio spectrum method, energy detection method, time-frequency domain filtering method and fractional Fourier domain filtering method in the anti-interference method library.

3. The radar anti-interference decision generation method based on experience replay and policy gradient according to claim 2 is characterized in that: The online anti-interference environment is modeled as a Markov decision process, which is represented by a five-tuple (S, A, P, R, γ), where S is the state set, A is the action set, P is the state transition probability, and γ is the discount factor.

4. The radar anti-interference decision generation method based on experience replay and policy gradient according to claim 3 is characterized in that: Use the radar system as an intelligent agent to interact with the online anti-interference environment to obtain an experience revisit pool, specifically including: At each moment, the interference signal in the interference signal library of the radar system is obtained, and the interference signal is identified using a preset interference identification model to obtain the interference identification result at each moment; The interference recognition result at each moment is regarded as the interference state at each moment. According to the interference state at each moment and the observation equation, anti-interference result equation and reward function of the Markov decision process, the observation result, anti-interference completion mark and reward at each moment are obtained. Based on the observation results and completion flag of the interference signal at each moment and the observation results, actions and rewards of the interference signal at the previous moment, an experience tuple is constructed; where the experience tuple is represented as<o,a,r,o',done> , o represents the observation result of the interference signal at the previous moment, a represents the action, r represents the reward, o' represents the observation result of the interference signal at the current moment, and done represents the completion flag; Based on the experience tuple, an experience replay pool is obtained.

5. The radar anti-interference decision generation method based on experience replay and policy gradient according to claim 4 is characterized in that: The interference recognition model is a pre-trained two-dimensional convolutional neural network for identifying interference category labels of interference signals; wherein the two-dimensional convolutional neural network includes four layers of convolution.

6. The radar anti-interference decision generation method based on experience replay and policy gradient according to claim 1 is characterized in that: The reinforcement learning GPPO model includes a policy network and a value network; The value network is defined as a neural network whose basic structure is a gated recurrent unit; in the value network, Where V π (s) represents the maximization value function, γ represents the discount factor, E[·] represents the expected value operation, π represents the optimization goal of the value network, that is, the optimal anti-interference strategy, T represents the total time step, t represents the moment, k represents the time step, r k represents the reward value obtained at time step k, s represents the interference state, s t represents the interference state at time t; Among them, the policy network is defined as a neural network whose basic structure is the gated recurrent unit (GRU); in the policy network, Where Q π (s,a) represents the action value function, γ represents the discount factor, E[·] represents the expected value operation, T represents the total time step, t represents the moment, k represents the time step, r k represents the reward value obtained at time step k, a represents the action, a t represents the action at time t; Among them, the advantage function of the reinforcement learning GPPO model is A π (s,a)=Q π (s,a)-V π (s,a); where A π (s,a) represents the advantage function; Preferably, the reinforcement learning GPPO model is constructed, including: The gated recurrent unit is integrated into the policy network and value network of the proximal policy optimization algorithm to obtain the reinforcement learning GPPO model.

7. The radar anti-interference decision generation method based on experience replay and policy gradient according to claim 1 is characterized in that: Use the experience replay pool to train the reinforcement learning GPPO model to obtain a trained reinforcement learning GPPO model, specifically including: Randomly extract a batch of experience tuples from the experience replay pool D; Input the experience tuple into the reinforcement learning GPPO model, calculate the gradient and update the parameters of the policy network and the value network in the reinforcement learning GPPO model; Based on the updated policy network, the optimal anti-interference strategy a' is selected from the pre-built anti-interference method library; based on the updated value network and the optimal anti-interference strategy a', a new interference state is generated; Based on the new interference state, determine whether the reinforcement learning GPPO model meets the preset convergence conditions; If so, output the trained reinforcement learning GPPO model; If not, then based on the new interference state and the observation equation, anti-interference result equation and reward function of the Markov decision process, new observation results, anti-interference completion flag and reward are obtained; Based on the new observation results, anti-interference completion flag and reward, a new batch of experience tuples is called and the reinforcement learning GPPO model is iteratively trained until the reinforcement learning GPPO model reaches the preset convergence condition; Preferably, the parameters of the policy network and the value network are updated as follows: Update the parameters of the policy network by optimizing the objective function of the policy network; The parameters of the value network are updated by minimizing the loss function of the value network.

8. A radar anti-interference decision generation system based on experience replay and policy gradient, characterized in that: include: Experience replay pool construction module, used to build the experience replay pool; Model building module, used to build reinforcement learning GPPO model; The training model is used to train the reinforcement learning GPPO model using the experience replay pool to obtain a trained reinforcement learning GPPO model; The strategy generation module is used to generate the optimal anti-interference strategy for the interference signal to be suppressed using the trained reinforcement learning GPPO model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the radar anti-interference decision-making method based on experience replay and policy gradient are implemented as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the radar anti-interference decision-making method based on experience replay and policy gradient are implemented as described in any one of claims 1 to 7.

Images