Adaptive generation method and device for cognitive radar anti-interference strategy based on PPO

Abstract

The present invention discloses a method and device for adaptively generating an anti-interference strategy for a cognitive radar based on PPO, wherein the method comprises: obtaining an echo spectrum and inputting it into an anti-interference strategy selection model to obtain anti-interference parameters; the model is trained by: constructing a cognitive radar anti-interference model, wherein the cognitive radar model continuously transmits radar pulses with variable frequency, and the jammer model intercepts the radar pulses and releases interference; using the PPO algorithm to train an actor network and a commentator network; the actor network outputs the expectation and standard deviation of the behavior based on the echo spectrum; the behavior corresponds one-to-one with the center frequency of the radar pulse; the commentator network outputs the reward value based on the echo spectrum; and constructing an anti-interference strategy selection model based on the trained actor network. The present invention solves the problem that existing anti-interference strategies are overly dependent on rules and experience and are difficult to meet the anti-interference requirements in complex electromagnetic interference environments, enabling cognitive radars to quickly and accurately achieve anti-interference.

Description

Technical Field

[0001] The present invention belongs to the field of radar technology, and in particular relates to a method and device for adaptively generating an anti-interference strategy for a cognitive radar based on PPO (Proximal Policy Optimization). Background Art

[0002] Cognitive radar, which autonomously improves its performance by interacting with its environment, is widely used in increasingly complex cognitive electronic warfare. Its anti-interference capabilities are also constantly being tested. This capability is determined by interference recognition, anti-interference strategy selection, and interference suppression. Anti-interference strategy selection refers to the radar's ability to select the appropriate anti-interference strategy when faced with various interference strategies.

[0003] However, in a complex real-world confrontation environment, even if cognitive radar has a high ability to select appropriate anti-interference strategies, it is still difficult for cognitive radar to achieve anti-interference quickly and accurately. Summary of the Invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a method and device for adaptively generating an anti-interference strategy for a cognitive radar based on PPO.

[0005] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] A method for adaptively generating an anti-interference strategy for a cognitive radar based on PPO, comprising:

[0007] Acquire a first echo spectrum; wherein the first echo spectrum is an echo spectrum of a cognitive radar;

[0008] inputting the first echo spectrum into an anti-interference strategy selection model, so that the anti-interference strategy selection model adapts and outputs anti-interference parameters according to the first echo spectrum;

[0009] The anti-interference strategy selection model is trained in the following way:

[0010] Constructing a cognitive radar anti-interference model; wherein the cognitive radar anti-interference model includes a jammer model and a cognitive radar model; the cognitive radar model is configured to continuously transmit radar pulses with variable frequency; the jammer model is configured to intercept the radar pulses transmitted by the cognitive radar model and release interference of an adaptive type and adaptive parameters based on the intercepted radar pulses;

[0011] Constructing an actor network and a commentator network; wherein the actor network is used to output the expectation and standard deviation of the behavior implemented by the cognitive radar model based on a second echo spectrum; and the commentator network is used to output the reward value that the cognitive radar model can obtain by implementing the behavior based on the second echo spectrum; wherein the second echo spectrum is the echo spectrum obtained by the jammer model intercepting the radar pulse transmitted by the cognitive radar model; the behavior implemented by the cognitive radar model and the center frequency of the radar pulse it transmits are in one-to-one correspondence;

[0012] The actor network and the commentator network are trained using the PPO algorithm, and the anti-interference strategy selection model is constructed based on the trained actor network.

[0013] Optionally, the jammer model is specifically used to:

[0014] During a first duration of each working cycle, radar pulses are continuously intercepted and stored; during a second duration of each working cycle, adaptive interference is released according to the intercepted radar pulses; the length of the working cycle is equal to the sum of the first duration and the second duration;

[0015] Among them, if the continuously intercepted N cap If the average center frequency of each radar pulse is interfered by the interference released in the previous working cycle, then in the subsequent working cycle, the first duration is reduced by N with a probability of γ1. step radar pulse duration, and increasing the second duration by N step Radar pulse duration; where N cap is an integer greater than 1, N step Adjust the steps for duration.

[0016] Optionally, the jammer model releases adaptive jamming according to the intercepted radar pulse, including:

[0017] If the continuously intercepted N cap The range of the center frequency of a radar pulse is less than B tar , then the release center frequency is Bandwidth is B tar The aiming interference; among them, B tar is the aiming interference bandwidth; f c is the center frequency of the radar pulse emitted by the cognitive radar model;

[0018] If the continuously intercepted N cap The range of the center frequency of each radar pulse is not less than B tar , then the release center frequency is Bandwidth is B blk The blocking interference ofblk is the blocking interference bandwidth.

[0019] Optionally, the cognitive radar model emits radar pulses according to time steps; and the training of the actor network and the commentator network using the PPO algorithm includes:

[0020] In each time step i, the current second echo spectrum is used as the state s at the time step i i are respectively input into the actor network so that the actor network outputs the expectation and standard deviation of the behavior implemented by the cognitive radar model; a normal distribution behavior probability density function is constructed according to the expectation and the standard deviation, and a behavior a is randomly extracted based on the probability density function. i ; for behavior a i Conduct interference suppression effect evaluation to get reward r i , and determine the state s of the next time step i+1 i+1 ;

[0021] The state s of each time step i is i 、Behavior a i , reward r i and the state s at the next time step i+1 i+1 As a state combination (s i ,a i ,r i ,s i+1 ) is stored in the experience buffer; the experience buffer can store up to N cap state combinations;

[0022] A state combination is randomly extracted from the experience buffer, and the actor network and the critic network are trained according to the extracted state combination.

[0023] Optionally, training the actor network and the critic network according to the extracted state combination includes:

[0024] Calculating a first loss of the actor network using a first loss function and a second loss of the critic network using a second loss function according to the extracted state combination and the reward value output by the critic network;

[0025] The second loss function is:

[0026] L2(ω)=(r i +γ2V ω (s i+1 )-V ω (s i )) 2 ;

[0027] Among them, γ2 represents the learning rate, ω represents the network weight parameter of the critic network, V ω (s i ) is the input state s to the commentator network with the network weight parameter ω i When the critic network outputs the reward value, V ω (s i+1 ) is the input state s to the commentator network with the network weight parameter ω i+1 When , the reward value output by the commentator network, L2(ω) is the second loss of the commentator network with the network weight parameter ω;

[0028] The first loss function is:

[0029]

[0030] in, Indicates the gradient, log indicates the logarithm, p β (a i |s i ) is the cognitive radar model in state s i When choosing behavior a i The probability, p β (a i |s i ) is calculated using the target probability density function, which is the actor network with a network weight parameter β and an input of s i The behavior probability density function is constructed by the expectation and standard deviation of the corresponding output; L1(β) is the first loss of the actor network with the network weight parameter β.

[0031] Optionally, randomly extracting behavior a based on the probability density function i ,include:

[0032] Generate random numbers and calculate dynamic exploration rate;

[0033] If the dynamic exploration rate is greater than the random number, the probability density function is used to randomly extract behavior a i ;

[0034] If the dynamic exploration rate is not greater than the random number, the random selection behavior a i ;

[0035] The calculation formula of the dynamic exploration rate is:

[0036] ε=ε end +(ε start -ε end )×exp(-i / 64);

[0037] Among them, εstart is the preset initial exploration rate, ε end is the preset termination exploration rate, exp(·) is the natural exponential function, and ε is the dynamic exploration rate.

[0038] The present invention also provides a PPO-based cognitive radar anti-interference strategy adaptive generation device, comprising: an echo acquisition module and an anti-interference strategy selection model;

[0039] The echo acquisition module is used to acquire a first echo spectrum; wherein the first echo spectrum is an echo spectrum of a cognitive radar;

[0040] The anti-interference strategy selection model is used to adapt and output an anti-interference strategy according to the first echo spectrum;

[0041] The anti-interference strategy selection model is trained in the following way:

[0042] Constructing a cognitive radar anti-interference model; wherein the cognitive radar anti-interference model includes a jammer model and a cognitive radar model; the cognitive radar model is configured to continuously transmit radar pulses with variable frequency; the jammer model is configured to intercept the radar pulses transmitted by the cognitive radar model and release interference of an adaptive type and adaptive parameters based on the intercepted radar pulses;

[0043] Constructing an actor network and a commentator network; wherein the actor network is used to output the expectation and standard deviation of the behavior implemented by the cognitive radar model based on a second echo spectrum; and the commentator network is used to output the reward value that the cognitive radar model can obtain by implementing the behavior based on the second echo spectrum; wherein the second echo spectrum is the echo spectrum obtained by the jammer model intercepting the radar pulse transmitted by the cognitive radar model; the behavior implemented by the cognitive radar model and the center frequency of the radar pulse it transmits are in one-to-one correspondence;

[0044] The actor network and the commentator network are trained using the PPO algorithm, and the anti-interference strategy selection model is constructed based on the trained actor network.

[0045] Optionally, the jammer model is specifically used to:

[0046] Continuously detecting and storing radar pulses during a first duration of each operating cycle; and releasing adaptive interference based on the intercepted radar pulses during a second duration of each operating cycle; wherein the length of the operating cycle is equal to the sum of the first duration and the second duration;

[0047] Among them, if the continuously intercepted N capIf the average center frequency of each radar pulse is interfered by the interference released in the previous working cycle, then in the subsequent working cycle, the first duration is reduced by N with a probability of γ1. step radar pulse duration, and increasing the second duration by N step Radar pulse duration; where N cap is an integer greater than 1, N step Adjust the steps for duration.

[0048] Optionally, the jammer model releases adaptive jamming according to the intercepted radar pulse, including:

[0049] If the continuously intercepted N cap The range of the center frequency of a radar pulse is less than B tar , then the release center frequency is Bandwidth is B tar The aiming interference; among them, B tar is the aiming interference bandwidth; f c is the center frequency of the radar pulse emitted by the cognitive radar model;

[0050] If N is intercepted continuously cap The range of the center frequency of each radar pulse is not less than B tar , then the release center frequency is Bandwidth is B blk The blocking interference of blk is the blocking interference bandwidth.

[0051] Optionally, the cognitive radar model emits radar pulses according to time steps; and the training of the actor network and the commentator network using the PPO algorithm includes:

[0052] In each time step i, the current second echo spectrum is used as the state s at the time step i i are respectively input into the actor network so that the actor network outputs the expectation and standard deviation of the behavior implemented by the cognitive radar model; a normal distribution behavior probability density function is constructed according to the expectation and the standard deviation, and a behavior a is randomly extracted based on the probability density function. i ; for behavior a i Conduct interference suppression effect evaluation to get reward r i , and determine the state s of the next time step i+1 i+1 ;

[0053] The state s of each time step i is i 、Behavior a i , reward r i and the state s at the next time step i+1 i+1 As a state combination (si ,a i ,r i ,s i+1 ) is stored in the experience buffer; the experience buffer can store up to N cap state combinations;

[0054] A state combination is randomly extracted from the experience buffer, and the actor network and the critic network are trained according to the extracted state combination.

[0055] The PPO-based adaptive generation method for cognitive radar anti-interference strategies provided by the present invention has the following beneficial effects:

[0056] (1) In the present invention, the first echo spectrum of the cognitive radar is directly input into the anti-interference strategy selection model, and the anti-interference strategy selection model can adaptively output anti-interference parameters according to the first echo spectrum, ensuring that the cognitive radar can quickly and accurately achieve anti-interference. Specifically, when training the anti-interference strategy selection model, a cognitive radar anti-interference model including a jammer model and a cognitive radar model was constructed; the cognitive radar model was configured to continuously transmit radar pulses with variable frequency; the jammer model was configured to intercept the radar pulses emitted by the cognitive radar model and release interference with adaptation type and adaptation parameters according to the intercepted radar pulses; on this basis, an actor network and a commentator network were constructed, and the actor network was used to output the expectation and standard deviation of the behavior implemented by the cognitive radar model according to the second echo spectrum; the commentator network was used to output the reward value that can be obtained by implementing the behavior of the cognitive radar model according to the second echo spectrum; thus, the actor network and the commentator network were trained using the PPO algorithm, and an anti-interference strategy selection model was constructed based on the trained actor network. When the cognitive radar model implements anti-interference by emitting radar pulses, a better center frequency can be selected as the anti-interference parameter for the radar pulses emitted by the cognitive radar model, so that the cognitive radar can quickly and accurately achieve anti-interference.

[0057] (2) When modeling an electromagnetic countermeasure environment, the present invention provides two operating modes for the jammer model: a reconnaissance mode that continuously intercepts and stores radar pulses, and a jamming mode that releases adaptive jamming based on the intercepted radar pulses. As a result, the environmental modeling of the present invention is closer to a real-world countermeasure environment, with a smaller difference from the real-world countermeasure environment, making it suitable for application in real-world countermeasure environments.

[0058] (3) The present invention sets the action space of the cognitive radar model to the center frequency of the radar pulse, realizing the selection of anti-interference parameters for different interference strategies.

[0059] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Schematic diagram of the training process of the anti-interference strategy selection model provided by the present invention;

[0061] Figure 2 It is a schematic diagram of the jammer model change working mode provided by the present invention;

[0062] Figure 3 It is a schematic diagram of the interaction between the jammer model and the cognitive radar model provided by the present invention;

[0063] Figure 4 Schematic diagram of the anti-interference effect of Example 1 provided by the present invention;

[0064] Figure 5 Schematic diagram of the anti-interference effect of Example 2 provided by the present invention;

[0065] Figure 6 It is a schematic diagram of the anti-interference effect of Example 3 provided by the present invention. DETAILED DESCRIPTION

[0066] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0067] To address the problem that existing anti-interference strategies rely too much on rules and experience and are unable to meet the anti-interference requirements in complex electromagnetic interference environments, and to enable cognitive radar to quickly and accurately implement anti-interference, an embodiment of the present invention provides a method for adaptively generating an anti-interference strategy for a cognitive radar based on PPO. The method includes the following steps:

[0068] Step 1: Obtain a first echo spectrum; wherein the first echo spectrum is an echo spectrum of a cognitive radar.

[0069] Step 2: input the first echo spectrum into the anti-interference strategy selection model, so that the anti-interference strategy selection model adapts and outputs anti-interference parameters according to the first echo spectrum.

[0070] Here, the anti-interference parameter specifically refers to the center frequency of the radar pulse emitted by the cognitive radar.

[0071] In the embodiment of the present invention, the anti-interference strategy selection model is obtained by pre-training, and the training process includes the following steps:

[0072] (1) Constructing a cognitive radar anti-interference model; wherein the cognitive radar anti-interference model includes a jammer model and a cognitive radar model; the cognitive radar model is configured to continuously transmit radar pulses with variable frequency; the jammer model is configured to intercept the radar pulses transmitted by the cognitive radar model and release interference with adaptive types and adaptive parameters according to the intercepted radar pulses.

[0073] In the embodiment of the present invention, the cognitive radar model is modeled as a pulse-frequency agile cognitive radar, which continuously transmits a central frequency of f c Linear frequency modulated pulse with bandwidth B, f c ∈[f start ,f end ], f start Indicates the minimum operating frequency of the radar, f end Indicates the highest operating frequency, the minimum step size of frequency agility is f step The linear frequency modulated pulse emitted by the cognitive radar model can be characterized as:

[0074]

[0075] Where t is the time variable, Represents the rectangular window signal, T represents the signal pulse width, f c is the center frequency of the radar pulse emitted by the cognitive radar model, B is the bandwidth of the radar pulse, exp(·) is the exponential function, j is the sign of the imaginary part, and Sig is the linear frequency modulation pulse emitted by the cognitive radar model.

[0076] The jammer model is specifically used to: continuously intercept and store radar pulses during the first duration of each working cycle; and release adaptive jamming according to the intercepted radar pulses during the second duration of each working cycle; the length of the working cycle is equal to the sum of the first duration and the second duration.

[0077] For example, see Figure 2 , assuming that when the jammer mode is initialized, the length of its working cycle is set to be equal to N per radar pulse length, then its working time in reconnaissance mode is N l The radar pulse length is N, and its working time in jamming mode is j Radar pulse length, N per =N l +N j .

[0078] It's understandable that the jammer model has two operating modes: reconnaissance and jamming. During the first duration of each operating cycle, the jammer model operates in reconnaissance mode; in reconnaissance mode, it continuously intercepts and caches radar pulses emitted by the cognitive radar model. During the second duration of each operating cycle, the jammer model operates in jamming mode; in jamming mode, it releases blocking or targeting jamming at different frequencies based on the radar pulses intercepted during the first duration. This allows the environmental modeling of the cognitive radar anti-jamming model to more closely resemble a realistic adversarial environment.

[0079] Among them, if the jammer model continuously intercepts N capThe average center frequency of each radar pulse is interfered by the interference released in the previous working cycle, which means that the interference released by the jammer model before can effectively interfere with the cognitive radar model. Therefore, in the subsequent working cycle, the first duration is reduced by N with a probability of γ1. step radar pulse duration, and increase the second duration by N step Radar pulse duration; where N cap is an integer greater than 1, N step Adjust the steps for duration.

[0080] It can be understood that reducing the first duration and increasing the second duration with a probability of γ1 means that this will be done γ1×100 times out of 100 times, rather than reducing the first duration and increasing the second duration every time, 0<γ1<1.

[0081] Therefore, the duration of the jammer model's detection and interference on the cognitive radar model can be adjusted in real time according to the jamming effect.

[0082] The cache space of the jammer model can store a maximum of N cap radar pulses, the corresponding jammer model releases adaptive jamming according to the intercepted radar pulses, including: if N consecutive intercepted cap The range of the center frequency of a radar pulse is less than B tar , then the release center frequency is Bandwidth is B tar The aiming interference; among them, B tar is the aiming interference bandwidth; f c is the center frequency of the radar pulse emitted by the cognitive radar model; if N cap The range of the center frequency of each radar pulse is not less than B tar , then the release center frequency is Bandwidth is B blk The blocking interference of blk is the blocking interference bandwidth.

[0083] Therefore, the interference emitted by the jammer model can be variable in not only the type of interference but also the frequency.

[0084] Among them, the interference signal released by the jammer model in the jamming mode can be expressed by the following formula:

[0085]

[0086] Among them, U0 represents the modulation signal amplitude, f j represents the interference center frequency, represents the initial phase that obeys the uniform distribution [0,2π], U n(t) represents amplitude modulated noise, WGN represents environmental noise, and Jam is the interference signal emitted by the jammer model, which is a noise amplitude modulated signal.

[0087] (2) Construct an actor network and a commentator network; wherein, the actor network is used to output the expectation and standard deviation of the behavior implemented by the cognitive radar model based on the second echo spectrum; the commentator network is used to output the reward value that can be obtained by the cognitive radar model implementing the behavior based on the second echo spectrum; wherein, the second echo spectrum is the echo spectrum obtained by the jammer model intercepting the radar pulse emitted by the cognitive radar model; the behavior implemented by the cognitive radar model corresponds to the center frequency of the radar pulse it emits; that is, the behavior implemented by the cognitive radar model is defined as the center frequency of the radar pulse emitted by the cognitive radar model.

[0088] In practice, both the actor network and the commentator network can be implemented by constructing a CNN (convolutional neural network), which is not limited in this embodiment of the present invention.

[0089] (3) The actor network and the critic network are trained using the PPO algorithm, and an anti-interference strategy selection model is constructed based on the trained actor network.

[0090] Specifically, the cognitive radar model emits radar pulses according to time steps. Accordingly, the PPO algorithm is used to train the actor network and the commentator network, including:

[0091] (3-1) In each time step i, the current second echo spectrum is taken as the state s at the time step i. i Input them into the actor network respectively, so that the actor network outputs the expectation and standard deviation of the behavior implemented by the cognitive radar model; construct the normal distribution behavior probability density function according to the expectation and standard deviation, and randomly extract behavior a based on the probability density function i ; for behavior a i Conduct interference suppression effect evaluation to get reward r i , and determine the state s of the next time step i+1 i+1 .

[0092] like Figure 2 As shown in Figure 1, at each time step i, the actor network receives the second echo spectrum as its state, then generates an action based on the transmitted radar pulse and generates a corresponding reward. For the jammer model, if it is operating in reconnaissance mode, it receives and stores the radar pulse without releasing the jamming signal. At this time, the stored radar pulse is used as the state for the next time step. If the jammer is operating in jamming mode, it releases the jamming signal according to the jamming strategy set for the jammer model in the cognitive radar anti-jamming model. The new echo generated by the superposition of the jamming signal and the radar pulse is used as the state for the next time step.

[0093] Therefore, after defining the states and behaviors in the training process, we can combine the Markov idea and use reinforcement learning to model the anti-interference strategy generation process as a mapping process of reinforcement learning agents, and use the PPO algorithm to train the actor network and the commentator network.

[0094] In one implementation, the behavior a is randomly selected based on the probability density function. i , which may include:

[0095] (a1) Generate random numbers and obtain a pre-configured exploration rate;

[0096] (b1) If the pre-configured exploration rate is greater than the random number, the probability density function is used to randomly extract behavior a i ;

[0097] (c1) If the pre-configured exploration rate is not greater than the random number, the random sampling behavior a i .

[0098] In another implementation, in order to improve the convergence speed when training the model, the behavior a is randomly extracted based on the probability density function. i , which may include:

[0099] (a2) Generate random numbers and calculate the dynamic exploration rate;

[0100] (b2) If the dynamic exploration rate is greater than the random number, use the probability density function to randomly extract behavior a i ;

[0101] (c2) If the dynamic exploration rate is not greater than the random number, randomly select behavior a i .

[0102] The calculation formula for the above dynamic exploration rate is:

[0103] ε=ε end +(ε start -ε end )×exp(-i / 64);

[0104] Among them, ε start is the preset initial exploration rate, ε end is the preset termination exploration rate, exp(·) is the natural exponential function, ε is the dynamic exploration rate, and i represents the time step.

[0105] In this implementation, an improved ε-greedy strategy is used to provide a wide and accurate behavior space, which can accelerate the convergence of the model.

[0106] In the embodiment of the present invention, the behavior a i Conduct interference suppression effect evaluation to get reward r i, can be set according to whether the cognitive radar model is successfully jammed and by what kind of jamming signal. For example, if the cognitive radar model is not jammed by behavior a i The corresponding interference signal interferes with the reward r i =5, if the cognitive radar model is a i The corresponding blocking interference signal is interfered with, then the reward r i = -5, if the cognitive radar model is a i The corresponding aiming interference signal is interfered with, then the reward r i =-10.

[0107] (3-2) The state s of each time step i i 、Behavior a i , reward r i and the state s at the next time step i+1 i+1 As a state combination (s i ,a i ,r i ,s i+1 ) is stored in the experience buffer; the experience buffer can store up to N cap state combinations;

[0108] Specifically, a first-in-first-out strategy can be used to store state combinations (s i ,a i ,r i ,s i+1 ).

[0109] (3-3) Randomly extract state combinations from the experience buffer and train the actor network and critic network based on the extracted state combinations.

[0110] In this embodiment of the present invention, a random sampling strategy can be used to extract state combinations from the experience buffer to train the actor and critic networks. It will be appreciated that with this experience buffer, there is no need to evaluate the loss of the actor and critic networks and adjust network parameters at every time step. Instead, a large number of state combinations can be generated in the early stages, and then the loss of the actor and critic networks and the network parameters can be evaluated and adjusted based on these large number of generated state combinations, which can improve training efficiency.

[0111] When modeling the anti-interference strategy generation process as a Markov process, it is hoped to quantify the reward value obtained from the behavior implemented by the cognitive radar model. In theory, this reward value can be calculated using a state value function, which is expressed as:

[0112]

[0113] Among them, γ2 represents the learning rate, π represents the strategy of selecting the implementation behavior in the state, V π (·) indicates the return value; s i is the state of the current time step i, r i refers to the reward at the current time step i, Ε π [·] represents the expected return value, S represents the state space, A represents the action (behavior) space, π(a|s) represents the probability that strategy π chooses to implement behavior a in state s, r(s,a) represents the reward obtained by choosing to implement behavior a in state s, and p(s'|s,a) represents the probability that the next state is s' after choosing to implement behavior a in state s.

[0114] However, in practice, it is not possible to obtain V π The specific value of (s') is determined by the commentator network. Therefore, in this embodiment of the present invention, this reward value is quantified using the commentator network. Specifically, the commentator network is constructed to predict the reward value of the current state based on the input state (the second echo spectrum). To ensure that the commentator network accurately predicts the reward value, a second loss function is used to evaluate the commentator network's prediction performance, thereby achieving network convergence. Simultaneously, the first loss function is used to evaluate the actor network's prediction performance.

[0115] Specifically, the actor network and the critic network are trained according to the extracted state combinations, including: calculating a first loss of the actor network using a first loss function according to the extracted state combinations and the reward value output by the critic network, and calculating a second loss of the critic network using a second loss function.

[0116] The second loss function mentioned above is:

[0117] L2(ω)=(r i +γ2V ω (s i+1 )-V ω (s i )) 2 ;

[0118] Among them, γ2 represents the learning rate, ω represents the network weight parameter of the critic network, V ω (s i ) is the input state s to the commentator network with the network weight parameter ω i When the reward value output by the critic network is ω (s i+1 ) is the input state s to the commentator network with the network weight parameter ω i+1 When , the reward value output by the critic network, L2(ω) is the second loss of the critic network with the network weight parameter ω.

[0119] The first loss function mentioned above is:

[0120]

[0121] in, Indicates the gradient, log indicates the logarithm, p β (a i |s i ) is the cognitive radar model in state s i When choosing behavior a i The probability, p β (a i |s i ) is calculated using the target probability density function, which is based on the actor network with network weight parameter β and input s i The behavior probability density function is constructed by the expectation and standard deviation of the corresponding output; L1(β) is the first loss of the actor network with the network weight parameter β.

[0122] Therefore, based on the above two loss functions, the back-propagation algorithm is used to continuously update the actor network and the critic network until both losses converge below the corresponding threshold or the iteration is stopped when a sufficient number of iterations is reached.

[0123] Then, an anti-interference strategy selection model is constructed based on the trained actor network. Specifically, the anti-interference strategy selection model includes the trained actor network and a behavior extraction module. The trained actor network is used to output the expected and standard deviation of the cognitive radar's behavior based on the first echo spectrum. The behavior extraction module constructs a normally distributed behavior probability density function based on this expectation and standard deviation, randomly extracts behaviors based on this probability density function, determines the corresponding anti-interference parameters based on these behaviors, and outputs the anti-interference parameters. This enables the selection of anti-interference parameters for different interference strategies.

[0124] In the PPO-based adaptive generation method for cognitive radar anti-interference strategies provided in an embodiment of the present invention, the first echo spectrum of the cognitive radar is directly input into the anti-interference strategy selection model. The anti-interference strategy selection model can then adaptively output anti-interference parameters based on the first echo spectrum, ensuring that the cognitive radar can quickly and accurately achieve anti-interference. Specifically, when training the anti-interference strategy selection model, a cognitive radar anti-interference model including a jammer model and a cognitive radar model was constructed; the cognitive radar model was configured to continuously transmit radar pulses with variable frequency; the jammer model was configured to intercept the radar pulses emitted by the cognitive radar model and release interference with adaptation type and adaptation parameters according to the intercepted radar pulses; on this basis, an actor network and a commentator network were constructed, and the actor network was used to output the expectation and standard deviation of the behavior implemented by the cognitive radar model according to the second echo spectrum; the commentator network was used to output the reward value that can be obtained by implementing the behavior of the cognitive radar model according to the second echo spectrum; thus, after using the PPO algorithm to train the actor network and the commentator network, an anti-interference strategy selection model was constructed based on the trained actor network, so that when the cognitive radar model implements anti-interference by emitting radar pulses, a better center frequency can be selected as the anti-interference parameter for the radar pulses emitted by the cognitive radar model, so that the cognitive radar can quickly and accurately achieve anti-interference.

[0125] When modeling an electromagnetic countermeasure environment, the embodiment of the present invention sets two operating modes for the jammer model, including a reconnaissance mode that continuously intercepts and stores radar pulses, and an interference mode that releases adaptive interference based on the intercepted radar pulses. As a result, the environmental modeling of the present invention is closer to the real countermeasure environment, with a smaller gap from the real countermeasure environment, making it suitable for application in real countermeasure environments. The embodiment of the present invention sets the action space of the cognitive radar model to the center frequency of the radar pulse, realizing the selection of anti-interference parameters for different interference strategies.

[0126] The beneficial effects of the embodiments of the present invention are further illustrated below through simulation experiments.

[0127] Constructing a cognitive radar anti-interference model: The cognitive radar model's radar transmit pulses have a center frequency of 8 GHz to 10 GHz, a minimum frequency step of 15.625 MHz, and a bandwidth of 40 MHz. The jammer model operates in two modes: reconnaissance and jamming. In reconnaissance mode, it intercepts all radar signals in the 8 GHz to 10 GHz frequency band. In jamming mode, it transmits targeting jamming or blocking jamming based on the radar signals intercepted in reconnaissance mode. If the center frequency of the intercepted radar signal varies within 160 MHz, the model transmits targeting jamming with a bandwidth of 160 MHz, centered around the average frequency of the intercepted radar signals. Otherwise, the model transmits blocking jamming with a bandwidth of 800 MHz, centered around the average frequency of the intercepted radar signals.

[0128] Example 1: Based on the above-mentioned cognitive radar anti-interference model, an actor network and a commentator network are constructed. The actor network and the commentator network are trained using the PPO algorithm according to the training method given in the embodiment of the present invention. An anti-interference strategy selection model is constructed based on the trained actor network. The anti-interference strategy selection model is used to perform anti-interference evaluation. The effect is as follows: Figure 4 shown.

[0129] from Figure 4 It can be seen that the radar pulses generated by the generated radar anti-interference strategy can effectively deceive the jammer. The jammer gradually increases the working time in reconnaissance mode to obtain more radar pulse information, but it still cannot avoid being deceived by the above strategy. At the same time, the strategy is mature. Only two frequency points are exposed during the radar operation. The jammer only releases aiming interference and the radar pulses are not interfered with.

[0130] Example 2 is a comparative example: Based on the above-mentioned cognitive radar anti-interference model, the DQN algorithm is used to train the Q network. Among them, the DQN algorithm is a reinforcement learning algorithm based on deep learning, and its full name is Deep Q-Network. Specifically, the second echo spectrum is input as the state into the Q network, so that the Q network directly outputs the behavior, and the center frequency of the radar pulse emitted by the cognitive radar model is modified according to the behavior. After that, the reward is calculated by judging whether the cognitive radar model is interfered, and then the next state is entered; similarly, the state combination (s i ,a i ,r i ,s i+1 ), and stored in the experience buffer, the state combination stored in the experience buffer is used to train the Q network, and the trained Q network is used as the anti-interference strategy selection model for anti-interference evaluation. The effect is as follows Figure 5 shown.

[0131] from Figure 5As can be seen in the figure, the radar pulses generated by the generated strategy have a tendency to deceive the jammer, causing the jammer to gradually increase the time it operates in reconnaissance mode to obtain more radar pulse information and avoid being deceived by the above strategy. Due to the immaturity of the strategy, the radar is exposed to a large number of operating frequencies, so the jammer releases targeting jamming and blocking jamming to interfere with the radar pulses, and a small number of radar pulses are blocked by the jammer.

[0132] Example 3: Using the jammer model in the above cognitive radar anti-interference model, we conduct an anti-interference experiment on the random frequency agile cognitive radar. At each time step i, the random frequency agile cognitive radar is in the range [f i-1 -400MHz,f i-1 +400MHz] randomly select the center frequency of the radar pulse, f i-1 Represents the center frequency of the radar pulse in the previous time step. The anti-interference effect of this random frequency agile cognitive radar is as follows: Figure 6 As shown. Figure 6 It can be seen that the random frequency agile cognitive radar is frequently interfered by the blocking interference emitted by the jammer model. It can be seen that under the interference of the same jammer model, the cognitive radar adopting the anti-interference strategy selection model of the embodiment of the present invention has better anti-interference capability.

[0133] In summary, the PPO-based adaptive generation method for cognitive radar anti-interference strategies provided by the embodiments of the present invention outperforms existing anti-interference strategy generation algorithms in terms of strategy generation speed and quality.

[0134] The method provided in the embodiments of the present invention can be applied to electronic devices. Specifically, the electronic devices can be desktop computers, portable computers, smart mobile terminals, servers, etc. This is not limited here; any electronic device that can implement the present invention falls within the scope of protection of the present invention.

[0135] Based on the same inventive concept, an embodiment of the present invention further provides a device for adaptively generating an anti-interference strategy for a cognitive radar based on PPO, comprising: an echo acquisition module and an anti-interference strategy selection model;

[0136] An echo acquisition module, configured to acquire a first echo spectrum; wherein the first echo spectrum is an echo spectrum of a cognitive radar;

[0137] an anti-interference strategy selection model, used for adapting and outputting an anti-interference strategy according to the first echo spectrum;

[0138] Among them, the anti-interference strategy selection model is trained in the following way:

[0139] Constructing a cognitive radar anti-interference model; wherein the cognitive radar anti-interference model includes a jammer model and a cognitive radar model; the cognitive radar model is configured to continuously transmit radar pulses with variable frequency; the jammer model is configured to intercept the radar pulses transmitted by the cognitive radar model and release interference with an adaptive type and adaptive parameters based on the intercepted radar pulses;

[0140] Construct an actor network and a commentator network. The actor network is used to output the expectation and standard deviation of the cognitive radar model's behavior based on the second echo spectrum. The commentator network is used to output the reward value that the cognitive radar model can obtain from implementing the behavior based on the second echo spectrum. The second echo spectrum is the echo spectrum obtained by the jammer model intercepting the radar pulse emitted by the cognitive radar model. The behavior implemented by the cognitive radar model corresponds one-to-one to the center frequency of the radar pulse it emits.

[0141] The actor network and critic network are trained using the PPO algorithm, and an anti-interference strategy selection model is constructed based on the trained actor network.

[0142] Optionally, the jammer model is specifically used to:

[0143] Continuous radar pulses are detected and stored during a first duration of each working cycle; adaptive interference is released according to the intercepted radar pulses during a second duration of each working cycle; the length of the working cycle is equal to the sum of the first duration and the second duration;

[0144] Among them, if the continuously intercepted N cap If the average center frequency of each radar pulse is interfered by the interference released in the previous working cycle, then in the subsequent working cycle, the first duration is reduced by N with a probability of γ1. step radar pulse duration, and increase the second duration by N step Radar pulse duration; where N cap is an integer greater than 1, N step Adjust the steps for duration.

[0145] Optionally, a jammer model that releases adaptive jamming based on intercepted radar pulses includes:

[0146] If the continuously intercepted N cap The range of the center frequency of a radar pulse is less than B tar , then the release center frequency is Bandwidth is B tar The aiming interference; among them, B tar is the aiming interference bandwidth; f c is the center frequency of the radar pulse transmitted by the cognitive radar model;

[0147] If N is intercepted continuouslycap The range of the center frequency of each radar pulse is not less than B tar , then the release center frequency is Bandwidth is B blk The blocking interference of blk is the blocking interference bandwidth.

[0148] Optionally, the cognitive radar model emits radar pulses according to time steps; and the actor network and the critic network are trained using the PPO algorithm, including:

[0149] In each time step i, the current second echo spectrum is used as the state s at the time step i i Input them into the actor network respectively, so that the actor network outputs the expectation and standard deviation of the behavior implemented by the cognitive radar model; construct the normal distribution behavior probability density function according to the expectation and standard deviation, and randomly extract behavior a based on the probability density function i ; for behavior a i Conduct interference suppression effect evaluation to get reward r i , and determine the state s of the next time step i+1 i+1 ;

[0150] The state s of each time step i is i 、Behavior a i , reward r i and the state s at the next time step i+1 i+1 As a state combination (s i ,a i ,r i ,s i+1 ) is stored in the experience buffer; the experience buffer can store up to N cap state combinations;

[0151] State combinations are randomly sampled from the experience buffer, and the actor network and critic network are trained based on the sampled state combinations.

[0152] It should be noted that, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0153] It should be noted that the terms "first," "second," and the like are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure.

[0154] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0155] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings and the disclosed content. In the description of the present invention, the word "comprising" does not exclude other components or steps, "one" or "a" does not exclude multiple situations, and "multiple" means two or more, unless otherwise clearly and specifically limited. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0156] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for adaptively generating cognitive radar anti-interference strategies based on PPO, characterized in that: include: Acquire a first echo spectrum; wherein the first echo spectrum is an echo spectrum of a cognitive radar; inputting the first echo spectrum into an anti-interference strategy selection model, so that the anti-interference strategy selection model adapts and outputs anti-interference parameters according to the first echo spectrum; The anti-interference strategy selection model is trained in the following way: Constructing a cognitive radar anti-interference model; wherein the cognitive radar anti-interference model includes a jammer model and a cognitive radar model; the cognitive radar model is configured to continuously transmit radar pulses with variable frequency; the jammer model is configured to intercept the radar pulses transmitted by the cognitive radar model and release interference of an adaptive type and adaptive parameters based on the intercepted radar pulses; Constructing an actor network and a commentator network; wherein the actor network is used to output the expectation and standard deviation of the behavior implemented by the cognitive radar model based on a second echo spectrum; and the commentator network is used to output the reward value that the cognitive radar model can obtain by implementing the behavior based on the second echo spectrum; wherein the second echo spectrum is the echo spectrum obtained by the jammer model intercepting the radar pulse transmitted by the cognitive radar model; the behavior implemented by the cognitive radar model and the center frequency of the radar pulse it transmits are in one-to-one correspondence; The actor network and the commentator network are trained using the PPO algorithm, and the anti-interference strategy selection model is constructed based on the trained actor network.

2. The adaptive generation method of cognitive radar anti-interference strategy based on PPO according to claim 1 is characterized in that: The jammer model is specifically used for: During a first duration of each working cycle, radar pulses are continuously intercepted and stored; during a second duration of each working cycle, adaptive interference is released according to the intercepted radar pulses; the length of the working cycle is equal to the sum of the first duration and the second duration; Among them, if the continuous interception If the average center frequency of each radar pulse is interfered by the interference released in the previous working cycle, then in the subsequent working cycle, The probability of reducing the first duration radar pulse duration, and increasing the second duration by radar pulse duration; where is an integer greater than 1, Adjust the steps for duration.

3. The adaptive generation method of cognitive radar anti-interference strategy based on PPO according to claim 2 is characterized in that: The jammer model releases adaptive jamming based on the intercepted radar pulses, including: If the continuous interception The range of the center frequency of a radar pulse is less than , then the release center frequency is , bandwidth is of aiming interference; among which, To target the interference bandwidth; is the center frequency of the radar pulse emitted by the cognitive radar model; If the continuous interception The range of the center frequency of each radar pulse is not less than , then the release center frequency is , bandwidth is The blocking interference of is the blocking interference bandwidth.

4. The adaptive generation method of cognitive radar anti-interference strategy based on PPO according to claim 1 is characterized in that: The cognitive radar model emits radar pulses according to time steps; and the PPO algorithm is used to train the actor network and the commentator network, including: At each time step The current second echo spectrum is used as the time step The state below input into the actor network respectively, so that the actor network outputs the expectation and standard deviation of the behavior implemented by the cognitive radar model; constructs a normal distribution behavior probability density function according to the expectation and the standard deviation, and randomly extracts behavior based on the probability density function ; for behavior Get rewarded for evaluating interference suppression effectiveness , and determine the next time step Status ; Each time step Status ,Behavior ,award and the next time step Status As a state combination Stored in the experience buffer; the experience buffer can store up to state combinations; A state combination is randomly extracted from the experience buffer, and the actor network and the critic network are trained according to the extracted state combination.

5. The adaptive generation method of cognitive radar anti-interference strategy based on PPO according to claim 4 is characterized in that: Training the actor network and the critic network according to the extracted state combination includes: Calculating a first loss of the actor network using a first loss function and a second loss of the critic network using a second loss function according to the extracted state combination and the reward value output by the critic network; The second loss function is: ； in, represents the learning rate, represents the network weight parameters of the critic network, The network weight parameter is The commentator network input state When , the reward value output by the critic network is, The network weight parameter is The commentator network input state When , the reward value output by the critic network is, The network weight parameter is The second loss of the commentator network; The first loss function is: ； in, represents the gradient, It means finding the logarithm, Is the cognitive radar model in state Select behavior The probability of It is calculated using the target probability density function, which is based on the network weight parameter The actor network takes input The behavioral probability density function constructed by the expectation and standard deviation of the corresponding output; The network weight parameter is The first loss of the actor network.

6. The adaptive generation method of cognitive radar anti-interference strategy based on PPO according to claim 4 is characterized in that: Randomly extract behavior based on the probability density function ,include: Generate random numbers and calculate dynamic exploration rate; If the dynamic exploration rate is greater than the random number, the behavior is randomly extracted using the probability density function. ; If the dynamic exploration rate is not greater than the random number, the random extraction behavior ; The calculation formula of the dynamic exploration rate is: ； in, is the preset initial exploration rate, is the preset termination exploration rate, is the natural exponential function, is the dynamic exploration rate.

7. A PPO-based cognitive radar anti-interference strategy adaptive generation device, characterized in that: include: Echo acquisition module and anti-interference strategy selection model; The echo acquisition module is used to acquire a first echo spectrum; wherein the first echo spectrum is an echo spectrum of a cognitive radar; The anti-interference strategy selection model is used to adapt and output an anti-interference strategy according to the first echo spectrum; The anti-interference strategy selection model is trained in the following way: Constructing a cognitive radar anti-interference model; wherein the cognitive radar anti-interference model includes a jammer model and a cognitive radar model; the cognitive radar model is configured to continuously transmit radar pulses with variable frequency; the jammer model is configured to intercept the radar pulses transmitted by the cognitive radar model and release interference of an adaptive type and adaptive parameters based on the intercepted radar pulses; Constructing an actor network and a commentator network; wherein the actor network is used to output the expectation and standard deviation of the behavior implemented by the cognitive radar model based on a second echo spectrum; and the commentator network is used to output the reward value that the cognitive radar model can obtain by implementing the behavior based on the second echo spectrum; wherein the second echo spectrum is the echo spectrum obtained by the jammer model intercepting the radar pulse transmitted by the cognitive radar model; the behavior implemented by the cognitive radar model and the center frequency of the radar pulse it transmits are in one-to-one correspondence; The actor network and the commentator network are trained using the PPO algorithm, and the anti-interference strategy selection model is constructed based on the trained actor network.

8. The device for adaptively generating cognitive radar anti-interference strategies based on PPO according to claim 7, characterized in that: The jammer model is specifically used for: Continuously detecting and storing radar pulses during a first duration of each operating cycle; and releasing adaptive interference based on the intercepted radar pulses during a second duration of each operating cycle; wherein the length of the operating cycle is equal to the sum of the first duration and the second duration; Among them, if the continuous interception If the average center frequency of each radar pulse is interfered by the interference released in the previous working cycle, then in the subsequent working cycle, The probability of reducing the first duration radar pulse duration, and increasing the second duration by radar pulse duration; where is an integer greater than 1, Adjust the steps for duration.

9. The PPO-based cognitive radar anti-interference strategy adaptive generation device according to claim 8, characterized in that: The jammer model releases adaptive jamming based on the intercepted radar pulses, including: If the continuous interception The range of the center frequency of a radar pulse is less than , then the release center frequency is , bandwidth is of aiming interference; among which, To target the interference bandwidth; is the center frequency of the radar pulse emitted by the cognitive radar model; If continuous interception The range of the center frequency of each radar pulse is not less than , then the release center frequency is , bandwidth is The blocking interference of is the blocking interference bandwidth.

10. The PPO-based cognitive radar anti-interference strategy adaptive generation device according to claim 7, characterized in that: The cognitive radar model emits radar pulses according to time steps; and the PPO algorithm is used to train the actor network and the commentator network, including: At each time step The current second echo spectrum is used as the time step The state below input into the actor network respectively, so that the actor network outputs the expectation and standard deviation of the behavior implemented by the cognitive radar model; constructs a normal distribution behavior probability density function according to the expectation and the standard deviation, and randomly extracts behavior based on the probability density function ; for behavior Get rewarded for evaluating interference suppression effectiveness , and determine the next time step Status ; Each time step Status ,Behavior ,award and the next time step Status As a state combination Stored in the experience buffer; the experience buffer can store up to state combinations; A state combination is randomly extracted from the experience buffer, and the actor network and the critic network are trained according to the extracted state combination.

Images