A method for generating FAR anti-active suppression jamming strategy based on MAB model

Abstract

This invention discloses a method for generating an active jamming countermeasure strategy for a Free AR (FAR) based on a MAB (Multi-Action Block) model, comprising: step S1, constructing a received echo signal model of the FAR under active jamming conditions; step S2, constructing a POMDP (Programmable Optical Device Detection Platform) model independent of environmental states in a dynamically changing operational electromagnetic environment; step S3, calculating the jammer state in the POMDP model based on an active jamming identifier; and step S4, calculating anti-jamming action decisions in the POMDP model based on the MAB model. The method proposed in this invention allows the FAR to learn the optimal active anti-jamming strategy based on the detection and feedback results of the jamming environment from each frequency transmission channel, and to execute active jamming avoidance actions, thereby improving the FAR's ability to compete with jamming suppression and achieving continuous performance in detecting target range and velocity.

Description

Technical Field

[0001] This invention belongs to the field of radar anti-jamming technology, and particularly relates to a method for generating FAR (Frequency Agility Radar) anti-active suppression jamming strategies based on the MAB (Multi-armed Bandit, MAB) model. Background Technology

[0002] The FAR (Frequency Arrestor) continuously changes the carrier frequency of the transmitted signal within the Coherent Processing Interval (CPI), giving it a strong advantage in active jamming countermeasures. If the enemy jammer fails to effectively track the FAR's frequency, targeted narrowband jamming almost completely loses its ability to interfere with the FAR, and the power of sweeping and wide-stopband jamming cannot be concentrated on the FAR's frequency, resulting in a significant reduction in jamming effectiveness. The FAR's ability to evade enemy active jamming greatly enhances its survivability in modern cognitive electronic warfare.

[0003] The key factor affecting the performance of FAR (Fault-Avoidance Response) jamming countermeasures is the carrier frequency agility strategy. Most current carrier frequency agility strategies are either random or based on offline frequency band selection using reinforcement learning. Random agility strategies fail to utilize environmental knowledge and learn the jamming strategy information of the attacker, resulting in weak anti-jamming capabilities. Reinforcement learning relies on continuous interaction and iteration between the agent (e.g., radar) and the environment (e.g., the battlefield electromagnetic space). Based on the current policy, the agent takes appropriate actions in its current state to maximize the total reward, and then adjusts the policy based on the reward obtained, ultimately obtaining the optimal policy for the agent in the environment. Although reinforcement learning theory has made significant progress in recent years, the following problems still urgently need to be addressed. 1. Reinforcement learning intelligent jamming countermeasures training and testing environments are demanding. The interaction between the agent and the environment needs to conform to the Markov decision process (MDP). In real combat environments, it is difficult to satisfy the Markov property in the observation of the opponent's actions and state transitions. 2. Reinforcement learning intelligent jamming countermeasures usually define the action and state space and design reward functions to learn and converge the strategy in an offline environment. It is difficult to respond to unknown jamming types and actions in complex combat environments. 3. Reinforcement learning intelligent jamming countermeasures have poor robustness. If there is uncertainty in the environment between the radar and the jammer, due to the mismatch between the offline training environment and the combat environment, the anti-jamming performance may decrease when using the optimal anti-jamming strategy obtained in the training environment in the actual combat environment. Summary of the Invention

[0004] To address the technical problems existing in the background art, this invention proposes a method for generating FAR anti-active suppression jamming strategy based on the MAB model. Specifically, the FAR learns the optimal active anti-jamming strategy based on the detection and feedback results of the interference environment from each frequency transmission channel, and executes active jamming avoidance actions to improve the FAR's ability to compete with suppression jamming, thereby continuously exerting its ability to detect the target's distance and speed.

[0005] This invention proposes a method for generating FAR (Active Amplifier) ​​anti-active suppression interference strategies based on the MAB (Multi-Action Block) model, comprising:

[0006] Step S1: Construct a model of the received echo signal of FAR under active interference environment;

[0007] Step S2: In a dynamically changing combat electromagnetic environment, construct a POMDP model that is independent of the environmental state.

[0008] Step S3: Calculate the state of the jammer in the POMDP model based on the active jamming identifier;

[0009] Step S4: Calculate the anti-interference action decision in the POMDP model based on the MAB model.

[0010] Preferably, in step S1, the specific process of constructing the received echo signal model of FAR under active interference environment is as follows:

[0011] Step S11: FAR transmits X sub-pulses within one CPI, each sub-pulse being a linear frequency modulated continuous wave signal. The time-domain expression for the x-th sub-pulse is:

[0012]

[0013] Where A represents the amplitude of the transmitted signal; rect(·) represents the window function: T p Indicates the sub-pulse width; T r Indicates the sub-pulse repetition interval; f x f represents the sub-pulse transmission frequency. x =f c +Δf x x = 1, 2, ..., X, where f c For radar carrier frequency, Δf x ∈[0,B] is the frequency hopping step size of the sub-pulse; t represents time, and j represents the imaginary unit;

[0014] Step S12, assuming there are K targets to be detected in the scene, the target echo signal of the x-th sub-pulse is:

[0015]

[0016] Where, β k R represents the scattering intensity of the k-th target; k This represents the distance to the k-th target; v k Let c represent the velocity of the k-th target; c represents the speed of light in a vacuum.

[0017] Step S13, in a dynamically changing combat electromagnetic environment, the FAR frequency band [f c ,f c +B] is divided into M sub-bands:

[0018] f = {f1, f2, ... f M}, (3)

[0019] Among them, f b ∈[f c +(b-1)Δf,f c +bΔf], b∈{1,2,...M}, Δf=B / M, f b This indicates that the M sub-bands are uniform, non-overlapping, and continuous; the FAR sub-pulse selects the m-th sub-band to transmit the signal, and the received echo signal model is divided into the following four cases:

[0020]

[0021] Among them, w xm Let w represent noise in the environment. xm ~N(0,σ 2 ), which has a mean of 0 and a variance of σ. 2 Gaussian noise, r xm It corresponds to the target echo signal, J xm It is an active interference signal.

[0022] Preferably, in step S2, the specific process of constructing a POMDP model independent of environmental conditions in a dynamically changing combat electromagnetic environment is as follows:

[0023] Composed of quintuples<S,A,R,O,γ> Construct the POMDP model;

[0024] Where S is the FAR state space, and s is the state of FAR at time y. y ∈S,s y Let y be the state corresponding to the frequency band where FAR detects the jammer's transmission probability at time y. Right now This indicates the state of the jammer at time y-1;

[0025] A is the FAR action space, and the action a at time y y ∈A={f1,f2,...,fM} indicates the sub-band selected by the FAR transmit signal, and the action. This indicates the sub-band selected by the jammer to transmit signals;

[0026] R represents the reward function, R(s) y ,a y ,s y+1 ) is achieved through the agent in state s y Take action a y The reward obtained, the reward value function is in the form of: Where |·| is the absolute value function;

[0027] O represents the observation space, and the observation o y ∈O represents the environmental state observed by the FAR sensor at time y, i.e., the received echo signal. ym ;

[0028] γ∈[0,1] is a discount parameter used to adjust the weight of the regret value;

[0029] Where Y represents the time when the interaction ends, and the time during the interaction is denoted as y∈{1,2,...,Y}.

[0030] Preferably, in step S3, the specific process of calculating the state of the jammer in the POMDP model based on the active jamming identifier is as follows:

[0031] Step S31, construct an active interference detector;

[0032] Step S32: Obtain the dataset of the active interference detector;

[0033] Step S33: Set the training parameters of the active interference identifier network;

[0034] Step S34: Output the posterior probability of the interference signal in each channel, and obtain the state of the jammer based on the posterior probability.

[0035] Preferably, in step S31, the specific process of constructing the active interference identifier is as follows:

[0036] The active interference detector is based on a CNN framework: The first layer is the network's input layer, which separates the real and imaginary parts of each point of the received echo signal, concatenates the separate real parts sequentially, and then concatenates the concatenated real and imaginary parts to form a new one-dimensional data. Next are three convolutional layers, with inputs increasing in increments of 32, 64, and 128 respectively. Each convolutional layer has a kernel size of 3×1 and a stride of 2. After processing by each convolutional layer, the data passes through a ReLU layer, a max-pooling layer, and a Dropout layer. After processing by these three convolutional layers, the data is input into a global average pooling layer. Finally, a fully connected layer classifies the data, obtaining any f from the observed data at time y. q ∈{f1,f2,...,f M The probability of active interference in a sub-band Right now:

[0037]

[0038] Among them, e (·) It is an exponential function, ε q ρ represents the weight used to calculate the probability of active interference in the q-th output channel. q This represents the deviation used to calculate the probability of active interference in the output channel q.

[0039] Preferably, in step S32, the specific process of obtaining the active interference identifier dataset is as follows:

[0040] Generate a dataset according to formula (4), and for each echo class... xm The signal-to-noise ratio, interference-to-signal ratio, and delay of the signal are randomly selected from 20–30 dB, 5–10 dB, and 20–30 μs, respectively. A single echo... xm It contains a variety of different signals, each labeled with the channel of the interfering signal; the signal data is divided into three parts: training set, validation set, and test set.

[0041] Preferably, the ratio of training set, validation set, and test set is 7:2:1.

[0042] Preferably, in step S33, the specific process of setting the training parameters of the active interference identifier network is as follows:

[0043] The minimum learning rate is lrMin = 0.0008 and the maximum learning rate is lrMax = 0.01. The learning rate decreases continuously according to formula (6):

[0044]

[0045] Where epoch represents the current training iteration and T represents the total number of training iterations;

[0046] Use the cross-entropy loss function as the network's loss function:

[0047]

[0048] Among them, the non-softmax output of the sample Let represent the belief state at time y; l∈{0,1,2,...,M} represent the label of the sample. A sample label l=0 indicates that the jammer did not transmit a jamming signal and there is no active interference in the received echo signal. The rest indicate that the jammer selected the sub-frequency band f. l Transmit jamming signals and receive echo signals in the neutron band f l There is active interference.

[0049] Preferably, in step S34, the specific process of outputting the posterior probability of the interference signal in each channel and obtaining the jammer state based on the posterior probability is as follows:

[0050] The received echo signal is input into the belief state p output by the active interference identifier. y Belief state p y The frequency band f of the maximum a posteriori probability l' Given the jamming frequency band of the jammer, FAR obtains the state of the jammer at time y based on the received echo signal.

[0051] Preferably, in step S4, the specific process of calculating the anti-interference action decision in the POMDP model based on the MAB model is as follows:

[0052] Step S41: During the electronic countermeasures process, at time y, FAR selects a sub-band f from a given number of sub-bands. y Transmit signals; the metric for evaluating the FAR band selection strategy is the regret value, specifically the regret value at time y. y It is about choosing the optimal action. Receive rewards With the actual selected action a y Actual rewards received The difference, i.e. The sum of FAR regret values ​​is defined as follows:

[0053]

[0054] Step S42: Generate an anti-interference strategy algorithm online based on the MAB model to obtain action a. y ;

[0055] Preferably, the algorithm for generating anti-interference strategies online based on the MAB model is as follows:

[0056] Step S421: Obtain the FAR motion set and initialize the basic arm parameters of the MAB;

[0057] Step S422: In each round, perform TS sampling for each base arm and update the base arm parameters with discounts.

[0058] Step S423: Select the largest base arm in the sampling results as the sub-band of the transmitted signal;

[0059] Step S424: Observe the reward value and calculate the regret value;

[0060] Step S425: Update the MAB base arm parameters;

[0061] Step S426, repeat steps S422 to S425 until the interactive round ends.

[0062] Compared with existing technologies, this invention has the following advantages: 1) The POMDP model used in this invention does not depend on the environmental state. Compared with the traditional Markov Decision Process (MDP), the POMDP model provides a more accurate description of the FAR and active interference adversarial process. 2) Using an active interference identifyer to calculate the belief state in the POMDP model can significantly reduce storage space and save computational resources and time. 3) The MAB strategy based on online learning fully balances the contradiction between exploration and exploitation in the online adversarial process of FAR, and quickly obtains the optimal anti-interference action decision for FAR. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of a method for generating FAR anti-active suppression interference strategy based on MAB model proposed in an embodiment of the present invention.

[0064] Figure 2 This is a schematic diagram of the FAR selection of the transmission frequency sequence in the method proposed in the embodiments of the present invention;

[0065] Figure 3 This is a schematic diagram of the POMDP model for FAR-environment interaction in the method proposed in this embodiment of the invention;

[0066] Figure 4 This is a schematic diagram of the network architecture of the active interference identifier in the method proposed in the embodiments of the present invention;

[0067] Figure 5 A schematic diagram of the OODA loop of the method proposed in this embodiment of the invention.

[0068] Figure 6 The graph shows the accuracy curves of different anti-interference strategies in the method proposed in the embodiments of the present invention under different interference strategies.

[0069] Figure 7This is a regret value curve of different anti-interference strategies in the method proposed in the embodiments of the present invention under different interference strategies;

[0070] Figure 8 This is a graph showing the action selection curves of different anti-interference strategies in the method proposed in the embodiments of the present invention within a CPI under a constant interference strategy;

[0071] Figure 9 This is a graph showing the action selection curves of different anti-interference strategies in the method proposed in the embodiments of the present invention within one CPI of the lower triangular frequency sweep interference strategy;

[0072] Figure 10 This is a graph showing the action selection curves of different anti-interference strategies in the method proposed in the embodiments of the present invention within one CPI of the upper triangular frequency sweeping interference strategy;

[0073] Figure 11 This is a graph showing the action selection curves of different anti-interference strategies in the method proposed in the embodiments of the present invention within one CPI under the triangular frequency sweep interference strategy. Detailed Implementation

[0074] like Figure 1 As shown, Figure 1 This is a schematic diagram of a method for generating FAR anti-active suppression interference strategy based on MAB model proposed in an embodiment of the present invention.

[0075] The present invention proposes a method for generating FAR anti-active suppression interference strategy based on MAB model, comprising the following steps:

[0076] Step S1: Construct a model of the received echo signal of FAR under active interference environment.

[0077] Specifically, such as Figure 2 As shown, in step S1, the specific process of constructing the received echo signal model of FAR under active interference environment is as follows:

[0078] Step S11: The FAR transmits X sub-pulses within one CPI. Each sub-pulse is a Linear Frequency Modulation Continuous Wave (LFMCW) signal. The time-domain expression of the x-th sub-pulse is:

[0079]

[0080] Where A represents the amplitude of the transmitted signal; rect(·) represents the window function: T p Indicates the sub-pulse width; T r Indicates the sub-pulse repetition interval; f xf represents the sub-pulse transmission frequency. x =f c +Δf x x = 1, 2, ..., X, where f c For radar carrier frequency, Δf x ∈[0,B] is the frequency hopping step size of the sub-pulse; t represents time, and j represents the imaginary unit.

[0081] Step S12, ignoring the effects of electromagnetic clutter and multipath effects, assuming there are K targets to be detected in the scene, the target echo signal of the x-th sub-pulse is:

[0082]

[0083] Where, β k R represents the scattering intensity of the k-th target; k This represents the distance to the k-th target; v k Let represent the velocity of the k-th target; c represents the speed of light in a vacuum.

[0084] Step S13: In a dynamically changing combat electromagnetic environment, the target echo signal received by the FAR will be affected by environmental noise and enemy active interference; the FAR's frequency band [f c ,f c +B] is divided into M sub-bands:

[0085] f = {f1, f2, ... f M}, (3)

[0086] Among them, f b ∈[f c +(b-1)Δf,f c +bΔf], b∈{1,2,...M}, Δf=B / M, f b This indicates that the M sub-bands are uniform, non-overlapping, and continuous; the FAR sub-pulse selects the m-th sub-band to transmit the signal, and the received echo signal model is divided into the following four cases:

[0087]

[0088] Among them, w xm Let w represent noise in the environment. xm ~N(0,σ 2 ), which has a mean of 0 and a variance of σ. 2 Gaussian noise, r xm It corresponds to the target echo signal, J xm It is an active interference signal.

[0089] In formula (4), the first two cases indicate the absence of active interference, while the latter two cases indicate the presence of active interference. Noise is present in all four cases. When active interference exists in the channel receiving the echo signal, the FAR target detection performance will be significantly reduced. To mitigate the impact of active interference on FAR detection performance, the FAR sub-pulse transmission frequency should meet the following requirements: Try to avoid the channels occupied by jammers.

[0090] Step S2: In a dynamically changing combat electromagnetic environment, construct a POMDP (Partially Observable Markov Decision Process) model that is independent of the environmental state.

[0091] Specifically, in step S2, the process of constructing a POMDP model independent of environmental conditions in a dynamically changing combat electromagnetic environment is as follows:

[0092] The POMDP model described is a model for agents to make sequential decisions in dynamically changing combat electromagnetic environments. Its core principle is that agents cannot accurately know their own state and need to interact with the environment using external sensors to obtain this information. Due to the influence of received noise, it is difficult to obtain the true state of enemy jammers in non-cooperative active jamming scenarios, and FAR anti-jamming actions need to be based on incomplete observation information obtained from sensors. Therefore, the POMDP model is more suitable than the MDP model for radar electronic warfare scenarios where the environmental state is dynamically uncertain.

[0093] like Figure 3 As shown, it consists of a quintuple.<S,A,R,O,γ> Construct the POMDP model;

[0094] Where S is the FAR state space, and s is the state of FAR at time y. y ∈S,s y Let y be the state corresponding to the frequency band where FAR detects the jammer's transmission probability at time y. Right now This indicates the state of the jammer at time y-1;

[0095] A is the FAR action space, and the action a at time y y ∈A={f1,f2,...,f M} indicates the sub-band selected by the FAR transmit signal, and the action. Indicates the sub-band selected by the FAR transmit signal;

[0096] R represents the reward function, R(s) y ,ay ,s y+1 ) is achieved through the agent in state s y Take action a y The reward obtained, the reward value function is in the form of: Where |·| is the absolute value function; the greater the geometric distance between the FAR action and the jammer action, the less likely the FAR signal is to be jammed, and the greater the reward obtained.

[0097] O represents the observation space, and the observation o y ∈O represents the environmental state observed by the FAR sensor at time y, i.e., the received echo signal. ym ;

[0098] γ∈[0,1] is a discount parameter used to adjust the weight of the regret value;

[0099] Where Y represents the time when the interaction ends, and the time during the interaction is denoted as y∈{1,2,...,Y}.

[0100] Step S3: Calculate the state of the jammer in the POMDP model based on the active jammer identifier.

[0101] In electronic warfare environments with non-cooperative active jamming, it is difficult to obtain the true environmental state, and the radar cannot directly obtain the state of the jammer at time y in step S2. The POMDP model needs to be solved based on historical observations and actions, which requires a lot of storage space and computing resources to remember historical observations and actions and to calculate anti-interference action decisions. To solve this problem, the POMDP model introduces belief states to represent historical information.

[0102] In existing technologies, according to the Bayesian probability criterion, the formula for updating the belief state is:

[0103]

[0104] Among them, P obs (o y+1 |s y+1 ) is in state s y+1 Under the conditions of observation o y+1 The probability of T(o) y+1 |s y+1 ,a y ) is in state s y Execute action a under the condition y Transition to state s y+1 The probability; η = 1 / P r (o y+1 |b y ,a y ) is the normalization constant, where:

[0105] In formula (8), the variable P is related to the knowledge of the environmental model. obs (o y+1 |s y+1 ) and T(s y+1 |s y ,a y It is difficult to obtain this information through calculation, so this patent introduces a belief state calculation method based on an active interference identifier.

[0106] Specifically, in step S3, the process of calculating the state of the jammer in the POMDP model based on the active jamming identifier is as follows:

[0107] Step S31: Construct an active interference identifier.

[0108] Specifically, such as Figure 4 As shown, the specific process of constructing the active interference identifier in step S31 is as follows:

[0109] The active interference detector is based on a CNN framework: The first layer is the network's input layer, which separates the real and imaginary parts of each point of the received echo signal, concatenates the separate real parts sequentially, and then concatenates the concatenated real and imaginary parts to form a new one-dimensional data. Next are three convolutional layers, with inputs doubling to 32, 64, and 128 respectively. Each convolutional layer has a kernel size of 3×1 and a stride of 2. After processing by each convolutional layer, the data passes through a ReLU layer, a max-pooling layer, and a Dropout layer to facilitate data processing in subsequent convolutional layers. After processing by these three convolutional layers, the data is input into a global average pooling layer to alleviate overfitting. Finally, a fully connected layer classifies the data, obtaining any f from the observed data at time y. q ∈{f1,f2,...,f M The probability of active interference in a sub-band Right now:

[0110]

[0111] Among them, e (·) It is an exponential function, ε q ρ represents the weight used to calculate the probability of active interference in the q-th output channel. q This represents the deviation used to calculate the probability of active interference in the output channel q.

[0112] Step S32: Obtain the dataset of the active interference detector.

[0113] Specifically, in step S32, the process of obtaining the active interference identifyr dataset is as follows:

[0114] Generate a dataset according to formula (4), and for each echo class... xm The signal-to-noise ratio, interference-to-signal ratio, and delay of the signal are randomly selected from 20–30 dB, 5–10 dB, and 20–30 μs, respectively. A single echo... xm It contains a variety of different signals, each labeled with the channel of the interfering signal; the signal data is divided into three parts: training set, validation set, and test set.

[0115] Specifically, a single y nm It contains 100 different signals.

[0116] Specifically, the ratio of the training set, validation set, and test set is 7:2:1.

[0117] Step S33: Set the training parameters for the active interference identifier network.

[0118] Specifically, in step S33, the process of setting the training parameters of the active interference identifier network is as follows:

[0119] The minimum learning rate is lrMin = 0.0008 and the maximum learning rate is lrMax = 0.01. The learning rate decreases continuously according to formula (6):

[0120]

[0121] Where epoch is the current training iteration and T is the total number of training iterations.

[0122] By setting the training parameters of the active interference recognizer network as described above, the active interference recognizer network has a large learning rate in the early stage, converges quickly, and has a small learning rate in the later stage, thus avoiding oscillations near the extreme value.

[0123] Use the cross-entropy loss function as the network's loss function:

[0124]

[0125] Among them, the non-softmax output of the sample Let represent the belief state at time y; l∈{0,1,2,...,M} represent the label of the sample. A sample label l=0 indicates that the jammer did not transmit a jamming signal and there is no active interference in the received echo signal. The rest indicate that the jammer selected the sub-frequency band f. l Transmit jamming signals and receive echo signals in the neutron band f l There is active interference.

[0126] Step S34: Output the posterior probability of the interference signal in each channel, and obtain the state of the jammer based on the posterior probability.

[0127] Specifically, in step S34, the process of outputting the posterior probability of the interference signal in each channel and obtaining the state of the jammer based on the posterior probability is as follows:

[0128] The received echo signal is input into the belief state p output by the active interference identifier. y Belief state p y The frequency band f of the maximum a posteriori probability l' For the jamming frequency band of the jammer, f l' With actual f l There may be deviations; FAR obtains the jammer's state at time y based on the received echo signal.

[0129] Step S4: Calculate the anti-interference action decision in the POMDP model based on the MAB model.

[0130] Specifically, in step S4, the process of calculating the anti-interference action decision in the POMDP model based on the MAB model is as follows:

[0131] Step S41: The MAB model is often used to solve online learning problems.

[0132] During electronic warfare, at time y, FAR selects a sub-band f from a given number of sub-bands. y Transmit signals; the metric for evaluating the FAR band selection strategy is the regret value, specifically the regret value at time y. y It is about choosing the optimal action. Receive rewards With the actual selected action a y Rewards The difference, i.e. The sum of FAR regret values ​​is defined as follows:

[0133]

[0134] FAR needs to reduce the regret value at each moment in order to reduce the total regret value.

[0135] Step S42: Based on the MAB model, generate an online anti-interference strategy algorithm to obtain action a. y ;

[0136] Specifically, the algorithm for generating anti-interference strategies online based on the MAB model (Discounted Thompson sampling (DTS) method) is as follows:

[0137]

[0138] The Discounted Thompson sampling (DTS) algorithm mentioned above evolved from the MAB model through the TS algorithm for learning the optimal arm reward.

[0139] Thompson sampling (TS) uses Bayesian posterior probability to represent the uncertainty of the model. Instead of directly referencing the average return to select a greedy action, it randomly generates a sampled model from the Bayesian estimate, executes the greedy strategy, and then observes the feedback value to update the MAB model parameters. If an action is selected multiple times, it means there is no more room for exploration, and actions with fewer exploration attempts should be chosen. TS achieves an intelligent exploration mechanism by using updated random sampling, effectively balancing the conflict between exploration and exploitation.

[0140] In the above algorithm, the parameters and These are important parameters in the MAB model, representing the selection of sub-band f. i The accumulated rewards and accumulated regret points are shown in lines 5-6. and Sampling is performed as a TS parameter, and then updated according to the discount factor γ. and This is the DTS method; in lines 4-8, the algorithm samples the model estimates from the posterior data, then slightly "discounts" the model parameters to the future and "forgets the past"; in line 9, the algorithm takes the maximum value from the samples to determine which superarm to select in this round, i.e., which radar action 'a' to launch. y .

[0141] The technical approach of this invention is as follows: For FAR (Fighter Aid) missions, electromagnetic signals are emitted into the environment, and the signal reception model is divided into four cases. Under dynamic and uncertain environmental conditions, a partially observable Markov Decision Process (POMDP) ​​model is constructed. An active interference identifier based on the principles of a Convolutional Neural Network (CNN) is used to calculate the belief state in the POMDP. Finally, an online rapid anti-interference strategy (POMDP-MAB) is designed based on a Multi-armed Bandit (MAB) model evolved from Thompson sampling (TS). The regret value is used as the strategy evaluation index, and the optimal action decision is calculated according to the different states of the target.

[0142] Reference Figure 5As shown, this invention proposes a method for generating anti-active suppression jamming strategies for FAR based on the MAB model. It constructs an interactive dynamic game model with an "observation-judgment-decision-action" (OODA) closed-loop structure to solve the problem of weak cognitive countermeasure capabilities of traditional FAR. It deduces the optimal action decision based on different states of the jamming environment, generates anti-jamming strategies online quickly, and accurately responds to the countermeasures of enemy jamming machines.

[0143] In summary, the FAR anti-active jamming strategy generation method based on the MAB model proposed in this invention has the following advantages: 1) The active jamming identifier is trained in a short time without requiring prior environmental state information, and can quickly and accurately obtain the environmental state, reducing storage space for remembering historical states and saving computing resources for calculating belief states; 2) The DTS method is an online learning method that does not require training and can adapt to changes in the dynamic electromagnetic environment of combat with non-cooperative active jamming, without the problem of mismatch between training and testing environments; 3) Compared with other offline anti-jamming methods, this method balances the contradiction between "exploration" and "exploitation," tending to explore unexplored actions in the early stage and stabilizing with correct actions in the later stage, resulting in low regret value and high accuracy.

[0144] Example 1:

[0145] To verify the superiority of the FAR online frequency agility anti-interference capability of the present invention in complex, dynamic, and uncertain electromagnetic environments, a set of simulation experiments were conducted to compare the present invention with traditional methods. The hardware platform parameters for the simulation test are shown in Table 1, and the software platform parameters are shown in Table 2.

[0146] Table 1 Hardware Platform Parameters

[0147] Memory 32GB Graphics card model NVIDIA GeForce RTX 3070 Ti Graphics card memory 8GB

[0148] Table 2 Software Platform Parameters

[0149] compiler PyCharm 2021.3.1 CUDA version 11.3.1

[0150] Simulation results and analysis:

[0151] The FAR carrier frequency is 10GHz, and the operating bandwidth is 1000MHz. The operating bandwidth is divided into 10 equal parts, numbered 1, 2, ..., 10. Each time, the FAR selects one sub-band to transmit a signal, and simultaneously, the interfering party also selects one sub-band to transmit an interfering signal. The results of the active interference identifier determining the interference channel are shown in Table 3.

[0152] Table 3. Identification Results of Active Interference Identifier

[0153]

[0154] After training, the active jamming identification device can identify the channel of non-cooperative active jamming with an accuracy of over 99% in a dynamically changing combat electromagnetic environment, without relying on prior environmental knowledge, and obtain the jamming status of the enemy jammer.

[0155] The jammer employed constant, upper triangle, lower triangle, and triangle sweep jamming strategies. To verify the superiority of the DTS method, random, greedy, deep Q-Network (DQN), and Thompson methods were used for comparative verification. Each anti-jamming strategy underwent 1000 online interactions with the jamming method. After each closed-loop interaction, 32 sub-pulses within a CPI were tested. The performance of different strategies was compared from three perspectives: accuracy, online selection regret value, and action selection. It was assumed that strategies other than DTS and TS had complete knowledge of the environmental state; when the environmental state was uncertain, the test performance was unknown.

[0156] like Figure 6 As shown, Figure 6 (a) shows the accuracy curves of different anti-interference strategies under constant interference; Figure 6 (b) shows the accuracy curves of different anti-interference strategies under triangular frequency sweep interference. Figure 6 (c) shows the accuracy curves of different anti-interference strategies under the lower triangular frequency sweep interference. Figure 6 (d) shows the accuracy curves of different anti-jamming strategies under upper triangular frequency sweeping interference. Among all jamming strategies, the random transmission frequency strategy performs poorly in all jamming scenarios due to the lack of utilization of environmental knowledge. In contrast, the anti-jamming strategy that learns through interaction with the environment has a significantly higher accuracy than the random anti-jamming strategy. The Egreedy strategy utilizes environmental knowledge for learning and outperforms the Random strategy, but its performance is unstable and its accuracy is lower than that of DQN, TS, and DTS algorithms. DQN uses deep network training, and its convergence speed is significantly slower than that of TS and DTS methods. It is also sensitive to parameters and may still experience fluctuations in accuracy in the later stages.

[0157] Compared to other anti-interference strategies, the DTS method achieves rapid convergence in the early stages and maintains stable accuracy in the later stages. TS is similar to DTS, but converges slowly in the lower triangular frequency sweep and fails to reach convergence after 800-1000 steps. DQN, trained using a deep network, converges significantly slower than DTS and is sensitive to parameters, exhibiting fluctuations in accuracy later on. Under all interference strategies, DTS focuses on exploration in the early stages and converges rapidly, while focusing on exploitation in the later stages, consistently maintaining the highest accuracy.

[0158] like Figure 7 As shown, Figure 7 (a) is a graph of the regret value of different anti-interference strategies under constant interference; Figure 7 (b) is a graph showing the regret value curves of different anti-interference strategies under triangular frequency sweep interference; Figure 7 (c) is a graph of regret value under different anti-interference strategies under lower triangular frequency sweep interference; Figure 7 (d) shows the regret value curves of different anti-jamming strategies under upper triangular frequency sweep interference. The regret value growth rates of DTS and TS tend to be consistent and are significantly lower than other anti-jamming strategies. Although TS remains almost unchanged in the constant interference strategy, DTS still shows an increasing trend. This is because DTS balances "utilizing history" and "exploring the unknown" in the early stage, without being lazy about the correct choices of the past. The regret value will be slightly higher than TS in the early stage. After 600 rounds, the regret value of DTS is lower than that of TS, and the growth of regret value slows down and tends to stabilize between 800-1000 rounds.

[0159] exist Figure 8 middle, Figure 8 (a) is a graph of action selection curves for the Random policy within one CPI under a constant disturbance policy; Figure 8 (b) is a graph of the action selection curve of the Egreedy policy within one CPI under a constant disturbance policy; Figure 8 (c) is a graph of the action selection curve of the DQN strategy within one CPI under a constant disturbance strategy; Figure 8 (d) is a graph of the action selection curve of the TS strategy within one CPI under a constant disturbance strategy; Figure 8 (e) is a graph showing the action selection curve of the DTS strategy within one CPI under a constant disturbance strategy. In the constant disturbance strategy, the other three knowledge-based methods besides DTS are lazy about past correct choices and always occupy the correct channel selected in the past. In contrast, DTS, in addition to occupying the same channel, still explores other channels. Compared with other anti-disturbance strategies, DTS is more exploratory in the early action selection.

[0160] exist Figure 9 middle, Figure 9 (a) is a graph showing the action selection curve of the Random strategy within one CPI under the lower triangular frequency sweeping interference strategy; Figure 9 (b) is a graph showing the action selection curve of the Egreedy strategy within one CPI under the lower triangular frequency sweeping interference strategy; Figure 9 (c) is a graph showing the action selection curve of the DQN strategy within one CPI under the lower triangular frequency sweeping interference strategy; Figure 9 (d) is a graph showing the action selection curve of the TS strategy within one CPI under the lower triangular frequency sweeping interference strategy; Figure 9(e) is a graph showing the action selection curve of the DTS strategy within one CPI under the lower triangular frequency sweeping interference strategy; Figure 10 middle, Figure 10 (a) is a graph showing the action selection curve of the Random strategy within one CPI under the upper triangular frequency sweeping interference strategy; Figure 10 (b) is a graph showing the action selection curve of the Egreedy strategy within one CPI under the upper triangular frequency sweeping interference strategy; Figure 10 (c) is a graph showing the action selection curve of the DQN strategy within one CPI under the upper triangular frequency sweeping interference strategy; Figure 10 (d) is a graph showing the action selection curve of the TS strategy within one CPI under the upper triangular frequency sweeping interference strategy; Figure 10 (e) is a graph showing the action selection curve of the DTS strategy within one CPI under the upper triangular frequency sweeping interference strategy; Figure 11 middle, Figure 11 (a) is a graph showing the action selection curve of the Random strategy within one CPI under the triangular frequency sweeping interference strategy; Figure 11 (b) is a graph showing the action selection curve of the Egreedy strategy within one CPI under the triangular frequency sweeping interference strategy; Figure 11 (c) is a graph showing the action selection curve of the DQN strategy within one CPI under the triangular frequency sweeping interference strategy; Figure 11 (d) is a graph showing the action selection curve of the TS strategy within one CPI under the triangular frequency sweeping interference strategy; Figure 11 (e) is a graph showing the action selection curve of the DTS strategy within one CPI under the triangular frequency sweep jamming strategy. Figure 9-11 In this context, the DTS method rarely overlaps with interference strategies, and it occupies the same channel for a shorter period of time than other methods.

[0161] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for generating FAR anti-active suppression interference strategy based on MAB model, characterized in that, include: Step S1: Construct a model of the received echo signal of FAR under active interference environment; Step S2: In a dynamically changing combat electromagnetic environment, construct a POMDP model that is independent of the environmental state. Step S3: Calculate the state of the jammer in the POMDP model based on the active jamming identifier; Step S4: Calculate the anti-interference action decision in the POMDP model based on the MAB model; In step S4, the specific process of calculating the anti-interference action decision in the POMDP model based on the MAB model is as follows: Step S41, during electronic warfare, At time FAR, select one sub-band from a given number of sub-bands. Transmit signals; the metric for evaluating the FAR band selection strategy is the regret value. Regret value at any moment It is about choosing the optimal action. Receive rewards With actual selection action Rewards The difference, i.e. The sum of FAR regret values ​​is defined as follows: Step S42: Generate an anti-interference strategy algorithm online based on the MAB model and obtain actions. ; The algorithm for generating anti-interference strategies online based on the MAB model is as follows: Step S421: Obtain the FAR motion set and initialize the basic arm parameters of the MAB; Step S422: In each round, perform TS sampling for each base arm and update the base arm parameters with discounts. Step S423: Select the largest base arm in the sampling results as the sub-band of the transmitted signal; Step S424: Observe the reward value and calculate the regret value; Step S425: Update the MAB base arm parameters; Step S426, repeat steps S422 to S425 until the interactive round ends.

2. The method for generating FAR anti-active suppression interference strategy based on MAB model according to claim 1, characterized in that, In step S1, the specific process of constructing the received echo signal model of FAR under active interference environment is as follows: Step S11, FAR is launched within one CPI. Each of the 10 sub-pulses is a linear frequency modulated continuous wave signal. The time-domain expression for each sub-pulse is: in, Indicates the amplitude of the transmitted signal; Representing window functions: ; Indicates the sub-pulse width; Indicates the sub-pulse repetition interval; Indicates the sub-pulse transmission frequency. ,in, For radar carrier frequency, It is the frequency hopping step size of the sub-pulse; Indicates time, Represents the imaginary unit; Step S12, assuming the scenario contains The target to be detected, launch the first The target echo signal of each sub-pulse is: in, Indicates the first The scattering intensity of each target; Indicates the first Distance to each target; Indicates the first The speed of the target; This represents the speed of light in a vacuum. Step S13: In a dynamically changing combat electromagnetic environment, the frequency band of FAR is... Divided into Sub-band: in, , express Each sub-band is uniform, non-overlapping, and continuous; FAR No. Subpulse selection The model for transmitting signals and receiving echo signals in a single frequency band can be divided into the following four cases: in, To represent noise in the environment, assume The mean is 0 and the variance is . Gaussian noise, It is the corresponding target echo signal. It is an active interference signal.

3. The method for generating FAR anti-active suppression interference strategy based on MAB model according to claim 1, characterized in that, In step S2, the specific process of constructing a POMDP model independent of environmental conditions in a dynamically changing combat electromagnetic environment is as follows: Composed of quintuples Construct the POMDP model; in, It is the FAR state space, FAR at time... status , for At any given moment, FAR detects the state corresponding to the frequency band with the highest probability of jammer transmission. ,Right now , express The status of the jammer at all times; It is the FAR action space, at any moment action Indicates the sub-band selected by the FAR transmit signal, and the action. This indicates the sub-band selected by the jammer to transmit signals; Represents the reward value function, Through the agent in the state Take action The reward obtained, the reward value function is in the form of: ,in It is an absolute value function; Represents the observation space, observation Indicates FAR sensor The constantly observed environmental conditions, i.e., the received echo signals ; Discount parameters used to adjust the weighting of regret values; in, The time at which the interaction ends is denoted as _____. The time during the interaction is denoted as _____. .

4. The method for generating FAR anti-active suppression interference strategy based on MAB model according to claim 1, characterized in that, In step S3, the specific process of calculating the state of the jammer in the POMDP model based on the active jamming identifier is as follows: Step S31, construct an active interference detector; Step S32: Obtain the dataset of the active interference identifyr network; Step S33: Set the training parameters of the active interference identifier network; Step S34: Output the posterior probability of the interference signal in each channel, and obtain the state of the jammer based on the posterior probability.

5. The method for generating FAR anti-active suppression interference strategy based on MAB model according to claim 4, characterized in that, In step S31, the specific process of constructing the active interference identifier is as follows: The active interference detector is based on a CNN framework: the first layer is the network's input layer, which separates the real and imaginary parts of each point of the received echo signal, concatenates the separate real parts sequentially, and then concatenates the concatenated real and imaginary parts to form a new one-dimensional data. Next are three convolutional layers, with inputs doubling to 32, 64, and 128 respectively. The kernel size of each convolutional layer is [missing information]. The convolution stride is 2; after each convolutional layer, the data goes through a ReLU layer, a max pooling layer, and a Dropout layer; after three convolutional layers, the data is input into a global average pooling layer; finally, a fully connected layer classifies the data, obtaining the... Any time observation data The probability of active interference in a sub-band ,Right now: Among them, among them, It is an exponential function. This indicates the method used to calculate the output of the first... Each channel has a weighted probability of active interference. This indicates the method used to calculate the output of the first... There is a deviation in the probability of active interference in each channel.

6. The method for generating FAR anti-active suppression interference strategy based on MAB model according to claim 4, characterized in that, In step S32, the specific process of obtaining the active interference identifyer dataset is as follows: Generate a dataset according to formula (4), for each class The signal-to-noise ratio, interference-to-signal ratio, and time delay of the signal are respectively at , and Randomly selected from, single It contains a variety of different signals, each labeled with the channel of the interfering signal; the signal data is divided into three parts: training set, validation set, and test set.

7. The method for generating FAR anti-active suppression interference strategy based on MAB model according to claim 6, characterized in that, The ratio of training set, validation set, and test set is: .

8. The method for generating FAR anti-active suppression interference strategy based on MAB model according to claim 4, characterized in that, In step S33, the specific process of setting the training parameters of the active interference identifier network is as follows: Minimum learning rate and maximum learning rate The learning rate decreases continuously according to formula (6): in Indicates the current number of training iterations. Indicates the total number of training iterations; Use the cross-entropy loss function as the network's loss function: Among them, the non-softmax output of the sample Indicates the first The state of belief at any given moment; The label representing the sample, sample label This indicates that the jammer did not transmit any jamming signal, and there was no active interference in the received echo signal. The remaining information indicates that the jammer selected a sub-band. Transmit jamming signals and receive echo signals in the neutron band. There is active interference.

9. The method for generating FAR anti-active suppression interference strategy based on MAB model according to claim 8, characterized in that, In step S34, the specific process of outputting the posterior probability of the interference signal on each channel and obtaining the jammer state based on the posterior probability is as follows: The received echo signal is input into the belief state output by the active interference identifier. Belief state The frequency band of the maximum a posteriori probability The jamming frequency band of the jammer is determined by FAR based on the received echo signal. state of time .

Images