Interference strategy generation method, device and equipment

Abstract

The invention provides an interference strategy generation method, device and equipment, and the method comprises the steps: carrying out the sampling of a confrontation process of a jammer and a radar in a confrontation scene of the jammer and the radar, and obtaining an estimation value of a state transition matrix of the radar; normalizing the sum of each row of data in the state transition matrix into 1 to obtain an original Markov environment; calculating a plurality of synthetic Markov environments based on the estimated value of the state transition matrix and the original Markov environment; running a Q-Learning algorithm in parallel in a plurality of synthetic Markov environments so as to respectively learn to obtain a Q value table; performing fusion processing on the plurality of Q value tables based on an adaptive weighting mechanism of JS divergence to obtain a first comprehensive Q value table; repeating the above steps to complete a preset number of iterative calculations; and in response to the completion of the iterative calculation, obtaining a target comprehensive Q value table, and determining an optimal interference strategy based on the target comprehensive Q value table. According to the scheme provided by the invention, the optimal interference strategy for the interference radar can be quickly and accurately generated.

Description

Technical Field

[0001] The present invention belongs to the technical field of electronic countermeasures, and in particular relates to a method, device and equipment for generating an interference strategy. Background Art

[0002] With the rapid development of information technology, the electromagnetic environment of modern arenas has become increasingly complex, and electronic countermeasures play an increasingly important role in it. As a key link in electronic countermeasures, interference decision-making aims to generate the best interference strategy in real time based on the results of perception and judgment, realize the recognition of the law of the target working mode and the generation of the best interference strategy for the mode. Driven by advanced electronic technologies such as digital phased array systems, radars are gradually developing in the direction of multi-function, with the ability of pulse waveform agility, and can adaptively adjust the working state according to the task and external environment. For example, in order to enhance the anti-interference capability, multi-function radars increase the pulse repetition frequency. If the jammer does not have sufficient prior knowledge, it will be difficult to establish the best correspondence between the target state and the existing interference pattern, and it will not be able to form a set of optimal interference strategies for different states of the target. Therefore, in the context of complex and changeable electromagnetic environments, it is of great significance to study an efficient interference strategy generation method.

[0003] Scholars have conducted extensive research on jamming decision-making from three aspects: template matching, game theory, and reasoning. However, these methods rely on a large amount of prior knowledge and are difficult to adapt to complex and changing electromagnetic environments. For new radars such as multi-function radars, prior information is difficult to obtain, and it is difficult to accurately jam them in actual combat. The emergence of reinforcement learning solves the problem that traditional jamming decision-making methods rely too much on prior data.

[0004] Among the existing solutions, a multifunctional radar cognitive jamming decision algorithm based on Q-Learning is proposed. By comparing cognitive ideas and jamming decision principles, Q-Learning is applied to cognitive jamming decision-making. The jammer obtains information such as the target radar status and threat level by sensing the external electromagnetic environment, and decides on the appropriate jamming style by integrating and analyzing this information, and evaluates the jamming effect in real time to provide a basis for the next decision. Specifically, the jammer calculates and selects the jamming style a corresponding to the maximum state-action value function Q(s,a) under the state according to the Q-Learning algorithm based on the detected target radar working state, and executes it; according to the actual jamming effect evaluation value obtained, the parameters in the Q-Learning algorithm are adjusted to select a new jamming style in real time.

[0005] Specifically, the jammer obtains the radar state at the current moment through reconnaissance; the ε-greedy strategy is used to select the jamming style, balancing the exploration phase and the utilization phase, that is, the jamming style with the largest state-action value Q(s,a) is selected with a probability of 1-ε, and the action is randomly selected with a probability of ε; the jamming effectiveness is evaluated according to the change of the radar state to obtain the evaluation value r; it is determined whether the Q value converges, if not, the Q value is updated according to the jamming effectiveness evaluation result, the radar state is re-reconnaissanced for training, and if the Q value converges, the trained strategy is output.

[0006] However, all existing solutions have a common technical problem, that is, the existing interference decision algorithm has a long convergence time, insufficient exploration ability in complex strategy space, weak stability and generalization ability, low sample utilization efficiency, and high error rate. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a jamming strategy generation method, device and equipment which can quickly and accurately generate an optimal jamming strategy for jamming radar.

[0008] The present invention includes a method for generating an interference strategy, comprising:

[0009] In a confrontation scenario between a jammer and a radar, sampling the confrontation process between the jammer and the radar to obtain an estimated value of a state transfer matrix of the radar;

[0010] Normalizing the sum of each row of data in the state transfer matrix to 1 to obtain an original Markov environment;

[0011] Calculating a plurality of synthetic Markov environments based on the estimated value of the state transfer matrix and the original Markov environment;

[0012] Running the Q-Learning algorithm in parallel in the plurality of synthetic Markov environments to respectively learn and obtain a Q value table, wherein the Q value table includes a plurality of Q values ​​respectively representing the interference degree of each interference action of the jammer on radars in different states, wherein the larger the Q value, the higher the interference efficiency of the interference action on the radar state;

[0013] The plurality of Q value tables are fused and processed by an adaptive weighting mechanism based on JS divergence to obtain a first comprehensive Q value table;

[0014] Repeat the above steps to complete a preset number of iterative calculations;

[0015] In response to the completion of the iterative calculation, a target comprehensive Q value table is obtained, and an optimal interference strategy is determined based on the target comprehensive Q value table, in which the optimal interference actions for interfering with radars in different states are recorded.

[0016] In some embodiments, in the confrontation scenario between the jammer and the radar, sampling the confrontation process between the jammer and the radar to obtain an estimated value of the state transfer matrix of the radar includes:

[0017] In a confrontation scenario between a jammer and a radar, the working mode of the radar is obtained based on the reconnaissance of the jammer, and each working mode corresponds to a state of the radar;

[0018] Randomly applying a jamming action to the radar to detect whether the working mode of the radar is switched;

[0019] Collect the jamming action of each jamming and the status information of the radar before and after the jamming;

[0020] estimating element values ​​of a state transfer matrix of the radar based on the collected information to obtain the estimated value;

[0021] Update the estimated value of the state transfer matrix estimated based on the information collected last time based on the currently obtained estimated value;

[0022] Repeat the above steps until the sampling times of each state transition pair of the radar meets the threshold requirement, and obtain a state transfer matrix with completed assignment, wherein the state transition pair includes the state before the radar interference and the state after the interference, and the states in different state transition pairs are different.

[0023] In some embodiments, the calculation based on the estimated value of the state transfer matrix and the original Markov environment to obtain multiple synthetic Markov environments includes:

[0024] Calculating the nth matrix power of the estimated value of the state transfer matrix, where n is a preset positive integer;

[0025] The estimated value is represented by an n-th matrix power based on the original Markov environment to obtain n synthetic Markov environments, and the n synthetic Markov environments correspond to different time scales.

[0026] In some embodiments, the Q-Learning algorithm is run in parallel in the plurality of synthetic Markov environments to respectively learn a Q value table, including:

[0027] Running the Q-Learning algorithm in parallel in the plurality of synthetic Markov environments to respectively learn the Q function of each state-action pair (s, a), wherein the state-action pair is formed by combining the state s of the radar and the jamming action a applied by the jammer;

[0028] The Bellman optimal equation is solved based on the Q function to obtain the Q value table containing each of the Q values.

[0029] In some embodiments, the method further comprises:

[0030] When running the Q-Learning algorithm for learning, a greedy strategy is applied to balance the relationship between exploration and utilization in the learning process, and the learned Q function is updated based on the following formula until the Q value converges with probability 1:

[0031]

[0032] Among them, α∈(0,1) is the learning rate, γ is the discount factor, and r a (s) is the benefit brought by using interference action a to change the radar state s, Q(s′,a′) is the Q function corresponding to the state-action pair (s′,a′), Q(s,a) is the Q function corresponding to the state-action pair (s,a), and A is the set of interference actions.

[0033] In some embodiments, the Q function includes an optimal policy function and an optimal value function:

[0034] The optimal strategy function is:

[0035] The optimal value function is:

[0036] Among them, v π is the value function under strategy π, v * is the optimal value function, π * is the optimal strategy, and Q* is the optimal Q function.

[0037] In some embodiments, the adaptive weighting mechanism based on JS divergence fuses and processes multiple Q value tables to obtain a first comprehensive Q value table, including:

[0038] Calculate the difference value between the two Q value tables based on the JS divergence algorithm;

[0039] Based on the difference value, a weight is configured for each of the Q value tables, wherein the weight represents the contribution made by the Q value table when the first comprehensive Q value table is updated;

[0040] The first comprehensive Q value table is obtained based on the weight and each of the Q value tables.

[0041] In some embodiments, the method further comprises:

[0042] At the preset update rate u t The second comprehensive Q value table generated by the new iteration updates the first comprehensive Q value table generated by the previous iteration:

[0043]

[0044] Said is the weight of the Q value table, is the probability distribution corresponding to the first comprehensive Q value table, K is the number of preset Markov environments, and u t is the update rate, The Q-value table obtained by running the Q-learning algorithm for the nth Markov environment, For each Q value table corresponding to the weight, is the first comprehensive Q value table updated at the t+1 time step, The first comprehensive Q value table updated for time step t.

[0045] Another embodiment of the present invention also provides an interference strategy generating device, including:

[0046] A sampling module is used to sample the confrontation process between the jammer and the radar in a confrontation scenario between the jammer and the radar, so as to obtain an estimated value of the state transfer matrix of the radar;

[0047] A normalization module, used for normalizing the sum of each row of data in the state transfer matrix to 1, so as to obtain an original Markov environment;

[0048] A first calculation module, used for calculating a plurality of synthetic Markov environments according to the estimated value of the state transfer matrix and the original Markov environment;

[0049] A learning module is used to run the Q-Learning algorithm in parallel in a plurality of the synthetic Markov environments to respectively learn and obtain a Q value table, wherein the Q value table contains a plurality of Q values ​​respectively representing the interference degree of each interference action of the jammer on radars in different states, and the larger the Q value, the higher the interference efficiency of the interference action on the radar state;

[0050] A first processing module, configured to fuse and process the plurality of Q value tables according to an adaptive weighting mechanism of JS divergence to obtain a first comprehensive Q value table;

[0051] A control module, used for controlling the above steps to be repeated to complete a preset number of iterative calculations;

[0052] The response module is used to obtain a target comprehensive Q value table in response to the completion of the iterative calculation, and determine the optimal interference strategy based on the target comprehensive Q value table, wherein the target comprehensive Q value table records the optimal interference actions for interfering with radars in different states.

[0053] Another embodiment of the present invention also provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the interference strategy generation method described in any one of the embodiments above are implemented.

[0054] The beneficial effects of the present invention include that by adopting the scheme of the present application, the jammer can accelerate the understanding of the environment, discover new valuable state-action pairs, and reduce unnecessary interactions with the radar by considering state transitions and reward signals for a longer period of time. This helps to improve the sample efficiency and decision-making performance of the jammer. Secondly, the scheme can simulate the potential trajectories that cannot be directly observed, allowing the jammer to learn the state transition laws of the radar from indirect experience. This simulation-based method makes up for the lack of actual environmental data and enables the jammer to draw knowledge from a wider range of experience. In addition, the scheme can also encourage the jammer to consider longer-term returns and adopt jamming strategies that may not have direct returns but are beneficial to long-term benefits. This forward-looking decision-making method helps to improve the long-term performance of the jammer in a dynamic radar environment. In general, the scheme of the present application can help the jammer better handle complex and changeable radar environments, learn valuable lessons from past experience, improve its decision-making efficiency and adaptability, reduce the convergence time of the jamming decision algorithm, enhance the exploration ability in a complex strategy space, improve the utilization efficiency of samples, and reduce the strategy error rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 The figure is a flow chart of the interference strategy generation method of the present invention.

[0057] Figure 2 The figure is a schematic diagram of the application flow of the interference strategy generation method of the present invention.

[0058] Figure 3 The figure is another application flow chart of the interference strategy generation method of the present invention.

[0059] Figure 4 Schematic diagram of radar operating mode switching rules of the present invention.

[0060] Figure 5 It is a structural block diagram of the interference strategy generating device of the present invention. DETAILED DESCRIPTION

[0061] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings, but are not intended to limit the present invention.

[0062] It should be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the following description should not be considered as limiting, but merely as an example of an embodiment. Other modifications within the scope of the present disclosure will occur to those skilled in the art.

[0063] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the general description of the present disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the present disclosure.

[0064] These and other characteristics of the invention will become apparent from the following description of a preferred form of embodiment given as a non-limiting example, with reference to the accompanying drawings.

[0065] It should also be understood that, although the invention has been described with reference to some specific examples, those skilled in the art will be able to realize many other equivalent forms of the invention that have the characteristics recited in the claims and are therefore within the scope of protection defined thereby.

[0066] The features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0067] Specific embodiments of the present disclosure are described hereinafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid obscuring the present disclosure with unnecessary or redundant details. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present disclosure in a variety of ways with substantially any suitable detailed structure.

[0068] This specification may use the phrases "in one embodiment," "in another embodiment," "in a further embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present disclosure.

[0069] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0070] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides an interference strategy generation method, including:

[0071] S1: In a jammer-radar confrontation scenario, sampling the jammer-radar confrontation process to obtain an estimated value of the radar state transfer matrix;

[0072] S2: normalize the sum of each row of data in the state transfer matrix to 1 to obtain the original Markov environment;

[0073] S3: Calculating multiple synthetic Markov environments based on the estimated value of the state transfer matrix and the original Markov environment;

[0074] S4: running the Q-Learning algorithm in parallel in the plurality of synthetic Markov environments to respectively learn and obtain a Q value table, wherein the Q value table comprises a plurality of Q values ​​respectively representing the interference degree of each interference action of the jammer on radars in different states, wherein the larger the Q value, the higher the interference efficiency of the interference action on the radar state;

[0075] S5: fusing the multiple Q value tables based on the adaptive weighting mechanism of JS divergence to obtain a first comprehensive Q value table;

[0076] S6: Repeat the above steps to complete a preset number of iterative calculations;

[0077] S7: In response to completion of the iterative calculation, a target comprehensive Q value table is obtained, and an optimal interference strategy is determined based on the target comprehensive Q value table, wherein the target comprehensive Q value table records optimal interference actions for interfering with radars in different states.

[0078] Based on the above content, it can be known that this embodiment will first control the jammer to detect a radar and interact with the radar to build a confrontation scenario. Then, the information detected by the jammer can be used to complete the sampling of the confrontation process, which mainly includes sampling the radar state transition and the interference action imposed by the jammer, and based on this, the value of the state transfer matrix of the radar is estimated, that is, the estimated value of the element in the state transfer matrix is ​​estimated. Next, the system will construct a Markov environment, specifically normalizing the sum of each row of data in the state transfer matrix to 1 to obtain the original Markov environment. After that, in order to obtain multiple random Markov environments for learning and calculation, the system will calculate multiple synthetic Markov environments based on the estimated value of the state transfer matrix and the original Markov environment, that is, multiple random Markov environments. Control multiple synthetic Markov environments to run the Q-Learning algorithm in parallel to learn and obtain a Q value table respectively. The Q value in the Q value table specifically reflects the influence of different interference operations imposed by the jammer on the changes in different states of the radar. The greater the influence, the higher the Q value. After obtaining multiple Q value tables, the system will fuse and process multiple Q value tables based on the adaptive weighting mechanism of JS divergence to obtain the first comprehensive Q value table. The comprehensive Q value table contains candidate interference actions corresponding to the conversion of each state of the radar, and a candidate interference strategy can be formed based on the multiple interference actions. However, this candidate interference strategy is not the optimal interference strategy and needs to be optimized. Therefore, the number of iterations is set to iteratively calculate the above steps, continuously optimize the comprehensive Q value table, and finally obtain the target comprehensive Q value table. Based on the information in the target comprehensive Q value table, the optimal interference strategy can be obtained, which records the optimal interference actions used to interfere with the radar to achieve different state conversions.

[0079] Based on the scheme of the above embodiment, the jammer can accelerate the understanding of the environment, discover new valuable state-action pairs, and reduce unnecessary interactions with the radar by considering longer state transitions and reward signals. This helps to improve the sample efficiency and decision-making performance of the jammer. Secondly, the scheme can simulate the potential trajectories that cannot be directly observed, allowing the jammer to learn the state transition rules of the radar from indirect experience. This simulation-based method makes up for the lack of actual environmental data and enables the jammer to draw knowledge from a wider range of experience. In addition, the scheme can also encourage the jammer to consider longer-term returns and adopt jamming strategies that may not have direct returns but are beneficial to long-term benefits. This forward-looking decision-making method helps to improve the long-term performance of the jammer in a dynamic radar environment. In general, the scheme of this embodiment can help the jammer better handle complex and changing radar environments, learn valuable lessons from past experience, improve its decision-making efficiency and adaptability, reduce the convergence time of the jamming decision algorithm, enhance the exploration ability in a complex strategy space, improve the utilization efficiency of samples, and reduce the strategy error rate.

[0080] Specifically, in one embodiment, Figure 3 As shown in the figure, the application scenario is a scenario where an aircraft carries a jammer for self-defense jamming, specifically a one-to-one confrontation between the jammer and the radar. The multi-function radar located on the first aircraft tracks the target aircraft. The target aircraft carries a jammer and a radar warning receiver, which can intercept radar signals in real time for reconnaissance and evaluate the jamming effectiveness based on the changes in the intercepted radar signals. The jammer masters the transfer rules of the radar working mode through continuous confrontation with the radar, and learns the jamming strategy at the same time. According to the jamming strategy, it selects the optimal jamming style and releases the jamming signal. The goal of the jammer is to continuously reduce the threat level of the radar working mode, thereby protecting the target from being locked by the radar. The jamming strategy generation method is essentially a sequential decision problem, which can be represented by a Markov decision process (MDP) model. MDP is represented by a quaternion {S, A, P, R}, where S, A, P, and R are composed of the radar working mode, jamming style, radar state transition probability, and instantaneous reward, respectively.

[0081] The state space in this embodiment is composed of different working modes of the multifunctional radar, which is defined as a total of N radar states in the state space. The working mode of the multifunctional radar may include, but is not limited to, velocity search (VelocitySearch, VS) mode, range while searching (Range While Search, RWS) mode, track while scanning (Track While Search, TWS) mode, track and search (Track and Search, TAS) mode, multi-target tracking (MultiTarget Tracking, MTT) mode and single target tracking (Single Target Tracking, STT) mode, and different working modes can be switched flexibly. The present embodiment designs the working mode switching rules of the multifunctional radar in the absence of interference according to the change of the target and the threat level or distance. When the fighter carrying the multifunctional radar is far away from the target and the target aircraft is not found, it first works in the VS mode and searches the entire airspace of the mission area. At this time, it does not pose a threat to the target, so the threat level of this working mode is the lowest; if the target is detected at this time, the multifunctional radar will select the RWS mode to continue to follow up the detected target according to its own energy and time resource constraints. When the target is closer, the multi-function radar enters the coarse tracking stage, and the fighter selects TWS or TAS mode to coarsely track the target according to the real-time situation. When the fighter is very close to the target and the threat level of the target is high, the air target is switched to MTT or STT mode and enters the continuous fine tracking stage to provide strong support for subsequent precision weapon strikes. Therefore, the threat level of the multi-function radar reaches the highest at this time, and the jammer should avoid the multi-function radar working in this mode.

[0082] The action space consists of different interference patterns, defined as J = [j1, j2, …, j M ], there are M types of jamming styles in the action space. Including narrowband noise jamming, broadband noise jamming, swept frequency noise jamming, comb spectrum jamming, dense false target jamming, slice forwarding jamming, distance dragging jamming, speed dragging jamming and Doppler noise jamming. It mainly includes three types of suppression jamming, deception jamming and suppression-deception combined jamming, which will produce different jamming effects on the radar. Effective jamming will reduce the threat level of the radar working mode.

[0083] The state transition probability p(s′s,a) represents the probability that the radar will transfer from state s to s′ after being interfered with by a in state s. The switching of radar working modes has certain rules. Different interferences will have different effects on the transfer of radar working modes, and choosing effective interference will reduce the threat level of the working mode. For example, when the radar is working in TWS mode, after being interfered with by the distance dragging interference imposed by the jammer, the target position changes, and the radar will re-retrieve the target position and enter the RWS mode. After being interfered with by the sweep frequency, if the radar finds a new target at this time, the radar can still enter the TAS working mode normally. If multiple targets are confirmed, it can enter the MTT working mode normally. If the radar is interfered with by narrowband noise and the target is just suppressed, the radar loses the target and needs to search again, thus entering the VS mode.

[0084] Furthermore, in this embodiment, the return is defined by the transition of the radar state, r(s t+1 |s t ) indicates that the radar status changes from s t Transfer to t+1 The difference in threat level between two states is defined as:

[0085] Δ TL=TL(s t+1 )-TL(s t )

[0086] Establish a function f between the threat level difference and the benefit, then the benefit r(s t+1 |s t )for:

[0087] r(s t+1 |s t )=f(ΔTLs t+1 ,s t )

[0088] In this embodiment, if Figure 4As shown in the figure, the radar state transition diagram is established with the six working modes of the multi-function radar as the states. S1 to S6 represent STT, MTT, TAS, TWS, RWS, and VS modes respectively, and their threat levels are defined as 10, 8, 6, 5, 3, and 1 respectively. The VS mode has the lowest threat level and is the radar target state. During training, STT is set as the radar initial state. ij , such as, a 34 , a 42 etc. means that the radar status S i Transfer from S j The interference pattern, which has probability p ij The radar state is transferred. In actual application, three different radar state transfer models env_1, env_2, and env_3 can be defined.

[0089] The state transition probabilities of the radars in env_1, env_2, and env_3 are set to 0.6, 0.8, and 0.9, respectively. Therefore, the uncertainty of the radar state transition in env_1 is the largest, that is, the environment is the most unstable and poses the greatest challenge to the jammer.

[0090] Furthermore, in a confrontation scenario between a jammer and a radar, the system samples the confrontation process between the jammer and the radar to obtain an estimated value of a state transfer matrix of the radar, including:

[0091] S8: In a confrontation scenario between a jammer and a radar, obtaining a working mode of the radar based on reconnaissance by the jammer, each working mode corresponding to a state of the radar;

[0092] S9: randomly applying a jamming action to the radar to detect whether the working mode of the radar is switched;

[0093] S10: collecting the jamming action of each jamming and the state information of the radar before and after the jamming;

[0094] S11: estimating element values ​​of the state transfer matrix of the radar based on the collected information to obtain the estimated value;

[0095] S12: updating the estimated value of the state transfer matrix estimated based on the information collected last time based on the currently obtained estimated value;

[0096] S13: Repeat the above steps until the sampling times of each state transition pair of the radar meets the threshold requirement, and obtain a state transfer matrix with completed assignment, wherein the state transition pair includes the state before the radar interference and the state after the interference, and the states in different state transition pairs are different.

[0097] Exemplarily, since the state transfer matrix of the radar is unknown to the jammer, if one wants to obtain the state transfer matrix of the radar, one first needs to sample the environment and create multiple synthetic Markov environments. represents the estimated PTT (probability transition tensor, consisting of |A| PTMs, |A| represents the number of interference patterns), Indicates the PTM (probability transition matrix) corresponding to each interference pattern / interference action a in PTT. Each interference pattern corresponds to a probability transition matrix, which indicates the probability of the radar transferring from the current working mode to the next working mode when the interference pattern is adopted. Initialize each element to have a value of , and then record the number of times each state transition pair has been experienced in the confrontation between the jammer and the radar. In this embodiment, it is required to ensure that each state transition pair has been experienced at least v times. During the confrontation between the jammer and the radar, the jammer first detects the working mode s of the radar, and then randomly applies a jamming action. The radar switches the working mode according to its own scheduling mechanism. At this time, the jammer detects the working mode s′ after the radar is switched, records the data {s, a, s′} of this confrontation process, and updates the value of the estimated state transition matrix accordingly:

[0098]

[0099] Repeat the above process until the number of sampling times for each state transition pair (s, s′) exceeds v times.

[0100] In another embodiment, the multiple synthetic Markov environments are calculated based on the estimated value of the state transfer matrix and the original Markov environment, including:

[0101] S14: Calculate the nth matrix power of the estimated value of the state transfer matrix, where n is a preset positive integer;

[0102] S15: Representing the estimated value by the n-th matrix power based on the original Markov environment to obtain n synthetic Markov environments, wherein the n synthetic Markov environments correspond to different time scales.

[0103] Specifically, in order to improve the algorithm’s exploration capability at the environmental level, we first need to create multiple Markov environments, with the radar working mode transition probability p under different environments. a(s, s′) need to be related, and the probability transfer tensor (PTT) is row-random (that is, the probability rate transfer matrix (PTM) corresponding to each interference pattern is row-random), or it can be converted into a row-random environment by normalization without changing the original structure. This embodiment uses n-hop PTTs (P n ): n-hop Ensemble Q-Learning, n-hop integrated Q learning is a reinforcement learning algorithm that combines the ideas of n-hop state transfer and ensemble learning to improve the exploration and learning efficiency of Q-Learning. The n-hop state transfer is used to describe the process in which the system transfers from one state to another after n steps. By adopting this method, the n-step transition probability between radar operating modes can be described, where n is a random value. Different numbers of transfer steps correspond to multiple time scales, so the state transition of the radar has the advantage of being explainable. In addition, the PTT of different time scale environments can be quickly calculated by using matrix multiplication. In this embodiment, when performing n-power calculations, each power calculation obtains a synthetic Markov environment. After completing the n-power calculations, n synthetic Markov environments can be obtained, that is, n random Markov environments. The value of n is not fixed and can be determined according to the number of Markov environments required.

[0104] When the system obtains multiple Markov environments, it controls the multiple synthetic Markov environments to run the Q-Learning algorithm in parallel to learn and obtain a Q value table respectively, including:

[0105] S16: running the Q-Learning algorithm in parallel in the plurality of synthetic Markov environments to respectively learn the Q function of each state-action pair (s, a), wherein the state-action pair is formed by combining the state s of the radar and the jamming action a imposed by the jammer;

[0106] S17: Solving the Bellman optimal equation based on the Q function to obtain the Q value table containing each of the Q values.

[0107] Specifically, as mentioned above, the solution of this embodiment is to model the interference strategy generation problem as an MDP (Markov Decision Process, a mathematical model used to describe how to maximize a cumulative reward by choosing different actions in the decision-making process). Its essence is to find the optimal interference style corresponding to each working mode of the radar, and finally switch the working mode of the radar to the lowest threat level, that is, to find the optimal interference strategy. The radar state transition probability is stored in the probability transition tensor (PTT) P. P consists of |A| probability transition matrices (probability transition, PTM) of size |S|×|S|. The goal is to solve the Bellman optimal equation to obtain the optimal strategy:

[0108]

[0109] For all s∈S, v π is the value function under strategy π, v * is the optimal value function, π * is the optimal strategy, and γ∈(0,1) represents the discount factor. The interference strategy π can define deterministic actions and random actions in each state. Deterministic actions mean that each state has the optimal specific action, while random actions mean that each state will generate a distribution in the action space, and actions are selected based on the distribution. By controlling multiple synthetic Markov environments for Q learning, the Q function of each state-action pair (s,a) can be obtained, and the Q value can be calculated based on the function, and then the optimal strategy can be found based on the Q value.

[0110] Furthermore, since multiple iterative calculations are required in this embodiment, the system will update the previous data based on the Q function and Q value table obtained by each iterative calculation, specifically including:

[0111] S18: When running the Q-Learning algorithm for learning, a greedy strategy is applied to balance the relationship between exploration and utilization in the learning process, such as selecting a random action with a probability of ε, selecting the current optimal action with a probability of 1-ε, and updating the learned Q function based on the following formula until the Q value converges with a probability of 1:

[0112]

[0113] Among them, α∈(0,1) is the learning rate, γ is the discount factor, and r a(s) is the benefit of using interference action a to change the state s of the radar, Q(s′,a′) is the Q function of the corresponding state action pair (s′,a′), Q(s,a) is the Q function of the corresponding state action pair (s,a), and A is the set of interference actions. Based on the above content, it can be seen that the system is to calculate and update the Q function by interacting with the jammer and the radar and collecting samples {s,a,s′,r}. Until the Q function converges to the optimal value with probability 1, the Q function includes the optimal strategy function and the optimal value function, where:

[0114] The optimal strategy function is:

[0115] The optimal value function is:

[0116] Among them, v π is the value function under strategy π, v * is the optimal value function, π * is the optimal strategy, and Q* is the optimal Q function.

[0117] In another embodiment, the adaptive weighting mechanism based on JS divergence fuses and processes multiple Q value tables to obtain a first comprehensive Q value table, including:

[0118] S19: Calculating the difference value between the two Q value tables based on the JS divergence algorithm;

[0119] S20: configuring a weight for each of the Q value tables based on the difference value, wherein the weight represents the contribution made by the Q value table when the first comprehensive Q value table is updated;

[0120] S21: Obtain the first comprehensive Q value table based on the weight and each of the Q value tables.

[0121] In this embodiment, the average Jensen-Shannon divergence (AJSD) is used to calculate the probability distribution and Distance between:

[0122]

[0123] in, Denotes a probability vector of size |A|. The JSD between two probability distributions is calculated as follows:

[0124]

[0125] KL(@) represents the KL divergence between two distributions. AJSD is used to measure the difference between two distributions. The greater the difference between the two Q value tables, the smaller the weight they take when updating, so we use to update the weight vector.

[0126] Furthermore, the method further comprises:

[0127] S22: At the preset update rate u t The second comprehensive Q value table generated by the new iteration updates the first comprehensive Q value table generated by the previous iteration:

[0128]

[0129] Said is the weight of the Q value table, is the probability distribution corresponding to the first comprehensive Q value table, K is the number of preset Markov environments, and u t is the update rate, The Q-value table obtained by running the Q-learning algorithm for the nth Markov environment, For each Q value table corresponding to the weight, is the first comprehensive Q value table updated at the t+1 time step, The first comprehensive Q value table updated for time step t.

[0130] For example, after the system completes a round of training for each synthetic Markov environment and obtains each weight vector, it uses the softmax function to normalize the weight vector. it Update, the specific formula is as above.

[0131] After the update is completed, the system will determine whether the number of iterations has been reached. If not, the next round of training will be continued. If it has been reached, the optimal interference strategy will be output. The optimal interference strategy formula in this embodiment is:

[0132] In specific applications, the ε-Greedy algorithm can be used when selecting the interference style, setting the learning rate of the Q-Learning algorithm α = 0.1, the discount factor γ = 0.9, and the number of training rounds E num = 12000, the total number of Markov environments K = 5, the update rate u t =0.5. After each round of training, the sum of all elements in the Q table is calculated, which is defined as follows:

[0133] Where |S| represents the number of radar states, and |A| represents the number of interference patterns. In a stable environment, the Q learning algorithm or the nEQL algorithm can be used to complete the learning, while in an unstable environment, the nEQL algorithm is preferably used to complete the learning.

[0134] like Figure 5 As shown, the embodiment of the present invention also provides an interference strategy generating device 100, including:

[0135] A sampling module is used to sample the confrontation process between the jammer and the radar in a confrontation scenario between the jammer and the radar, so as to obtain an estimated value of the state transfer matrix of the radar;

[0136] A normalization module, used for normalizing the sum of each row of data in the state transfer matrix to 1, so as to obtain an original Markov environment;

[0137] A first calculation module, used for calculating a plurality of synthetic Markov environments according to the estimated value of the state transfer matrix and the original Markov environment;

[0138] A learning module is used to run the Q-Learning algorithm in parallel in a plurality of the synthetic Markov environments to respectively learn and obtain a Q value table, wherein the Q value table contains a plurality of Q values ​​respectively representing the interference degree of each interference action of the jammer on radars in different states, and the larger the Q value, the higher the interference efficiency of the interference action on the radar state;

[0139] A first processing module, configured to fuse and process the plurality of Q value tables according to an adaptive weighting mechanism of JS divergence to obtain a first comprehensive Q value table;

[0140] A control module, used for controlling the above steps to be repeated to complete a preset number of iterative calculations;

[0141] The response module is used to obtain a target comprehensive Q value table in response to the completion of the iterative calculation, and determine the optimal interference strategy based on the target comprehensive Q value table, wherein the target comprehensive Q value table records the optimal interference actions for interfering with radars in different states.

[0142] In some embodiments, in the confrontation scenario between the jammer and the radar, sampling the confrontation process between the jammer and the radar to obtain an estimated value of the state transfer matrix of the radar includes:

[0143] In a confrontation scenario between a jammer and a radar, the working mode of the radar is obtained based on the reconnaissance of the jammer, and each working mode corresponds to a state of the radar;

[0144] Randomly applying a jamming action to the radar to detect whether the working mode of the radar is switched;

[0145] Collect the jamming action of each jamming and the status information of the radar before and after the jamming;

[0146] estimating element values ​​of a state transfer matrix of the radar based on the collected information to obtain the estimated value;

[0147] Update the estimated value of the state transfer matrix estimated based on the information collected last time based on the currently obtained estimated value;

[0148] Repeat the above steps until the sampling times of each state transition pair of the radar meets the threshold requirement, and obtain a state transfer matrix with completed assignment, wherein the state transition pair includes the state before the radar interference and the state after the interference, and the states in different state transition pairs are different.

[0149] In some embodiments, the calculation based on the estimated value of the state transfer matrix and the original Markov environment to obtain multiple synthetic Markov environments includes:

[0150] Calculating the nth matrix power of the estimated value of the state transfer matrix, where n is a preset positive integer;

[0151] The estimated value is represented by an n-th matrix power based on the original Markov environment to obtain n synthetic Markov environments, and the n synthetic Markov environments correspond to different time scales.

[0152] In some embodiments, the Q-Learning algorithm is run in parallel in the plurality of synthetic Markov environments to respectively learn a Q value table, including:

[0153] Running the Q-Learning algorithm in parallel in the plurality of synthetic Markov environments to respectively learn the Q function of each state-action pair (s, a), wherein the state-action pair is formed by combining the state s of the radar and the jamming action a applied by the jammer;

[0154] The Bellman optimal equation is solved based on the Q function to obtain the Q value table containing each of the Q values.

[0155] In some embodiments, the learning module is further configured to:

[0156] When running the Q-Learning algorithm for learning, a greedy strategy is applied to balance the relationship between exploration and utilization in the learning process, and the learned Q function is updated based on the following formula until the Q value converges with probability 1:

[0157]

[0158] Among them, α∈(0,1) is the learning rate, γ is the discount factor, and r a (s) is the benefit brought by using interference action a to change the radar state s, Q(s′,a′) is the Q function corresponding to the state-action pair (s′,a′), Q(s,a) is the Q function corresponding to the state-action pair (s,a), and A is the set of interference actions.

[0159] In some embodiments, the Q function includes an optimal policy function and an optimal value function:

[0160] The optimal strategy function is:

[0161] The optimal value function is:

[0162] Among them, v π is the value function under strategy π, v * is the optimal value function, π * is the optimal strategy, and Q* is the optimal Q function.

[0163] In some embodiments, the adaptive weighting mechanism based on JS divergence fuses and processes multiple Q value tables to obtain a first comprehensive Q value table, including:

[0164] Calculate the difference value between the two Q value tables based on the JS divergence algorithm;

[0165] Based on the difference value, a weight is configured for each of the Q value tables, wherein the weight represents the contribution made by the Q value table when the first comprehensive Q value table is updated;

[0166] The first comprehensive Q value table is obtained based on the weight and each of the Q value tables.

[0167] In some embodiments, the apparatus further comprises:

[0168] Update module, used to update at a preset update rate u t The second comprehensive Q value table generated by the new iteration updates the first comprehensive Q value table generated by the previous iteration:

[0169]

[0170] Said is the weight of the Q value table, is the probability distribution corresponding to the first comprehensive Q value table, K is the number of preset Markov environments, and u t is the update rate, The Q-value table obtained by running the Q-learning algorithm for the nth Markov environment, For each Q value table corresponding to the weight, is the first comprehensive Q value table updated at the t+1 time step, The first comprehensive Q value table updated for time step t.

[0171] Another embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the interference strategy generation method as described in any one of the above embodiments when executing the computer program.

[0172] Another embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the interference strategy generation method described in any one of the above embodiments are implemented.

[0173] It should be noted that the computer storage medium of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program configured to be used by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, antenna, optical cable, RF, etc., or any suitable combination of the above.

[0174] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of protection of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

Claims

1. A method for generating an interference strategy, characterized in that: include: In a confrontation scenario between a jammer and a radar, sampling the confrontation process between the jammer and the radar to obtain an estimated value of a state transfer matrix of the radar; Normalizing the sum of each row of data in the state transfer matrix to 1 to obtain an original Markov environment; Calculating a plurality of synthetic Markov environments based on the estimated value of the state transfer matrix and the original Markov environment; Running the Q-Learning algorithm in parallel in the plurality of synthetic Markov environments to respectively learn and obtain a Q value table, wherein the Q value table includes a plurality of Q values ​​respectively representing the interference degree of each interference action of the jammer on radars in different states, wherein the larger the Q value, the higher the interference efficiency of the interference action on the radar state; The plurality of Q value tables are fused and processed by an adaptive weighting mechanism based on JS divergence to obtain a first comprehensive Q value table; Repeat the above steps to complete a preset number of iterative calculations; In response to the completion of the iterative calculation, a target comprehensive Q value table is obtained, and an optimal interference strategy is determined based on the target comprehensive Q value table, in which the optimal interference actions for interfering with radars in different states are recorded.

2. The interference strategy generation method according to claim 1, characterized in that: In the confrontation scenario between the jammer and the radar, sampling the confrontation process between the jammer and the radar to obtain an estimated value of the state transfer matrix of the radar includes: In a confrontation scenario between a jammer and a radar, the working mode of the radar is obtained based on the reconnaissance of the jammer, and each working mode corresponds to a state of the radar; Randomly applying a jamming action to the radar to detect whether the working mode of the radar is switched; Collect the jamming action of each jamming and the status information of the radar before and after the jamming; estimating element values ​​of a state transfer matrix of the radar based on the collected information to obtain the estimated value; Update the estimated value of the state transfer matrix estimated based on the information collected last time based on the currently obtained estimated value; Repeat the above steps until the sampling times of each state transition pair of the radar meets the threshold requirement, and obtain a state transfer matrix with completed assignment, wherein the state transition pair includes the state before the radar interference and the state after the interference, and the states in different state transition pairs are different.

3. The interference strategy generation method according to claim 1, characterized in that: The multiple synthetic Markov environments are calculated based on the estimated value of the state transfer matrix and the original Markov environment, including: Calculating the nth matrix power of the estimated value of the state transfer matrix, where n is a preset positive integer; The estimated value is represented by an n-th matrix power based on the original Markov environment to obtain n synthetic Markov environments, and the n synthetic Markov environments correspond to different time scales.

4. The interference strategy generation method according to claim 1, characterized in that: The Q-Learning algorithm is run in parallel in the plurality of synthetic Markov environments to respectively learn a Q value table, including: Running the Q-Learning algorithm in parallel in the plurality of synthetic Markov environments to respectively learn the Q function of each state-action pair (s, a), wherein the state-action pair is formed by combining the state s of the radar and the jamming action a applied by the jammer; The Bellman optimal equation is solved based on the Q function to obtain the Q value table containing each of the Q values.

5. The method for generating an interference strategy according to claim 4, characterized in that: The method further comprises: When running the Q-Learning algorithm for learning, a greedy strategy is applied to balance the relationship between exploration and utilization in the learning process, and the learned Q function is updated based on the following formula until the Q value converges with probability 1: Among them, α∈(0,1) is the learning rate, γ is the discount factor, r a (s) is the benefit brought by using interference action a to change the radar state s, Q(s′,a′) is the Q function corresponding to the state-action pair (s′,a′), Q(s,a) is the Q function corresponding to the state-action pair (s,a), and A is the set of interference actions.

6. The method for generating an interference strategy according to claim 5, characterized in that: The Q function includes the optimal strategy function and the optimal value function: The optimal strategy function is: The optimal value function is: Among them, v π is the value function under strategy π, v * is the optimal value function, π * is the optimal strategy, and Q* is the optimal Q function.

7. The method for generating an interference strategy according to claim 1, characterized in that: The adaptive weighting mechanism based on JS divergence fuses and processes the multiple Q value tables to obtain a first comprehensive Q value table, including: Calculate the difference value between the two Q value tables based on the JS divergence algorithm; Based on the difference value, a weight is configured for each of the Q value tables, wherein the weight represents the contribution made by the Q value table when the first comprehensive Q value table is updated; The first comprehensive Q value table is obtained based on the weight and each of the Q value tables.

8. The interference strategy generation method according to claim 1, characterized in that: The method further comprises: At the preset update rate u t The second comprehensive Q value table generated by the new iteration updates the first comprehensive Q value table generated by the previous iteration: Said is the weight of the Q value table, is the probability distribution corresponding to the first comprehensive Q value table, K is the number of preset Markov environments, and u t is the update rate, The Q-value table obtained by running the Q-learning algorithm for the nth Markov environment, For each Q value table corresponding to the weight, is the first comprehensive Q value table updated at the t+1 time step, The first comprehensive Q value table updated for time step t.

9. An interference strategy generating device, characterized in that: include: A sampling module is used to sample the confrontation process between the jammer and the radar in a confrontation scenario between the jammer and the radar, so as to obtain an estimated value of the state transfer matrix of the radar; A normalization module, used for normalizing the sum of each row of data in the state transfer matrix to 1, so as to obtain an original Markov environment; A first calculation module, used for calculating a plurality of synthetic Markov environments according to the estimated value of the state transfer matrix and the original Markov environment; A learning module is used to run the Q-Learning algorithm in parallel in a plurality of the synthetic Markov environments to respectively learn and obtain a Q value table, wherein the Q value table contains a plurality of Q values ​​respectively representing the interference degree of each interference action of the jammer on radars in different states, and the larger the Q value, the higher the interference efficiency of the interference action on the radar state; A first processing module, configured to fuse and process the plurality of Q value tables according to an adaptive weighting mechanism of JS divergence to obtain a first comprehensive Q value table; A control module, used for controlling the above steps to be repeated to complete a preset number of iterative calculations; The response module is used to obtain a target comprehensive Q value table in response to the completion of the iterative calculation, and determine the optimal interference strategy based on the target comprehensive Q value table, wherein the target comprehensive Q value table records the optimal interference actions for interfering with radars in different states.

10. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in the interference strategy generation method according to any one of claims 1 to 8 are implemented.

Images