Multi-jammer collaborative optimization method and system based on IH-MADDPG algorithm, electronic equipment and storage medium

Abstract

The invention discloses a radar intelligent interference decision-making method and system based on an IH-MADDPG algorithm, electronic equipment and a storage medium. The method comprises the following steps: S1, randomly initializing an Actor network, a Critic network and a target network of an upper-layer multi-agent depth deterministic gradient algorithm agent and a lower-layer multi-agent depth deterministic gradient algorithm agent; s2, initializing an upper layer experience playback pool {D1, D2} and a lower layer experience playback pool {D3}; s3, obtaining an initial state st, and obtaining an optimal combined interference pattern (a1t, a2t) from the st by using the depth deterministic gradient of the upper-layer multi-agent; s4, obtaining an optimal combined interference parameter # imgabs0 # S5 from (st, a1t) and (st, a2t) by using a lower-layer multi-agent depth deterministic gradient algorithm, executing an action, changing an environment state into st + 1, calculating environment rewards rt1 and rt2 and a curiosity reward rtb, and storing experience samples into experience playback pools D1, D2 and D3; and S6, when the number of samples reaches a set value, sampling is supervised to obtain uniform samples, and the upper and lower layer multi-agent depth deterministic gradient algorithm and the curiosity reward are updated. According to the method, the radar interference decision performance is improved under the condition of radar networking.

Description

Technical Field

[0001] The present invention belongs to the field of radar cognitive electronic warfare technology, and specifically relates to a multi-jammer collaborative optimization method, system, electronic equipment and storage medium based on an improved hierarchical multi-agent deep deterministic policy gradient (IH-MADDPG) algorithm, and relates to a new radar jamming scenario modeling method. The present invention can be used in the field of cognitive electronic warfare. Background Art

[0002] On the modern battlefield, with increasingly complex combat requirements, the development of radar networking has become an irreversible trend. Compared to a single radar, a radar network integrates multiple radars to form a comprehensive perception system with wide coverage and strong anti-interference capabilities. In this situation, traditional single jammers are no longer sufficient for electronic countermeasures. Therefore, the strategy of multiple jammers working in coordination has gradually become a key means of countering radar networks.

[0003] In recent years, multi-agent systems have been applied in radar countermeasures due to their ability to improve overall system performance and efficiency. However, they also face challenges such as the "curse of dimensionality" caused by the exponential growth of state and action spaces, as well as resource competition and inefficiency caused by communication and coordination difficulties in dynamic environments. Based on this, this paper combines the concept of layering with multi-agent algorithms to propose an IH-MADDPG method to address the challenges posed by radar networks. This method dynamically perceives battlefield situations and allocates jamming resources in real time. Furthermore, using supervised sampling and a curiosity-based optimized reward function, the model's training samples are more balanced and its exploration capabilities are enhanced. Summary of the Invention

[0004] In order to solve the above-mentioned problems existing in the prior art, the present invention proposes a multi-jammer collaborative optimization method, system, electronic device and storage medium based on IH-MADDPG. The present invention improves the radar jamming decision-making performance under the condition of radar networking.

[0005] The application scenarios of the present invention are as follows:

[0006] A jamming countermeasure scenario is established in which two jammers interfere with three radars: the radar working modes of a single radar include search (S), acquisition (Acquisition, Acq), non-adaptive tracking (NAT), range resolution (RR), and track maintenance (TM). In each working mode, the radar signal parameters such as pulse repetition interval (PRI), pulse width (PW), carrier frequency (CF), and peak power (PP) have specific selection ranges. j = ... j}, j = 1, 2, 3 represents the set of all radar working states. Radar parameter information consists of PRI, PW, RF, PP, and is represented by the symbol Signal = {signal1 i ,signal2 i}, i = 1, 2, 3 represents the set of all radar parameter information, where signal11 = [PRI1, PW1, RF1, PP1] represents the signal parameters when radar 1 performs the radar mission on jammer 1. Signal21 is similar. The state space of the final jamming scenario is expressed as:

[0007] S={s t =(Radar,Signal) t}

[0008] Assuming that a jammer can jam two different radars at most, a 1t Indicates the interference mode selected by the first jammer at time t. The optional actions are: RF noise interference (a1), noise amplitude modulation interference (a2), noise frequency modulation interference (a3), a 2t Indicates the jamming style selected by the second jammer at time t. The optional actions are: range deception jamming (a4), range-speed combined false target jamming (a5), spectrum dispersion (SMSP) jamming (a6), slice combination (C&I) jamming (a7), and range dragging jamming (a8). For example: a 1t =[a1,a2,0] means the first jammer uses RF noise to jam the first radar, uses noise amplitude modulation to jam the second radar, and does nothing to the third radar. t 1 =r t 11 +r t 12 +rt 13 Indicates that the first radar reward value (r t 11 ), the reward value of the second radar (r t 12 ) and the reward value of the third radar (r t 13 ) as the total reward for the first jammer. Indicates the interference power, interference timing, and interference duration selected after the first jammer interferes with the first and second radars to generate interference signals. The third radar chooses not to interfere due to limited interference resources, so there is no interference parameter selection.

[0009] This paper draws on the idea of force field to design a new reward function. Assuming that the radar operating mode with the lowest threat level is the jammer's ultimate target, the closer the current radar operating mode is to the ultimate target, the greater the "force" or "reward" on the jammer. Continuous rewards can provide more refined feedback, helping the agent converge to the optimal strategy more quickly. The calculation formula of the jammer's k-reward function is:

[0010]

[0011] Among them, (PRI t ,PW t ,CF t ,PP t ) is the state the radar transitions to after being interfered with. (PRI, PW, CF, PP) is the radar state with the lowest threat level. z is a constant. Because a radar operating mode encompasses countless radar states, we only need to ensure that the radar reward is greater than 0.8 × z to determine that the radar has transitioned to the ideal state. (ω1, ω2, ω3, ω4) are the weights of these dimensions.

[0012] The present invention specifically adopts the following technical solutions:

[0013] A multi-jammer collaborative optimization method based on the IH-MADDPG algorithm includes the following steps:

[0014] S1. Randomly initialize the Actor network, Critic network and corresponding target network of the upper and lower multi-agent deep deterministic gradient MADDPG algorithm agents;

[0015] S2. Initialize the upper and lower network experience replay pools {D1, D2, D3};

[0016] S3. Get the initial state s t , by s t The optimal combined interference pattern (a1t ,a 2t );

[0017] S4, by (s t ,a 1t ) and (s t ,a 2t ) Use the lower layer MADDPG algorithm to obtain the optimal combined interference parameters

[0018] S5, execute the action, the environment state changes to s t+1 , computing environment reward r t 1 、r t 2 and curiosity reward r t b , store the experience samples in the experience replay pools D1, D2, and D3;

[0019] S6. When the number of samples is sufficient (reaches the set value), supervised sampling is performed to obtain uniform samples, and the upper and lower layer MADDPG algorithms and curiosity rewards are updated.

[0020] Furthermore, in step S3, the policy gradient update formula of the upper MADDPG algorithm is:

[0021]

[0022] The loss function is:

[0023]

[0024] Among them, k=1,2 represents the number of agents that need to be trained in the upper-level MADDPG algorithm. represents the total reward of the kth jammer.

[0025] Furthermore, in step S4, the policy gradient update formula of the lower-level MADDPG algorithm is:

[0026]

[0027] The loss function is:

[0028]

[0029] Where i, c = 1, 2, 3, respectively represent the number of agents and radar labels that need to be trained in the lower-level MADDPG algorithm. Further, in step S5, the curiosity reward is calculated by the curiosity module, which includes: t 、 s t+1 , the curiosity module will be based on s t and at To predict

[0030]

[0031] This predicted value is consistent with the true s t+1 Perform differential operation, and the reward is r t b :

[0032]

[0033] Among them, η>0 is a proportional factor, r t b Indicates the future state. The more difficult it is to predict, the greater the reward.

[0034] Furthermore, in step S6, the decision samples stored in the experience replay pool can generally be expressed in the following form:

[0035] D1:

[0036] D2:

[0037] D3:

[0038] Among them, D1, D2 and D3 are the upper and lower experience replay pools of the IH-MADDPG algorithm respectively.

[0039] When extracting data, the interfering party first classifies the samples with the same starting state (the first column value of the sample) in the experience replay pool D1, D2, and D3 into one category. The set of these classified sample subsets is recorded as:

[0040]

[0041] In the formula, m(s t ) function means that only the radar working mode information in the radar state sample at time t is taken. Indicates that the state before the upper experience replay pool was The same goes for D2 and D3.

[0042] Based on the data of completed state classification, samples with the same interference style or interference parameters are further classified into one category based on action information. For samples in a specific state category, after secondary division based on action, the result can be expressed as:

[0043]

[0044] Where, express The interference actions taken in The sample set, Similarly, Discrete processing: the interference timing, interference duration, and interference power are each divided into 20 intervals.

[0045] The present invention also discloses a radar intelligent jamming decision system based on IH-MADDPG, which is used to execute the above method and includes the following modules:

[0046] Initialization module: randomly initializes the Actor network, Critic network, and corresponding target network of the upper and lower MADDPG algorithm agents and the upper and lower experience replay pools;

[0047] Decision module: Get the initial state s t , by s t The optimal combined interference pattern (a 1t ,a 2t ), by (s t ,a 1t ) and (s t ,a 2t ) Use the lower layer MADDPG algorithm to obtain the optimal combined interference parameters Execute the action and the environment state changes to s t+1 , computing environment reward r t 1 、r t 2 and curiosity reward r t b ;

[0048] Sampling module: stores experience samples into experience replay pools D1, D2, and D3;

[0049] Training module: When the number of samples is sufficient (reaches the set value), supervised sampling is performed to obtain uniform samples, and the upper and lower layer MADDPG algorithms and curiosity rewards are updated.

[0050] The present invention also discloses an electronic device, comprising a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the above method or system.

[0051] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor to execute the above method or system.

[0052] Compared with the prior art, the advantages of the present invention are as follows:

[0053] (1) An interference scenario that is closer to the actual electromagnetic countermeasure environment is constructed. In order to achieve a better interference effect, while considering the selection of interference style, the optimization selection of multiple parameters such as interference timing, interference power and interference duration in continuous space is considered.

[0054] (2) In the many-to-many jamming countermeasure scenario, considering the limited jamming resources of the jammer formation and the problem of jamming parameter optimization, a resource allocation and multi-dimensional optimization method is proposed.

[0055] (3) As the dimension of the action space is further expanded, the difficulty of interference optimization is increased. The following methods are adopted: a. The design of the optimized reward function is used to enhance the algorithm's exploration ability; b. The supervised sampling method is used to solve the problem of unbalanced training samples, thereby improving training efficiency and reducing the impact of the enlarged action space on the algorithm's convergence. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of a multi-jammer collaborative optimization method based on the IH-MADDPG algorithm in a preferred embodiment of the present invention;

[0057] Figure 2 This is the principle diagram of the IH-MADDPG algorithm;

[0058] Figure 3 for Figure 2 A magnified image of the IH-MADDPG portion;

[0059] Figure 4 for Figure 2 A partial enlarged view of the reward function;

[0060] Figure 5 Multi-function radar task scheduling flow chart;

[0061] Figure 6 This is a performance comparison chart of the H-MADDPG algorithm in Example 1, MADDPG, and MAPPO algorithms;

[0062] Figure 7 1H-MADDPG algorithm ablation experiment results;

[0063] Figure 8 This is a graph showing the results of an experimental comparison of the effectiveness of the H-MADDPG algorithm in Example 1.

[0064] Figure 9 This is a block diagram of a multi-interference machine collaborative optimization system based on the IH-MADDPG algorithm in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following describes the implementation of the present invention using preferred embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the details in this specification may be modified or altered based on different perspectives and applications without departing from the spirit of the present invention.

[0066] See also Figure 1 This embodiment provides a multi-jammer collaborative optimization method based on the IH-MADDPG algorithm, which specifically includes the following steps:

[0067] S1, randomly initialize the Actor network, Critic network and corresponding target network of the upper and lower MADDPG algorithm agents;

[0068] S2. Initialize the upper and lower network experience replay pools;

[0069] S3. Get the initial state s t , by s t The optimal combined interference pattern (a 1t ,a 2t );

[0070] The policy gradient update formula of the upper MADDPG algorithm is:

[0071]

[0072] The loss function is:

[0073]

[0074] Among them, k=1,2 represents the number of agents that need to be trained in the upper layer MADDPG. represents the total reward of the kth jammer.

[0075] S4, by (s t ,a 1t ) and (s t ,a 2t ) Use the lower layer MADDPG algorithm to obtain the optimal combined interference parameters

[0076] The lower-level MADDPG algorithm policy gradient update formula is:

[0077]

[0078] The loss function is:

[0079]

[0080] Where i, c = 1, 2, and 3, respectively, represent the number of agents and radar labels that need to be trained in the lower-level MADDPG algorithm.

[0081] S5, execute the action, the environment state changes to s t+1 , computing environment reward r t 1 、r t 2 and curiosity reward r t b , store the experience samples in the replay pools D1, D2, and D3.

[0082] The curiosity reward is calculated by the curiosity module, which includes: t 、 s t+1 , the curiosity module will be based on s t and a t To predict

[0083]

[0084] This predicted value is consistent with the true s t+1 Perform differential operation, and the reward is r t b :

[0085]

[0086] Among them, η>0 is a proportional factor, r t b It means that the more difficult it is to predict the future state, the greater the reward will be.

[0087] S6. When the number of samples is sufficient, supervised sampling is performed to obtain uniform samples, and the upper and lower layer MADDPG algorithms and curiosity rewards are updated.

[0088] The decision samples stored in the IH-MADDPG algorithm experience replay pool can generally be expressed in the following form:

[0089] D1:

[0090] D2:

[0091] D3:

[0092] Among them, D1, D2 and D3 are the upper and lower experience replay pools of the IH-MADDPG algorithm respectively.

[0093] When extracting data, the interfering party first classifies the samples with the same starting state (the first column value of the sample) in the experience replay pool D1, D2, and D3 into one category. The set of these classified sample subsets is recorded as:

[0094]

[0095] In the formula, m(s t ) function means that only the radar working mode information in the radar state sample at time t is taken. Indicates that the state before the upper experience replay pool was The same goes for D2 and D3.

[0096] Based on the data of completed state classification, samples with the same interference style or interference parameters are further classified into one category based on action information. For samples in a specific state category, after secondary division based on action, the result can be expressed as:

[0097]

[0098] Where, express The interference actions taken in The sample set, Similarly, Discrete processing: the interference timing, interference duration, and interference power are each divided into 20 intervals.

[0099] The overall principle is as follows Figure 2-4 As shown in the figure, after the signal generated by the radar network is intercepted, the jammer formation uses the IH-MADDPG algorithm to select the optimal combined jamming signal, and then Figure 5 The transfer calculation of the radar network state after interference is rewarded, and the reward and radar network state are stored in the experience replay pool. When the number of samples is sufficient, the supervised sampling method is used to obtain experience samples to train the IH-MADDPG algorithm.

[0100] In order to verify the effectiveness of the present invention, the jammer uses IH-MADDPG, MADDPG, and MAPPO algorithms for comparative experiments. The simulation results are as follows: Figure 6 As shown in the figure, it can be seen that the IH-MADDPG algorithm converges after about 180 rounds of training. Compared with the unimproved MADDPG (370 rounds) and MAPPO (420 rounds) algorithms, the convergence speed is increased by 51% and 57% respectively. At the same time, it finds the fastest interference strategy with the shortest time step to complete the interference task, with the best timeliness.

[0101] The IH-MADDPG algorithm without curiosity mechanism is named H-MADDPG2, and the traditional H-MADDPG is named H-MADDPG1. Figure 7 As shown in Figure 3, the convergence speed of IH-MADDPG is 28% faster than that of H-MADDPG2 (250 rounds) and 40% faster than that of H-MADDPG1 (300 rounds).

[0102] In order to determine the impact of the choice of interference parameters on the final interference effect, the algorithm with fixed interference parameters is set to IH-MADDPG-2, and the algorithm with parameter optimization capability is set to IH-MADDPG-1. The performance comparison is as follows: Figure 8 As shown in the figure, the curve of the IH-MADDPG-1 algorithm also includes the interference parameters in the optimization range. Compared with the IH-MADDPG-2 algorithm obtained by only optimizing the interference pattern under the condition of fixed interference parameters, the reward value after convergence is increased by 17%, which shows that the interference effect of the interference signal on the radar has been improved after the optimization of the parameters. At the same time, the time step required for the optimal strategy of the algorithm is reduced by 10%, which means that the time required to achieve the optimal interference effect is greatly shortened, and the timeliness of the jammer is effectively improved.

[0103] like Figure 9 As shown, this embodiment discloses a multi-interference machine collaborative optimization system based on IH-MADDPG, which is used to execute the above method embodiment, including the following modules:

[0104] Initialization module: randomly initializes the Actor network and Critic network of the upper and lower MADDPG algorithm agents, as well as the corresponding target network and the upper and lower network experience replay pools;

[0105] Decision module: Get the initial state s t , by s t The optimal combined interference pattern (a 1t ,a 2t ), by (s t ,a 1t ) and (s t ,a 2t ) Use the lower layer MADDPG algorithm to obtain the optimal combined interference parameters Execute the action and the environment state changes to s t+1 , computing environment reward r t 1 、r t 2 and curiosity reward r t b ;

[0106] Sampling module: stores experience samples into experience replay pools D1, D2, and D3;

[0107] Training module: When the number of samples is sufficient, supervised sampling is performed to obtain uniform samples, and the upper and lower layer MADDPG algorithms and curiosity rewards are updated.

[0108] For other contents of this embodiment, please refer to the above method embodiment.

[0109] A preferred embodiment of the present invention further discloses an electronic device, comprising a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the above method or system.

[0110] A preferred embodiment of the present invention further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor to execute the above method or system.

[0111] The above are merely preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions are possible for those skilled in the art without departing from the scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. The multi-jammer collaborative optimization method based on the IH-MADDPG algorithm is characterized by: The following steps are involved: S1. Randomly initialize the Actor network, Critic network, and target network of the upper and lower multi-agent deep deterministic gradient algorithm agents; S2. Initialize the upper and lower layer experience replay pools {D1, D2, D3}; S3. Get the initial state s t , by s t The optimal combined interference pattern (a 1t ,a 2t ); S4, by (s t ,a 1t ) and (s t ,a 2t ) Use the lower-level multi-agent deep deterministic gradient algorithm to obtain the optimal combined interference parameters S5, execute the action, the environment state changes to s t+1 , computing environment reward r t 1 、r t 2 and curiosity reward r t b , store the experience samples in the experience replay pools D1, D2, and D3; S6. When the number of samples reaches the set value, supervised sampling is performed to obtain uniform samples, and the upper and lower multi-agent deep deterministic gradient algorithms and curiosity rewards are updated.

2. The multi-interference machine collaborative optimization method based on the IH-MADDPG algorithm according to claim 1 is characterized in that: In step S3, the upper-layer multi-agent deep deterministic gradient algorithm policy gradient update formula is: The loss function is: Where k = 1, 2, represents the number of agents that need to be trained in the upper-layer multi-agent deep deterministic gradient algorithm. represents the total reward of the kth jammer, For the current network, is the target network, E is the expectation, is to find the gradient, is the policy function, θ k is the network parameter, s t is the state of the radar at time t, a 1t 、a 2t The jamming signals selected by the two jammers for the radar status, r t b 、r t k It is the environmental reward obtained according to the radar state transition after interference and the calculated curiosity reward.

3. The multi-interference machine collaborative optimization method based on the IH-MADDPG algorithm according to claim 2 is characterized in that: In step S4, the policy gradient update formula of the lower-layer multi-agent deep deterministic gradient algorithm is: The loss function is: Among them, i, c = 1, 2, 3, respectively represent the number of agents and radar labels that need to be trained in the lower-level multi-agent deep deterministic gradient algorithm.

4. The multi-interference machine collaborative optimization method based on the IH-MADDPG algorithm according to any one of claims 1 to 3, characterized in that: In step S5, the curiosity reward is calculated by the curiosity module, which includes: t 、 s t+1 , the curiosity module is based on s t and a t predict This predicted value is consistent with the true s t+1 Perform differential operation and the reward is r t b : Where η>0 is the scaling factor, and φ(·) represents the feature extraction function.

5. The multi-interference machine collaborative optimization method based on the IH-MADDPG algorithm according to claim 4 is characterized in that: In step S6, the decision samples stored in the experience replay pool are represented in the following form: Among them, D1 and D2 are the upper experience replay pool of the IH-MADDPG algorithm, and D3 is the lower experience replay pool of the IH-MADDPG algorithm; When extracting data, the interfering party first classifies the samples with the same starting state in the experience replay pools D1, D2, and D3 into one category. The set of classified sample subsets is recorded as: In the formula, m(s t ) function means that only the radar working mode information in the radar state sample at time t is taken. Indicates that the state before the upper experience replay pool was The sample set of Based on the action information, samples with the same interference style or interference parameters are classified into one category. For samples in a specific state category, after secondary division based on action, the result is expressed as: Where, express The interference actions taken in The sample set will Discretization processing divides the interference opportunity, interference duration, and interference power into a set number of intervals.

6. A multi-interference machine collaborative optimization system based on the IH-MADDPG algorithm, used to execute the method according to any one of claims 1 to 5, characterized in that: Includes the following modules: Initialization module: randomly initializes the Actor network, Critic network, target network, and upper and lower layer experience replay pools of the upper and lower layer multi-agent deep deterministic gradient algorithm agents; Decision module: Get the initial state s t , by s t The optimal combined interference pattern (a 1t ,a 2t ), by (s t ,a 1t ) and (s t ,a 2t ) Use the lower-level multi-agent deep deterministic gradient algorithm to obtain the optimal combined interference parameters Execute the action and the environment state changes to s t+1 , computing environment reward r t 1 、r t 2 and curiosity reward r t b ; Sampling module: stores experience samples into experience replay pools D1, D2, and D3; Training module: When the number of samples reaches the set value, supervised sampling is performed to obtain uniform samples, and the upper and lower layer multi-agent deep deterministic gradient algorithms and curiosity rewards are updated.

7. An electronic device, characterized in that: The system comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the method according to any one of claims 1 to 5 or the system according to claim 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, where the instructions are suitable for being loaded by a processor to execute the method according to any one of claims 1 to 5 or the system according to claim 6.

Images