Radar adaptive interference decision-making method based on confidence coefficient upper bound search

Abstract

The invention discloses a self-adaptive radar interference decision-making method based on confidence coefficient upper bound search, which realizes self-adaptive interference decision-making in a radar confrontation process. According to the implementation scheme, interference decision confidence search and deep reinforcement learning are adopted, and the method comprises the steps that 1, a radar self-adaptive confrontation decision environment is constructed; 2, constructing an interference action confidence tree, an adversarial parameter generator and a decision performance evaluator; 3, updating interference decision income confidence through confrontation action confidence search; and 4, updating the interference parameter generator and the decision performance evaluator through reinforcement learning training. According to the method, the confrontation decision income confidence coefficient is evaluated, the interference strategy is adjusted according to the confrontation decision income confidence coefficient, the optimal or approximately optimal decision path can be found under the uncertain condition, and the self-adaptive decision capability of radar confrontation in the battlefield dynamic environment is improved.

Description

Technical Field

[0001] The present application relates to the field of electronic countermeasure technology, and in particular to a radar adaptive interference decision method based on confidence upper bound search. Background Art

[0002] Radar countermeasures are an important part of modern electronic countermeasures. In order to counter the enemy's radar detection, active jamming is one of the typical countermeasures. Traditional radar active jamming methods often rely on preset rules or fixed jamming style combinations, which are difficult to cope with the complex and changeable electromagnetic environment of the battlefield, especially for multifunctional advanced physical radars. Fixed jamming strategies are difficult to guarantee the effectiveness of the countermeasures.

[0003] In recent years, artificial intelligence technology has achieved rapid development. Among them, artificial intelligence technology represented by deep learning has been widely used in the field of electronic countermeasures. The current intelligent methods of radar jamming decision-making mainly include:

[0004] (1) Interference decision based on Q-learning: Based on the radar prior knowledge base, the researchers constructed a radar working state transition model, set the interference style benefit matrix, and selected the interference style under different working modes of the radar through the iteration of the Q-learning algorithm. The above method has two main shortcomings. First, the use of the preset radar state transition model makes the interference decision lack of generalization and scalability, and the interference effectiveness is difficult to guarantee; second, the interference decision only involves the selection of interference style, and lacks the joint optimization of interference parameters.

[0005] (2) Interference decision-making based on knowledge graph: Based on radar models, jammer models, and confrontation scenarios, researchers established an interference confrontation entity relationship graph database through knowledge mining, and formed a comprehensive confrontation strategy based on real-time confrontation situation analysis. The main disadvantage of the above method is that the modeling of the knowledge graph relies on the prior knowledge of a large number of experts. In the face of unknown and highly dynamic confrontation situations, the adaptive correction ability of interference decision-making is weak.

[0006] Therefore, how to adaptively adjust the jamming strategy in real time based on the jamming countermeasure effect and find the optimal or approximately optimal jamming strategy combination under uncertain conditions has become a current research hotspot. Summary of the invention

[0007] The present application provides a radar adaptive jamming decision method based on confidence upper bound search, which can be used to solve the technical problem of how to adaptively adjust the jamming strategy in real time according to the jamming countermeasure effect, and find the optimal or approximately optimal jamming strategy combination under uncertain conditions.

[0008] The present application discloses an adaptive radar jamming decision method based on confidence upper bound search, the method comprising the following steps:

[0009] Step 1: Construct an adaptive interference decision-making countermeasure environment and standardize the input state space of interference decision-making And the output action space

[0010] Step 2: Construct the interference action confidence tree Tr(A) and parameter generator G θ (s), performance evaluator Q ω (s,a);

[0011] Step 3: Calculate the confidence of the interference action UCB(A) and generate the interference action parameter G according to the current state s θ (s), execute the interference action with the maximum confidence and the corresponding parameters, feedback the actual interference action benefit r and the adversarial transition state s', record the adversarial process data (s, a, r, s'), and form an adversarial data set;

[0012] Step 4: Randomly sample the adversarial data set (s, a, r, s′) and use reinforcement learning to train the parameter generator G θ (s), performance evaluator Q ω (s, a); Repeat steps 3 and 4 for multiple rounds of interaction with the adversarial environment until the interference action confidence, parameter generator, and performance evaluator converge.

[0013] Furthermore, the radar countermeasure environment is constructed, the jamming action confidence tree is constructed, the countermeasure parameter generator is constructed, and the decision effectiveness evaluator is constructed. This includes step 1 and step 2.

[0014] Furthermore, a jammer decision-making confrontation environment is constructed, including both radar and jammer. During the confrontation process, the jammer receives the radar electromagnetic signal to form the jammer decision input. The jammer implements the jamming action of the adaptive jamming decision output Generates actual adversarial waveforms.

[0015] Furthermore, the interference decision state space and action space as follows:

[0016] Disturbance decision input state space Contains the threat target pulse description information PDW and the radiation source electromagnetic description information EDW; wherein PDW and EDW contain the carrier frequency RF, pulse repetition interval PRI, pulse width PW, pulse amplitude PA, bandwidth BW, azimuth DoA, intra-pulse modulation IPM and other information of the countermeasure radar;

[0017] Interference decision output action space It includes repetition frequency tracking decision and active interference decision. The specific repetition frequency tracking decision includes tolerance setting, capture mode, and corresponding tracking parameters; the interference styles include broadband blocking, narrowband aiming, comb spectrum, leading edge replication, towed false targets, dense false targets, etc., as well as the corresponding waveform parameters.

[0018] Furthermore, the interference action confidence tree Tr(A) is constructed, including the repetition frequency tracking A track Active jammer; repetition tracking action includes threat pulse filter tolerance setting Pulse capture mode setting The repetition tracking action contains several specific parameters Active jamming action includes jamming gate advance pattern Wave gate delay style Wave gate movement style The interference style action contains several specific parameters Using the generalized table, the interference action confidence tree is expressed as follows:

[0019]

[0020] Construct the interference action parameter generator G θ (s), the input is the interference confrontation state s, and the output is the interference action parameter a=[a 1 ,…,a k ,…,a M ], specifically, a k is the specific parameter associated with a certain interference action in the confidence tree Tr(A), expressed as {a k |Tr(A)}, the generator adopts a typical neural network structure, the network weight parameter is θ, and it is randomly initialized;

[0021] Constructing the interference action effectiveness evaluator Q ω (s,a), the input is the interference confrontation state s, the interference action parameter a, and the output is the interference action effectiveness evaluation q = [q 1 ,…,q k ,....,q N ], specifically, q k It is the effectiveness evaluation value associated with a certain interference action in the confidence tree Tr(A). The evaluator adopts a typical neural network structure, the network weight parameter is ω, and it is randomly initialized.

[0022] Furthermore, in the adaptive interference countermeasure process, the interference action confidence is updated, the countermeasure parameter generator and the decision effectiveness evaluator are updated, including step 3 and step 4.

[0023] Furthermore, the confidence UCB(A) of the interference action is calculated in step 3, and the interference action parameter G is generated according to the current state s. θ(s), execute the interference action with the maximum confidence and the corresponding parameters, feedback the actual interference action benefit r and the adversarial transition state s′, record the adversarial process data (s, a, r, s′), and form an adversarial data set, including:

[0024] Step 3.1: Calculate the interference action confidence UCB(A) by the following method:

[0025]

[0026] Among them A parent With A child Represents the upper and lower actions in the interference action confidence tree Tr(A). Specifically, for repetition frequency tracking and active interference, it is represented as {A track ,A jam |A}, the specific action setting for repetition frequency tracking is expressed as The specific style setting for active interference is expressed as N(A parent ) and N(A child ) represents the number of executions of the upper-level interference action and the lower-level action, R(A child ) represents the cumulative benefit of the interference action execution, and the coefficient c is the preset value of the interference action exploration;

[0027] Step 3.2: According to the confidence values ​​of the interference actions at each level, the interference action decision chain with the highest benefit is obtained, which is expressed as follows:

[0028] A opt = argmax{A parent →A child |UCB(A)}

[0029] Step 3.3: Input the current adversarial state s, according to the interference parameter generator G θ (s), generate interference action parameter a, associated with the highest benefit interference action A in step 3.2 opt , and obtain the corresponding interference action implementation parameters {a i |A opt};

[0030] Step 3.4: Execute the current optimal interference action A opt With the corresponding parameter {a i |A opt}, based on the radar confrontation effect, the current interference action benefit r and confrontation state transition s' are fed back, and the confrontation process data (s, a, r, s′) is recorded to form a confrontation data set.

[0031] Furthermore, the randomly sampled adversarial data set (s, a, r, s′) described in step 4 is trained using reinforcement learning parameter generator G θ(s), performance evaluator Q ω (s, a); Repeat steps 3 and 4 to interact with the adversarial environment for multiple rounds until the interference action confidence, parameter generator, and effectiveness evaluator converge, including:

[0032] Step 4.1: Randomly sample the interference adversarial data set (s, a, r, s′) to form the reinforcement learning data of the current generator and evaluator;

[0033] Step 4.2: Fix the generator parameters θ and update the interference action effectiveness evaluator Q ω (s, a), calculate the expected return of the current interference action, including the current return and the future estimated return. The calculation method is as follows:

[0034]

[0035] Using the minimum mean square error, the loss function of the performance evaluator is minimized by gradient descent to train the network parameters ω as follows:

[0036]

[0037] Where Q ω (s,a) is the estimated benefit of the current interference action;

[0038] Step 4.3: Fix the performance evaluator parameter ω and update the interference action parameter generator G θ (s), maximize the benefit of the interference action by gradient descent, and train the network parameters θ as follows:

[0039] maximize∑Q(G θ (s))

[0040] Step 4.4: Interact with the adversarial environment, repeat steps 3 and 4, update the interference action confidence UCB(A), and update the interference action effectiveness evaluator Q ω (s,a), update the interference parameter generator G θ (s), until the interference action confidence, parameter generator, and effectiveness evaluator converge.

[0041] Compared with the prior art, the beneficial effects of the present invention are as follows: constructing an interference decision action confidence tree, a confrontation parameter generator, and a decision effectiveness evaluator, and improving the adaptive interference decision-making capability in the radar confrontation process through interference action confidence search and reinforcement learning training. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the process of radar adaptive interference decision in the present invention;

[0043] Figure 2A schematic diagram of constructing an interference action confidence tree in the present invention;

[0044] Figure 3 It is a schematic diagram of the construction of the interference parameter generator and the interference effectiveness evaluator in the present invention. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0046] The following first introduces the embodiments of the present application in conjunction with the accompanying drawings.

[0047] The present invention proposes a radar adaptive jamming countermeasure method based on jamming strategy confidence search, which combines multi-level jamming action confidence search with parameter generation and effectiveness evaluation reinforcement learning algorithm, and adjusts the jamming strategy adaptively in real time according to the jamming countermeasure effect. It can find the optimal or approximately optimal jamming strategy combination under uncertain conditions, and provide a new research idea for adaptive radar countermeasure decision-making in a dynamic battlefield environment.

[0048] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. The accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0049] A specific embodiment of the present invention, combined with Figure 1 The adaptive decision-making process of radar jamming confrontation includes the construction of confrontation environment, confrontation parameter generation network G_NET, decision effectiveness evaluation network Q_NET, and interference action confidence search UCB. The adaptive update of the jamming confrontation strategy is realized through reinforcement learning. The specific implementation process is as follows:

[0050] Step 1: Construct an adaptive interference decision-making adversarial environment and standardize the input state space of interference decision-making And the output action space The specific implementation is as follows:

[0051] Constructing a jamming decision-making confrontation environment: including both radar and jammer. During the confrontation process, the jammer receives the radar electromagnetic signal to form a jamming decision input. The jammer implements the jamming action of the adaptive jamming decision output Generate actual adversarial waveforms to specifically interfere with the decision state space and action space as follows:

[0052] Normalized interference decision input state space Contains threat target pulse description information (PDW) and radiation source electromagnetic description information (EDW). Specifically, PDW and EDW contain information such as carrier frequency (RF), pulse repetition interval (PRI), pulse width (PW), pulse amplitude (PA), bandwidth (BW), direction (DoA), intra-pulse modulation (IPM) of the countermeasure radar. The specific PDW information and EDW information are as follows:

[0053] Table 1: Radar PDW state space table

[0054] Field Name Field meaning ToA Arrival time ns RF Carrier frequency value MHz DoA True bearing, 0 degrees - 359.9 degrees PA Pulse amplitude value dBm PW Pulse width us BW Bandwidth MHz

[0055] Table 2: Radar EDW state space table

[0056] Field Name Field meaning THL Target Threat Level RFT Carrier frequency type RF

[32] Carrier frequency value MHz DoA True bearing, 0 degrees - 359.9 degrees PA Pulse amplitude value dBm PWT Pulse width type PW

[32] Pulse width value us PRIT Pulse repetition interval type PRI

[32] Pulse repetition interval value us PRIGRP

[32] Number of pulse groups InPulseMod Intra-pulse modulation type BW Modulation bandwidth MHz

[0057] Normalized interference decision output action space It includes repetition frequency tracking decision and active interference decision. The specific repetition frequency tracking includes tolerance setting, capture mode, and corresponding tracking parameters; the specific interference styles include broadband blocking, narrowband aiming, comb spectrum, leading edge replication, towed false targets, dense false targets, etc., as well as the corresponding waveform parameters.

[0058] Step 2: Construct interference action confidence tree Tr(A) and parameter generator G θ (s), performance evaluator Q ω (s,a), combined Figure 2 and Figure 3 , the specific implementation is as follows;

[0059] Step 2.1: Construct the interference action confidence tree Tr(A), and track A by repetition frequency track With active jammer A jam Composition, repetition tracking action includes threat pulse filter tolerance setting Pulse capture mode setting The repetition tracking action contains several specific parameters Active jamming action includes jamming gate advance pattern Wave gate delay style Wave gate movement style The interference style action contains several specific parameters Using the generalized table, the above interference action confidence tree is expressed as follows:

[0060]

[0061] Step 2.2: Construct the perturbation action parameter generator G θ (s), the input is the interference confrontation state s, and the output is the interference action parameter a=[a 1 ,…,a k ,…,aM ], specifically, a k is the specific parameter associated with a certain interference action in the confidence tree Tr(A), expressed as {a k |Tr(A)}, the generator adopts a typical neural network structure, the network weight parameter is θ, and it is randomly initialized;

[0062] Step 2.3: Construct the interference action effectiveness evaluator Q ω (s,a), the input is the interference confrontation state s, the interference action parameter a, and the output is the interference action effectiveness evaluation q = [q 1 ,…,q k ,....,q N ], specifically, q k It is the effectiveness evaluation value associated with a certain interference action in the confidence tree Tr(A). The evaluator adopts a typical neural network structure, the network weight parameter is ω, and it is randomly initialized.

[0063] Step 3: Calculate the confidence of the interference action UCB(A) and generate the interference action parameter G according to the current state s θ (s), execute the interference action with the maximum confidence and the corresponding parameters, feedback the actual interference action benefit r and the adversarial transition state s', record the adversarial process data (s, a, r, s'), and form an adversarial data set, as follows:

[0064] Step 3.1: Calculate the interference action confidence UCB(A), as follows:

[0065]

[0066] Among them A parent With A child Represents the upper and lower actions in the interference action confidence tree Tr(A). Specifically, for repetition frequency tracking and active interference, it is represented as {A track ,A jam |A}, the specific action setting for repetition frequency tracking is expressed as The specific style setting for active interference is expressed as N(A parent ) and N(A child ) represents the number of executions of the upper-level interference action and the lower-level action, R(A child ) represents the cumulative benefit of the interference action execution, and the coefficient c is the preset value of the interference action exploration;

[0067] Step 3.2: According to the confidence values ​​of the interference actions at each level, the interference action decision chain with the highest benefit is obtained, which is expressed as follows:

[0068] A opt = argmax{A parent →Achil d|UCB(A)}

[0069] Step 3.3: Input the current adversarial state s, according to the interference parameter generator G θ (s), generate interference action parameter a, associated with the highest benefit interference action A in step 3.2 opt , and obtain the corresponding interference action implementation parameters {a i |A opt};

[0070] Step 3.4: Execute the current optimal interference action A opt With the corresponding parameter {a i |A opt}, based on the radar confrontation effect, the current interference action benefit r and confrontation state transition s' are fed back, and the confrontation process data (s, a, r, s′) is recorded to form a confrontation data set.

[0071] Step 4: Randomly sample the adversarial dataset (s, a, r, s′) and use reinforcement learning to train the parameter generator G θ (s), performance evaluator Q ω (s, a); Repeat steps 3 and 4 to interact with the adversarial environment for multiple rounds until the interference action confidence, parameter generator, and effectiveness evaluator converge. Figure 1 and Figure 3 , as follows:

[0072] Step 4.1: Randomly sample the interference adversarial data set (s, a, r, s′) to form the reinforcement learning data of the current generator and evaluator;

[0073] Step 4.2: Fix the generator parameters θ and update the interference action effectiveness evaluator Q ω (s, a), calculate the expected return of the current interference action, including the current return and the estimated future return, as follows:

[0074]

[0075] The minimum mean square error is used to minimize the loss function of the performance evaluator through gradient descent to train the network parameters ω, as follows:

[0076]

[0077] Where Q ω (s,a) is the estimated benefit of the current interference action;

[0078] Step 4.3: Fix the performance evaluator parameter ω and update the interference action parameter generator G θ (s), maximize the interference action benefit by gradient descent, and train the network parameters θ as follows:

[0079] maximize∑Q(G θ (s))

[0080] Step 4.4: Repeat steps 3 and 4 to interact with the adversarial environment, and update the interference action confidence UCB(A) and the interference action effectiveness evaluator Q through the confidence formula and the loss function reverse gradient propagation. ω (s,a), update the interference parameter generator G θ (s), until the interference action confidence, parameter generator, and performance evaluator converge. Specifically, after the pre-model training is completed, each interference action q k Whether the value tends to converge, if it reaches the preset convergence standard, save the current parameter generation network and performance evaluation network, otherwise update the current radar state s t+1 , and jump to step 3 for a new round of interference decision and strategy learning.

[0081] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

[0082] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. An adaptive radar jamming decision method based on confidence upper bound search, characterized in that: The method comprises the following steps: Step 1: Construct an adaptive interference decision-making confrontation environment and standardize the input state space S and output action space of the interference decision Step 2: Construct the interference action confidence tree Tr(A) and parameter generator G θ (s), performance evaluator Q ω (s, a); Step 3: Calculate the confidence of the interference action UCB(A) and generate the interference action parameter G according to the current state s θ (s), execute the interference action with the maximum confidence and the corresponding parameters, feedback the actual interference action benefit r and the adversarial transition state s′, record the adversarial process data (s, a, r, s′), and form an adversarial data set; Step 4: Randomly sample the adversarial data set (s, a, r, s′) and use reinforcement learning to train the parameter generator G θ (s), performance evaluator Q ω (s, a); Repeat steps 3 and 4 for multiple rounds of interaction with the adversarial environment until the interference action confidence, parameter generator, and performance evaluator converge.

2. The adaptive radar interference decision-making method for confidence upper bound search according to claim 1 is characterized in that: Interference Decision State Space and action space as follows: Disturbance decision input state space Contains threat target pulse description information PDW and radiation source electromagnetic description information EDW; wherein PDW and EDW contain the carrier frequency RF, pulse repetition interval PRI, pulse width PW, pulse amplitude PA, bandwidth BW, azimuth DoA, and intra-pulse modulation IPM of the countermeasure radar; Interference decision output action space It includes repetition frequency tracking decision and active interference decision. The specific repetition frequency tracking decision includes tolerance setting, capture mode, and corresponding tracking parameters; the interference styles include broadband blocking, narrowband aiming, comb spectrum, leading edge replication, towed false targets, dense false targets, etc., as well as the corresponding waveform parameters.

3. The adaptive radar interference decision-making method for confidence upper bound search according to claim 1, characterized in that: Construct the interference action confidence tree Tr(A), including repetition frequency tracking A track Active jammer; repetition tracking action includes threat pulse filter tolerance setting Pulse capture mode setting The repetition tracking action contains several specific parameters Active jamming action includes jamming gate advance pattern Wave gate delay style Wave gate movement style The interference style action contains several specific parameters Using the generalized table, the interference action confidence tree is expressed as follows: Construct the interference action parameter generator G θ (s), the input is the interference confrontation state s, and the output is the interference action parameter a=[a1,...,a k , ..., a M ], specifically, a k is the specific parameter associated with a certain interference action in the confidence tree Tr(A), expressed as {a k |Tr(A)), the generator adopts a typical neural network structure, the network weight parameter is θ, and is randomly initialized; Constructing the interference action effectiveness evaluator Q ω (s, a), the input is the interference confrontation state s, the interference action parameter a, and the output is the interference action effectiveness evaluation q = [q1, ..., q k , …, q N ], specifically, q k It is the effectiveness evaluation value associated with a certain interference action in the confidence tree Tr(A). The evaluator adopts a typical neural network structure, the network weight parameter is ω, and it is randomly initialized.

4. The adaptive radar interference decision-making method of confidence upper bound search according to claim 1, characterized in that: Step 3 calculates the confidence UCB(A) of the interference action and generates the interference action parameter G according to the current state s θ (s), execute the interference action with the maximum confidence and the corresponding parameters, feedback the actual interference action benefit r and the adversarial transition state s′, record the adversarial process data (s, a, r, s′), and form an adversarial data set, including: Step 3.1: Calculate the interference action confidence UCB(A) by the following method: Among them A parent With A child The interference action confidence tree Tr(A) represents the upper and lower actions. For repetition frequency tracking and active interference, it is represented as {A track , A jam |A}, the specific action setting for repetition frequency tracking is expressed as The specific style setting for active interference is expressed as N(A parent ) and N(A child ) represents the number of executions of the upper-level interference action and the lower-level action, R(A child ) represents the cumulative benefit of the interference action execution, and the coefficient c is the preset value of the interference action exploration; Step 3.2: According to the confidence values ​​of the interference actions at each level, the interference action decision chain with the highest benefit is obtained, which is expressed as follows: A opt =argmax{A parent →A child |UCB(A)) Step 3.3: Input the current adversarial state s, according to the interference parameter generator G θ (s), generate interference action parameter a, associated with the highest benefit interference action A in step 3.2 opt , and obtain the corresponding interference action implementation parameters {a i |A opt }; Step 3.4: Execute the current optimal interference action A opt With the corresponding parameter {a i |A opt }, based on the radar confrontation effect, the current interference action benefit r and confrontation state transition s′ are fed back, and the confrontation process data (s, a, r, s′) are recorded to form a confrontation data set.

5. The adaptive radar interference decision-making method of confidence upper bound search according to claim 1, characterized in that: The randomly sampled adversarial data set (s, a, r, s′) described in step 4 uses reinforcement learning to train the parameter generator G θ (s), performance evaluator Q ω (s, a); Repeat steps 3 and 4 for multiple rounds of interaction with the adversarial environment until the interference action confidence, parameter generator, and effectiveness evaluator converge, including: Step 4y1: Randomly sample the interference adversarial data set (s, a, r, s′) to form the reinforcement learning data of the current generator and evaluator; Step 4.2: Fix the generator parameters θ and update the interference action effectiveness evaluator Q ω (s, a), calculate the expected return of the current interference action, including the current return and the future estimated return, the calculation method is as follows: Using the minimum mean square error, the loss function of the performance evaluator is minimized by gradient descent to train the network parameters ω as follows: Where Q ω (s,a) is the estimated benefit of the current interference action; Step 4.3: Fix the performance evaluator parameter ω and update the interference action parameter generator G θ (s), maximize the benefit of the interference action by gradient descent, and train the network parameters θ as follows: maximize∑Q(G θ (s)) Step 4.4: Interact with the adversarial environment, repeat steps 3 and 4, update the interference action confidence UCB(A), and update the interference action effectiveness evaluator Q ω (s,a), update the interference parameter generator G θ (s), until the interference action confidence, parameter generator, and effectiveness evaluator converge.

Images