Communication interference strategy generation method based on multistage pruning lightweight network

Through the multi-level pruning and PDQN algorithm of the stratified reinforcement learning network, the problem of insufficient computing power of small reconnaissance jamming equipment is solved, and the communication interference strategy generation under low computing power is realized, which improves interference effect and resource utilization optimization.

CN118984206APending Publication Date: 2024-11-19THE 54TH RESEARCH INSTITUTE OF CHINA ELECTRONICS TECHNOLOGY GROUP CORPORATION
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411067906.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Due to computing power and power limitations, small reconnaissance jamming equipment such as small drones cannot effectively use interference strategies based on reinforcement learning to generate networks, resulting in slow operation and poor real-time performance.

Method used

The pre-trained hierarchical reinforcement learning network is compressed by using multi-level pruning technology to form a lightweight network, reducing model size and resource requirements, and hierarchical interaction and feedback of interference types and parameters are realized through the PDQN hierarchical reinforcement learning algorithm. Improve exploration capabilities and avoid local optimum.

Benefits of technology

Under low computing power conditions, the autonomous interference signal parameter control of communication interference equipment is realized, which improves the optimal solution for interference effect and resource utilization, avoids performance losses, and is suitable for communication confrontation fields.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984206A_ABST
    Figure CN118984206A_ABST
Patent Text Reader

Abstract

The invention discloses a communication interference strategy generation method based on a multistage pruning lightweight network, and belongs to the technical field of artificial intelligence and reinforcement learning. The method comprises the steps that a PDQN-based multi-layer reinforcement learning network model is used for communication interference strategy generation modeling, multi-stage pruning is used for model lightweight so as to conform to deployment on a low-computing-power platform, the lightweight model comprises Q-Network and Actor, interference types and corresponding parameters are output respectively, the training speed is increased by adopting a dual-order threshold strategy, and the model weight is reduced. And finally, a reward function based on an adversarial strategy is constructed for guiding frequency alignment, and the model realizes the target bit error rate with less interference resources. Through the communication interference strategy generation method based on the multistage pruning lightweight network, intelligent cognitive interference of a communication countermeasure party to a communication party is realized, the deployment problem of the communication countermeasure party on a platform with relatively low computing power is solved through a lightweight method, and meanwhile, a plurality of updating acceleration methods are designed for the model, so that the learning efficiency is improved, and the communication countermeasure requirement is met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and reinforcement learning, and in particular refers to a communication interference strategy generation method based on a multi-level pruning lightweight network. Background Art

[0002] Some communication equipment has cognitive anti-countermeasures, which can judge whether it is being interfered with and automatically adjust the signal pattern according to a certain strategy. In this case, reconnaissance and jamming equipment is needed to complete perception and automatic jamming.

[0003] The reconnaissance jammer selects the jamming signal at the next moment based on the communication signal detected at the previous moment. This process can be regarded as a Markov process, which can be predicted through reinforcement learning. At the same time, reinforcement learning can improve the communication bit error rate after interference and reduce the interference power by designing a specific reward function, thereby achieving the optimal solution for interference effect and resource utilization.

[0004] However, some small reconnaissance jamming equipment, such as the reconnaissance jamming payloads of small drones, can only provide relatively small computing power and power. The computing power requirements of the jamming strategy generation network based on reinforcement learning cannot be met, resulting in slow network operation and poor real-time performance.

[0005] Therefore, it is necessary to use lightweight methods to compress the interference strategy generation network model, reduce the model size, and reduce the demand for computing power and power, so that it can be predicted on small payloads. Summary of the invention

[0006] The purpose of the present invention is to provide a communication jamming strategy generation method based on a multi-level pruning lightweight network, which can be deployed in a small reconnaissance jamming payload and can autonomously select jamming signal parameters according to reconnaissance information to achieve the optimal solution for jamming effect and resource utilization.

[0007] The technical solution adopted by the present invention is:

[0008] A communication interference strategy generation method based on a multi-level pruning lightweight network includes the following steps:

[0009] Step 1, setting up a communication interference countermeasure environment, including a communication reconnaissance aircraft, a communication jammer, a communication transmitter and a communication receiver, wherein the communication transmitter is used to transmit communication signals, the communication receiver is used to receive communication signals and interference signals, and calculate the bit error rate, and the communication transmitter and the communication receiver change the communication strategy according to the interference situation of the signal; the communication reconnaissance aircraft is used to conduct reconnaissance on the communication signal transmitted by the communication transmitter, and obtain the frequency, bandwidth, symbol rate, modulation style and signal level; the communication jammer is used to adjust the interference strategy according to the parameters obtained by reconnaissance and the communication bit error rate, and transmit continuous wave interference signals, pulse interference signals, comb spectrum interference signals and swept frequency interference signals;

[0010] Step 2: Set up communication strategy When the communication party is interfered, randomly select one of the three adjustment measures: reducing the modulation order, increasing the communication transmission power, and reducing the communication symbol rate. When all three adjustment measures fail to work, randomly change the communication channel; and set the initial interference strategy To pre-train the hierarchical reinforcement learning network m base The generated hierarchical strategy, pre-trained hierarchical reinforcement learning network m base The parameters of are pre-trained in an adversarial environment; the pre-trained hierarchical reinforcement learning network m base Including Q-Network and Actor network, the resulting hierarchical strategy is t ,x t >, a t For the selected interference type, Q-Network output, x t The interference parameters corresponding to each interference type, output by the Actor network;

[0011] Step 3: Pre-training hierarchical reinforcement learning network m base Initialize the hyperparameters of the learning rate, batch size, experience replay storage size, and discount factor;

[0012] Step 4: Pre-trained hierarchical reinforcement learning network m base Each layer is evaluated according to the importance E k Calculate the pruning rate and use the pruning rate as the ratio of each layer mask of the network model to perform multi-level pruning to form a lightweight network m p ;

[0013] Step 5: Prune the lightweight network m p Retraining, in which the communication transmitter transmits the communication signal, and the communication signal parameters at time t obtained by the communication reconnaissance aircraft are used as the state space s t , input to m p In the retraining strategy ​Get the action value at time t, including the interference type a t and the disturbance parameter x t ;

[0014] Step 6: According to the interference type a t and the disturbance parameter x t The communication jammer transmits the corresponding interference signal, superimposes the interference signal with the communication signal, and the communication receiver receives the superimposed signal and decodes the signal to obtain the bit error rate BER t , according to BER t The relationship between the size of the set threshold and the completion of the interference task is obtained. t , and calculate the reward value r through the reward function t ;

[0015] Step 7: According to the bit error rate BER t , according to the communication strategy Update the signal parameters of the communication transmitter and obtain the state space s at the next moment t+1 , and {a t ,x t ,s t ,s t+1 ,D t ,r t}Deposit into the experience pool;

[0016] Step 8: Sampling {a b ,x b ,s b ,s b+1 ,D b ,r b}, and s b+1 Input into the Actor network to get the action value x b+1 , then s b+1 With x b+1 Input into the Q-Network network and get a b+1 ; will s b With x b Input into the Q-Network network and get Finally, a b+1 , and r b Substitute into the loss function l Q , update the network parameters;

[0017] Step 9: b Input into the Actor network to get the action value x' b , then s b With x' b Input into the Q-Network network and get a'b , a' b Substitute into the loss function l θ , update the network parameters;

[0018] Step 10: Repeat steps 5 to 9 until the number of repetitions reaches the maximum number of iterations or the bit error rate reaches a threshold, and the training stops.

[0019] Furthermore, in step 2, the pre-trained hierarchical reinforcement learning network m base The PDQN hierarchical reinforcement learning model is adopted, including the Actor-Q-Network dual network structure. The Actor network takes the state value as input and the continuous parameter as output, and the Q-Network takes the state value and the output of the Actor network as input and the discrete parameter as output.

[0020] Furthermore, in step 4, the importance E is evaluated. k Through network depth Network Width and inter-layer importance Three indicators are used for comprehensive evaluation, E k The calculation method is:

[0021]

[0022] Where k is the network layer number, network depth The calculation method is:

[0023]

[0024] d k =k-(n+1) / 2

[0025] Where sqrt is the root-finding symbol, and n is the total number of network layers;

[0026] Network Width The calculation method is:

[0027]

[0028] in is the total number of parameters of the kth layer, sum is the summation symbol, sigmoid is the activation function, and the function form is:

[0029] sigmoid(x)=1 / (1+e -x )

[0030] Importance of layers for:

[0031]

[0032] Among them, s k is the sum of all parameter values ​​of the kth layer.

[0033] Furthermore, in step 5, the retraining strategy a t The interference type is randomly sampled from a uniform distribution with a probability of ε and output through the Q-Network with a probability of 1-ε.

[0034] Furthermore, in step 6, the reward function r is calculated as:

[0035]

[0036] Among them, c1~c5 are set constants, f j is the interference frequency of decision making, f c is the communication frequency, and p is the interference power.

[0037] Furthermore, in step 8, the loss function l Q The calculation formula is:

[0038]

[0039] Furthermore, in step 9, the loss function l θ The calculation formula is:

[0040]

[0041] Among them, K represents all interference types, a' b,k Indicates a' b The output value of the kth interference type in .

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] (1) The present invention is applicable to the field of communication confrontation and uses an improved PDQN-based hierarchical reinforcement learning algorithm to achieve hierarchical interaction and feedback of interference types and interference parameters, thereby improving exploration capabilities and avoiding falling into local optimality.

[0044] (2) The present invention adopts a multi-level pruning method to improve the network model in a lightweight way, and measures the pruning rate through multi-dimensional features, thereby reducing resource requirements and avoiding large performance losses. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a communication interference countermeasure scenario diagram of the present invention.

[0046] Figure 2 It is a flow chart of the communication interference strategy generation method of the present invention.

[0047] Figure 3It is a structural diagram of the PDQN model of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0049] A communication jamming strategy generation method based on a multi-level pruning lightweight network is used to generate jamming strategies for communication signals under low computing power conditions. It enables communication jamming equipment to autonomously control jamming signal parameters based on reconnaissance information, and achieve intelligent confrontation with communication equipment with anti-jamming strategies.

[0050] The present invention is applied to communication interference countermeasure scenarios. Figure 1 This is a diagram of a communication confrontation scenario. During the communication confrontation process, the communicating parties are the communication transmitter and the communication receiver, and the confronting parties are the communication reconnaissance aircraft and the communication jammer. The communicating parties have fixed anti-interference strategies to ensure communication, and the confronting parties have interference strategies based on reconnaissance cognition to adjust their attack methods.

[0051] A communication jamming strategy generation method based on multi-level pruning lightweight network, the specific process is as follows Figure 2 As shown, the following steps are included:

[0052] Step 1: Model building

[0053] Set up a communication interference confrontation environment. The environment includes two parties, the red party is the communication party, which has a communication transmitter and a communication receiver; the blue party is the interference party, which has a communication reconnaissance aircraft and a communication jammer. Among them, the communication transmitter transmits communication signals; the communication receiver receives the signals in the environment, including communication signals and interference signals, decodes them, and calculates the bit error rate. The communication transmitter and the communication receiver can change the communication strategy according to the interference situation of the signal; the communication reconnaissance aircraft can detect the signals transmitted by the communication transmitter and obtain parameters such as frequency, bandwidth, symbol rate, modulation style, signal level, etc.; the communication jammer can adjust the interference strategy according to the parameters obtained by the reconnaissance and the communication bit error rate, and can transmit continuous wave interference signals, pulse interference signals, comb spectrum interference signals and swept frequency interference signals.

[0054] Step 2: Initial strategy formulation

[0055] Setting up Red Team’s Communication Strategy When the communication party is interfered, randomly select one of the three measures: reducing the modulation order, increasing the communication transmission power, and reducing the communication symbol rate. When the above three adjustment measures are ineffective, randomly change the communication channel. Set the initial interference strategy To pre-train the hierarchical reinforcement learning network m base The resulting classification strategy, m base Using the PDQN hierarchical reinforcement learning model, such as Figure 3 As shown, it includes the Actor-Q-Network dual network structure, where θ Q is the network parameter of Q-Network, θ x are the network parameters of Actor, Q1,,,Q k is the output of Q-Network, and the sequence number k of the action value a is obtained through the argmax (maximum value) function; the Actor network takes the state value s as input and the continuous parameters x = x1,,,X k As output, Q-Network takes the state value s and the output x of the Actor network as input and a discrete parameter as output; the resulting classification strategy is divided into two parts t ,x t >, where a t For the selected interference type, Q-Network output, x t The interference parameters corresponding to each interference type, Actor network output. Pre-trained hierarchical reinforcement learning network m base The parameters of are pre-trained in an adversarial environment.

[0056] Step 3: Pre-training hierarchical reinforcement learning network m base Initialize the hyperparameters of the model, including learning rate, batch size, experience replay storage size, discount factor, etc.

[0057] Step 4: Model pruning

[0058] Pre-trained hierarchical reinforcement learning network m base Each layer is evaluated according to the importance E k Calculate the pruning rate, and perform multi-level pruning according to the pruning rate as the ratio of each layer mask of the network model to form a lightweight network m p ;

[0059] Evaluation importance E k Through network depth Network Width and inter-layer importance Three indicators are used for comprehensive evaluation, E k The calculation method is:

[0060]

[0061] Where k is the network layer number, network depth The calculation method is:

[0062]

[0063] d k =k-(n+1) / 2​

[0064] Where sqrt is the root-finding symbol, and n is the total number of network layers;

[0065] Network Width The calculation method is:

[0066]

[0067] in is the total number of parameters of the kth layer, sum is the summation symbol, sigmoid is the activation function, and the function form is:

[0068]

[0069] Importance of layers for:

[0070]

[0071]

[0072] Among them, s k is the sum of all parameter values ​​of the kth layer.

[0073] Step 5: Obtaining lightweight model action values

[0074] For lightweight network p Retraining is performed, in which the communication transmitter transmits a communication signal, and the communication signal parameters at time t obtained by the communication reconnaissance aircraft are used as the state space s t , input to m p In the retraining strategy Get the action value at time t, including the interference type a t and the disturbance parameter x t ;

[0075] The retraining strategy Initial interference strategy The difference is that the retraining strategy a t There are two ways to calculate . One is to randomly explore the parameters in the uniform distribution with a probability of ε to generate interference signals, and the rest are output through the Q-Network network with a probability of 1-ε.

[0076] In random exploration, according to the characteristics of communication confrontation, partial random exploration will be selected with a probability of 90%, and completely random exploration will be selected with a probability of 10%.

[0077] Partial random exploration will select ① the interference signal frequency as the currently detected communication signal frequency; ② the interference signal modulation style as the currently detected communication signal modulation style; ③ the interference signal bandwidth is randomly selected; ④ the symbol rate; ⑤ the interference signal level is randomly selected. This can increase the number of effective interferences during the exploration process.

[0078] Completely random sampling will randomly select all parameters, which can improve the diversity of signals during the exploration process and avoid falling into local optimality.

[0079] Step 6: Calculate the reward

[0080] According to the action value: interference type a t and the disturbance parameter x t The communication jammer transmits the corresponding interference signal, superimposes the interference signal with the communication signal, and the communication receiver receives the superimposed signal and decodes the signal to obtain the bit error rate BER t , according to BER t The relationship between the size of the set threshold and the completion of the interference task is obtained. t , and calculate the reward value r through the reward function t .

[0081] The reward function r is calculated as:

[0082]

[0083] Where c1~c5 are constants, selected based on experience and experiments, f j is the interference frequency of decision making, f c is the communication frequency, and p is the interference power. The meaning of the reward function is that when BER<0.03, the closer the interference frequency is to the communication frequency, the greater the reward value is, guiding the frequency alignment in the early stage; when BER≥0.03, the larger the BER is and the smaller p is, the greater the reward value is, guiding the model to achieve the target bit error rate with less interference resources in the later stage.

[0084] Step 7: Experience pool storage and sampling

[0085] According to BER t , according to the communication strategy Update the signal parameters of the communication transmitter and obtain the state space s at the next moment t+1 , and {a t ,x t ,s t ,s t+1 ,D t ,r t} is stored in the experience pool buffer in the form of tuples, and the experience pool buffer is managed using a two-stage threshold strategy.

[0086] The dual-stage threshold strategy will pre-set dual buffer space thresholds α1 and α2, where the buffer capacity is α2. When the number of tuples stored in the buffer is less than the threshold α1, the buffer will not be sampled, in order to ensure the diversity of model training data; when the number of tuples stored in the buffer is greater than the threshold α1 and less than the threshold α2, in order to improve the exploration efficiency, the buffer is sampled and trained in advance, and the buffer continues to be expanded; when the number of tuples stored in the buffer is equal to the threshold α2, the buffer is sampled and trained, but the buffer is no longer expanded, and the buffer is only updated using the queue.

[0087] Step 8, Network Update

[0088] Sampling {a b ,x b ,s b ,s b+1 ,D b ,r b}, and s b+1 Input into the Actor network to get the action value x b+1 , then s b+1 With x b+1 Input into the target Q-Network network and get a b+1 ; will s b With x b Input into the Q-Network network and get Finally, a b+1 , and r b Substitute into the loss function l Q In the process, update the network parameters; the loss function l Q The calculation formula is:

[0089]

[0090] Step 9: b Input into the Actor network to get the action value x' b , then s b With x' b Input into the Q-Network network and get a' b , a' b Substitute into the loss function l θ , update the network parameters;

[0091] Loss function l θ The calculation formula is:

[0092]

[0093] Among them, K represents all interference types, a' b,k Indicates a' b The output value of the kth interference type in .

[0094] Step 10: Repeat steps 5 to 9 until the number of repetitions reaches the maximum number of iterations and the training stops.

Claims

1. A communication interference strategy generation method based on a multi-level pruning lightweight network, characterized in that: The following steps are involved: Step 1, setting up a communication interference countermeasure environment, including a communication reconnaissance aircraft, a communication jammer, a communication transmitter and a communication receiver, wherein the communication transmitter is used to transmit communication signals, the communication receiver is used to receive communication signals and interference signals, and calculate the bit error rate, and the communication transmitter and the communication receiver change the communication strategy according to the interference situation of the signal; the communication reconnaissance aircraft is used to conduct reconnaissance on the communication signal transmitted by the communication transmitter, and obtain the frequency, bandwidth, symbol rate, modulation style and signal level; the communication jammer is used to adjust the interference strategy according to the parameters obtained by reconnaissance and the communication bit error rate, and transmit continuous wave interference signals, pulse interference signals, comb spectrum interference signals and swept frequency interference signals; Step 2: Set up communication strategy When the communication party is interfered, randomly select one of the three adjustment measures: reducing the modulation order, increasing the communication transmission power, and reducing the communication symbol rate. When all three adjustment measures fail to work, randomly change the communication channel; and set the initial interference strategy To pre-train the hierarchical reinforcement learning network m base The generated hierarchical strategy, pre-trained hierarchical reinforcement learning network m base The parameters of are pre-trained in an adversarial environment; the pre-trained hierarchical reinforcement learning network m base Including Q-Network and Actor network, the resulting hierarchical strategy is t ,x t >, a t For the selected interference type, Q-Network output, x t The interference parameters corresponding to each interference type, output by the Actor network;​ Step 3: Pre-training hierarchical reinforcement learning network m basse Initialize the hyperparameters of the learning rate, batch size, experience replay storage size, and discount factor; Step 4: Pre-trained hierarchical reinforcement learning network m base Each layer is evaluated according to the importance E k Calculate the pruning rate and use the pruning rate as the ratio of each layer mask of the network model to perform multi-level pruning to form a lightweight network m p ; Step 5: Prune the lightweight network m p Retraining, in which the communication transmitter transmits the communication signal, and the communication signal parameters at time t obtained by the communication reconnaissance aircraft are used as the state space s t , input to m p In the retraining strategy Get the action value at time t, including the interference type a t and the disturbance parameter x t ; Step 6: According to the interference type a t and the disturbance parameter x t The communication jammer transmits the corresponding interference signal, superimposes the interference signal with the communication signal, and the communication receiver receives the superimposed signal and decodes the signal to obtain the bit error rate BER t , according to BER t The relationship between the size of the set threshold and the completion of the interference task is obtained. t , and calculate the reward value r through the reward function t ; Step 7: According to the bit error rate BER t , according to the communication strategy Update the signal parameters of the communication transmitter and obtain the state space s at the next moment t+1 , and {a t , x t ,s t ,s t+1 , D t , r t }Deposit into the experience pool; Step 8: Sampling {a b , x b ,s b ,s b+1 , D b , r b }, and s b+1 Input into the Actor network to get the action value x b+1 , then s b+1 With x b+1 Input into the Q-Network network and get a b+1 ; will s b With x b Input into the Q-Network network and get Finally, a b+1 , and r b Substitute into the loss function l Q , update the network parameters; Step 9: b Input into the Actor network to get the action value x′ b , then s b and x′ b Input into the Q-Network network and get a′ b , a′ b Substitute into the loss function l θ , update the network parameters; Step 10: Repeat steps 5 to 9 until the number of repetitions reaches the maximum number of iterations or the bit error rate reaches a threshold, and the training stops.

2. According to claim 1, a communication interference strategy generation method based on a multi-level pruning lightweight network is characterized in that: In step 2, the pre-trained hierarchical reinforcement learning network m base The PDQN hierarchical reinforcement learning model is adopted, including the Actor-Q-Network dual network structure. The Actor network takes the state value as input and the continuous parameter as output, and the Q-Network takes the state value and the output of the Actor network as input and the discrete parameter as output.

3. According to claim 1, a communication interference strategy generation method based on a multi-level pruning lightweight network is characterized in that: In step 4, evaluate the importance E k Through network depth Network Width and inter-layer importance Three indicators are used for comprehensive evaluation, E k The calculation method is: Where k is the network layer number, network depth The calculation method is: d k =k-(n+1) / 2 Where sqrt is the root-finding symbol, and n is the total number of network layers; Network Width The calculation method is: in is the total number of parameters of the kth layer, sum is the summation symbol, sigmoid is the activation function, and the function form is: sigmoid(x)=1 / (1+e -x ) Importance of layers for: Among them, s k is the sum of all parameter values ​​of the kth layer.

4. According to claim 1, a communication interference strategy generation method based on a multi-level pruning lightweight network is characterized in that: In step 5, retrain the strategy a t The interference type is randomly sampled from a uniform distribution with a probability of ε and output through the Q-Network with a probability of 1-ε.

5. According to claim 1, a communication interference strategy generation method based on a multi-level pruning lightweight network is characterized in that: In step 6, the reward function r is calculated as: Among them, c1~c5 are set constants, f j is the interference frequency of decision making, f c is the communication frequency, and p is the interference power.

6. According to claim 1, a communication interference strategy generation method based on a multi-level pruning lightweight network is characterized in that: In step 8, the loss function l Q The calculation formula is:

7. According to claim 1, a communication interference strategy generation method based on a multi-level pruning lightweight network is characterized in that: In step 9, the loss function l θ The calculation formula is: Among them, K represents all interference types, a' b,k Indicates a' b The output value of the kth interference type in .

Citation Information

Cited By

  • Communication interference decision-making method based on reinforcement learning

    CN119788237A

  • Multi-target interference effect real-time analysis method and system in electronic warfare simulation

    CN120030803A

  • A Real-time Analysis Method and System for Multi-target Interference Effects in Electronic Warfare Simulation

    CN120030803B