A communication interference strategy generation method based on reinforcement learning
By constructing a Markov decision process and the WoLF-PHC algorithm, real-time fast interference without prior information is achieved, which solves the problems of flexibility and targeting of traditional interference methods under unknown signals and improves interference effectiveness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NAVAL UNIV OF ENG PLA
- Filing Date
- 2024-06-19
- Publication Date
- 2026-06-02
AI Technical Summary
Traditional jamming methods require relatively complete information about the communication target. When prior information is lacking, jamming decisions lack flexibility and specificity, making it difficult to effectively jam unknown signals.
A communication interference strategy generation method based on reinforcement learning is adopted. By constructing a Markov decision process and the WoLF-PHC algorithm, an interference decision model is designed to achieve real-time and fast interference without prior information.
It improves the learning rate and convergence speed, enabling it to quickly find the optimal interference strategy under unknown signal conditions and improve interference effectiveness.
Smart Images
Figure CN118828592B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of communication countermeasures, specifically a method for generating communication jamming strategies based on reinforcement learning. Background Technology
[0002] In recent years, with the development of cognitive radio, new signal modulation styles, and new communication anti-jamming technologies, traditional jamming methods have become increasingly inadequate to adapt to flexible and ever-changing communication targets, resulting in a continuous decline in jamming effectiveness. Traditional jamming methods require relatively complete information about the communication target before implementing jamming based on existing jamming templates. For unknown signals lacking relevant prior information, only manual experience in selecting jamming parameters or high-power suppression can be employed, resulting in a lack of flexibility and specificity in jamming decisions. Therefore, to effectively jam unknown communication signals, it is urgent to improve the intelligence of jamming algorithms. Reinforcement learning theory can autonomously learn optimal strategies through continuous interaction with the environment without requiring prior information. Therefore, applying reinforcement learning theory to the problem of jamming strategy selection has significant research value and application prospects. Summary of the Invention
[0003] The purpose of this invention is to provide a communication jamming strategy generation method based on reinforcement learning. This method does not require knowledge of the specific signal characteristics of the communication target signal, but comprehensively considers the basic principles of jamming and changes in the behavior of the communication target as indicators of jamming effect, thereby achieving the goal of real-time and rapid jamming.
[0004] This invention is achieved using the following technical solution:
[0005] A method for generating communication interference policies based on reinforcement learning includes the following steps:
[0006] Step 1: Construct system models for the communicating party and the interfering party;
[0007] Step 2: Based on the system models of the communicating party and the interfering party constructed in Step 1, the win-or-learning hill-climbing algorithm is used to learn the anti-interference scheme of the communicating party and design the corresponding interference decision model.
[0008] Step 3: Using the interference decision model obtained in Step 2, learn the anti-interference strategy of the communication target and implement interference according to the decision-making process of "observation-adjustment-decision-action".
[0009] Furthermore, step one, constructing the system model for the communicating party and the interfering party, specifically includes:
[0010] The adversarial process between the communication jammer and the communication provider is modeled as a Markov decision process, which consists of a 4-tuple of state, action, transition probability, and reward. Description; where the state This indicates the operating status of the jammer and the communication target, consisting of the jammer's channel and power, and the communication target's channel and power. (Note: The last part, "", is a typographical notation and doesn't need a direct translation, but can be left as is.) The state at time is ;action The action taken by the jammer is determined by its channel and power, which in turn determine the jammer's channel and power at the next moment. The action at a given moment is Transition probability Let denot be the probability that the jammer will transition to the next state after taking a certain action in the current state. The transition probability at time t is ;award This represents the immediate reward obtained after the jammer takes action; in actual communication jamming, the interference status of the communication target is obtained by estimating whether the interference-to-noise ratio at the receiver reaches a threshold; the jamming effect is estimated by observing whether the communicating party takes anti-jamming measures; the jamming power is included as a factor affecting the reward, the higher the jamming power, the smaller the reward value, denoted as... The instant reward for each moment is:
[0011]
[0012] in
[0013]
[0014]
[0015]
[0016] The effectiveness of the interference, whether the communication target changes frequency after being interfered with, and the situation where the transmission power is increased after the communication target is interfered with are all measured respectively. The level of communication interference power is measured. Indicates the maximum interference power. These are the weights of the four reward influencing factors mentioned above;
[0017] A jammer learns to counter the communication target's jamming avoidance strategies, thereby obtaining the optimal jamming strategy. Let the jamming strategy of the jammer be denoted as . Its goal is to find the optimal strategy. This maximizes the cumulative interference reward or return:
[0018]
[0019] in, This is the discount factor.
[0020] Furthermore, step two specifically includes:
[0021] When the jammer is in a "winning" state, a slow learning rate is adopted. When the jammer is in an "input" state, a rapid learning rate is adopted. This allows the interference strategy to converge quickly to the optimal strategy; the criteria for "winning" and "losing" are determined by the current strategy. and averaging strategy To determine; to introduce Indicates the current state The frequency of occurrence and the average strategy update rule are as follows:
[0022]
[0023] Current Strategy The initial value is , Given the size of the action space, the update rule for the current policy is:
[0024]
[0025] in
[0026]
[0027] Learning rate The adjustment rules are as follows:
[0028] .
[0029] Furthermore, step three specifically includes:
[0030] Observation: By observing the interference-to-noise ratio and the change in the state of the communication target after interference, the interference effect is estimated, and the influence of interference power is comprehensively considered, based on the formula... Calculate instant reward value ;
[0031] Adjustment: Based on instant reward value For input, adjust the Q-value table according to the formula. Calculate the averaging strategy And compare the advantages and disadvantages of the current strategy with the average strategy, according to the formula Japanese style Adjust current strategy According to the formula Adjusting the learning rate ;
[0032] Decision: Order In state Next, according to the strategy Decide on action ;
[0033] Action: To perform a movement Transition to the next state Then, proceed to the observation step, and continuously observe, adjust, decide, and act on the communication target.
[0034] The present invention has the following beneficial effects:
[0035] 1. In cases where the communication jammer cannot obtain direct feedback on the communication quality of the communication target, this invention proposes a communication jamming strategy generation method based on reinforcement learning. It adopts the WoLF-PHC intelligent learning algorithm to improve the learning rate and accelerate the convergence speed.
[0036] 2. Q-learning exhibits a clear "step-like" convergence because it is a pure policy algorithm that selects the current optimal policy with a high probability and explores randomly with a low probability, resulting in policy updates only after a certain number of iterations. In contrast, the WoLF-PHC algorithm used in this invention is a hybrid policy algorithm that explores all possible policies with a certain probability in the initial stage. Therefore, its convergence curve is smooth and steep, enabling it to learn the optimal policy more quickly. Attached Figure Description
[0037] Figure 1 This is a diagram illustrating the relationship between the communication transmitter, communication receiver, and communication jammer in an embodiment of the present invention.
[0038] Figure 2 This is a schematic diagram illustrating the communication interference implementation process of an embodiment of the present invention;
[0039] Figure 3 This is a schematic diagram illustrating the change in interference report according to an embodiment of the present invention. Detailed Implementation
[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0041] This invention provides a method for generating communication interference strategies based on reinforcement learning, comprising the following steps:
[0042] Step 1: Construct system models for the communicating party and the interfering party, see formula. - .
[0043] exist At any given time, the available communication channels for setting the communication target are: One, respectively Available power levels include One, respectively The number of channels that the jammer can select is also... One, respectively Available power levels include One, respectively Let the channel gain of the communication link be... The channel gain of the interference link is The noise power is Then the communication receiver is The noise-to-signal ratio at time t is:
[0044]
[0045] in
[0046]
[0047] That is, if the interference channel is the same as the communication channel, then Otherwise, it is 0.
[0048] To maintain normal communication, the signal-to-interference-plus-noise ratio (SNR) of the communication receiver must not exceed a certain threshold. Otherwise, information cannot be received correctly. In this case, the communication target needs to take evasive measures, such as... As shown, communication can be restored by changing the frequency or increasing the transmission power to reduce the signal-to-interference ratio at the receiver.
[0049] Furthermore, the adversarial process between the communication jammer and the communicator is modeled as a Markov decision process. A Markov decision process consists of a 4-tuple comprising state, action, transition probability, and reward. Description. In this problem, the state... This indicates the operating status of the jammer and the communication target, consisting of the jammer's channel and power, and the communication target's channel and power. (Note: The last part, "", is a typographical notation and doesn't need a direct translation, but can be left as is.) The state at time is ;action The action taken by the jammer is determined by its channel and power, which in turn determine the jammer's channel and power at the next moment. The action at a given moment is Transition probability Let denot be the probability that the jammer will transition to the next state after taking a certain action in the current state. The transition probability at time t is ;award This represents the immediate reward obtained by the jammer after taking action. In actual communication jamming, the jammer has difficulty obtaining the interference status of the communication target (such as bit error rate, packet loss rate, etc.), but can estimate the jamming effect by estimating whether the interference-to-noise ratio at the receiver reaches a threshold and observing whether the communication target's state changes, i.e., whether the communicating party takes anti-jamming measures. Meanwhile, considering that excessive jamming power would lead to energy waste and increase the risk of location exposure, jamming power is included as a factor affecting the reward; the higher the jamming power, the smaller the reward value. The instant reward for each moment is:
[0050]
[0051] in
[0052]
[0053]
[0054]
[0055] The effectiveness of the interference, whether the communication target changes frequency after being interfered with, and the situation where the transmission power is increased after the communication target is interfered with are all measured respectively. The level of communication interference power is measured. This indicates the maximum interference power. The weights of the four reward influencing factors mentioned above can be flexibly set according to the actual situation.
[0056] A jammer learns to find the optimal jamming strategy by engaging with the communication target and exploring its jamming evasion strategies. Let the jamming strategy of the jammer be denoted as [equation missing]. Its goal is to find the optimal strategy. This maximizes the cumulative interference reward or return, i.e.:
[0057]
[0058] in, This serves as a discount factor, balancing short-term and long-term returns.
[0059] Step 2: Based on the system models of the communicating party and the interfering party constructed in Step 1, the Win or Learn Fast-Policy Hill Climb (WoLF-PHC) algorithm is used to learn the anti-interference scheme of the communicating party, and a corresponding interference decision model is designed, as shown in the formula. - ;
[0060] The WoLF-PHC algorithm is a learning algorithm that combines the "Win-Or-Learn (WoLF)" rule with the "Policy Hill Climbing (PHC)" method. The PHC algorithm is a simple extension of Q-learning, using a learning rate... The probability of selecting the maximum Q-value is adjusted to improve the policy. The Q-value update rule of this algorithm is the same as that of the Q-learning algorithm, which is:
[0061]
[0062] The WoLF-PHC algorithm employs a hybrid strategy to guide the jammer in making communication jamming decisions. Its main idea is that when the jammer is in a "winning" state, it adopts a slow learning rate. When the jammer is in an "input" state, a rapid learning rate is adopted. This allows the interference strategy to quickly converge to the optimal strategy. The criteria for "winning" and "losing" can be determined through the current strategy. and averaging strategy To determine this. To calculate the average policy, we introduce... Indicates the current state The frequency of occurrence and the average strategy update rule are as follows:
[0063]
[0064] Current Strategy The initial value is , This represents the size of the action space. The current policy update rule is:
[0065]
[0066] in
[0067]
[0068] Learning rate The adjustment rules are as follows:
[0069]
[0070] Step 3: Using the interference decision-making model derived in Step 2, implement interference according to the decision-making process of "observation-adjustment-decision-action".
[0071] The jammer uses the WoLF-PHC algorithm to learn the anti-jamming strategies of communication targets through "observation-adjustment-decision-action" and effectively implement jamming. The specific steps are as follows:
[0072] Step 1 Observation: By observing the interference-to-noise ratio and the change in the state of the communication target after interference, estimate the interference effect, and comprehensively consider the influence of interference power, according to the formula... Calculate instant reward value .
[0073] Step 2 Adjustment: With instant reward value As input, according to the formula Adjust the Q-value table according to the formula Calculate the averaging strategy And compare the advantages and disadvantages of the current strategy with the average strategy, according to the formula Japanese style Adjust current strategy According to the formula Adjusting the learning rate .
[0074] Step 3 Decision: Let In state Next, according to the strategy Decide on action .
[0075] Step 4 Action: Perform the action Transition to the next state Jump to step 1 and continue to "observe-adjust-decide-act" on the target.
[0076] like Figure 1 The diagram illustrates the relationship between a communication transmitter, a communication receiver, and a communication jammer. The transmitter and receiver establish communication and transmit information via a communication link. The jammer interferes with the receiver, blocking information transmission; the link between the jammer and the receiver is called the jamming link.
[0077] The communication interference "observation-adjustment-decision-action" implementation process in step three of this embodiment of the invention is as follows: Figure 2 As shown.
[0078] The communication interference strategy generation method proposed in this invention is verified. In the simulation experiment, it is assumed that the number of communication channels of the communication target is... At any given time, it can only communicate on one of the channels. At the target receiver, depending on the transmitter's power, the received signal power is 7mW, 14mW, 21mW, or 28mW respectively; correspondingly, the jammer has the following number of jamming channels: At the target receiver, the received interference signal power is 2mW, 4mW, 6mW, 8mW, 10mW, and 12mW, depending on the jammer's transmit power. It's important to note that in real-world environments, while the jammer cannot accurately measure the transmit and interference power at the receiver, these values can be effectively estimated based on the positional relationships between the transmitter, receiver, and jammer, as well as the signal propagation path characteristics. For simulation purposes, the transmit and interference power values at the receiver are given directly. The noise power at the receiver is set to 1mW. It is assumed that the jammer and target can only change the channel and power once per communication time slot. Other simulation parameters are set as shown in Table 1.
[0079] Table 1 Simulation Parameter Settings
[0080]
[0081] The jammer employs two intelligent algorithms proposed in this paper, WoLF-PHC and Q-learning, to make communication jamming decisions. Specifically, Q-learning uses... - A greedy algorithm searches for the optimal strategy, setting... Consider that communication targets will employ certain avoidance strategies to restore communication after being interfered with.
[0082] Figure 3 The algorithm proposed in this invention is compared with the Q-learning algorithm in terms of reward. It can be clearly seen that the algorithm proposed in this invention does not have "step-like" convergence. The convergence process of the entire algorithm is relatively steep and smooth, and the reward value after convergence is higher.
[0083] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for generating communication interference strategies based on reinforcement learning, characterized in that: Includes the following steps: Step 1: Construct system models for the communicating party and the interfering party; Step 2: Based on the system models of the communicating party and the interfering party constructed in Step 1, the hill climbing algorithm of the win-or-learning strategy is used to learn the anti-interference scheme of the communicating party and design the corresponding interference decision model. Step 3: Using the interference decision model obtained in Step 2, learn the anti-interference strategy of the communication target and implement interference according to the decision-making process of "observation-adjustment-decision-action"; Step one, constructing the system model for the communicating party and the interfering party, specifically includes: The adversarial process between the communication jammer and the communication provider is modeled as a Markov decision process, which consists of a 4-tuple of state, action, transition probability, and reward. Description; where the state This indicates the operating status of the jammer and the communication target, consisting of the jammer's channel and power, and the communication target's channel and power. (Note: The last part, "", is a typographical notation and doesn't need a direct translation, but can be left as is.) The state at time is ;action The action taken by the jammer is determined by its channel and power, which in turn determine the jammer's channel and power at the next moment. The action at a given moment is Transition probability Let denot be the probability that the jammer will transition to the next state after taking a certain action in the current state. The transition probability at time t is ;award This represents the immediate reward obtained after the jammer takes action; in actual communication jamming, the interference status of the communication target is obtained by estimating whether the interference-to-noise ratio at the receiver reaches a threshold; the jamming effect is estimated by observing whether the communicating party takes anti-jamming measures; the jamming power is included as a factor affecting the reward, the higher the jamming power, the smaller the reward value, denoted as... The instant reward for each moment is: ； in ； ； ； The effectiveness of the interference, whether the communication target changes frequency after being interfered with, and the situation where the transmission power is increased after the communication target is interfered with are all measured respectively. The level of communication interference power is measured. Indicates the maximum interference power. These are the weights of the four reward influencing factors mentioned above; A jammer learns to counter the communication target's jamming avoidance strategies, thereby obtaining the optimal jamming strategy. Let the jamming strategy of the jammer be denoted as . Its goal is to find the optimal strategy. This maximizes the cumulative interference reward or return: ； in, This is the discount factor.
2. The communication interference strategy generation method based on reinforcement learning as described in claim 1, characterized in that: Step two specifically includes: When the jammer is in a "winning" state, a slow learning rate is adopted. When the jammer is in an "input" state, a rapid learning rate is adopted. This allows the interference strategy to converge quickly to the optimal strategy; the criteria for "winning" and "losing" are determined by the current strategy. and averaging strategy To determine; to introduce Indicates the current state The frequency of occurrence and the average strategy update rule are as follows: ； Current strategy The initial value is , Given the size of the action space, the update rule for the current policy is: ； in ； Learning rate The adjustment rules are as follows: 。 3. The communication interference strategy generation method based on reinforcement learning as described in claim 2, characterized in that: Step three specifically includes: Observation: By observing the interference-to-noise ratio and the change in the state of the communication target after interference, the interference effect is estimated, and the influence of interference power is comprehensively considered, based on the formula... Calculate instant reward value ; Adjustment: Based on instant reward value For input, adjust the Q-value table according to the formula. Calculate the averaging strategy And compare the advantages and disadvantages of the current strategy with the average strategy, according to the formula Japanese style Adjust current strategy According to the formula Adjusting the learning rate ; Decision: Order In state Next, according to the strategy Decide on action ; Action: To perform a movement Transition to the next state Then, proceed to the observation step and continuously "observe-adjust-decide-act" on the communication target.