An anti-interference communication method based on actor-critic deep reinforcement learning

Abstract

The application relates to the technical field of communication, and particularly discloses an anti-interference communication method based on an Actor-Critic deep reinforcement learning, which comprises the following steps: building a wireless communication scene; calculating a wireless signal power spectrum density function received by a receiving end and a signal power spectrum density function at a jammer based on the wireless communication scene, and constructing an anti-interference optimization model; solving the anti-interference optimization model based on a double Actor-Critic deep reinforcement learning method to obtain an optimal anti-interference strategy; and performing anti-interference based on the optimal anti-interference strategy. The method realizes efficient anti-interference communication in a dynamic interference environment through parallel decision-making of a distributed Actor network, unified evaluation of a Critic network, and combination of an offline training and an online access two-stage framework.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and specifically to an anti-interference communication method based on Actor-Critic deep reinforcement learning. Background Technology

[0002] Cognitive radio technology effectively alleviates the problem of spectrum resource shortage by improving spectrum utilization. However, in complex electromagnetic environments, the open communication channels in wireless communication systems are susceptible to malicious interference, leading to a decrease in spectrum utilization, reduced communication quality, and decreased anti-interference performance.

[0003] With the application of deep reinforcement learning in the field of anti-jamming, researchers have improved the performance of communication systems by modifying traditional algorithms (such as DQN and DDPG). However, joint domain anti-jamming methods still face challenges such as complex decision spaces, slow convergence speeds, and insufficient adaptability to intelligent jamming. Currently, traditional single-domain anti-jamming methods are difficult to cope with multi-dimensional jamming and have limited anti-jamming effects. Existing anti-jamming methods based on deep reinforcement learning have slow convergence due to their large decision spaces. Moreover, they currently lack the ability to adapt to the dynamic policies of intelligent jamming machines, resulting in prominent policy overfitting problems and insufficient robustness. Summary of the Invention

[0004] To address the aforementioned problems, the purpose of this invention is to provide an anti-interference communication method based on Actor-Critic deep reinforcement learning. This method achieves efficient anti-interference communication in dynamic interference environments by using a distributed Actor network for parallel decision-making and a unified Critic network for evaluation, combined with a two-stage framework of offline training and online access. At the same time, a parameter reset mechanism is introduced to avoid policy overfitting, improve the adaptability against intelligent interference, and ultimately improve the communication success rate, convergence speed, and robustness.

[0005] The technical solution adopted in this invention is: an anti-interference communication method based on Actor-Critic deep reinforcement learning, comprising the following steps: S100: Establish a wireless communication scenario; and calculate the power spectral density function of the wireless signal received by the receiver and the power spectral density function of the signal at the jammer based on the wireless communication scenario; S200: Construct an anti-interference optimization model based on the power spectral density function of the wireless signal received by the receiver and the power spectral density function of the signal at the jammer; S300: Solve the anti-interference optimization model based on the dual Actor-Critic deep reinforcement learning method to obtain the optimal anti-interference strategy; and perform anti-interference based on the optimal anti-interference strategy.

[0006] Preferably, the power spectral density function of the wireless signal received by the receiver in step S100 is calculated using the following formula: ; In the formula, Let be the power spectral density function of the wireless signal received at the receiving end; This refers to the channel gain from the transmitting end to the receiving end of the communication. This represents the channel gain from the jammer to the receiver. Let be the power spectral density function of the communication signal; The power spectral density function of the interference signal; Let be the power spectral density function of the noise; and These are the center frequencies of the communicating signal and the interfering signal, respectively. and These are the antenna transmit gains of the communicating party and the jammer, respectively. This represents the antenna receiving gain at the receiving end.

[0007] Preferably, the signal power spectral density function at the jammer in step S100 is calculated using the following formula: ; In the formula, Let be the signal power spectral density function at the jammer; The receiving antenna gain of the interfering party's intelligent decision-making equipment. Let be the channel gain from the communicator to the jammer at time t; This refers to the antenna transmit gain of the jammer; Let be the power spectral density function of the communication signal; Let be the power spectral density function of the noise.

[0008] Preferably, the anti-interference optimization model in step S200 is expressed by the following formula: ; In the formula, For communication strategy; This is a strategy for the interfering party; This is the state value function.

[0009] Preferably, step S300 includes: solving the joint domain anti-interference problem in the anti-interference optimization model based on the dual Actor-Critic deep reinforcement learning method; the joint domain includes the power domain and the frequency domain.

[0010] Preferably, the dual Actor-Critic deep reinforcement learning method adds another Actor network to the Actor-Critic network; the dual Actor-Critic deep reinforcement learning method includes a power-first Actor network, a frequency-first Actor network, and a Critic network.

[0011] Preferably, step S300 includes: During the training phase of the dual Actor-Critic deep reinforcement learning method, both the frequency-priority Actor network and the power-priority Actor network perform actions and return the rewards to the Critic network. The Critic network scores the actions performed by the frequency-priority Actor network and the power-priority Actor network, guiding them to continuously improve so that they reach the optimal network under the decision-making approach.

[0012] Preferably, step S300 includes: In the access phase of the dual Actor-Critic deep reinforcement learning method, the power-priority Actor network and the frequency-priority Actor network obtain the power-priority optimal action and the frequency-priority optimal action respectively based on the electromagnetic environment. The Critic network evaluates the power-priority optimal action and the frequency-priority optimal action respectively, and selects the action with more reward between the power-priority optimal action and the frequency-priority optimal action as the optimal anti-interference strategy.

[0013] Preferably, the Actor network is trained using a policy gradient algorithm.

[0014] Preferably, the Critic network is trained using a temporal difference learning algorithm.

[0015] The beneficial effects of the above technical solution are as follows: (1) This invention addresses the problems of slow convergence, poor robustness, and difficulty in coping with dynamic intelligent interference in existing methods. It discloses an anti-interference communication method based on Actor-Critic deep reinforcement learning. This method utilizes a distributed Actor network for parallel decision-making and a unified Critic network for evaluation, combined with a two-stage framework of offline training and online access, to dynamically optimize anti-interference strategies in the frequency and power domains, achieving efficient anti-interference communication under dynamic interference environments. Simultaneously, a parameter reset mechanism is introduced to avoid policy overfitting and improve adaptability against intelligent interference, thereby effectively increasing communication success rate, convergence speed, and robustness. It is suitable for joint domain anti-interference communication in intelligent interference scenarios.

[0016] (2) This invention aims to provide a communication anti-interference method based on deep reinforcement learning to solve the problem of slow convergence in the face of intelligent interference. Simulation results show that the deep reinforcement learning anti-interference algorithm based on dual Actor-Critic has good effectiveness, robustness and convergence. Attached Figure Description

[0017] Figure 1 A schematic diagram illustrating a wireless communication scenario provided in one embodiment of the present invention; Figure 2 This is a schematic diagram of the time slot structure of a base station and a communication party according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the jammer time slot structure provided in one embodiment of the present invention; Figure 4 A flowchart of the Actor-Critic process provided in one embodiment of the present invention; Figure 5 This is a schematic diagram of the training phase of the Dual Actor-Critic Method deep reinforcement learning method provided in one embodiment of the present invention; Figure 6 This is a schematic diagram of the access phase of the dual Actor-Critic Method deep reinforcement learning method provided in one embodiment of the present invention; Figure 7 This is a schematic diagram illustrating the communication success rate when the jammer uses the DNQ algorithm, according to one embodiment of the present invention. Figure 8 A comparison diagram showing the results of different anti-interference methods provided in one embodiment of the present invention; Figure 9 This is a comparison chart of communication success rates provided for one embodiment of the present invention. Detailed Implementation

[0018] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following detailed description of the embodiments and the accompanying drawings are used to illustrate the principles of the present invention by way of example, but should not be used to limit the scope of the present invention. That is, the present invention is not limited to the described preferred embodiments, and the scope of the present invention is defined by the claims.

[0019] In the description of this invention, it should be noted that, unless otherwise stated, "a plurality of" means two or more; the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance; those skilled in the art can understand the specific meaning of the above terms in this invention as appropriate.

[0020] Example 1 One embodiment of the present invention provides an anti-interference communication method based on Actor-Critic deep reinforcement learning, comprising the following steps: S100: Establish a wireless communication scenario; and calculate the power spectral density function of the wireless signal received by the receiver and the power spectral density function of the signal at the jammer based on the wireless communication scenario; (1) Establish a wireless communication scenario; like Figure 1 As shown, the wireless communication scenario constructed by this invention includes an jammer and a communication party (transmitter-receiver pair). The communication link in the wireless communication scenario contains multiple channels, and the channel set is defined as M={1,2,…,M}, where M is the maximum number of available channels. Considering the time-slot-based communication scenario, the communication party occupies only 1 channel for data transmission in each time slot. The receiver uses an omnidirectional antenna.

[0021] To minimize interference during communication, the communicating party (i.e., the user) deploys an intelligent decision-making device. This device continuously monitors the spectrum environment and runs intelligent anti-interference algorithms, selecting the optimal communication strategy at the beginning of each time slot. The jammer (i.e., the jammer) also deploys an intelligent decision-making device, which runs intelligent jamming algorithms but can only release jamming signals to one channel per time slot. The transmitter's maximum transmission power... Maximum transmission of jammer The available state vector of the channel is: , A value of 0 indicates that the channel is unavailable, and a value of 1 indicates that the channel is available.

[0022] (2) Calculate the power spectral density function of the wireless signal received at the receiver and the power spectral density function of the signal at the jammer; Since the success of communication depends on the signal quality at the receiving end, the intelligent decision-making device of the communicating party is deployed at the receiving end to guide the transmission based on the signal quality at the receiving end; the time slot structure of the receiving end and the communicating party is as follows: Figure 2 As shown. The intelligent decision-making device at the receiving end continuously senses the entire communication frequency band and decides the communication channel for each communication party at the beginning of the communication time slot. The decision result is fed back to each communication party through the downlink. Considering that the downlink is a low-rate control link, it can be assumed that the downlink can guarantee reliable transmission. The communication party then transmits data to the receiving end on the designated channel.

[0023] make , , , Let these represent the decision-making time, feedback time, reception time, and benefit calculation time, respectively. , This invention represents the time spent receiving feedback and the time spent transmitting data. It assumes that the time spent on decision-making, feedback, and calculating benefits is much shorter than the time spent receiving data. And the time spent receiving feedback is much shorter than the time spent transmitting it. It can be ignored.

[0024] At time t, let and Let represent the channel gain from the transmitter to the receiver and from the jammer to the receiver, respectively. , and Let PSD represent the power spectral density functions of the communicating signal, the interfering signal, and the noise, respectively; and let PSD represent the power spectral density function of the wireless signal received at the receiver, considering the presence of all signals. This can be expressed by the following formula: ; In the formula, Let be the power spectral density function of the wireless signal received at the receiving end; This refers to the channel gain from the transmitting end to the receiving end of the communication. This represents the channel gain from the jammer to the receiver. Let be the power spectral density function of the communication signal; The power spectral density function of the interference signal; Let be the power spectral density function of the noise; and These are the center frequencies of the communicating signal and the interfering signal, respectively. and These are the antenna transmit gains of the communicating party and the jammer, respectively. This represents the antenna receiving gain at the receiving end.

[0025] The signal-to-noise ratio (SNR) of the signal transmitted by one party to the receiver is defined as: ; like Figure 3 As shown, the intelligent decision-making device at the jammer continuously senses the entire frequency band and decides which channel to jam at the beginning of the time slot; the jammer randomly releases jamming signals on the channel and evaluates the jamming effect at the end of the time slot (assuming that the intelligent jammer can obtain the benefits of the communicating party in some way, thereby evaluating its jamming effect).

[0026] Unlike the deployment of intelligent decision-making equipment by the communication party, the jammer's intelligent decision-making equipment is deployed at the jammer. While the intelligent decision-making equipment continuously senses the spectrum, the jammer needs to release jamming signals, thus the jammer faces the problem of self-interference from its internal transmitter and receiver. To better focus on uncovering the frequency usage patterns of the communication signal and thus improve the jamming effect, the jammer needs to remove the jamming signal from the sensing results of the intelligent decision-making equipment. Since the jamming signal released by the jammer is known to the intelligent decision-making equipment, self-interference cancellation technology can be used to eliminate the influence of the jamming signal. At time t, the signal power spectral density function at the jammer is: ; In the formula, Let be the signal power spectral density function at the jammer; The receiving antenna gain of the interfering party's intelligent decision-making equipment. Let be the channel gain from the communicator to the jammer at time t; This refers to the antenna transmit gain of the jammer; Let be the power spectral density function of the communication signal; Let be the power spectral density function of the noise.

[0027] S200: Construct an anti-interference optimization model based on the power spectral density function of the wireless signal received by the receiver and the power spectral density function of the signal at the jammer; A game-theoretic model is used to model the joint domain's anti-interference capabilities using mathematical models. To accurately characterize the problem, the characteristics of the anti-interference optimization problem are summarized as follows: There is no strict order between the two parties: there is no strict sequence of decisions between the communicating party and the interfering party. Randomness in strategy selection: In communication warfare, in order to make it difficult for the adversary to grasp the pattern, the strategy selection should have a certain degree of randomness; The two sides in the confrontation have completely competitive objectives: As the two sides in the confrontation, the communicating party and the interfering party are in a completely competitive relationship with conflicting optimization objectives. The communicating party hopes to avoid interference attacks as much as possible and improve the communication effect, while the interfering party wants to disrupt the communication process and reduce the communication effect as much as possible. Information uncertainty: The decision-making environment in a two-way contest is inherently uncertain. It is impossible to fully understand the other party's strategy. For both parties in the game, information is uncertain, and their perception of the spectrum environment is also prone to error. Large decision space: In the joint domain anti-interference optimization problem, the adversarial domain includes spatial domain, power domain, frequency domain, time domain, code domain, etc., and the decision space to be selected is large.

[0028] Generally, in game theory, dynamic and static games do not depend on whether the adversaries play simultaneously, but rather on whether the party that acts later knows more information. In communication anti-interference, the party that acts later can obtain more information based on spectrum sensing. Therefore, the anti-interference optimization problem of this invention is a dynamic game problem. In summary, this anti-interference optimization problem can be modeled based on a zero-sum game model and a dynamic game model. This invention models the joint domain adversarial optimization problem between the communicating party and the interfering party as an adversarial zero-sum dynamic game model. This model is an imperfect information game with dynamic and zero-sum characteristics. The adversarial zero-sum dynamic game model can be represented as: ; In the formula, For adversarial zero-sum dynamic game models; The environment refers to the electromagnetic spectrum environment, and the environmental state refers to the spectrum state. The number of sub-models is defined as follows: sub-models include frequency-priority sub-models and power-priority sub-models, and each sub-model is a decision network. For the action decision space of the communicating party; For each participant (communicator and interferer) The belief function describes a participant's beliefs about the types of other participants after knowing their own type; For the benefit of the communication party; For observation space.

[0029] Benefits of the communication party for: ; In the formula, For the benefit of the communicating party; This is an action by the interfering party; Environmental state; For channel capacity; The signal-to-dryness ratio at the receiving end of the communication; It is a given threshold, which is related to the type of service, such as data, image, and voice; if the signal-to-interference-plus-noise ratio at the receiving end of the communication party... If the value is greater than the given threshold, the transmitted data packet can be correctly demodulated; otherwise, the return value is 0.

[0030] For the interfering party, due to the conflict of interests between the opposing sides, the interfering party's gains are... for: ; Introducing future cumulative rewards, future cumulative rewards for: ; In the formula, Accumulated rewards for the future; The discount factor is borrowed from the discount factor in the reward function of reinforcement learning. The discount factor can gradually reduce the proportion of the impact of future prediction value on current returns. , , , All are random variables; due to the actions Based on strategy It can also be said to be based on the probability distribution of beliefs, and and Relevant, therefore in time; , , , These are all random variables, making it difficult to assess the current situation. Therefore, we can... To find the expected value, we use an integral product to eliminate the randomness and obtain a value that can assess the current situation.

[0031] ; In the formula, To follow the strategy The expected cumulative return that can be obtained is the state-action value function. function), It can evaluate the effectiveness of the current strategy in the current state; For a moment The state random variable; For the current moment The state; For a moment The action random variable; For the current moment The action; It expresses expectation.

[0032] ; In the formula, This is a strategy for the interfering party; for Expectations for the action; Strategies for the communicating parties In state Take action The probability of; To follow the strategy Expected cumulative returns that can be obtained; For the state value function, Communication strategy Interference Strategy and state Related, if strategy It is fixed. It can reflect whether the current state is good or bad, if the strategy Not fixed. It can also reflect strategy Good or bad.

[0033] The problem of anti-interference optimization in this invention aims to obtain a better strategy. Therefore, the optimization goal of the communication party is to maximize... The optimization objective of the communicating party is expressed by the following formula: ; Since the interfering party's intention is to disrupt the communicating party's communication as much as possible, which is completely in conflict with the communicating party's interests, the interfering party's optimization objective is opposite to that of the communicating party; that is, the interfering party attempts to optimize its strategy. To minimize the state value function Therefore, the anti-interference optimization model constructed in this invention is expressed by the following formula: ; In the formula, For communication strategy; This is a strategy for the interfering party; This is the state value function.

[0034] S300: Solve the anti-interference optimization model based on the dual Actor-Critic deep reinforcement learning method to obtain the optimal anti-interference strategy; and perform anti-interference based on the optimal anti-interference strategy.

[0035] To find the equilibrium solution of the game, this invention constructs a dual Actor-Critic deep reinforcement learning method based on Actor-Critic to solve the joint domain anti-interference problem in the anti-interference optimization model. "Dual" refers to solving the joint domain problem, which includes the power domain and the frequency domain. Each domain utilizes an Actor network (i.e., a policy neural network), and all domains share a single Critic network (i.e., a value neural network). Taking the power domain and frequency domain as examples, this invention has two Actor networks: a power-priority Actor network and a frequency-priority Actor network. Since the models of each domain are similar, the theoretical analysis of one domain will suffice below.

[0036] In the general Actor-Critic algorithm, the agent makes decisions in response to the environment. In this invention, the agent not only faces the environment but also a jamming machine, so a game-theoretic jamming machine is added. The dual Actor-Critic deep reinforcement learning method proposed in this invention adds another Actor network (i.e., decision network) to the Actor-Critic algorithm. That is, the dual Actor-Critic deep reinforcement learning method includes a power-first Actor network, a frequency-first Actor network, and a Critic network. The dual Actor-Critic deep reinforcement learning method includes a training phase and an access adversarial phase. In the training phase, a parameter reset mechanism is introduced. During the training phase, the communicating party and the jammer with intelligent decision-making conduct adversarial training to continuously optimize the parameters of the two Actor networks and the Critic network to obtain anti-jamming strategies. In the access adversarial phase, the parameters of the two Actor networks and the Critic network remain unchanged.

[0037] ; To obtain the optimal anti-interference strategy and get the largest possible The dual Actor-Critic deep reinforcement learning method uses neural networks to approximate... , , among which Figure 4 As shown, an Actor network (i.e., a policy neural network) is used. approximate Using a Critic network (i.e., a value neural network). To approximate This allows us to solve the anti-interference optimization model and obtain the optimal anti-interference strategy.

[0038] like Figure 5As shown, during the training phase, two Actor networks make decisions about the same environment using different approaches. One Actor network prioritizes the power of the communication channel (i.e., prioritizing the channel with the optimal power), hence the term "power-priority Actor network." The other Actor network prioritizes the frequency of the communication channel (i.e., prioritizing the channel with the optimal frequency), hence the term "frequency-priority Actor network." During training, both the frequency-priority and power-priority Actor networks take actions and return the rewards to the Critic network (i.e., the value network). The Critic network scores the actions taken by the frequency-priority and power-priority Actor networks, guiding them to continuously improve and reach their optimal network state under their respective decision-making approaches. Simultaneously, the frequency-priority and power-priority Actor networks optimize their evaluation systems based on the rewards.

[0039] During training, the communicating parties rely on the Actor network. Making decisions, Critic Network By analyzing the Actor network The decisions made are scored to help the Actor network continuously improve; after training, the communicating parties still rely on the Actor network. It makes decisions, but after extensive training, it does not require the assistance of an Actor network.

[0040] The Actor network is trained using a policy gradient algorithm, which is expressed by the following formula: ; In the formula, State value function For policy neural network parameters The gradient; These are the parameters of the policy neural network; This is a strategy (Given state) ,parameter Choose action The probability of (the probability of) with respect to the parameter The gradient; The selected action; To follow the strategy The expected cumulative returns that can be obtained.

[0041] Once the gradient is calculated, performing gradient ascent once can improve the state value function. .

[0042] The Critic network is trained using the TD (Temporal Difference) learning algorithm. The principle of the TD learning algorithm is as follows: First, use a Critic network to predict the expected action reward: ; ; , Both are estimates of expected returns, but the value of the action-value function... The prediction is performed entirely using a Critic network (i.e., a value network), based on the state at time t. Actions (i.e., decisions) An estimate of the return; and middle The real reward is the decision made by the policy neural network (Actor network) at time t+1 based on the state. Add the estimated return to the actual reward value This forms an estimate of the return at time t; therefore More reliable, this invention... As the objective function of the TD learning algorithm, As a loss function, the loss function affects the neural network parameters of the Critic network. Taking the derivative yields the value gradient, i.e.: ; Then perform a gradient descent operation, let Closer .

[0043] The specific process for training the Critic network is as follows:

[0044] The specific algorithm of the dual Actor-Critic deep reinforcement learning method is as follows: Initialize Actor and Critic network parameters: Parameters for the frequency-priority Actor network for communication parties. Parameters of the power-priority Actor network for communication Parameters of the interfering Actor network Parameters of the frequency-priority Critic network for communication parties Parameters of the power-priority Critic network for communication Parameters of the interfering party's Critic network ; Input: Update interval of the communicating party The update interval of the interfering party Learning rate , .

[0045] Loop: The frequency-first sub-model (i.e., the frequency-first Actor network) and the power-first sub-model (i.e., the power-first Actor network) each execute the following loop: ; 1. The communicating party obtains the current observation status. Through the policy network of the communicating parties The probability distribution of the communicating parties' beliefs at this time was calculated. The communicating parties randomly sample and make communication actions. The jammer operates according to the jammer's strategy network. The same method was used to perform the interference action. ; 2. Perform communication actions Interference actions The environment presents a new state. The receiver gives the communication party a reward. and jammer rewards ; 3. Obtained by random sampling and ; Indicates based on the current state and strategy parameters Through communication strategy Sampling yields the action to be performed. It is used to interact with the environment to obtain rewards, and then update the strategy parameters. This allows the strategy to be gradually optimized. Indicates based on the current state and strategy parameters Through interference strategy Sampling yields the action to be performed. It is used to interact with the environment to obtain rewards, and then update the strategy parameters. This allows the strategy to be gradually optimized. 4. Calculate the values ​​of the action value function at time t and time t+1. and ; 5. The communication policy network and the interference policy network respectively calculate the error of the temporal difference learning: ; 6. Calculate the gradient of the Critic network: ; 7. Update the communication method using the temporal difference learning algorithm. Network parameters : ; 8. Calculate the gradients of the communication policy network and the interference policy network respectively: , ; 9. The policy networks of the communicating party and the interfering party are updated using gradient ascent: .

[0046] Judgment: When the Critic network parameters satisfy... Less than the set threshold And the duration is greater than the time of one round. Time: Training phase complete, Critic network parameters no longer updated. Critic network parameters in frequency-priority Actor networks and power-priority Actor networks Perform normalization and allow the two Actor networks to share a single Critic network parameter. .

[0047] Note: Among them Representative: Through policy networks The probability distribution of the communicating parties' beliefs at this time was calculated. Randomly sample and make actions ; These are just hypothetical actions, the purpose of which is to calculate. .

[0048] like Figure 6 As shown, after the Actor network and Critic network are fully trained, they can enter the access phase. In the access phase, the power-priority Actor network and the frequency-priority Actor network obtain the power-priority optimal action and the frequency-priority optimal action respectively according to the electromagnetic environment. The Critic network evaluates the power-priority optimal action and the frequency-priority optimal action respectively, and selects the action with more reward between the power-priority optimal action and the frequency-priority optimal action as the optimal anti-interference strategy. At this time, the neural network parameters will be corrected with a small exploration rate.

[0049] The communication success rate obtained through simulation implementation of this invention is as follows: Figure 7 As shown, by Figure 7 It can be seen that, during the simulated access phase, the anti-interference communication method based on Actor-Critic deep reinforcement learning disclosed in this invention has a high communication success rate.

[0050] When testing the robustness of the dual Actor-Critic deep reinforcement learning method, such as Figure 8As shown, this invention considers different jamming methods used by the jammer: frequency sweeping jamming, DQN-based intelligent decision jamming, and AC-based intelligent decision jamming, and statistically analyzes the average communication success rate during the simulated access phase; Figure 8 It can be seen that, under different interference methods, the anti-interference communication method based on Actor-Critic deep reinforcement learning disclosed in this invention has a high communication success rate.

[0051] Figure 9 This paper presents the communication success rate of the proposed dual Actor-Critic deep reinforcement learning method under the DNQ intelligent interference algorithm, compared with the contrasting methods DNQ and Actor-Critic, as the number of simulated rounds changes. The first 300 rounds represent the training process of the proposed method, during which the parameters are continuously optimized and gradually converge. In the last 200 rounds, the model parameters are kept constant, simulating the access phase. During this phase, the communication success rate of the proposed algorithm remains at its maximum value, demonstrating that the trained decision model can learn the optimal anti-interference method and avoid or suppress interference signals.

[0052] It is evident from the curves that the ordinary Actor-Critic algorithm reaches its optimal convergence in 250 rounds, while the proposed dual Actor-Critic deep reinforcement learning method takes approximately 180 rounds, exhibiting faster convergence speed. Furthermore, the proposed dual Actor-Critic deep reinforcement learning method shows less jitter in the simulated access phase curve. In contrast, the DNQ algorithm exhibits significant jitter during simulation and is difficult to converge, demonstrating that while the standalone DQN algorithm has good applications in channel access, it struggles to converge in joint domain anti-interference.

[0053] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0054] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An anti-interference communication method based on Actor-Critic deep reinforcement learning, characterized in that, Includes the following steps: S100: Establish a wireless communication scenario; and calculate the power spectral density function of the wireless signal received by the receiver and the power spectral density function of the signal at the jammer based on the wireless communication scenario; S200: Construct an anti-interference optimization model based on the power spectral density function of the wireless signal received by the receiver and the power spectral density function of the signal at the jammer; S300: Solve the anti-interference optimization model based on the dual Actor-Critic deep reinforcement learning method to obtain the optimal anti-interference strategy; and perform anti-interference based on the optimal anti-interference strategy.

2. The anti-interference communication method according to claim 1, characterized in that, The power spectral density function of the wireless signal received by the receiver in step S100 is calculated using the following formula: ； In the formula, Let be the power spectral density function of the wireless signal received at the receiving end; This refers to the channel gain from the transmitting end to the receiving end of the communication. This represents the channel gain from the jammer to the receiver. Let be the power spectral density function of the communication signal; The power spectral density function of the interference signal; Let be the power spectral density function of the noise; and These are the center frequencies of the communicating signal and the interfering signal, respectively. and These are the antenna transmit gains of the communicating party and the jammer, respectively. This represents the antenna receiving gain at the receiving end.

3. The anti-interference communication method according to claim 1, characterized in that, The signal power spectral density function at the jammer in step S100 is calculated using the following formula: ； In the formula, Let be the signal power spectral density function at the jammer; The receiving antenna gain of the interfering party's intelligent decision-making equipment. Let be the channel gain from the communicator to the jammer at time t; This refers to the antenna transmit gain of the jammer; Let be the power spectral density function of the communication signal; Let be the power spectral density function of the noise.

4. The anti-interference communication method according to claim 1, characterized in that, The anti-interference optimization model described in step S200 is expressed by the following formula: ； In the formula, For communication strategy; This is a strategy for the interfering party; This is the state value function.

5. The anti-interference communication method according to claim 1, characterized in that, Step S300 includes: solving the joint domain anti-interference problem in the anti-interference optimization model based on the dual Actor-Critic deep reinforcement learning method; the joint domain includes the power domain and the frequency domain.

6. The anti-interference communication method according to claim 5, characterized in that, The dual Actor-Critic deep reinforcement learning method adds another Actor network to the Actor-Critic network; the dual Actor-Critic deep reinforcement learning method includes a power-first Actor network, a frequency-first Actor network, and a Critic network.

7. The anti-interference communication method according to claim 6, characterized in that, Step S300 includes: During the training phase of the dual Actor-Critic deep reinforcement learning method, both the frequency-priority Actor network and the power-priority Actor network perform actions and return the rewards to the Critic network. The Critic network scores the actions performed by the frequency-priority Actor network and the power-priority Actor network, guiding them to continuously improve so that they reach the optimal network under the decision-making approach.

8. The anti-interference communication method according to claim 6, characterized in that, Step S300 includes: In the access phase of the dual Actor-Critic deep reinforcement learning method, the power-priority Actor network and the frequency-priority Actor network obtain the power-priority optimal action and the frequency-priority optimal action respectively based on the electromagnetic environment. The Critic network evaluates the power-priority optimal action and the frequency-priority optimal action respectively, and selects the action with more reward between the power-priority optimal action and the frequency-priority optimal action as the optimal anti-interference strategy.

9. The anti-interference communication method according to claim 6, characterized in that, The Actor network is trained using a policy gradient algorithm.

10. The anti-interference communication method according to claim 6, characterized in that, The Critic network is trained using a temporal difference learning algorithm.

Images