Fast anti-jamming communication method and system based on state-action similarity weighted reward mechanism

Abstract

The application provides a fast anti-interference communication method and system based on a state-action similarity weighted reward mechanism, the method optimizes a reward function, designs a reward function capable of comprehensively considering instant rewards and state-action similarity, simultaneously introduces a similarity measurement method of state-action pairs, constructs a state-action pair sample database by using successful anti-interference experience, ensures that high-value and high-similarity experience samples can be preferentially used, and combines an optimized exploration-exploitation mechanism, so that the intelligent agent can more effectively use historical experience, improve learning efficiency and decision quality, simplify calculation complexity, and enhance the anti-interference ability and transmission efficiency of the wireless communication system in a complex electromagnetic environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication anti-interference technology, in particular to a fast anti-interference communication system and method based on state-action similarity weighted reward mechanism. BACKGROUND

[0002] With the rapid development of wireless communication technology, the electromagnetic environment is becoming increasingly complex, and communication systems are facing various forms of interference, such as malicious interference, co-channel interference, and environmental noise. These interferences can seriously affect the communication quality, and even cause communication interruption. Traditional anti-interference technology mainly relies on fixed spectrum allocation and pre-set anti-interference strategy, which is difficult to adapt to the dynamic changes of electromagnetic environment in time.

[0003] In recent years, with the rapid development of artificial intelligence technology, deep reinforcement learning (DRL) combines the powerful feature extraction ability of deep learning and the decision optimization ability of reinforcement learning, especially the introduction of deep Q-network (DQN), which makes it become the mainstream way to solve the problem of communication anti-interference. But this kind of algorithm is often complex, and it takes a long time to reach convergence, which is not conducive to the optimization and performance improvement of wireless communication system, therefore, how to improve the convergence speed of anti-interference algorithm, reduce the complexity of operation, and maintain a high success rate of confrontation is of great significance. SUMMARY

[0004] In view of the above problems, the present application provides a fast anti-interference communication method and system based on state-action similarity weighted reward mechanism, by optimizing the reward function, a reward function is designed which can consider the immediate reward and state-action similarity, and the similarity measurement method of state-action pair is introduced, the state-action pair sample database is constructed by using the successful anti-interference experience in history, to ensure that the high-value and high-similarity experience samples can be used preferentially, combined with the optimized exploration-exploitation mechanism, so that the agent can use the historical experience more effectively, improve the learning efficiency and decision quality, simplify the calculation complexity, and enhance the anti-interference ability and transmission efficiency of wireless communication system in complex electromagnetic environment.

[0005] The present application provides a fast anti-interference communication method based on state-action similarity weighted reward mechanism, applied to wireless communication system, comprising the following steps:

[0006] Step 1, model the communication anti-jamming problem as a Markov decision process, wherein the environment state of different time slots is defined based on the electromagnetic environment faced by the wireless communication system, including the instantaneous signal power observed by the receiver in each time slot, and the transmission action selected by the wireless communication system in different time slots is defined, including the transmission channel, the transmission power and the transmission rate selected by the transmitter;

[0007] Step 2, build a deep Q network and initialize the network parameters, wherein the deep Q network includes a policy Q network and a target Q network, the structure of the target Q network is the same as that of the policy Q network, and the target Q network is used to provide stable target Q values during the training process to calculate and update the action value of the agent;

[0008] Step 3, calculate the immediate reward of the environment state and the transmission action of the last time slot (t-1), and combine the environment state of the last time slot (t-1), the transmission action and the environment state of the current time slot (t) to obtain a complete state-action sample which is stored in the database;

[0009] Step 4, use the historical samples in the state-action pair database to calculate the similarity measure value of the current state-action pair, and generate a weighted reward through similarity weighting;

[0010] Step 5, form a set of experience samples by combining the environment state of the last time slot (t-1), the transmission action, the weighted reward, the environment state of the current time slot (t) and the termination flag, and store them in the experience pool for subsequent training;

[0011] Step 6, adjust the loss function of the deep Q network based on the weighted reward, and update the weights of the Q network through the back propagation algorithm;

[0012] Step 7, use the updated policy Q network to select the transmission action of the current time slot (t) and feed it back to the wireless communication system; if the communication has not ended, return to step 3 to continue the transmission action selection and system interaction of the next time slot (t+1) until the communication process ends.

[0013] In some embodiments, step 1 includes the following sub-steps:

[0014] Step 101, define the environment state of the wireless communication system with average received power;

[0015] Step 102, the transmission action of the receiver in T time slot is composed of the transmission channel, the transmission power and the transmission rate of T+1 time slot;

[0016] Step 103, define the reward function of the immediate reward based on the weight coefficient of the transmission rate, the cost factor of the transmission power, the average SJNR of T+1 time slot and the SJNR demodulation threshold of the receiver.

[0017] In some embodiments, the higher the transmission rate selected by the agent, the higher the weight coefficient of the transmission rate; the lower the transmission power selected by the agent, the lower the cost factor of the transmission power of the wireless communication system.

[0018] In some embodiments, step 2 comprises the following sub-steps:

[0019] Step 201, defining a policy Q network for estimating the state-action value vector Q(s, a), which is a multi-layer feedforward neural network;

[0020] Step 202, defining a target Q network, which has the same structure as the policy Q network, for providing stable target Q values during the training process to calculate and update the action value of the agent;

[0021] Step 203, initializing the network parameters of the policy Q network and the target Q network, including weights and bias terms;

[0022] Step 204, selecting an optimization algorithm and a loss function for updating the network parameters of the policy Q network;

[0023] Step 205, setting the hyperparameters in the training process;

[0024] Step 206, initializing an experience pool for storing experience samples obtained by the agent during the interaction process;

[0025] Step 207, initializing a step counter;

[0026] Step 208, during the training process, periodically copying the network parameters of the policy Q network to the target Q network, i.e., performing a target network update operation.

[0027] In some embodiments, step 3 comprises the following sub-steps:

[0028] Step 301, at each time slot, the receiver observes the environment state s t of the current time slot (t);

[0029] Step 302, the transmitter performs a transmission action a t-1 in the previous time slot (t-1);

[0030] Step 303, according to the environment state s t and the transmission action a t-1 , the immediate reward r(s t , a t-1 ) is calculated using a pre-set reward function;

[0031] Step 304, the receiver observes the environment state s t+1, the environment state of the next time slot (t+1);

[0032] Step 305, obtaining the environment state s t of the current time slot; t-1 Step 306, judging the current sample, if the immediate reward meets the requirement and there is no similar sample in the database, storing the state-action pair in the database. t t-1 Step 307, obtaining the environment state s t+1 of the next time slot; t t-1 Step 308, combining the current sample to form a complete state-action pair sample (s t , a t-1 , r(s t+1 , a f ), s f ).

[0033] Step 306, judging the current sample, if the immediate reward meets the requirement and there is no similar sample in the database, storing the state-action pair in the database.

[0034] In some embodiments, step 4 comprises the following sub-steps:

[0035] Step 401, dynamically updating the state space and the state-action pair database;

[0036] Step 402, extracting historical samples from the state-action pair sample database, including the previous state-action pair (s′,a′) and its corresponding immediate reward r(s′,a′);

[0037] Step 403, for the current state-action pair (s,a), comparing it with each historical sample, if the difference between the immediate reward r(s,a) of the current state-action pair (s,a) and the immediate reward r(s′,a′) of the historical sample state-action pair (s′,a′) is less than or equal to the similarity threshold, and the state transition probability P f (s,a) of the current state-action pair is equal to the state transition probability P f (s′,a′) of the historical sample, then it is determined that the two state-action pairs meet the similarity condition;

[0038] Step 404, for the samples successfully finding similar samples in the state-action pair sample database, weighting the immediate reward according to the similarity, obtaining the weighted reward.

[0039] In some embodiments, step 401 specifically comprises: recording all historical observed states from the initial transmission to the current state, obtaining the statistical state set According to the historical data and model prediction, expanding the possible future state to form a more complete state space Then, the state-action pairs that successfully resist interference and their related information are stored in the state-action pair sample database D, including the environment state s′, the transmission action a′, the immediate reward r(s′, a′), the state-action value vector Q(s′, a′) and the transition probability P f (s′, a′); record the state at each time step and update the state-action pair sample database.

[0040] In some embodiments, step 6 includes the following sub-steps:

[0041] Step 601: Randomly sample a batch of experience samples (s i ,a i-1 ,R(s i ,a i-1 ),s i+1 ,done i ), where s i is the environmental state, a i For transmission action, s i+1 For the next environment state, done i It is the termination sign;

[0042] Step 602: For each sampled empirical sample, calculate the target Q value:

[0043]

[0044] Among them, Q target is the target Q network, used to calculate the next environment state s i+1 All possible transmission actions a i+1 The maximum Q value of ;γ is the discount factor;

[0045] Step 603: Calculate the predicted value Q(s) of the current strategy Q network. i ,a i ; θ), where θ is the network parameter of the strategy Q network;

[0046] Step 604: Define the loss function L(θ) of the deep Q network as:

[0047]

[0048] Among them, N is the batch size, which indicates the number of samples used in each training;

[0049] Step 605: Use the back propagation algorithm to calculate the gradient of the loss function with respect to the network parameters θ of the policy Q network.

[0050] Step 606: Update the network parameters θ of the Q network according to the gradient;

[0051] Step 607: Periodically copy the updated network parameters θ of the policy Q network to the network parameters of the target Q network.

[0052] In some embodiments, step 7 includes the following sub-steps:

[0053] Step 701: At the current time slot (t), use the updated strategy Q network Q (s t ,a t ;θ), according to the current environmental state s t , select the optimal transmission action

[0054] Step 702: The selected transmission action Applied to wireless communication systems, that is, the transmitter performs transmission in the current time slot (t)

[0055] Step 703: The transmitter sends a signal according to the selected transmission parameters, and the receiver receives the signal synchronously in the same time slot and performs demodulation and decoding according to the current electromagnetic environment state;

[0056] Step 704: The wireless communication system obtains the transmission result of the current time slot based on the feedback from the receiver and updates the environment state s t+1 ;

[0057] Step 705: The environmental state s of the current time slot (t) t , transfer action Instant reward r(s,a), environment state s at the next time slot (t+1) t+1 and the termination symbol done to form a new experience sample And stored in the experience pool.

[0058] Step 706: If the communication is not completed, return to step 3 and continue to perform the transmission action selection and system interaction for the next time slot until the communication process is completed.

[0059] The present invention also provides a wireless communication system, comprising a transmitter and a receiver, wherein the transmitter and the receiver are synchronized according to time slots and execute the above-mentioned fast anti-interference communication method based on the state-action similarity weighted reward mechanism during the communication process.

[0060] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0061] 1. By introducing a reward mechanism based on similarity measurement, the present invention enables intelligent agents to more effectively identify and utilize historical successful experiences, thereby selecting more optimal transmission actions in complex electromagnetic environments, significantly improving the anti-interference performance of wireless communication systems, and enhancing the stability and reliability of data transmission.

[0062] 2、The application adopts a similarity weighted reward function, so that the agent can fully utilize the experience of similar state-action pairs during the training process, accelerate the convergence speed of DQN, reduce the time and computing resources required for training, and improve the overall learning efficiency.

[0063] 3、The application combines a dynamically adjusted exploration-exploitation mechanism, so that the agent can flexibly balance exploring new strategies and utilizing known optimal strategies in different interference environments, improve the intelligent level of decision-making, and ensure that the optimal transmission scheme is always selected in a variable environment.

[0064] 4、The application stores both immediate rewards and weighted rewards, so that the system can maintain high adaptability and robustness when facing diversified and dynamically changing interference sources, reduce communication interruption and performance degradation caused by environmental changes, and ensure the continuous and stable operation of the communication system.

[0065] 5、The application stores both immediate rewards and weighted rewards in the experience replay pool, simplifies the implementation logic of similarity calculation and training process, avoids data redundancy and complex synchronization problems, makes the system design more simple and efficient, and is convenient for practical application and further optimization. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 The flowchart of the fast anti-interference communication method based on the state-action similarity weighted reward mechanism in the embodiment of the application.

[0067] Figure 2 The flowchart of constructing and training the deep Q network in the embodiment of the application.

[0068] Figure 3 The flowchart of generating a weighted reward by similarity weighting in the embodiment of the application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments. The components of the embodiments of the application described and shown in the drawings can be arranged and designed in various different configurations.

[0070] Therefore, the following detailed description of the embodiments of the application provided in the drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the application.

[0071] As Figure 1 shown in the figure, the embodiment proposes a fast anti-jamming communication method based on state-action similarity weighted reward mechanism, applied to a wireless communication system, the wireless communication system comprising a transmitter and a receiver, the transmitter and the receiver being synchronized by time slots. The method comprises the following steps:

[0072] Step 1, model the communication anti-jamming problem as a Markov Decision Process (MDP), wherein the environment state of different time slots is defined based on the electromagnetic environment faced by the wireless communication system, including the instantaneous signal power observed by the receiver in each time slot, and the transmission action selected by the wireless communication system in different time slots is defined, including the transmission channel, the transmission power and the transmission rate selected by the transmitter;

[0073] Step 2, construct a Deep Q-Network (DQN) and initialize the network parameters, wherein the Deep Q-Network comprises a policy Q-network and a target Q-network, the structure of the target Q-network being the same as that of the policy Q-network, used to provide stable target Q-values in the training process to calculate and update the action value of the agent;

[0074] Step 3, calculate the immediate reward of the environment state and the transmission action of the last time slot (t-1), and obtain a complete state-action sample by combining the environment state of the last time slot (t-1), the transmission action and the environment state of the current time slot (t), and store the sample in a database;

[0075] Step 4, use the state-action pair database to calculate the similarity measure value of the current state-action pair, and generate a weighted reward through similarity weighting;

[0076] Step 5, form an experience sample by using the environment state of the last time slot (t-1), the transmission action, the weighted reward, the environment state of the current time slot (t) and the termination flag, and store the sample in an experience pool for subsequent training;

[0077] Step 6, adjust the loss function of the Deep Q-Network based on the weighted reward, and update the weights of the Q-network through a backpropagation algorithm;

[0078] Step 7, use the updated policy Q-network to select the transmission action of the current time slot (t) and feed it back to the wireless communication system; if the communication has not ended, return to step 3 to continue to select the transmission action of the next time slot (t+1) and interact with the system until the communication process ends.

[0079] In some embodiments, the communication anti-jamming problem is modeled as a Markov Decision Process in step 1, which comprises the following sub-steps:

[0080] Step 101, define the state of the environment of the wireless communication system with average received power, denoted as:

[0081]

[0082] where S T denotes the state of the environment of the T-slot wireless communication system, the average received power consists of the transmitter signal power and the interference signal power received on the channel m, denoted as:

[0083]

[0084] where δ(·) is the indicator function, i.e., δ(x) = 1 if x is true, otherwise δ(x) = 0; p T-1 is the transmission power of the transmitter, g T-1 is the transmission channel gain, denotes the average power of the received interference plus noise.

[0085] Step 102, the action space in the Markov decision process contains the set of all possible execution actions that the agent can take. The transmission action of the receiver at T slot decision consists of the transmission channel of T+1 slot, the transmission power and the transmission rate, denoted as:

[0086] a T = (f T+1 , p T+1 , v 1+1 ) ∈ A (3)

[0087] where a T denotes the transmission action of T slot decision, f T+1 is the transmission channel of T+1 slot, p T+1 is the transmission power of T+1 slot, v T+1 is the transmission rate of T+1 slot;

[0088] Step 103, based on the reward function defining the immediate reward, denoted as:

[0089]

[0090] where, is the weight coefficient of the transmission rate, the higher the transmission rate selected by the agent, the higher the weight coefficient of the transmission rate; is the cost factor of the transmission power, the lower the transmission power selected by the agent, the lower the cost factor of the transmission power of the wireless communication system. θ T+1 is the average SJNR at T+1 slot, and For the SJNR demodulation threshold of the receiver, the greater the transmission rate, the higher the demodulation threshold. δ(·) is an indicator function, which is 1 only when i.e. demodulation is successful, otherwise 0.

[0091] In some embodiments, the deep Q network is constructed and the network parameters are initialized in step 2, which specifically includes the following sub-steps:

[0092] Step 201, define a policy Q network (QNetwork) for estimating the state-action value vector Q(s, a), which is a multi-layer feedforward neural network, including:

[0093] Input layer: receiving state vector s, the dimension of which is the number of channels M;

[0094] Hidden layer: including one or more fully connected layers, each hidden layer containing a number of neurons, and the activation function being a nonlinear function;

[0095] Output layer: outputting the state-action value vector Q(s, a), the dimension of which is the size of the action space;

[0096] Step 202, define a target Q network (Target QNetwork), which has the same structure as the policy Q network, and is used to provide stable target Q values during the training process to calculate and update the action value of the agent;

[0097] Step 203, initialize the network parameters of the policy Q network and the target Q network, including weights and bias terms, which can be obtained by random initialization or pre-training;

[0098] Step 204, select an optimization algorithm and a loss function for updating the network parameters of the policy Q network;

[0099] Step 205, set the hyperparameters in the training process, including: learning rate α, discount factor γ, initial ε value of exploration strategy, ε decay rate ε decay , batch size and capacity of experience replay buffer;

[0100] Step 206, initialize the experience pool (experience replay buffer) for storing the experience samples obtained by the agent during interaction with the environment, including the environment state, the transmission action, the weighted reward, the next state and the termination flag;

[0101] Step 207, initialize the step counter for recording the total number of steps of the agent interacting with the environment, so as to adjust the exploration rate ε according to the number of steps in the ε-greedy strategy;

[0102] Step 208, in the training process, the network parameters of the policy Q network are copied into the target Q network at a certain step or period, that is, the target network update operation is performed, to ensure that the target Q network provides stable target values and reduces the oscillation and instability in the training process;

[0103] The target Q network is used to provide a stable reference value when calculating the target Q value, and the network parameters thereof are copied from the policy Q network at a certain step or period in the training process. By introducing the target Q network, the instability in the training process can be effectively alleviated, and the learning efficiency and decision performance of the agent in a dynamic interference environment can be improved.

[0104] In some embodiments, in step 3, the immediate reward of the environment state and the transmission action of the last time slot (t-1) is calculated, and a complete state-action sample is obtained by combining the environment state of the last time slot (t-1), the transmission action and the environment state of the current time slot (t), and stored in the database, which includes the following sub-steps:

[0105] Step 301, in each time slot, the receiver observes the environment state s t of the current time slot (t);

[0106] Step 302, the transmitter performs a transmission action a t-1 in the last time slot (t-1), including the selected transmission channel, the transmission power and the transmission rate;

[0107] Step 303, according to the environment state s t and the transmission action a t-1 , the immediate reward r(s t ,a t-1 ) is calculated by using a preset reward function, which comprehensively considers the communication success rate, channel quality and energy consumption and other factors;

[0108] Step 304, the receiver observes the environment state s t+1 of the current time slot (t) as the environment state of the next time slot (t+1);

[0109] Step 305, the environment state s t of the current time slot, the transmission action a t-1 , the immediate reward r(s t ,a t-1 ) and the environment state s t+1 of the next time slot are combined to form a complete state-action pair sample (s t ,a t-1 ,r(s t ,a t-1 ),s t+1 ).

[0110] Step 306: The current sample is judged. If the immediate reward meets the requirements and there is no sample in a similar state in the database, it is stored in the state-action pair database.

[0111] In some embodiments, step 4 uses historical samples in the state-action pair database to calculate the similarity metric of the current state-action pair, and generates a similarity weighted reward by similarity weighting, which specifically includes the following sub-steps:

[0112] Step 401: Dynamically update the state space and state-action pair database:

[0113] In the case that the complete state space cannot be modeled in advance, the accuracy of the similarity measurement is gradually improved by dynamically expanding the state space. Record all historical observed states from the initial transmission to the current state to obtain a statistical state set Expand possible future states based on historical data and model predictions to form a more complete state space Then, the state-action pairs that successfully resist interference and their related information are stored in the state-action pair sample database D, including the environment state s′, the transmission action a′, the immediate reward r(s′, a′), the state-action value vector Q(s′, a′) and the transition probability P f (s′, a′); record the state at each time step and update the state-action pair sample database;

[0114] Step 402: Extract historical samples from the state-action pair sample database, including previous state-action pairs (s′, a′) and their corresponding immediate rewards r(s′, a′);

[0115] Step 403: For the current state-action pair (s, a), compare it with each historical sample. If the difference between the immediate reward r(s, a) of the current state-action pair (s, a) and the immediate reward r(s′, a′) of the historical sample state-action pair (s′, a′) is less than or equal to the similarity threshold, and the state transition probability P of the current state-action pair is f (s,a) and the state transition probability P of historical samples f When (s′, a′) are equal, it can be determined that the two state-action pairs meet the similarity condition; where:

[0116] The similarity determination condition is expressed as:

[0117] |r(s,a)-r(s′,a′)|≤θ R (5)

[0118] Among them, θ R is the similarity threshold;

[0119] The state transition probability determination condition is expressed as:

[0120] P f (s,a)=P f (s′,a′) (6)

[0121] Step 404, for the sample successfully found in the state-action pair sample database Similar samples, weight their immediate rewards for similarity, get weighted rewards, expressed as:

[0122] R(s,a)=r(s,a)*β+λ (7)

[0123] Where, β represents the scaling factor of the immediate reward r(s, a), by adjusting β, the influence weight of the immediate reward can be increased or decreased according to the similarity measurement result of the state-action pair; λ represents an offset value (translation amount) added to the scaled reward value, by adjusting λ, the final reward value can be moved up or down by a benchmark value, to realize the fine tuning and compensation of the reward signal.

[0124] Through the above steps, the wireless communication system can use historical successful experience to calculate the similarity measurement value of the current state-action pair in a dynamically complex electromagnetic environment, and optimize the decision strategy of the intelligent agent by adjusting the reward function, guide the intelligent agent to prefer to select the verified effective strategy, improve the performance and learning efficiency of communication anti-jamming, and ensure the fairness between different state-action pairs through weighted reward, and simplify the calculation complexity.

[0125] In some embodiments, the environment state s t , the transmission action a t-1 , the weighted reward R(s t ,a t-1 ), the environment state s t+1 and the termination flag done t of the last time slot (t-1) in step 5 constitute a set of experience samples, which are stored in the experience pool for subsequent training, which specifically includes the following:

[0126] The experience sample (s t ,a t-1 ,R(s t ,a t-1 ),s t+1 ,done t ) is stored in the experience pool (experience replay buffer) for subsequent training and learning process;

[0127] In some embodiments, the loss function of the deep Q network is adjusted based on the similarity weighting reward in step 6, and the weights of the Q network are updated by a back propagation algorithm, specifically including the following sub-steps:

[0128] Step 601, randomly sample a batch of experience samples (s i ,a i-1 ,R(s i ,a i-1 ),s i+1 ,done i ) from the experience pool, wherein s i is the environment state, a i is the transmission action, s i+1 is the next environment state, done i is the termination flag;

[0129] Step 602, for each experience sample, calculate the target Q value:

[0130] y i =R(s i ,a i )+γ·max a′ Q target (s i+1 ,a i+1 )·(1-done i ) (8)

[0131] Wherein Q target is the target Q network, used to calculate the maximum Q value of all possible transmission actions a i+1 under the next environment state s i+1 ; γ is the discount factor;

[0132] Step 603, calculate the predicted value Q(s i ,a i ; θ) of the current policy Q network, wherein θ is the network parameter of the policy Q network;

[0133] Step 604, define the loss function L(θ) of the deep Q network as:

[0134]

[0135] Wherein N is the batch size, indicating the number of samples used for each training;

[0136] Step 605, calculate the gradient of the loss function to the network parameter θ of the policy Q network using the back propagation algorithm

[0137] Step 606, update the network parameter θ of the policy Q network according to the gradient:

[0138]

[0139] wherein η is the learning rate;

[0140] Step 607, periodically copy the network parameters θ of the updated policy Q network to the network parameters of the target Q network:

[0141] θ target ←θ (11)

[0142] Thus, it is ensured that the target Q network provides stable target values and reduces fluctuations in the training process.

[0143] Through the above steps, the similarity weighted reward is used to adjust the loss function of the deep Q network, so that the policy Q network pays more attention to the state-action pairs similar to the historical successful experience in the training process, and the learning efficiency and anti-interference ability of the agent are improved. At the same time, the target Q network, as a delayed copy of the policy Q network, provides stable target Q values to assist the training of the policy Q network, and enhances the decision-making performance of the agent in a dynamic interference environment.

[0144] In some embodiments, the updated policy Q network is used to select the transmission action of the current time slot (t) in step 7, and is fed back to the wireless communication system, which specifically includes the following sub-steps:

[0145] Step 701, in the current time slot (t), the updated policy Q network Q(s t ,a t ; θ) is used to select the optimal transmission action t according to the current environment state s That is:

[0146]

[0147] wherein θ is the network parameter of the updated policy Q network, and a is the set of feasible actions, including different combinations of transmission channels, transmission power and transmission rate.

[0148] Step 702, the selected transmission action is applied to the wireless communication system, that is, the transmitter performs the transmission action

[0149] Step 703, the transmitter transmits signals according to the selected transmission parameters, and the receiver synchronously receives signals in the same time slot and demodulates and decodes according to the current electromagnetic environment state;

[0150] Step 704, the wireless communication system obtains the transmission result of the current time slot according to the feedback of the receiver, and updates the environment state s t+1 ;

[0151] Step 705, the environment state s t , the transmission action The immediate reward r(s, a), the environment state s t+1 of the next time slot (t+1) and the termination flag done constitute a new experience sample and are stored in the experience pool.

[0152] Step 706, if the communication is not over, return to step 3 and continue to perform the transmission action selection and system interaction of the next time slot until the communication process is over.

[0153] Wherein, by using the updated policy Q network, the agent can select the optimal transmission action t based on the current environment state s and the learned policy in real time and apply it to the wireless communication system, thereby improving the anti-interference performance and transmission efficiency of the wireless communication system in a complex electromagnetic environment.

[0154] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A fast anti-interference communication method based on a state-action similarity weighted reward mechanism, applied to wireless communication systems, characterized in that: The following steps are involved: Step 1: Model the communication anti-interference problem as a Markov decision process. Based on the electromagnetic environment faced by the wireless communication system, the environmental state of each time slot is defined, including the instantaneous signal power observed by the receiver in each time slot. The transmission action selected by the wireless communication system in each time slot is also defined, including the transmission channel, transmission power, and transmission rate selected by the transmitter. Step 2: Build a deep Q network and initialize network parameters. The deep Q network includes a policy Q network and a target Q network. The target Q network has the same structure as the policy Q network and is used to provide a stable target Q value during training to calculate and update the action value of the agent. Step 3: Calculate the environment state and the immediate reward of the transmission action at the previous time slot (t-1). Combine the environment state and transmission action at the previous time slot (t-1) and the environment state at the current time slot (t) to obtain a complete set of state-action samples and store them in the database. Step 4: Use historical samples in the state-action pair database to calculate the similarity measure of the current state-action pair and generate a weighted reward by weighting the similarity. Step 5: The environment state, transmission action, weighted reward of the previous time slot (t-1), the environment state of the current time slot (t) and the termination flag are combined into a set of experience samples and stored in the experience pool for subsequent training; Step 6: Adjust the loss function of the deep Q network based on the weighted reward and update the weights of the Q network through the backpropagation algorithm; Step 7: The network uses the updated strategy Q to select the transmission action for the current time slot (t) and feeds it back to the wireless communication system. If the communication is not completed, it returns to step 3 and continues to select the transmission action for the next time slot (t+1) and interact with the system until the communication process ends.

2. The fast anti-interference communication method based on the state-action similarity weighted reward mechanism according to claim 1 is characterized in that: Step 1 includes the following sub-steps: Step 101: Define the environmental state of the wireless communication system using average received power; Step 102: The transmission channel, transmission power, and transmission rate of the T+1 time slot constitute the transmission action decided by the receiver in the T time slot; Step 103: Define a reward function for the immediate reward based on the weight coefficient of the transmission rate, the cost factor of the transmission power, the average SJNR of the receiving end in the T+1 time slot, and the SJNR demodulation threshold of the receiver.

3. The fast anti-interference communication method based on the state-action similarity weighted reward mechanism according to claim 2 is characterized in that: The higher the transmission rate selected by the agent, the higher the weight coefficient of the transmission rate; the lower the transmission power selected by the agent, the lower the cost factor of the transmission power of the wireless communication system.

4. The fast anti-interference communication method based on the state-action similarity weighted reward mechanism according to claim 1 is characterized in that: Step 2 includes the following sub-steps: Step 201: define a policy Q network for estimating a state-action value vector Q(s,a), wherein the policy Q network is a multi-layer feedforward neural network; Step 202: Define a target Q network. The target Q network has the same structure as the policy Q network and is used to provide a stable target Q value during training to calculate and update the action value of the agent. Step 203: Initialize the network parameters of the strategy Q network and the target Q network, including weights and bias terms; Step 204: Select an optimization algorithm and loss function to update the network parameters of the policy Q network; Step 205: setting hyperparameters during training; Step 206: Initialize the experience pool to store the experience samples obtained by the agent during the interaction process; Step 207: Initialize the step counter; Step 208: During the training process, the network parameters of the policy Q network are periodically copied to the target Q network, that is, the target network update operation is performed.

5. The fast anti-interference communication method based on the state-action similarity weighted reward mechanism according to claim 1 is characterized in that: Step 3 includes the following sub-steps: Step 301: In each time slot, the receiver observes the environmental state s of the current time slot (t) t ; Step 302: The transmitter performs transmission action a in the previous time slot (t-1). t-1 ; Step 303: According to the environmental state s t and transfer action a t-1 , use the preset reward function to calculate the instant reward r(s t ,a t-1 ); Step 304: The receiver observes the environmental state s at the current time slot (t) t+1 , as the environmental state of the next time slot (t+1); Step 305: The current time slot environment state s t , transmission action a t-1 , immediate reward r(s t ,a t-1 ), the environmental state s of the next time slot t+1 Combined to form a complete state-action pair sample (s t ,a t-1 ,r(s t ,a t-1 ),s t+1 ); Step 306: The current sample is judged. If the immediate reward meets the requirements and there is no sample in a similar state in the database, it is stored in the state-action pair database.

6. The fast anti-interference communication method based on the state-action similarity weighted reward mechanism according to claim 1 is characterized in that: Step 4 includes the following sub-steps: Step 401: Dynamically update the state space and state-action pair database; Step 402: Extract historical samples from the state-action pair sample database, including previous state-action pairs (s′, a′) and their corresponding immediate rewards r(s′, a′); Step 403: For the current state-action pair (s, a), compare it with each historical sample. If the difference between the immediate reward r(s, a) of the current state-action pair (s, a) and the immediate reward r(s′, a′) of the historical sample state-action pair (s′, a′) is less than or equal to the similarity threshold, and the state transition probability P of the current state-action pair is f (s,a) and the state transition probability P of historical samples f (s ′ ,a ′ ) are equal, then the two state-action pairs are determined to meet the similarity condition; Step 404: For samples that successfully find similar samples in the state-action pair sample database, perform similarity weighting on their immediate rewards to obtain weighted rewards.

7. The fast anti-interference communication method based on the state-action similarity weighted reward mechanism according to claim 6 is characterized in that: Step 401 specifically includes: recording all historically observed states from the initial transmission to the current state, and obtaining a statistical state set Expand possible future states based on historical data and model predictions to form a more complete state space Then, the state-action pairs that successfully resist interference and their related information are stored in the state-action pair sample database D, including the environment state s′, the transmission action a′, the immediate reward r(s′, a′), the state-action value vector Q(s′, a′) and the transition probability P f (s ′ ,a ′ ); record the state at each time step and update the state-action pair sample database.

8. The fast anti-interference communication method based on the state-action similarity weighted reward mechanism according to claim 1 is characterized in that: Step 6 includes the following sub-steps: Step 601: Randomly sample a batch of experience samples (s i ,a i-1 ,R(s i ,a i-1 ),s i+1 ,done i ), where s i is the environmental state, a i For transmission action, s i+1 For the next environment state, done i It is the termination sign; Step 602: For each sampled empirical sample, calculate the target Q value: Among them, Q target is the target Q network, used to calculate the next environment state s i+1 All possible transmission actions a i+1 The maximum Q value of ;γ is the discount factor; Step 603: Calculate the predicted value Q(s) of the current strategy Q network. i ,a i ; θ), where θ is the network parameter of the strategy Q network; Step 604: Define the loss function L(θ) of the deep Q network as: Among them, N is the batch size, which indicates the number of samples used in each training; Step 605: Use the back propagation algorithm to calculate the gradient of the loss function with respect to the network parameters θ of the policy Q network. Step 606: Update the network parameters θ of the Q network according to the gradient; Step 607: Periodically copy the updated network parameters θ of the policy Q network to the network parameters of the target Q network.

9. The fast anti-interference communication method based on the state-action similarity weighted reward mechanism according to claim 1 is characterized in that: Step 7 includes the following sub-steps: Step 701: At the current time slot (t), use the updated strategy Q network Q (s t ,a t ;θ), according to the current environmental state s t , select the optimal transmission action Step 702: The selected transmission action Applied to wireless communication systems, that is, the transmitter performs transmission in the current time slot (t) Step 703: The transmitter sends a signal according to the selected transmission parameters, and the receiver receives the signal synchronously in the same time slot and performs demodulation and decoding according to the current electromagnetic environment state; Step 704: The wireless communication system obtains the transmission result of the current time slot based on the feedback from the receiver and updates the environment state s t+1 ; Step 705: The environmental state s of the current time slot (t) t , transfer action Instant reward r(s,a), environment state s at the next time slot (t+1) t+1 and the termination symbol done to form a new experience sample And store it in the experience pool; Step 706: If the communication is not completed, return to step 3 and continue to perform the transmission action selection and system interaction for the next time slot until the communication process is completed.

10. A wireless communication system, characterized in that: The invention comprises a transmitter and a receiver, wherein the transmitter and the receiver are synchronized according to time slots and execute the fast anti-interference communication method based on the state-action similarity weighted reward mechanism as described in any one of claims 1 to 9 during the communication process.

Images