Method for avoiding multi-user spectrum access conflict in interference environment

Abstract

The invention belongs to the technical field of wireless communication, and particularly relates to a method for avoiding multi-user spectrum access conflicts in an interference environment. The invention provides a dynamic feedback sectional reward mechanism combining cognitive user spectrum access accuracy and resource allocation, which is applied to a centralized training distributed decision deep reinforcement learning framework so as to cope with multi-user spectrum access conflicts. External malicious interference can be avoided in a complex electromagnetic environment, the defect that a centralized algorithm is prone to malicious attacks is overcome, and the robustness of a communication system is improved while the communication requirement is guaranteed. A dynamic feedback sectional reward mechanism can adjust decisions of the cognitive users according to environmental changes, collision and mutual interference of the cognitive users are avoided under the condition that spectrum utilization is high in load, and the spectrum resource utilization rate of the system is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communications, and in particular relates to a method for avoiding conflicts in multi-user spectrum access in an interference environment. Background Art

[0002] With the rapid increase in the number of communication users, especially in scenarios such as the Internet of Vehicles and the Internet of Things, competition for spectrum resources has become more intense. When multiple users access the same frequency band at the same time, signal interference or data packet loss will occur, resulting in reduced communication quality or even communication interruption, which in turn causes a waste of spectrum resources. Due to the open nature of wireless channels, there may be illegal base stations and illegal mobile users maliciously occupying the spectrum in the spectrum environment, further exacerbating the shortage of spectrum environment resources and causing frequency interference to communication users. Frequent spectrum access conflicts may cause some users to be unable to access spectrum resources for a long time, resulting in unfair communication services.

[0003] Traditional multi-user spectrum access conflict avoidance methods mainly include statically allocated time division multiple access, frequency division multiple access methods, and relatively dynamic code division multiple access, orthogonal frequency division multiple access, etc. However, these methods are difficult to cope with the problem of highly tight spectrum resources in high-density user scenarios. The dynamic spectrum access method of cognitive radio based on reinforcement learning can gain experience by continuously interacting with the environment, detect spectrum holes and opportunistic spectrum access in unknown environments, thereby effectively alleviating the shortage of spectrum resources. However, the existing dynamic spectrum access technology enabled by reinforcement learning is mostly centralized decision-making, while in multi-user distributed access scenarios, a single agent using the same method and observing the same spectrum environment will make independent decisions and result in a large number of access conflicts. At the same time, the multi-user spectrum access method only considers the access situation of a single user during the training process, and does not comprehensively consider the decision-making performance and resource utilization fairness of all users in the system. Summary of the invention

[0004] To address the above problems, the present invention proposes a dynamic feedback segmented reward mechanism that combines cognitive user spectrum access accuracy and resource allocation, which is applied to a centralized training distributed decision-making deep reinforcement learning framework to deal with multi-user spectrum access conflicts.

[0005] The technical solution of the present invention is:

[0006] A method for avoiding conflicts in multi-user spectrum access in an interference environment is provided. The interference environment is defined as N independently operated jammers, which respectively interfere with N sub-bands in the current wireless environment to varying degrees. The interference environment has K cognitive users, i.e., K intelligent agents, where K>N. The method comprises the following steps:

[0007] S1. Cognitive users observe network environment information in each time slot and obtain their own observation vectors. The environment state information obtained by observing the tth time slot is defined as St :

[0008]

[0009] in represents the local environment state observed by user k in the tth time slot, A flag indicating whether the user observes sub-band n in the tth time slot and is subject to malicious interference, and a value of 0 indicates no interference, and a value of 1 indicates interference;

[0010] S2. For each cognitive user, i.e., an agent, a dynamic spectrum access model based on a multi-agent deep deterministic policy gradient algorithm is constructed to make dynamic spectrum access decisions in a centralized training and distributed execution manner, including a policy network μ θ and the evaluation network Q ω ;

[0011] S3, the agent based on the environmental state information S of the tth time slot t , get action a through the policy network t , that is, the spectrum access strategy, the agent uses a t Interacting with the environment, we can obtain the state of the channel selected by each cognitive user in the current time slot, the cumulative number of successful access to the spectrum to achieve data transmission, and calculate r through the reward function. t , S t 、a t 、r t and the observation vector S of the next time slot t+1 Composition of experience group (S t ,a t ,r t ,S t+1 ) is stored in the experience pool;

[0012] Among them, action represents the action of cognitive user k in time slot t, which is defined as:

[0013]

[0014] represents the sub-band selected for access by user k in the tth time slot;

[0015] Among them, rewards represents the reward obtained by user k in the tth time slot, is defined as:

[0016]

[0017] It means that cognitive user k selects the nth channel for access in time slot t, n∈1,2,…,N, Indicates that the data transmission in time slot t is successful. Indicates that the data transmission in time slot t failed. Indicates that no cognitive user i among all cognitive users in time slot t chooses to access the channel, Indicates that in time slot t there is a cognitive user i who chooses to access the channel, i∈1,2,…,K,i≠k, Indicates that cognitive user k does not make an access action in time slot t, k = argmax (Factors t ) indicates that the dynamic feedback coefficient value of cognitive user k is the maximum value of the dynamic feedback coefficients of all cognitive users. Factors t represents the dynamic feedback coefficient of each cognitive user at time slot t:

[0018]

[0019] is the dynamic feedback coefficient of cognitive user k:

[0020]

[0021] in, represents the cumulative number of times that cognitive user k chooses to access the non-interfered channel and successfully transmits data in the past t time slots, I(~) represents the indicator function, α and β are weight coefficients, represents the probability that cognitive user k has selected an undisturbed channel in the past t time slots, i.e., the decision accuracy performance feedback of cognitive user k. represents the probability that the cognitive user selects an undisturbed channel and successfully transmits data in the first t time slots;

[0022] When the data stored in the experience pool meets the set initial training experience scale, each agent is centrally trained after interacting with the environment (training can only start when the experience pool accumulates to a certain amount, and training is performed while the experience pool is updated interactively); when the number of training rounds reaches the set maximum number of training rounds, a trained dynamic spectrum access model is obtained;

[0023] S4. Obtain the environmental status information of the cognitive user in the current time slot in real time, input it into the trained dynamic spectrum access model, and obtain the spectrum access strategy of the current user through the intelligent agent.

[0024] The beneficial effects of the present invention are: it can avoid external malicious interference in a complex electromagnetic environment, overcome the disadvantage of centralized algorithms being vulnerable to malicious attacks, and improve the robustness of the communication system while ensuring communication needs. The dynamic feedback segmented reward mechanism can adjust the decision-making of cognitive users according to environmental changes, avoid collisions and mutual interference among cognitive users under high spectrum utilization load conditions, and improve the utilization rate of system spectrum resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the overall flow of the system of the present invention.

[0026] Figure 2 This is a flowchart of the algorithm of the multi-user dynamic spectrum access iterative model process of the present invention. DETAILED DESCRIPTION

[0027] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings:

[0028] As attached Figure 1 As shown, the method of the present invention specifically comprises:

[0029] S1. Cognitive users observe network environment information in each time slot and obtain their own observation vectors. The environment state information obtained by observing the tth time slot is defined as S t :

[0030]

[0031] in represents the local environment state observed by user k in the tth time slot, A flag indicating whether the user observes sub-band n in the tth time slot and is subject to malicious interference, and a value of 0 indicates no interference, and a value of 1 indicates interference;

[0032] S2. For each cognitive user, i.e., an agent, a dynamic spectrum access model based on the MADDPG multi-agent deep deterministic policy gradient algorithm is constructed to make dynamic spectrum access decisions in a centralized training and distributed execution manner, including the policy network μ θ and the evaluation network Q ω ;

[0033] Centralized training and distributed execution refer to: in the training state, the evaluation network Q in each agent ω It can obtain its own actions and the action strategies of other agents, and calculate the centralized action value of all cognitive user decisions; in the execution state, the strategy network μ θ It only needs to get actions based on the local environment state it observes.

[0034] The strategy network of each agent includes an input layer, a DNN neural network, a softmax output layer, and a one-hot encoding module connected in sequence; the evaluation network includes an input layer and a DNN neural network connected in sequence;

[0035] Among them, the policy network of agent k The decision function is expressed as:

[0036]

[0037] Evaluation network of agent k The centralized action value function is expressed as:

[0038] Q(S t ,a t ;ω k )

[0039] S3, as attached Figure 2 As shown in the figure, the multi-user dynamic spectrum access iterative model process is as follows: the agent converts the environmental state information S of the tth time slot into t , get action a through the policy network t , that is, the spectrum access strategy, the agent uses a t Interacting with the environment, we can obtain the state of the channel selected by each cognitive user in the current time slot, the cumulative number of successful access to the spectrum to achieve data transmission, and calculate r through the reward function. t , S t 、a t 、r t and the observation vector S of the next time slot t+1 Composition of experience group (S t ,a t ,r t ,S t+1 ) is stored in the experience pool;

[0040] Among them, action represents the action of cognitive user k in time slot t, which is defined as:

[0041]

[0042] represents the sub-band selected for access by user k in the tth time slot;

[0043] Among them, rewards represents the reward obtained by user k in the tth time slot, is defined as:

[0044]

[0045] It means that cognitive user k selects the nth channel for access in time slot t, n∈1,2,…,N, Indicates that the data transmission in time slot t is successful. Indicates that the data transmission in time slot t failed. Indicates that no cognitive user i among all cognitive users in time slot t chooses to access the channel, Indicates that in time slot t there is a cognitive user i who chooses to access the channel, i∈1,2,…,K,i≠k, Indicates that cognitive user k does not make an access action in time slot t, k = argmax (Factors t ) indicates that the dynamic feedback coefficient value of cognitive user k is the maximum value of the dynamic feedback coefficients of all cognitive users. Factors t represents the dynamic feedback coefficient of each cognitive user at time slot t:

[0046]

[0047] is the dynamic feedback coefficient of cognitive user k:

[0048]

[0049] in, represents the cumulative number of times that cognitive user k chooses to access the non-interfered channel and successfully transmits data in the past t time slots, I(~) represents the indicator function, α and β are weight coefficients, represents the probability that cognitive user k has selected an undisturbed channel in the past t time slots, i.e., the decision accuracy performance feedback of cognitive user k. represents the probability that the cognitive user selects an undisturbed channel and successfully transmits data in the first t time slots;

[0050] represents the number of cognitive users that choose to access the channel and collide with cognitive user k in the current time slot t;

[0051] When the data stored in the experience pool meets the set initial training experience scale, each agent is trained centrally after the agent interacts with the environment;

[0052] The specific steps of centralized training are as follows:

[0053] Agent k randomly extracts several samples (S t ,a t ,r t ,S t+1 ) participates in training and updates the evaluation network Q by minimizing the loss function ω The network parameters ω k , the loss function calculation formula is:

[0054]

[0055] in, represents the expected action of agent k in the next time slot, represents the target network parameters of the agent k strategy network in the next time slot, ωk represents the network parameters of the evaluation network in agent k, γ is the discount factor, represents the expected value of agent k in the next time slot, represents the network parameters of the evaluation network in agent k in the next time slot, η Q Indicates the size of the evaluation network learning rate, Indicates the L(ω k ) Perform gradient descent update;

[0056] Agent k updates the policy network μ by minimizing the policy gradient θ Network parameters θ k , the loss function calculation formula is:

[0057]

[0058] Among them, μ k represents the network parameters of the policy network in agent k, represents the target network parameter of the network evaluated by agent k in the next time slot, η μ Indicates the learning rate of the policy network, Indicates the L(ω k ) Perform gradient descent update;

[0059] Update the target network parameters of the policy network and evaluation network in agent k. The target network update formula is:

[0060] θ k - ←τθ k +(1-τ)θ k -

[0061] ω k - ←τω k +(1-τ)ω k -

[0062] Where, τ is the soft update rate;

[0063] When the number of training rounds meets the set maximum number of training rounds, a trained distributed dynamic spectrum access model is obtained;

[0064] S4. Obtain the environmental status information independently observed by the cognitive user in the current time slot in real time, input it into the trained distributed dynamic spectrum access model, and obtain the spectrum access strategy of the current user through the intelligent agent.

Claims

1. A method for avoiding conflicts in multi-user spectrum access in an interference environment, wherein the interference environment is defined as N independently operated jammers, each of which interferes with N sub-bands in the current wireless environment to varying degrees. The interference environment has K cognitive users, i.e., K agents, K>N; the method is characterized in that: The method comprises the following steps: S1. Cognitive users observe network environment information in each time slot and obtain their own observation vectors. The environment state information obtained by observing the tth time slot is defined as S t : in represents the local environment state observed by user k in the tth time slot, A flag indicating whether the user observes sub-band n in the tth time slot and is subject to malicious interference, and a value of 0 indicates no interference, and a value of 1 indicates interference; S2. For each cognitive user, i.e., an agent, a dynamic spectrum access model based on a multi-agent deep deterministic policy gradient algorithm is constructed to make dynamic spectrum access decisions in a centralized training and distributed execution manner, including a policy network μ θ and the evaluation network Q ω ; S3, the agent based on the environmental state information S of the tth time slot t , get action a through the policy network t , that is, the spectrum access strategy, the agent uses a t Interacting with the environment, we can obtain the state of the channel selected by each cognitive user in the current time slot, the cumulative number of successful access to the spectrum to achieve data transmission, and calculate r through the reward function. t , S t 、a t 、r t and the observation vector S of the next time slot t+1 Composition of experience group (S t ,a t ,r t ,S t+1 ) is stored in the experience pool; Among them, action represents the action of cognitive user k in time slot t, which is defined as: represents the sub-band selected for access by user k in the tth time slot; Among them, rewards represents the reward obtained by user k in the tth time slot, is defined as: It means that cognitive user k selects the nth channel for access in time slot t, n∈1,2,…,N, Indicates that the data transmission in time slot t is successful. Indicates that the data transmission in time slot t failed. Indicates that no cognitive user i among all cognitive users in time slot t chooses to access the channel, Indicates that in time slot t there is a cognitive user i who chooses to access the channel, i∈1,2,…,K,i≠k, Indicates that cognitive user k does not make an access action in time slot t, k = argmax (Factors t ) indicates that the dynamic feedback coefficient value of cognitive user k is the maximum value of the dynamic feedback coefficients of all cognitive users. Factors t represents the dynamic feedback coefficient of each cognitive user at time slot t: is the dynamic feedback coefficient of cognitive user k: in, represents the cumulative number of times that cognitive user k chooses to access the non-interfered channel and successfully transmits data in the past t time slots, I(~) represents the indicator function, α and β are weight coefficients, represents the probability that cognitive user k has selected an undisturbed channel in the past t time slots, i.e., the decision accuracy performance feedback of cognitive user k. represents the probability that the cognitive user selects an undisturbed channel and successfully transmits data in the first t time slots; When the data stored in the experience pool meets the set initial training experience scale, each agent is centrally trained after the agent interacts with the environment; when the number of training rounds reaches the set maximum number of training rounds, a trained dynamic spectrum access model is obtained; S4. Obtain the environmental status information of the cognitive user in the current time slot in real time, input it into the trained dynamic spectrum access model, and obtain the spectrum access strategy of the current user through the intelligent agent.

Images