A multi-agent collaborative spectrum sensing and access method for cognitive radio

Abstract

The present invention discloses a multi-agent collaborative spectrum sensing and access method for cognitive radio, including: Step S101: Obtain the environmental parameters of each SU n at the beginning of the current time frame; Step S102: Input the environmental parameters of each SU n at the beginning of the current time frame into the deterministic policy deep gradient MADDPG model; Step S103: Obtain the sensing policy of each SU n in the current time frame output by the MADDPG model, where the sensing policy is to select a certain channel to detect whether it is idle; Step S104: Fuse the sensing results of SUn and its partners' sensing results into the global state S(t), and then send the global state S(t), the global action A(t), the reward r(t), and the next moment state S(t+1) to the experience replay buffer of the deterministic policy deep gradient MADDPG model; Use the global state S(t) as the observation of the next moment, and so on in a loop until the training step ends. It solves the problem of how SU can quickly find an idle channel while avoiding collisions with other users.

Description

Technical Field

[0001] The present invention belongs to the technical field of Internet of Things communications, and in particular relates to a multi-agent collaborative spectrum sensing and access method for cognitive radio. Background Art

[0002] With the development of wireless communication technology and the increase of wireless multimedia applications, wireless spectrum has become an increasingly scarce resource. Studies have shown that the existing spectrum management and allocation strategy is one of the main reasons for the shortage of spectrum resources. The emergence of cognitive radio has created a new situation for alleviating the shortage of spectrum resources, realizing dynamic spectrum management and improving spectrum utilization.

[0003] Cognitive radio, in a broad sense, refers to radio terminals that have sufficient intelligence and cognitive capabilities to detect, analyze, learn, reason and plan the historical and current conditions of the surrounding wireless environment, and use the corresponding results to select the appropriate frequency band for transmission.

[0004] Cognitive radio consists of two types of users: primary users, who can occupy any frequency band to transmit information at any time; and secondary users, who can sense spectrum "holes" and use the frequency band for transmission when the primary user does not occupy the frequency band. Cognitive radio networks can greatly improve temporary and spatial spectrum utilization by identifying and exploring spectrum holes of authorized primary users. However, since the occupancy of the primary channel changes dynamically, fast and accurate spectrum perception is crucial.

[0005] Multi-agent system is a new distributed computing technology. In a multi-agent system, each agent learns to improve its own strategy by interacting with the environment to obtain rewards, so as to obtain the best strategy in the environment. In single-agent reinforcement learning, the environment where the agent is located is stable and unchanging, but in multi-agent reinforcement learning, the environment is complex and dynamic.

[0006] In a multi-agent system, there are at least two or more agents, and there are certain relationships between the agents, such as cooperative relationships, competitive relationships, or both. In the example of the present invention, the agents not only cooperate with each other and share perception results, but also compete with each other, that is, compete for spectrum resources, and there is an access collision problem. In a multi-agent system, the reward obtained by each agent is not only related to its own actions, but also to the actions of other agents, that is, the agents influence each other.

[0007] Generally, when an intelligent agent, that is, a secondary user SU, has an incoming call demand, it will scan all channels in sequence until it detects an idle channel or fails. This channel sensing and access method will cause certain transmission delays and low spectrum utilization. Due to the influence of fading and shadowing effects, each SU can only detect one channel in a time slot. When the number of channels in the communication network is greater than the number of SUs, this detection method is inefficient and greatly reduces spectrum utilization. Summary of the invention

[0008] The purpose of the present invention is to provide a multi-agent cooperative spectrum sensing and access method for cognitive radio to solve the problem of how SU can quickly find an idle channel while avoiding collision with other users.

[0009] The present invention adopts the following technical scheme: a multi-agent cooperative spectrum sensing and access method for cognitive radio, based on a cognitive radio network, wherein the cognitive radio network has M primary users PU, N secondary users SU, N fb orthogonal channels, the transmit power of PU is fixed at P, and the channel noise is additive white Gaussian noise;

[0010] The sensing and access method comprises:

[0011] Step S101: Obtain each SU in the N SUs n The environmental parameters at the beginning of the current time frame, where n=1, 2, ..., N, include the environmental parameters of each SU n Observation S n ;

[0012] Step S102: Each SU n The environment parameters at the beginning of the current time frame are input into a deterministic policy deep gradient MADDPG model;

[0013] Step S103: Obtain each SU output by the MADDPG model n A sensing strategy in the current time frame, wherein the sensing strategy is to select a certain channel to detect whether it is idle;

[0014] Step S104: The perception results of SUn and its partners are integrated into the global state S(t), and the global state S(t), the global action A(t), the reward r(t) and the next state S(t+1) are sent to the experience replay buffer of the deterministic policy deep gradient MADDPG model;

[0015] The global state S(t) is used as the observation at the next moment, and the cycle repeats until the training step is completed.

[0016] Furthermore, step S103 also includes: at the beginning of each time frame, all SUs simultaneously select their own strategies according to their own observations, that is, select a certain channel to detect whether the channel is idle. Each SU has only one chance until all SUs find an idle channel or fail, and then a time frame ends.

[0017] Furthermore, in step S104, the method of fusing the perception results of SUn and its partners into the global state S(t) is as follows:

[0018] Step S1041: Current SU n The credibility is calculated based on the rewards obtained by other SUs after performing actions at historical moments. If an SU has served as the current SU n If you are a good partner, you will have a higher level of credibility;

[0019] Step S1042: Calculate corresponding weights according to credibility;

[0020] Step S1043: Calculate the number of times the remaining SUs are selected as partners;

[0021] Step S1044: sort the remaining SUs by combining their weights and times, and select the best one or several as the current SUs. n partners;

[0022] Step S1045: The detection results are shared among partners and integrated into a global state.

[0023] Furthermore, the specific method for selecting partners is:

[0024] Define the reliability variable:

[0025]

[0026] The reliability in step S1041 is calculated according to formula (1), where n represents SU, k represents partner, t' represents the initial time, and t s represents the termination time, t represents the current time, ρ n,k (t) represents the current time under SU n Select SU k As the reliability of its partners, it is calculated by the cumulative discounted reward from the initial moment to the current moment, where γ is the decay rate, r n For SU n The reward obtained, G n is the set of partners, I(·) is the indicator function, if k∈G n ∪n, then I(·)=1, otherwise I(·)=0;

[0027]

[0028] The weight in step S1042 is calculated according to formula (2), ω n,k (t) represents the current SU n Select SU k As its partner fusion test results when SU k The weight of n,k (t) represents the partner reliability at the current moment, G n is a partner set, the weight is the current SU k The credibility and G n The ratio of the sum of the credibility of all SUs in, and the remaining SU numbers are represented by k';

[0029]

[0030] The number of cooperation times in step S1043 is calculated according to formula (3), and the number of cooperation times from the start time to the current time t is accumulated, G n is the set of partners, I(·) is the indicator function, if k∈G n , then I(·)=1, otherwise I(·)=0;

[0031]

[0032] According to formula (4), the remaining SUs are sorted and one or more SUs with the largest value are selected as partners. n,k is the partner weight, η n,k is the number of collaborations, and ε is a fixed constant.

[0033] Furthermore, in step S104, the SU uses an energy detection method to determine whether the channel is idle to obtain a corresponding detection result.

[0034] The second technical solution adopted by the present invention is a multi-agent cooperative spectrum sensing and access method for cognitive radio, which trains the MADDPG model with a tuple (S, A, R) consisting of state, action and reward, wherein the state S includes each SU n Fusion of its partners' perception results, the action A includes each SU n In the perception strategy of the current time frame, the reward R is calculated based on the n The reward received for the action taken.

[0035] Further, including:

[0036] The acquisition module can be used to acquire each SU in the N SUs nThe environmental parameters at the beginning of the current time frame, where n=1, 2, ..., N, include the environmental parameters of each SU n Self-observation n ;

[0037] The DDPG module can be used to convert each SU n The environment parameters at the beginning of the current time frame are input into the deterministic policy depth gradient MADDPG model, and each SU output by the MADDPG model is obtained n A sensing strategy in the current time frame, wherein the sensing strategy includes selecting which channel to detect;

[0038] A control module can be used to control each of the SU n In the current time frame, according to the spectrum sensing strategy, the spectrum sensing strategy is to select a certain channel for detection.

[0039] Further, including:

[0040] A memory for storing program codes, environmental parameters of wireless devices and perception strategies, wherein the environmental parameters include location information of the agent in the cognitive radio network CRN, observations, and number of incoming calls;

[0041] A deterministic policy deep gradient MADDPG model, used to determine the perception strategy according to the environmental parameters;

[0042] A processor, configured to call the program code and the MADDPG model to execute a multi-agent cooperative spectrum sensing and access method for cognitive radio according to any one of claims 1 to 6.

[0043] The beneficial effect of the present invention is that compared with a single-agent system, the mutual influence between agents in a multi-agent system cannot be ignored. In the embodiment of the present invention, each SU acts as an agent and selects an action based on its own observation in each time slot, that is, it senses a channel. Therefore, the collision between SUs, that is, the selection of the same channel for sensing, cannot be ignored. In addition, the SU does not know the preference of the PU for occupying the channel, so it needs to learn how to avoid collision with the PU in the process of continuous trial and error. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the system architecture of the CRN network in the present invention;

[0045] Figure 2 A schematic diagram of channel transfer probability according to an embodiment of the present invention;

[0046] Figure 3A schematic diagram of a flow chart of a multi-agent collaborative spectrum sensing and access method for cognitive radio according to the present invention;

[0047] Figure 4 A schematic diagram of the process flow of the partner selection algorithm in the present invention;

[0048] Figure 5 A schematic diagram of the structure of a multi-agent cooperative spectrum sensing and access device for cognitive radio according to the present invention;

[0049] Figure 6 Another structural schematic diagram of a multi-agent cooperative spectrum sensing and access device for cognitive radio of the present invention;

[0050] Figure 7 A comparison chart of the number of partners in an embodiment of the present invention;

[0051] Figure 8 4 is a comparison diagram of algorithms in an embodiment of the present invention.

[0052] Among them, 201. acquisition module, 202. DDPG module, 203. control module, 301. memory, 302. deterministic policy deep gradient MADDPG model, 303. processor. DETAILED DESCRIPTION

[0053] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] The present invention provides a multi-agent cooperative spectrum sensing and access method for cognitive radio. The method is based on a cognitive radio network. The cognitive radio network has M primary users PU, N secondary users SU, and N fb orthogonal channels, the transmit power of PU is fixed at P, and the channel noise is additive white Gaussian noise. Figure 1 As shown, the system of the CRN network includes primary users (PUs) and multiple secondary users SU1, SU2, ... and SU N , where N is an integer not less than 2. There are N fb Orthogonal channels, PUs can occupy the channel at any time, and SUs can use the channel randomly without interfering with the primary user. Each SU has the same communication distance, and those within the communication distance are called neighbors of the SU, and they can communicate with each other. In addition, the transmission power of PUs is fixed, and the channel noise is additive Gaussian white noise.

[0055] Figure 2 The figure shows the schematic diagram of channel transition probability. The channel state can be modeled by a two-state Markov chain. Each channel has two states: "0" indicates that the channel is occupied by PU; "1" indicates that the channel is idle; the transition probability P10 Represents the probability that the channel changes from idle to occupied. The mode of the primary user occupying the channel is fixed. It is assumed that the channel follows quasi-static block fading, that is, the channel state remains unchanged in each time slot. In fact, the secondary user does not know the state of the channel and the mode of the primary user occupying the channel. It needs to select the channel that is most likely to be idle at the current moment according to the belief vector to detect, and learn the occupancy mode of the primary user from its detection results and access results.

[0056] O(t)=[o1(t),o2(t)......o fb (t)] represents the channel state vector at time slot t, o f (t) = 1 means the channel is idle at that moment, o f (t) = 0 means that the channel is occupied by PU at that moment. SU does not know the value of O(t) at each moment, and each SU maintains its own belief vector s n (t) = [b n,1 ,b n,2 ,......b n,fb ], each component b n,f It is a continuous variable with values ​​between [0,1]. The larger the value, the more likely the current SU thinks the channel is idle. The value of each component is updated with the perception and access results of the SU and its partners.

[0057] Based on the current state n (t), the current SU performs action a n (t), and obtain the corresponding reward r n (t), and update the belief vector b based on the perception results n,f The flowchart for updating the belief vector is as follows:

[0058]

[0059]

[0060] For each channel, if the current SU selects the channel for detection, if the channel state is idle, the component is assigned a value of 1; otherwise, the component is assigned a value of 0. The state components of other channels change according to their values. If their values ​​are greater than 0.5, they are updated to max{0.5,αb n,f}; if its value is less than 0.5, it is updated to min{0.5,1-α(1-b n,f )}; where α is a fixed constant.

[0061] Since one SU can only detect one channel at a time, when there are too many channels, this detection method is obviously too inefficient. Therefore, fusing the detection results of its partners can obtain more channel status information. Due to the different positions of each SU and the time-varying nature of the channel, the perception ability of the SU sensing unit changes dynamically and is unknown to other SUs. Each SU prefers to choose a SU with strong detection ability as its partner. Therefore, we define a variable, denoted as reliability:

[0062]

[0063] If the SU k Those who have been partners of the current SU and helped it obtain higher rewards will be given priority as partners. I(·) is the indicator function. If k∈G n ∪n, then I(·)=1, otherwise I(·)=0. Before selecting a partner, the reliability of all remaining SUs should be calculated.

[0064] When updating the channel belief vector by integrating the access results of the partners, the weights of different partners are different. Different partner weights ω are set according to the reliability of historical rewards. n It is reasonable.

[0065]

[0066] The corresponding weight is calculated according to the reliability, as shown in formula (2). In addition, the channel belief vector is updated as follows:

[0067]

[0068] In the formula, the range of the summation subscript is the current SU itself and its partner set, ω n,k represents the corresponding weight of each partner, Indicates the test results of its partner.

[0069] In addition, to ensure that each SU has the opportunity to be selected as a partner, we will also calculate the number of times each SU has been selected as a partner up to the current moment η n,k :

[0070]

[0071] At the current time t, if the SU is selected as a partner, the value of I(·) is 1; otherwise, it is 0.

[0072] The criteria for each SU to select a partner are as follows: All SUs in its neighborhood are sorted and the one or several with the largest value are selected as partners:

[0073]

[0074] Among them, sortmax Ncoop () is a function that returns the partner number of the current SU in descending order. The independent variable of this function is the weight ω n,k and the number of times as a partner n,k Obviously, users with high historical reliability or fewer cooperation times are likely to be selected as new partners. ε is a decay coefficient.

[0075] Specifically, the algorithm flow chart shown in the following table may be used to describe the MADDPG-based collaborative spectrum sensing in detail.

[0076]

[0077]

[0078] like Figure 3 As shown, the present invention provides a multi-agent collaborative spectrum sensing and access method for cognitive radio, comprising:

[0079] Step S101: Obtain each SU in the N SUs n The environmental parameters at the beginning of the current time frame, where n=1, 2, ..., N, include the environmental parameters of each SU n Observation S n ;

[0080] This observation is a local observation, which is the probability that each agent believes that each channel may be idle at that moment. The agent makes a strategy based on its own observation and selects a channel for perception.

[0081] Step S102: Each SU n The environment parameters at the beginning of the current time frame are input into a deterministic policy deep gradient MADDPG model;

[0082] In some embodiments, the method further comprises: training the MADDPG model using a tuple (S, A, R, S') consisting of a state, an action, and a reward.

[0083] Among them, state S is the global state, which is the global state formed by the current SU fusing its own results with the perception results of its partners. S' is the next state. Action A is a joint action, that is, a global action formed by splicing the actions of SU and its partners together. Reward is its own reward. According to the access result, if the access is successful, a positive reward will be obtained, otherwise, a negative reward will be obtained.

[0084] Step S103: Obtain each SU output by the MADDPG modeln A sensing strategy in the current time frame, wherein the sensing strategy is to select a certain channel to detect whether it is idle;

[0085] Step S104: The perception results of SUn and its partners are fused into a global state S(t), and the global state S(t) is sent to the experience replay buffer of the deterministic policy deep gradient MADDPG model together with the global action A(t), the reward r(t) and the next state S(t+1). The global state S(t) is used as the observation at the next moment, and the cycle repeats until the training step is completed.

[0086] In some embodiments, step S103 also includes: at the beginning of each time frame, all SUs simultaneously select their own strategies according to their own observations, that is, select a channel to detect whether the channel is idle. Each SU has only one chance until all SUs find an idle channel or fail, and then a time frame ends.

[0087] In some embodiments, Figure 4 As shown, step S104 is specifically as follows:

[0088] Step S1041: Current SU n The credibility is calculated based on the rewards obtained by other SUs after performing actions at historical moments. If an SU has served as the current SU n If you are a good partner, you will have a higher level of credibility;

[0089] Step S1042: Calculate corresponding weights according to credibility;

[0090] Step S1043: Calculate the number of times the remaining SUs are selected as partners;

[0091] Step S1044: sort the remaining SUs by combining their weights and times, and select the best one or several as the current SUs. n partners;

[0092] Step S1045: The detection results are shared among partners and integrated into a global state.

[0093] In some embodiments, the specific method of selecting a partner is:

[0094] Define the reliability variable:

[0095]

[0096] The reliability in step S1041 is calculated according to formula (1), where n represents SU, k represents partner, t' represents the initial time, and t srepresents the termination time, t represents the current time, ρ n,k (t) represents the current time under SU n Select SU k As the reliability of its partners, it is calculated by the cumulative discounted reward from the initial moment to the current moment, where γ is the decay rate, r n For SU n The reward obtained, G n is the set of partners, I(·) is the indicator function, if k∈G n ∪n, then I(·)=1, otherwise I(·)=0;

[0097]

[0098] The weight in step S1042 is calculated according to formula (2), ω n,k (t) represents the current SU n Select SU k As its partner fusion test results when SU k The weight of n,k (t) represents the partner reliability at the current moment, G n is a partner set, the weight is the current SU k The credibility and G n The ratio of the sum of the credibility of all SUs in, and the remaining SU numbers are represented by k';

[0099]

[0100] The number of cooperation times in step S1043 is calculated according to formula (3), and the number of cooperation times from the start time to the current time t is accumulated, G n is the set of partners, I(·) is the indicator function, if k∈G n , then I(·)=1, otherwise I(·)=0;

[0101]

[0102] According to formula (4), the remaining SUs are sorted and one or more SUs with the largest value are selected as partners. n,k is the partner weight, η n,k is the number of collaborations, and ε is a fixed constant.

[0103] In some embodiments, in step S104, the SU uses an energy detection method to determine whether the channel is idle to obtain a corresponding detection result. Energy detection is a simple detection method. The SU needs to detect the energy value of the PU on the current channel and compare it with the corresponding threshold to determine whether the channel is occupied by the PU at that moment.

[0104] In some embodiments, the MADDPG model is trained with a tuple (S, A, R) consisting of a state, an action, and a reward, wherein the state S includes each SU n Fusion of its partners' perception results, the action A includes each SU n In the perception strategy of the current time frame, the reward R is calculated based on the n The reward received for the action taken.

[0105] The present invention also provides a multi-agent cooperative spectrum sensing and access device for cognitive radio, such as Figure 5 As shown, the spectrum sensing device 200 may include: an acquisition module 201, a MADDPG module 202 and a control module 203, wherein:

[0106] The acquisition module 201 may be used to acquire each SU in the N SUs. n The environmental parameters at the beginning of the current time frame, where n=1, 2, ..., N, include the environmental parameters of each SU n Self-observation n ;

[0107] The DDPG module 202 may be used to n The environment parameters at the beginning of the current time frame are input into the deterministic policy depth gradient MADDPG model, and each SU output by the MADDPG model is obtained n a sensing strategy in the current time frame, wherein the sensing strategy includes selecting which channel to detect; and

[0108] The control module 203 can be used to control each SU n In the current time frame, according to the spectrum sensing strategy, the spectrum sensing strategy is to select a certain channel for detection;

[0109] The spectrum sensing device is used to implement the aforementioned method embodiment, and its implementation principle and technical effect are similar and will not be described in detail here.

[0110] The present invention also provides another multi-agent cooperative spectrum sensing and access device for cognitive radio, such as Figure 6 As shown, the spectrum sensing device 300 may include: a memory 301, a MADDPG model 302, and a processor 303, which may be connected via a bus (such as Figure 6 The thick solid lines show the interconnection.

[0111] The memory 301 may be used to store program instructions, environmental parameters of the SU and spectrum sensing strategies. The environmental parameters include the environmental parameters including each SU n Self-observation n It should be noted that the memory 301 may also store other data or information for executing tasks, and the present invention does not limit this.

[0112] The MADDPG model 302 can be used to determine the spectrum sensing strategy according to the environmental parameters. The implementation of the MADDPG model can refer to the relevant description above, which will not be repeated here.

[0113] The processor 303 may be used to call the program code and the MADDPG model to execute the task method as described in the embodiment of the present invention.

[0114] The task offloading device in the embodiment of the present invention is used to implement the aforementioned method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0115] Figure 6 Only a schematic diagram of the structure of the task offloading device 300 disclosed in this article is shown. The task offloading device 300 can use all the components shown or only a subset of the components, and the structure may change for different situations. In addition, the task offloading device may include multiple instances of components, such as multiple processors, memories, transmitters, receivers, network interfaces, etc. The task offloading device may also include a processing unit equipped with one or more input / output devices such as a mouse, a touch screen, a keypad, a keyboard, a printer, a display, etc. The processing unit may include a central processing unit (CPU), a graphics processing unit (GPU), a memory, a mass storage device, a video adapter, an I / O interface, and a network interface connected to a bus.

[0116] The bus may be one or more of any type of multiple bus structures, including a memory bus or memory controller, a peripheral bus, a video bus, etc. The CPU may include any type of electronic data processor. The memory may include any type of system memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), and combinations thereof. In some embodiments, the memory may include a ROM for use at startup, and a DRAM for program and data storage used when executing a program.

[0117] The mass storage device may include any type of storage device for storing data, programs, and other information and making such data, programs, and other information accessible via a bus. The mass storage device may include, for example, one or more of a solid-state drive, a hard disk drive, a magnetic disk drive, an optical disk drive, or the like.

[0118] Video adapter and I / O interface provide the interface that external input and output equipment is coupled with processing unit.As shown in the figure, the example of input and output equipment comprises the display coupled with video adapter and the mouse / keyboard / printer coupled with I / O interface.Other equipment can be coupled with processing unit, and can use extra or less interface card.For example, can use serial interface such as universal serial bus (USB) (not shown) to provide interface for printer.

[0119] The processing unit also includes one or more network interfaces, which may include wired links, such as Ethernet cables, etc., and / or wireless links to access nodes or different networks. The network interface allows the processing unit to communicate with the remote unit via a network. For example, the network interface can provide wireless communication via one or more transmitters / transmitting antennas and one or more receivers / receiving antennas, wherein the transmitter and receiver can be integrated. In one embodiment, the processing unit is coupled to a local area network or a wide area network for data processing and communication with remote devices, such as other processing units, the Internet, remote storage devices, etc.

[0120] Example

[0121] The specific implementation process of the above method embodiment is described below: First, at the beginning of a time slot, all SUs take actions according to their own decisions at the same time. Each SU has one and only one chance. There are three results: an idle channel is detected and access is successful, an idle channel is not detected, and a collision occurs with the PU, and a collision occurs with other SUs.

[0122] After SU selects an action based on its own decision, it updates its own observations based on the results and obtains corresponding rewards. It calculates corresponding indicators based on the rewards, such as weight, credibility, etc. At the same time, it counts the number of times each SU is selected as a partner in the current SU's partner set, and selects partners based on several indicators.

[0123] After determining the partner, the SU fuses its own and its partner's perception results to form a global observation, and all SU's actions are spliced ​​together to form a global action, which is then sent to the experience replay buffer to train the network. Learning and training are carried out in iterations.

[0124] The states, actions, and rewards in the embodiment of the present invention are as follows:

[0125] State: Each SU maintains a state of 1*N fb The value of each component is a continuous variable between (0,1). The larger the value, the more likely the SU thinks the channel is idle at that moment.

[0126] Action: The action is discrete, and a channel is selected for detection at each moment.

[0127] Reward: The reward varies depending on the access result: if the access is successful, you will get a positive reward; if the access is unsuccessful, you will get a negative reward.

[0128] MADDPG has four networks:

[0129] Actor current network: responsible for iterative update of policy network parameters μ, selects current action A according to current state S to interact with the environment to generate next state S′ and reward R; in the embodiment of the present invention, it is reflected as follows: the current agent SU selects a perception strategy according to its own observation, that is, selects which channel to perceive, and obtains corresponding rewards according to the perception results.

[0130] Actor target network: responsible for selecting the next action A' from the next state S' sampled from the experience replay buffer. The network parameters μ' of this network are regularly updated by copying μ from the actor's current network; the experience replay buffer stores the agent's past experiences.

[0131] The critic current network is responsible for iteratively updating the value network parameter θ and calculating the current Q value. That is: y i =R+γQ′(S′,A′,θ′);

[0132] Critics target network: responsible for calculating the Q'(S', A', θ') part of the target Q value. The network parameters θ' of this network are regularly updated by copying θ from the critic current network.

[0133] MADDPG adopts a "soft" update method that updates only a little bit at a time, namely:

[0134] μ′ k+1 =τμ′ k +(1-τ)μ′ k (5)

[0135] θ′ k+1 =τθ k +(1-τ)θ′ k (6)

[0136] Where τ is the update coefficient. This update method can greatly improve the stability of learning.

[0137] The actor's current network uses a deterministic strategy to generate deterministic actions, and the loss gradient is:

[0138]

[0139] The loss function of the critic's current network uses mean square error:

[0140] J(θ)=E[(y k -Q(S,A|θ)) 2 ] (8),

[0141] The example of the present invention adopts the MADDPG method to learn the channel state. Each intelligent agent has an actor and a critic. The actor is based on the policy network π n The local observation state s n Mapped to an appropriate action a n , critic according to its value network Q n To evaluate the quality of the strategy. Both the actor and the critic have an online network and a target network to ensure the stability of learning and overcome over-optimism.

[0142] The actor network is trained using its own local observations and actions, while the critic network requires global observations and actions.

[0143] During the training process, Q n Output strategy π based on the actions and states of other agents n During execution, the well-trained π n It can independently select the optimal action based on its own state without considering other agents, resulting in less synchronization and communication overhead.

[0144] MADDPG further adds noise N to explore better strategies during training. Another basic technique is the experience replay buffer (RB). Each agent is equipped with a RB to store (S(t), A(t), r(t), S(t+1)), which will be randomly extracted to update the weights. In addition, experience replay can effectively avoid highly correlated actions in consecutive updates.

[0145] MADDPG inherits the DDPG method into the multi-agent domain. It not only eliminates the non-stationary characteristics of Q-Learning and policy gradients, but also retains the great advantage of DDPG, which can search the action space in a continuous rather than discrete manner.

[0146] In summary, the implementation of the embodiment of the present invention is that there are 3 SUs, 2 PUs, and 7 channels in the cognitive radio network. The channel state changes according to the transition probability, and the SU needs to opportunistically select a channel for transmission when the PU does not occupy the channel resources. The following is an analysis of the results of this embodiment:

[0147] Since there are 3 SUs in the CRN network, the maximum number of partners that each SU can choose is 2. Figure 7 The figure shows the change of access success rate with the change of the number of partners. The curves from top to bottom in the figure are 2-partner, 1-partner and 0-partner.

[0148] As the training step size and the number of partners increase, the access success probability is significantly improved. As MADDPG runs, the success probability converges from less than 50% at the beginning to about 90%, proving the effectiveness of the algorithm. At the same time, compared with no partner and 1 partner, the success probability of 2 partners increases by at least 5% and 3%, respectively, proving the effectiveness of the collaborative algorithm. As the number of SUs increases, the importance of the partner algorithm will become more significant.

[0149] In order to show the advantages of the MADDPG-based collaborative sensing algorithm, we further compared the one-shot success probability with traditional random access methods and DRL methods, such as DQN and Q-Learning. Figure 8 shown. Figure 8 The curves in the figure are MADDPG 2-partner, MADDPG 1-partner, DQN MLP2 2-partner, DQNMLP2 1-partner, DQN MLP1 2-partner, DQN MLP1 1-partner, Q-Learning, and Randomaccess from top to bottom. DQN uses a multi-layer perceptron (MLP) instead of a Q-table to map states to actions. DQN-MLP1 has one layer of neural network with 300 neurons, and DQN-MP2 has two hidden layers with 300 neurons each. Each algorithm is executed twice with 1 partner and 2 partners respectively. Compared with random access, which has the lowest success probability, channel belief fusion in the cooperative sensing algorithm is effective for both MADDPG and DQN. With the help of 2 partners, the success probability of MADDPG, DQN-MLP2, and DQN-MLP1 increases by nearly 1%, 2%, and 5%, respectively. MADDPG with 2 partners performs best.

[0150] However, Q-learning does not perform well in this scenario (it actually works well when there are only two SUs). Due to the lack of observations from other SUs, the state transitions of each individual are time-varying and unreliable, which makes the environment non-stationary. As the number of agents increases, the rapid growth of the Q-table makes it difficult for training to traverse all possible environmental states, and the users cannot directly use past experience to reply. Therefore, Q-Learning is not suitable for large-scale models.

[0151] In the embodiment of the present invention, there is no control center to uniformly schedule channel resources, so each agent must take actions according to its own decisions, which inevitably leads to collisions while reducing communication overhead. Due to the characteristics of centralized training and distributed execution of MADDPG, it is more suitable for multi-agent environments. Compared with traditional reinforcement learning methods such as DQN and Q-Learning, it has greater superiority and effectiveness in terms of one-time access success rate.

[0152] Secondly, in the embodiment of the present invention, cooperative spectrum sensing is added, and as the number of partners increases, the collision probability and access success rate performance are significantly improved.

[0153] Since the detection accuracy is affected by shadows and multipath fading, a single user unit cannot detect multiple channels at the same time. In order to enable each user unit to globally perceive the channel, the present invention proposes a partner selection and channel state fusion algorithm to improve perception efficiency and accuracy. Improve the accuracy of finding idle channels in one go, and reduce communication delays and perception overhead. The multi-agent collaborative spectrum sensing and access method for cognitive radio of the present invention performs better in a multi-agent system, that is, when the number of secondary users increases, the traditional machine learning methods, such as Q-Learning and DQN learning performance will decline. Due to the characteristics of centralized training and distributed execution of MADDPG, the probability of a trained SU finding an idle channel in one go can reach 90%, effectively reducing the probability of collision with PU and other SUs, while reducing communication delays and perception overhead.

[0154] The embodiment of the present invention provides a collaborative spectrum sensing method based on multi-agent reinforcement learning in a cognitive radio network, by inputting the environmental parameters of each agent at the beginning of the current time frame into a deterministic strategy deep gradient MADDPG model, wherein the environmental parameters include the observations of each agent at the beginning of the current time frame, so as to obtain the perception strategy of each agent in the current time frame output by the MADDPG model, wherein the perception strategy is to select which channel to detect whether the channel is idle at the current moment, and can access the channel to complete the transmission task, and each agent senses and accesses according to the determined strategy in each time frame. In addition, collaborative spectrum sensing is added on the basis of the original MADDPG algorithm, so that multiple users can share the spectrum sensing results, effectively improving the efficiency of spectrum sensing. When the number of users is large, spectrum holes can be quickly searched, idle channels can be sensed, and collisions between users can be avoided while successfully accessing, thereby reducing task delays and communication overheads while improving perception efficiency and accuracy.

Claims

1. A multi-agent collaborative spectrum sensing and access method for cognitive radio, characterized in that: Based on a cognitive radio network, there are M primary users PU, N secondary users SU, N fb orthogonal channels, the transmit power of PU is fixed at P, and the channel noise is additive white Gaussian noise; The sensing and access method comprises: Step S101: Obtain each SU in the N SUs n The environmental parameters at the beginning of the current time frame, where n=1, 2, ..., N, include the environmental parameters of each SU n Observation S n ; Step S102: Each SU n The environment parameters at the beginning of the current time frame are input into a deterministic policy deep gradient MADDPG model; Step S103: Obtain each SU output by the MADDPG model n A sensing strategy in the current time frame, wherein the sensing strategy is to select a certain channel to detect whether it is idle; Step S104: The perception results of SUn and its partners are integrated into the global state S(t), and the global state S(t), the global action A(t), the reward r(t) and the next state S(t+1) are sent to the experience replay buffer of the deterministic policy deep gradient MADDPG model; The global state S(t) is used as the observation at the next moment, and the cycle repeats until the training step is completed; In step S104, the method of fusing the perception results of SUn and its partners into the global state S(t) is: Step S1041: Current SU n The credibility is calculated based on the rewards obtained by other SUs after performing actions at historical moments. If an SU has served as the current SU n If you are a good partner, you will have a higher level of credibility; Step S1042: Calculate corresponding weights according to credibility; Step S1043: Calculate the number of times the remaining SUs are selected as partners; Step S1044: sort the remaining SUs by combining their weights and times, and select the best one or several as the current SUs. n partners; Step S1045: the partners share the test results and integrate the test results into a global state; The specific methods for selecting partners are as follows: Define the credibility variable: The credibility in step S1041 is calculated according to formula (1), where n represents SU, k represents partner, t' represents the initial time, and t s represents the termination time, t represents the current time, ρ n,k (t) represents the current time under SU n Select SU k As the credibility of its partners, it is calculated by the cumulative discounted reward from the initial moment to the current moment, where γ is the decay rate, r n For SU n The reward obtained, G n is the set of partners, I(·) is the indicator function, if k∈G n ∪n, then I(·)=1, otherwise I(·)=0; The weight in step S1042 is calculated according to formula (2), ω n,k (t) represents the current SU n Select SU k As its partner fusion test results when SU k The weight of n,k (t) represents the partner’s credibility at the current moment, G n is a partner set, the weight is the current SU k The credibility and G n The ratio of the sum of the credibility of all SUs in, and the remaining SU numbers are represented by k'; The number of cooperation times in step S1043 is calculated according to formula (3), and the number of cooperation times from the start time to the current time t is accumulated, G n is the set of partners, I(·) is the indicator function, if k∈G n , then I(·)=1, otherwise I(·)=0; According to formula (4), the remaining SUs are sorted and one or more SUs with the largest value are selected as partners. n,k is the partner weight, η n,k is the number of collaborations, and ε is a fixed constant.

2. A multi-agent collaborative spectrum sensing and access method for cognitive radio as claimed in claim 1, characterized in that: The step S103 also includes: at the beginning of each time frame, all SUs simultaneously select their own strategies according to their own observations, that is, select a channel to detect whether the channel is idle. Each SU has only one chance until all SUs find an idle channel or fail, and then a time frame ends.

3. A multi-agent collaborative spectrum sensing and access method for cognitive radio as claimed in claim 1 or 2, characterized in that: In step S104, the SU uses an energy detection method to determine whether the channel is idle to obtain a corresponding detection result.

4. A multi-agent collaborative spectrum sensing and access method for cognitive radio as claimed in claim 1 or 2, characterized in that: The MADDPG model is trained using a tuple (S, A, R) consisting of state, action, and reward, where the state S includes each SU n Fusion of its partners' perception results, the action A includes each SU n In the perception strategy of the current time frame, the reward R is calculated based on the n The reward received for the action taken.

5. A multi-agent cooperative spectrum sensing and access device for cognitive radio, characterized in that: The method for multi-agent collaborative spectrum sensing and access for cognitive radio according to any one of claims 1 to 4 comprises: The acquisition module (201) is used to acquire each SU in the N SUs. n The environmental parameters at the beginning of the current time frame, where n=1, 2, ..., N, include the environmental parameters of each SU n Self-observation n ; DDPG module (202), used to n The environment parameters at the beginning of the current time frame are input into the deterministic policy depth gradient MADDPG model, and each SU output by the MADDPG model is obtained n A sensing strategy in the current time frame, wherein the sensing strategy includes selecting which channel to detect; A control module (203) is used to control each of the SU n The spectrum sensing strategy is executed in the current time frame, and the spectrum sensing strategy is to select a certain channel for detection.

6. A multi-agent cooperative spectrum sensing and access device for cognitive radio, characterized in that: include: A memory (301) for storing program codes, environmental parameters of wireless devices and perception strategies, wherein the environmental parameters include location information of the agent in a cognitive radio network CRN, observations, and number of incoming calls; A deterministic policy deep gradient MADDPG model (302), used to determine the perception strategy according to the environmental parameters; A processor (303) is used to call the program code and the MADDPG model to execute a multi-agent collaborative spectrum sensing and access method for cognitive radio as described in any one of claims 1 to 4.

Images