Method for controlling optimal energy detection threshold value based on multi-agent reinforcement learning in distributed cognitive radio network and apparatus therefor

Abstract

A method for controlling an optimal energy detection threshold value based on multi-agent reinforcement learning in a distributed cognitive radio network comprises the steps of: (a) constructing a state space for a network environment including a plurality of primary terminals and a plurality of secondary terminals, wherein the state space includes each state regarding whether the primary terminals are occupied; (b) selecting, by an agent, an action according to a policy by applying a partial observation of each secondary terminal to a reinforcement learning-based actor-critic network model, and on the basis of the selected action in the environment, calculating a reward on the basis of a detection result for the primary terminals, wherein the action is a detection threshold value; (c) storing, as experience, the partial observation, the selected action, the reward, and a subsequent observation in a playback buffer; and (d) updating the actor-critic network model on the basis of the experience stored in the playback buffer.

Description

Method and device for optimal energy detection threshold control based on multi-agent reinforcement learning in distributed cognitive radio networks

[0001] The present invention relates to a method and device for controlling an optimal energy detection threshold based on multi-agent reinforcement learning in a distributed cognitive radio network.

[0002]

[0003] As the number of network devices increases, the demand for additional radio frequency spectrum bands is increasing, and the need for cognitive radio networks (CRN) technology is emerging to address the shortage of radio resources.

[0004] Through CRN, secondary users (SUs, hereinafter referred to as SUs) can opportunistically access spectrum bands licensed from primary users (PUs, hereinafter referred to as PUs).

[0005] Existing CRN methods require devices to accurately detect and utilize empty spectrum bands while avoiding interference, a challenge presented by dynamic and uncertain wireless environments that include factors such as multipath fading, shadowing, and receiver uncertainty.

[0006] Cooperative spectrum sensing (CSS) consists of two systems: centralized and distributed. Centralized CSS approaches face operational costs and potential bottleneck issues associated with FCs.

[0007]

[0008] The present invention provides a method and device for controlling an optimal energy detection threshold based on multi-agent reinforcement learning in a distributed cognitive radio network.

[0009] In addition, the present invention provides a method and device for controlling an optimal energy detection threshold based on multi-agent reinforcement learning in a distributed cognitive radio network capable of determining an optimal detection threshold that can maximize the detection probability of a primary terminal and minimize the false alarm probability.

[0010]

[0011] According to one aspect of the present invention, a method for controlling optimal energy detection threshold based on multi-agent reinforcement learning in a distributed cognitive radio network is provided.

[0012] According to one embodiment of the present invention, a method for controlling optimal energy detection thresholds based on multi-agent reinforcement learning in a distributed cognitive radio network may be provided, including: (a) constructing a state space for a network environment including a plurality of primary terminals and a plurality of secondary terminals, wherein the state space includes each state of whether the primary terminals are occupied; (b) applying partial observations of each secondary terminal to a reinforcement learning-based actor-critic network model to select an action according to a policy, and calculating a reward based on a detection result of the primary terminal based on the selected action in the environment, wherein the action is a detection threshold; (c) storing the partial observations, the selected action, the reward, and the next observation in a replay buffer as experience; and (d) updating the actor-critic network model based on experience stored in the replay buffer.

[0013] In the training phase, the replay buffer also stores the experiences of other agents, and in the step (d), each agent centrally learns the actor-critic network model of each agent by sharing the experiences of other agents, and in the execution phase, in the step (d), each agent can update the actor-critic network model of each agent using only the local experiences of each agent.

[0014] The above policy determines a detection threshold that maximizes the probability of correctly detecting the primary terminal and minimizes the cumulative false alarm probability up to time step t, and the above policy is formulated by the following mathematical formula:

[0015]

[0016] Here, is a specific channel at time step t selected by the ith secondary terminal. and sector Indicates the detection threshold for , represents the orthogonal channel owned by the primary terminal, represents the sector of the ith secondary terminal, K represents the number of orthogonal channels, and L represents the number of sectors. represents the cumulative detection probability of the primary terminal up to time step t, represents the cumulative false alarm probability up to time step t, and M represents the index of the secondary terminal.

[0017] The above compensation imposes zero (0) if the result of detecting the primary terminal based on the selected action in the environment is the same as the actual state, and imposes a penalty if the result of detecting the primary terminal based on the selected action in the environment is different from the actual state, but the actual state is either channel occupied or not occupied by the primary terminal.

[0018]

[0019] According to one embodiment of the present invention, a device and system capable of controlling an optimal energy detection threshold based on multi-agent reinforcement learning in a distributed cognitive radio network are provided.

[0020] According to one embodiment of the present invention, a computing device may be provided, comprising: a memory storing at least one instruction; and a processor executing the instruction stored in the memory, wherein the instruction executed by the processor performs the steps of: (a) constructing a state space for a network environment including a plurality of primary terminals and a plurality of secondary terminals, wherein the state space includes each state of whether the primary terminals are occupied; (b) executing each agent for each secondary terminal, each agent applying a partial observation of each secondary terminal to a reinforcement learning-based actor-critic network model to select an action according to a policy, and calculating a reward based on a detection result of the primary terminal based on the selected action in the environment, wherein the action is a detection threshold; (c) storing the partial observation, the selected action, the reward, and the next observation of each agent in a replay buffer as an experience; and (d) updating the actor-critic network model of each agent based on the experience stored in the replay buffer.

[0021]

[0022] According to another embodiment of the present invention, a system can be provided, comprising: a plurality of primary terminals; and a plurality of secondary terminals, wherein each of the plurality of secondary terminals comprises a step of constructing a state space for a network environment including the plurality of primary terminals and the plurality of secondary terminals, wherein the state space includes each state of whether the primary terminals are occupied; an agent applying a partial observation of each secondary terminal to a reinforcement learning-based actor-critic network model to select an action according to a policy, and calculating a reward based on a detection result of the primary terminal based on the selected action in the environment, wherein the action is a detection threshold; a step of storing the partial observation, the selected action, the reward, and the next observation in a replay buffer as an experience; and a step of updating the actor-critic network model based on the experience stored in the replay buffer.

[0023]

[0024] By providing a method and device for controlling an optimal energy detection threshold based on multi-agent reinforcement learning in a distributed cognitive radio network according to one embodiment of the present invention, an optimal detection threshold that can maximize the detection probability of a primary terminal and minimize the false alarm probability can be determined.

[0025]

[0026] FIG. 1 is a schematic diagram illustrating a distributed cognitive radio network system according to one embodiment of the present invention.

[0027] FIG. 2 is a flowchart illustrating a method for controlling an optimal energy detection threshold based on multi-agent reinforcement learning in a distributed cognitive radio network according to one embodiment of the present invention.

[0028] Figure 3 is a diagram illustrating pseudo code for Figure 2.

[0029] Figure 4 is a diagram illustrating a system simulation environment according to one embodiment of the present invention.

[0030] FIG. 5 is a diagram illustrating simulation parameters according to one embodiment of the present invention.

[0031] FIG. 6 is a diagram illustrating convergence results at different detection times according to one embodiment of the present invention.

[0032] Fig. 7 is a diagram comparing the convergence results of an energy detection threshold control method according to a conventional method and an embodiment of the present invention.

[0033] Figure 8 is a diagram showing the results of comparing the detection probability and false alarm probability according to a conventional method and an embodiment of the present invention.

[0034] Figure 9 is a block diagram schematically illustrating the internal configuration of a computing device according to one embodiment of the present invention.

[0035]

[0036] As used herein, singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "consist of" or "include" should not be construed to necessarily include all components or steps described in the specification, and should be construed to mean that some of the components or steps may not be included, or that additional components or steps may be included. In addition, terms such as "part" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software, or by a combination of hardware and software.

[0037] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0038]

[0039] FIG. 1 is a schematic diagram illustrating a distributed cognitive radio network system according to one embodiment of the present invention.

[0040] As illustrated in FIG. 1, it is assumed that a distributed cognitive radio network system according to one embodiment of the present invention has a primary network and an auxiliary ad hoc network.

[0041] The primary network may be a network between a base station (base station PBS: primary base station) and multiple PUs. Here, it is assumed that there are U PUs.

[0042] The auxiliary network may be an ad hoc network formed by M SUs.

[0043] According to one embodiment of the present invention, it is assumed that PU and SU are static within the network.

[0044] Traditional centralized cognitive networks require a coordinate node that integrates information from other nodes to make decisions. However, one embodiment of the present invention assumes a distributed environment where SUs can operate equally after sufficient time to collaborate and learn about the environment.

[0045] Let us assume that the PU is equipped with an omnidirectional antenna, as illustrated in Fig. 1. In addition, let us assume that the PU periodically broadcasts a pilot signal, as in DVB-T (Digital Video Broadcasting-Terrestrial) of IEEE 802.22, a standard for a wireless regional area network (WRAN) that uses the white space band, which is a TV frequency band.

[0046] In addition, we assume that there are a total of K orthogonal channels owned by the PU. That is, the PU has the highest priority in using the orthogonal channel, and the SU, as an unlicensed user, must wait until the PU releases the channel.

[0047] SUs, nodes that do not have permission to use the corresponding spectrum for wireless resources, must find and use spectrum not used by PUs. SUs that do not have priority for spectrum use must yield spectrum use to PUs even while transmitting data if the PUs wish to use the spectrum. Therefore, SUs must periodically perform spectrum sensing.

[0048] Each SU is equipped with a directional antenna, with L sectors, ideally non-overlapping. The SU can use the directional antenna to detect free channels and transmit data. Furthermore, with the help of the directional antenna, the SU can use the same channel as the PU without causing interference to the underlying network.

[0049] On the other hand, the PU is equipped with a traditional omnidirectional antenna for communication purposes. In one embodiment of the present invention, it is assumed that the network model of the system is Omn-Dir-CRN.

[0050] In one embodiment of the present invention, it is assumed that all SUs use an energy detection (ED)-based spectrum sensing method to detect the appearance of a PU and determine whether a specific channel is occupied by the PU.

[0051] ED has no requirement for historical information, so it is an inconsistent and very widely used detection method, often combined with common binary hypothesis testing.

[0052] and is the i-th SU( ) is a channel ( )) and sector( ) when detecting, It relatively indicates the presence and absence of PU under the observation of .

[0053] The i-th SU( ) can be expressed as in mathematical expression 1.

[0054]

[0055] Here, represents the channel gain, s(n) represents the signal of PU, and u(n) represents additive white Gaussian noise (AWGN) with mean zero.

[0056] The detection process passes the signal through an ideal band-pass filter to limit the noise bandwidth. can start with. The output is squared and integrated over the observation time interval. The final test statistics for can be given as in Equation 2.

[0057]

[0058] Here, N represents the number of received samples. The test statistic is the detection threshold to determine the presence or absence of PU. can be compared to.

[0059] If this is expressed as a mathematical formula, it can be expressed as in mathematical formula 3.

[0060]

[0061] The occupancy state of a PU for a channel is composed of a Markov chain model with two states: busy(1) and idle(0). Here, busy(1) indicates an occupied state, and idle(0) indicates an unoccupied, or idle, state. and Let us assume that is the probability of transitioning from busy state to busy state and from busy state to idle state. In addition, let us assume that is the probability of transitioning from idle state to idle state and from idle state to busy state. and Let's assume that.

[0062] The occupied state transition probability is the same for all channels and can be expressed as in Equation 4.

[0063]

[0064] SU's time frame structure does not require time for control compared to other centralized systems because each SU operates individually.

[0065] Each active period of SU includes two main parts: detection and transmission, T and represents the length of the active period and the detection period, respectively. These parameters can be kept constant across all SUs in the system.

[0066] The detection probability of PU according to the ED method can be expressed as in Equation 5. Here, the detection probability of PU is and Regardless of It means the probability of detecting PU.

[0067]

[0068] Here, represents the generalized Marcum Q function.

[0069] the other side, The false alarm probability for PU can be calculated as in Equation 6.

[0070]

[0071] Here, , represent the gamma function and the incomplete gamma function, respectively.

[0072] Mathematical expressions 5 and 6 are the detection thresholds It is widely used assuming that it will remain the same for a certain period of time.

[0073] However, in one embodiment of the present invention, the detection threshold Let us assume that this is a time-varying variable, i.e. the detection threshold. is probabilistic and can change over time. In such cases, The detection probability up to the point can be expressed as in mathematical expression 7.

[0074]

[0075] Here, represents an indicator that an event has occurred. Also, The false alarm probability up to the point in time can be calculated as in Equation 8.

[0076]

[0077] The present invention aims to determine the optimal energy detection threshold for a channel-sector pair at each time step for all SUs. In other words, by finding optimal parameters, the present invention can maximize the probability of correctly detecting the presence of a PU while minimizing the probability of false alarms, two key performance factors in detection methods.

[0078] Therefore, the problem at a specific time step t can be formulated as in Equation 9.

[0079]

[0080] Here, Is A specific channel at a time interval t selected by and sector It represents the detection threshold for . The problem in Equation 9 is a probabilistic sequential optimization problem, which is NP-hard, so we will solve it by reconstructing it in the form of a Markov decision process (MDP). This will be explained in more detail below.

[0081]

[0082] Figure 2 is a flowchart illustrating a method for optimal energy detection threshold control based on multi-agent reinforcement learning in a distributed cognitive radio network according to one embodiment of the present invention. It is assumed that each step performed below is performed by a computing device. Here, the computing device may be a server, an individual SU, or some device included in an individual SU.

[0083] In step 210, the computing device (200) configures a state space for a network environment including a plurality of primary terminals and a plurality of secondary terminals.

[0084] Here, the environment may be a distributed cognitive radio network system environment described in FIG. 1.

[0085] This is a schematic diagram of a distributed cognitive radio network system.

[0086] As illustrated in FIG. 1, it is assumed that a distributed cognitive radio network system according to one embodiment of the present invention has a primary network and an auxiliary ad hoc network.

[0087] Since each SU has no prior knowledge about the environment and the sensing information of other SUs, all decisions made by the SU must be based on its local knowledge. Therefore, the problem is can be transformed into a Dec-POMDP which can be defined using tuples of . Here, represents the set of all agents, S is the set representing the actual state of the environment, represents the partial observation space acknowledged by agent i, represents the action space of the i-th agent. In addition, represents the set of actions of all agents, represents the reward that agent i receives from the environment when taking action a from state S to a new state S'. In addition, represents the probability of transition to a new state given known states and actions, represents an observation channel that maps the actual state of the environment to the agent's observations, represents the discount rate. The goal of all agents is to maximize the expected long-term discounted reward by implementing the optimal policy. is to find.

[0088] The four main components of Dec-POMDP are the actual state of the environment, partial observation by SU, action space of SU, and reward space.

[0089] According to one embodiment of the present invention, the state space may be composed of states regarding whether each PU occupies a channel. For example, at each time step t, the state is can be defined as, where, Indicates the status of whether the PU occupies the channel. is a matrix of size K x L, There is at least one PU that uses the kth channel, Indicates that the lth sector is located in the area covered by the , otherwise it can be represented as 0.

[0090] In step 215, the computing device (200) executes each agent for each secondary terminal, and each agent applies partial observations of each secondary terminal to a reinforcement learning-based actor-critic network model to select an action according to a policy, and obtains a reward based on the detection results of the primary terminal based on the selected action in the environment.

[0091] Let me explain this in more detail.

[0092] Due to the physical limitations of SUs, it is impossible to fully grasp the actual state of the environment. Therefore, a single SU can only detect one channel and sector pair at each time step, providing a partial view of the environmental state.

[0093] When performing ED, the SU can estimate the received signal power for a selected channel-sector pair.

[0094] In one embodiment of the present invention, partial observation of the agent Let us define it as follows: Here, , represents the index of the selected channel and sector, represents the received signal power estimated by ED.

[0095] The action performed by each SU consists of a detection threshold, and the action (i.e., the detection threshold) can be used to determine whether a channel-sector pair is available. At each time step t, The action taken by can be expressed as, am.

[0096] That is, each agent can apply partial observations of the secondary terminal (i.e., specific channel-specific sector pairs, estimated received signal power) to a reinforcement learning-based actor-critic network model to select a policy-driven action. Here, the policy determines an optimal detection threshold that maximizes the detection probability of correctly detecting the presence of a PU while minimizing the cumulative false alarm probability, as defined in Equation 9.

[0097] Each agent can calculate a reward by performing actions on partial observations of the environment and comparing whether the primary terminal's sensing results match the actual state.

[0098] According to one embodiment of the present invention, a reward function for each SU (i.e., each agent) can be designed based on two possible cases. The first case is when the detection result of a channel-sector pair matches the actual state, and the second case is when the detection result of a channel-sector pair differs from the actual state.

[0099] The reward function should induce each agent to choose an action (i.e., a detection threshold) that provides accurate detection results, so it can be designed as in Equation 10.

[0100]

[0101] Here, p>0 represents a penalty score.

[0102] To solve Dec-POMDP, it is crucial to know the transition probability, a key component of the model, but this is often unknown. Therefore, in one embodiment of the present invention, we propose a solution called multi-agent distributed collaborative spectrum sensing (DCSS, Decentralized CSS).

[0103] In one embodiment of the present invention, a CTDE architecture is used in which the training phase and the execution phase are set differently.

[0104] During the training phase, we assume that all agents share their local knowledge with other agents, allowing them to learn from each other's experiences. During the training phase, the critique network is trained centrally, and observations acknowledged by all agents can be provided.

[0105] In contrast, during the execution phase, all agents acquire sufficient knowledge about the environment and can utilize the trained actor network to determine optimal actions based solely on local observations.

[0106] MA-DCSS adopts MADPPG, an extension of the DDPG algorithm that allows each agent to learn its own policy by considering the policies of other agents in the environment. Specifically, each agent is equipped with an actor network used to learn its own individual policy. The actor network can take current observations as input and return actions as output.

[0107] In step 220, the computing device (200) stores the partial observation, the selected action, the reward, and the next observation of each agent in a playback buffer as an experience.

[0108] At step 225, the computing device (200) can update the actor-critic network model of each agent based on the experience stored in the playback buffer.

[0109] During the training and execution phases, each agent's actor-critic network model can be updated differently.

[0110] During the training phase, each agent shares policies and partial observations with other agents, enabling them to learn an actor-critic network model. Therefore, during the training phase, each agent can centrally learn an actor-critic network model by sharing the experiences of other agents.

[0111] After sufficient learning is completed, in the execution phase, each agent can update the actor-critic network model using only its individual policy and local experience.

[0112] Let me explain this in more detail.

[0113] The actor network can be updated using a policy gradient expressed as in Equation 11.

[0114]

[0115] Here, B represents the batch sample data, represents the experience buffer, represents the joint action of all agents, may contain partial observations collected from all agents.

[0116] also, and are the deterministic policy and centralized policy of each agent, respectively. Represents a value estimation function.

[0117] In addition to the actor network, each agent has a critique network that is trained together during the training phase. The critique network uses all of the agents' observations and joint actions as input. The value can be generated as output. The loss function of the critic network can be expressed as Equation 12.

[0118]

[0119] Here, is as shown in mathematical formula 13.

[0120]

[0121] Here, and represents the target actor network and the target critic network.

[0122] During the training phase, the agent must maintain a balance between exploitation and exploration. To ensure sufficient exploration and enable the agent to find the optimal point, noise can be incorporated into the action output.

[0123] In one embodiment of the present invention, Ornstein-Uhlenbeck noise is used for action exploration, which can be defined as in mathematical expression 14.

[0124]

[0125] Here, represents the noise parameter.

[0126] A system according to one embodiment of the present invention aims to train all agents to learn an optimal detection threshold that maximizes detection performance for PUs.

[0127] The pseudocode of MA-DCSS for the training phase is as shown in Fig. 3.

[0128] During the training phase, each agent has its own actor and critic network that performs forward and backward propagation to update the weights, where the main operation is matrix multiplication. Therefore, the computational complexity for a single episode during the training phase is can be expressed as

[0129] Here, is the number of layers and N represents the size of the hidden layer. Also, I and J represent the sizes of the input layer and the output layer, respectively.

[0130] According to one embodiment of the present invention, I=4M, where M represents the number of SUs and 4 represents the total dimension of the observation and action space, and further, J is the output Q value and is equal to 1.

[0131] In the execution phase, only forward propagation is required for the actor network, so the computational complexity is That is, it is easy to see that the computational complexity is independent of the number of channels and sectors in both the training and execution stages.

[0132] However, it can be seen that the computational complexity increases linearly with the number of SUs in the training phase. Therefore, the proposed algorithm can be said to be fully scalable with respect to the number of channels, sectors, and SUs.

[0133] FIG. 4 is a diagram illustrating a system simulation environment according to one embodiment of the present invention. As illustrated in FIG. 4, a system environment with three PUs and five SUs is assumed.

[0134] In one embodiment of the present invention, it is assumed that each SU has three sectors for sensing and communication purposes, and can select a directional beam from one of the sectors at each time point t.

[0135] Also, all channels (e.g., ) are allocated to PUs locally and partially, and each sector of the SU (e.g., ) is assumed to be located within the coverage of the PU.

[0136] In one embodiment of the present invention, the simulation was run 10 times using random SU and PU locations, and the results were averaged. The simulation parameters are as shown in Fig. 5.

[0137] Figure 6 is a diagram showing the convergence results at different detection times. In Figure 6, the detection times are varied from 0.001 seconds to 0.03 seconds. This is a diagram illustrating the convergence of an optimal energy detection threshold control method based on multi-agent reinforcement learning in a distributed cognitive radio network according to one embodiment of the present invention. As shown in Fig. 6, as the detection time increases, the reward obtained by the agent increases, but it can be seen that each agent converges stably.

[0138] This positive correlation stems from the fact that a longer detection period allows the agent to gather more comprehensive information about the detected channel and sector. It's also worth noting that once a certain detection threshold is exceeded, further increasing the detection period has less of an impact on overall performance.

[0139] FIG. 7 is a diagram comparing the convergence results of an energy detection threshold control method according to a conventional method and an embodiment of the present invention.

[0140] As shown in FIG. 7, it can be seen that the optimal energy detection threshold control method based on multi-agent reinforcement learning in a distributed cognitive radio network according to one embodiment of the present invention converges at a faster rate than the conventional technology.

[0141] Figure 8 shows the results of comparing the detection probability and false alarm probability according to a conventional method and an embodiment of the present invention. The optimal energy detection threshold control method based on multi-agent reinforcement learning in a distributed cognitive radio network according to an embodiment of the present invention is approximately 500 episodes after about The detection probability of provided a false alarm probability of .

[0142] In contrast, conventional PPO-based RL algorithms achieve approximately The detection probability of It provided a false alarm probability, and in the case of DDPG-based RL algorithm, it showed very good performance in terms of detection probability. was achieved. However, this The worst false alarm probability in the vicinity was calculated.

[0143]

[0144] Figure 9 is a block diagram schematically illustrating the internal configuration of a computing device according to one embodiment of the present invention. Here, the computing device (200) may be a server or an SU.

[0145] Referring to FIG. 9, a computing device (200) according to one embodiment of the present invention is configured to include a communication unit (910), a memory (920), and a processor (930).

[0146] The communication unit (910) is a means for transmitting and receiving data with other devices through a communication network.

[0147] The memory (920) stores at least one command for performing a multi-agent reinforcement learning-based optimal energy detection threshold control method in a distributed cognitive radio network according to one embodiment of the present invention.

[0148] The processor (930) is a means for controlling internal components (e.g., communication unit (910), memory (920), etc.) of a computing device (200) according to one embodiment of the present invention.

[0149] In addition, the processor (930) can execute instructions stored in the memory (920). Instructions executed by the processor (930) can perform a series of processes including constructing a state space for a network environment including a plurality of primary terminals and a plurality of secondary terminals, executing each agent for each secondary terminal, each agent applying partial observations of each secondary terminal to a reinforcement learning-based actor-critic network model to select an action according to a policy, calculating a reward based on the detection result of the primary terminal based on the selected action in the environment, storing the partial observations, the selected action, the reward, and the next observation of each agent in a replay buffer as experience, and updating the actor-critic network model of each agent based on the experience stored in the replay buffer. This is the same as described with reference to FIG. 2, so a redundant description will be omitted.

[0150]

[0151] The device and method according to the embodiment of the present invention may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the computer-readable medium may be those specially designed and configured for the present invention or may be those known and usable by those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0152] The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0153] The present invention has been described above, focusing on specific embodiments thereof. Those skilled in the art will appreciate that the present invention can be implemented in modified forms without departing from its essential characteristics. Therefore, the disclosed embodiments should be considered illustrative rather than limiting. The scope of the present invention is set forth in the claims, not the foregoing description, and all differences within the scope equivalent thereto should be construed as being encompassed by the present invention.

Claims

1. (a) A step of constructing a state space for a network environment including a plurality of primary terminals and a plurality of secondary terminals, wherein the state space includes each state regarding whether the primary terminals are occupied; (b) the agent applies the partial observations of each secondary terminal to a reinforcement learning-based actor-critic network model to select an action according to the policy, and calculates a reward based on the detection results of the primary terminal based on the selected action in the environment, wherein the action is a detection threshold; (c) storing the partial observation, the selected action, the reward and the next observation in a playback buffer as an experience; and (d) A method for controlling optimal energy detection thresholds based on multi-agent reinforcement learning in a distributed cognitive radio network, comprising the step of updating the actor-critic network model based on the experience stored in the playback buffer.

2. In paragraph 1, In the training phase, the above replay buffer also stores the experience of other agents, and in the above step (d), each agent shares the experience of other agents and centrally trains the actor-critic network model of each agent. A method for controlling optimal energy detection thresholds based on multi-agent reinforcement learning in a distributed cognitive radio network, characterized in that in the execution phase, the step (d) updates the actor-critic network model of each agent using only the local experience of each agent.

3. In paragraph 1, A method for controlling an optimal energy detection threshold based on multi-agent reinforcement learning in a distributed cognitive radio network, characterized in that the above policy determines a detection threshold that maximizes the probability of correctly detecting the primary terminal and minimizes the cumulative false alarm probability up to time step t.

4. In paragraph 1, A method for controlling optimal energy detection thresholds based on multi-agent reinforcement learning in a distributed cognitive radio network, characterized in that the above policy is formulated by the following mathematical formula. Here, is a specific channel at time step t selected by the ith secondary terminal. and sector Indicates the detection threshold for , represents the orthogonal channel owned by the primary terminal, represents the sector of the ith secondary terminal, K represents the number of orthogonal channels, and L represents the number of sectors. represents the cumulative detection probability of the primary terminal up to time step t, represents the cumulative false alarm probability up to time step t, and M represents the index of the secondary terminal.

5. In paragraph 1, The above reward imposes zero (0) if the result of detecting the primary terminal based on the selected action in the environment is the same as the actual state, and imposes a penalty if the result of detecting the primary terminal based on the selected action in the environment is different from the actual state. A method for controlling optimal energy detection thresholds based on multi-agent reinforcement learning in a distributed cognitive radio network, wherein the above actual state is characterized in that the channel of the primary terminal is either occupied or unoccupied.

6. A computer-readable recording medium having recorded thereon a program code for performing the method according to paragraph 1.

7. Memory for storing at least one instruction; and Including a processor that executes instructions stored in the above memory, The instructions executed by the above processor are respectively: (a) a step of constructing a state space for a network environment including a plurality of primary terminals and a plurality of secondary terminals, wherein the state space includes each state regarding whether the primary terminals are occupied; (b) a step of executing each agent for each secondary terminal, each agent applying partial observations of each secondary terminal to a reinforcement learning-based actor-critic network model to select an action according to a policy, and calculating a reward based on the detection result of the primary terminal based on the selected action in the environment, wherein the action is a detection threshold; (c) storing the partial observations, the selected actions, the rewards and the next observations of each agent in a replay buffer as experience; and (d) A computing device characterized in that it performs a step of updating the actor-critic network model of each agent based on the experience stored in the playback buffer.

8. In paragraph 7, In the training phase, the above replay buffer also stores the experience of other agents, and in the above step (d), each agent shares the experience of other agents and centrally trains the actor-critic network model of each agent. A computing device characterized in that, in the execution phase, the step (d) above updates the actor-critic network model of each agent using only the local experience of each agent.

9. In paragraph 7, A computing device characterized in that the above policy determines a detection threshold that maximizes the probability of correctly detecting the primary terminal and minimizes the cumulative false alarm probability up to time step t.

10. In paragraph 7, A computing device characterized in that the above policy is formulated by the following mathematical formula. Here, is a specific channel at time step t selected by the ith secondary terminal. and sector Indicates the detection threshold for , represents the orthogonal channel owned by the primary terminal, represents the sector of the ith secondary terminal, K represents the number of orthogonal channels, and L represents the number of sectors. represents the cumulative detection probability of the primary terminal up to time step t, represents the cumulative false alarm probability up to time step t, and M represents the index of the secondary terminal.

11. In paragraph 7, The above reward imposes zero (0) if the result of detecting the primary terminal based on the selected action in the environment is the same as the actual state, and imposes a penalty if the result of detecting the primary terminal based on the selected action in the environment is different from the actual state. A computing device characterized in that the above actual state is one of channel occupancy and non-occupancy of the primary terminal.

12. Multiple primary terminals; and Including multiple secondary terminals, Each of the above secondary terminals is A step of constructing a state space for a network environment including a plurality of primary terminals and a plurality of secondary terminals, wherein the state space includes each state regarding whether the primary terminals are occupied; The agent applies the partial observations of each secondary terminal to a reinforcement learning-based actor-critic network model to select an action according to the policy, and calculates a reward based on the detection results of the primary terminal based on the selected action in the environment - the action is a detection threshold; A step of storing the above partial observation, the above selected action, the above reward and the next observation in a playback buffer as an experience; and A system characterized by comprising a step of updating the actor-critic network model based on the experience stored in the playback buffer.

Images