Throughput optimization method in multi-channel cognitive radio network under AoI constraint

Abstract

The invention discloses a throughput optimization method in a multi-channel cognitive radio network under AoI constraint. The method comprises the following steps: firstly, constructing a cognitive radio network comprising a main network and a secondary network; secondly, based on the cognitive radio network, defining a state space, an action space and a reward function of the improved soft actor-commentator C-SAC algorithm; constructing a reinforcement learning network model according to a state space, an action space and a reward function of the C-SAC algorithm, deploying the reinforcement learning network model on a receiver, and performing training according to a SAC algorithm mode; and finally, interacting with the environment according to the trained reinforcement learning network model, and performing training in stages to continuously improve the long-term total throughput of the secondary network. According to the invention, the time complexity is extremely low, the long-term throughput is optimized, the freshness of the received data is ensured, and the data mastered by the secondary receiver is prevented from being too old.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of network throughput optimization, and particularly relates to a long-term throughput optimization method in a multi-channel multi-user cognitive radio network under an AoI constraint. BACKGROUND

[0002] The Internet of Things facilitates people's daily life, but also brings some challenges. The increase of Internet of Things devices leads to a significant increase in demand for network resources. However, network resources are limited and scarce, and most of them are occupied by authorized users, and part of the network resources has low utilization rate. In addition, since wireless devices cannot carry large-capacity energy storage devices, they cannot be replaced frequently in some extreme scenarios, so the energy supply problem of wireless devices is also a challenge to be solved. In view of this, the background technology involved in the present application mainly includes Radio Frequency (RF) wireless energy supply technology, Cognitive Radio (CR) technology, and Ambient Backscatter (AB) technology, etc. These technologies effectively alleviate the problems of spectrum and energy shortage in wireless networks from different angles.

[0003] Compared with solar energy devices, RF energy supply technology can provide stable and wide-range wireless energy supply, but it has the disadvantages of expensive devices and unstable energy supply. CR technology can automatically detect available spectrum by interacting with the environment, and then adjust its operating parameters (frequency, transmission power, modulation technology, etc.) to effectively utilize the spectrum in real time. AB technology enables secondary users (SUs) to transmit their own data by modulating environmental RF signals (such as TV tower (TV) signals, broadcast tower (FM) signals), which not only alleviates the problem of insufficient available spectrum, but also alleviates the problem of energy supply difficulty, and also does not need to deploy a dedicated RF signal source for backscatter devices. These technologies have been widely used in academia and industry. SUMMARY

[0004] In the field of RF energy supply AB-assisted CR networks, few documents consider the multi-channel multi-user case. In addition, since only throughput indicators are considered, the freshness of data cannot be guaranteed. However, many time-sensitive scenarios, such as smart medical care, disaster monitoring, etc., need to guarantee the freshness of data. Therefore, the main purpose of the present application is to provide a long-term throughput optimization method in a multi-channel multi-user cognitive radio network under an AoI constraint, to guarantee the throughput and data freshness of the secondary network of cognitive radio.

[0005] In order to achieve the above-mentioned purpose, the technical scheme of the present application is as follows: A long-term throughput optimization method in a multi-channel multi-user cognitive radio network under AoI constraints. The cognitive radio network includes two sub-networks: a primary network and a secondary network. Among them, the primary network consists of M signal sources (PTs), which can be TV towers, WiFi access points, broadcast base stations or cellular base stations. Each PT occupies one channel, that is, there are M available channels, and the state of each channel can be different. In the secondary network, there are N secondary transmitters (STs) and a secondary receiver (Agent), where M < N. Each ST is equipped with a battery with a limited capacity E and a single antenna, and transmits data to the Agent on different channels. The Agent is equipped with multiple antennas and is powered by wired energy, as well as sufficient computing power. It can receive data transmitted from different channels simultaneously, and train and optimize the channel scheduling and energy allocation tasks of the STs according to the environmental state. To prevent the data of some STs from staying for a long time due to poor long-term state, the Agent is also responsible for recording and updating the AoI of each ST, and adjusting the channel scheduling and energy allocation of the STs according to the AoI situation. Therefore, the main task of the Agent is to be responsible for the channel scheduling and energy allocation of multiple STs according to the current state information, including the channel states, channel gains of multiple PTs, the energies and AoIs of multiple STs, so as to optimize the long-term throughput of the secondary network under AoI constraints.

[0006] The long-term throughput optimization method for the multi-channel multi-user secondary network under AoI constraints is mainly based on the improved Soft Actor-Critic (SAC) algorithm, that is, the Conditional-SAC (C-SAC) algorithm. The main process includes: 1. Based on the cognitive radio network, simplify and process relevant data, such as whether the channel is idle, channel gain, etc., and then define the state space, action space and reward function required by the C-SAC algorithm; 2. Construct a reinforcement learning network model according to the state space, action space and reward function of the C-SAC algorithm, deploy the initial neural network model on the Agent, and train it according to the SAC algorithm mode until the specified effect is achieved, that is, optimize the long-term total throughput of the secondary network and ensure the AoI constraint conditions; 3. Deploy the trained reinforcement learning network model on the Agent, interact with the environment, and then conduct small-scale training periodically to continuously improve the long-term total throughput of the secondary network.

[0007] For process 1, the state space includes an energy state space, a channel occupancy state space, a channel gain state space and an AoI state space, where: Energy state space: , where represents thei ST (ST i ) in the initial energy of the first t frame, E representing the maximum capacity of the battery; Channel occupancy state space: , where represents that the channel i is busy, represents that the channel i is idle; Channel gain state space: , where , , where represents the channel gain from the first j PT (PT j ) to the ST i , and represents the channel gain from the ST i to the Agent; in order to accelerate convergence, the channel gain state space is redefined in the present application , , where respectively represents the short-term channel attenuation from the ST to the Agent, and from the PT j to the ST, d i represents the distance from the ST i to the Agent, D i,j represents the distance from the PT j to the ST i , represents the path loss exponent), S gain can be updated as ; AoI state space: , where represents the AoI about the ST i , represents the maximum AoI that the Agent can tolerate; Therefore, the state space is .

[0008] For Process 1, the action space is mainly designed according to the channel and energy allocation strategy followed to optimize the algorithm target. The optimization target is to improve the channel and energy allocation strategy π under the condition of multiple channels, so as to realize the long-term throughput optimization of the secondary network and meet the constraint of AoI. Therefore, the action space is defined as follows: Channel allocation action space: , where represents the STi No channel was assigned. ST i The assigned channel number is k -1; Energy distribution action space: ,in ST i In the t The energy that a frame can consume is less than [amount missing]. This means that it is not allowed to allocate more energy than the initial available energy.

[0009] Therefore, the action space is .

[0010] For process 1, the reward function required for training the neural network model is designed primarily based on the optimization objective. The optimization objective is to achieve long-term throughput optimization of the secondary network under multi-channel conditions by improving channel and energy allocation strategies, while satisfying the AoI constraint. Therefore, the reward function mainly consists of the total throughput function and the AoI function.

[0011] Since ST performs different behaviors depending on whether a channel is allocated, whether the channel is occupied, and the various states of the current frame, the throughput function and AoI function also differ accordingly. Therefore, it is necessary to analyze the AoI and throughput under different conditions, as well as the changes in each state, where: Channel Model: In this application, the channel adopts a block fading model, meaning that the channel state and channel gain remain constant within a frame, but the state may differ between different frames. The changes in channel gain follow a Rayleigh distribution. Additive white Gaussian noise is used, with a noise variance of... .

[0012] Energy model: when At that time, i.e., ST i No channel assigned, ST i An energy capture operation will be performed to capture energy from all currently busy channels. The captured energy is shown in Equation (1).

[0013] (1) in, P j Indicates PT j The transmission power, ST i The energy conversion factor, where T is the frame length. To simplify the problem, this application assumes that the transmit power of each PT is the same, which is... P ,Right now Pj = P The energy conversion coefficient of each ST is the same, which is 1. ,Right now The frame length T = 1. Therefore, formula (1) can be transformed into formula (2).

[0014] (2) when ,and At that time, i.e., ST i A channel has been assigned, and the assigned channel is idle. ST i It will consume the allocated energy Transmit in overlay mode.

[0015] Therefore, the energy change follows formula (3).

[0016] (3) Among them, when hour, ,otherwise, .

[0017] Throughput model: If That is, the channel k- 1 is occupied, ST i The AB mode transfer will be executed, according to Shannon's formula, ST i The resulting throughput As shown in formula (4).

[0018] (4) Where W is the transmission bandwidth. The reflection coefficient, This represents the channel noise power.

[0019] like That is, the channel k- 1. Not occupied, ST i Overlay mode transmission will be performed, according to Shannon's formula, ST i The resulting throughput As shown in formula (5).

[0020] (5) Therefore, the total throughput function for each frame As shown in formula (6).

[0021] (6) AoI Model: AoI is a new metric for measuring the freshness of data received by the receiver. It is mainly used to track the time elapsed from the time the latest data is generated to the time when the receiver successfully receives the latest data. The AoI function is updated as shown in formula (7).

[0022] (7) in, R min This represents the minimum throughput constraint, when ST i When the throughput of each frame meets the minimum throughput constraint, it indicates that the data is being fully received, corresponding to the AoI value, i.e. Reduce to 1, otherwise increment by 1, until the maximum AoI threshold is reached. .

[0023] Based on the above definition, the reward function The design takes into account both the long-term throughput of the secondary network and AoI constraints, and is defined as follows: (8) in This represents the average total throughput. N This represents the total number of STs. Indicating that ST i The penalty term is primarily used to control the AoI of the secondary network to satisfy AoI constraints; when the AoI exceeds a threshold, a larger penalty is imposed. and In addition, the penalty also takes into account the energy allocated; the more energy allocated, the greater the penalty.

[0024] Beneficial effects of this invention: This application proposes a long-term throughput optimization method for multi-channel, multi-user secondary networks under AoI constraints. It considers the complexities of multi-channel cognitive radio networks with environmental scattering assistance, comprehensively utilizes available spectrum to improve spectrum utilization, and introduces an AoI metric to measure the data freshness required by the receiver. The algorithm has extremely low time complexity. It utilizes a continuous state and action space that does not require discretization, thus avoiding the loss of important spatial information. This invention has low time complexity, eliminates the need for spatial discretization to prevent the loss of important information, and introduces an AoI metric to measure the data freshness required by the receiver, resulting in extremely low time complexity. It not only optimizes long-term throughput but also ensures the freshness of received data, preventing secondary receivers from holding outdated data. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the structure of a cognitive radio network; Figure 2 This is a flowchart of the method described in this application; Figure 3 A schematic diagram of a neural network implemented using the C-SAC algorithm; Figure 4 The figure shows the experimental results of the C-SAC algorithm in a multi-channel, multi-user cognitive radio network. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0027] The primary objective of this application is to optimize multiple performance metrics in multi-channel, multi-user scenarios within the field of backscatter-assisted cognitive radio networks. Compared to single-channel scenarios, multiple available channels provide services, reducing conflicts between secondary users, increasing the range of choices, and improving secondary network performance. Multiple secondary users, based on their current state, comprehensively utilize available resources and perform different actions, which not only improves the utilization rate of scarce resources such as spectrum and energy but also enhances the overall performance of the secondary network. In practical scenarios, focusing solely on one network resource metric obviously neglects others. Some metrics even have a game-theoretic relationship; considering only one metric can lead to a loss in the performance of another. Therefore, simultaneously focusing on multiple network resource metrics can achieve multi-resource balance or joint optimization of the network. How to achieve multi-resource optimization is also a worthy research topic.

[0028] There are various traditional evaluation metrics for wireless networks (including cognitive radio networks), such as throughput, latency, fairness, energy consumption, and quality of service. Throughput is a straightforward indicator of network performance, and it can be used to optimize the throughput of independent networks, such as maximizing the throughput of secondary networks, or to optimize the throughput of cooperative networks, such as joint optimization of secondary and primary networks. Besides throughput optimization, AoI (Aspect of Interest) is a new and highly relevant performance metric, measuring the time elapsed from data generation to successful reception of the latest data by the receiver. Unlike traditional latency metrics, AoI determines the timeliness of received data from the perspective of the destination receiver. Optimizing the long-term throughput of secondary networks while maintaining AoI constraints is a worthwhile research topic. This not only improves the throughput of secondary networks but also ensures the freshness of received data, preventing secondary receivers from acquiring outdated data. This application considers multiple secondary receivers making decisions on channel scheduling and energy allocation for each secondary transmitter based on the channel status of the current frame, such as channel busyness, channel gain, available energy of each secondary transmitter, and the AoI status of the secondary transmitter in the secondary receiver.

[0029] like Figure 1The shown environment backscatter-assisted cognitive radio network mainly consists of two sub-networks: the primary network and the secondary network. Among them, the primary network consists of M signal sources (PTs), where the PTs can be TV towers, WiFi access points, broadcast base stations, or cellular base stations. Each PT occupies one channel, that is, there are M available channels, and the states of each channel can be different. In the secondary network, there are N secondary transmitters (STs) and a secondary receiver (Agent), where M < N. Each ST is equipped with a battery with a finite capacity E and a single antenna, and transmits data to the Agent over different channels. The Agent is equipped with multiple antennas, wired energy supply, and sufficient computing power, and can receive data transmitted over different channels simultaneously. According to the environmental state, it trains and optimizes the channel scheduling and energy allocation tasks of the STs. To prevent the data of some STs from being detained for a long time due to their poor state for a long time, the Agent is also responsible for recording and updating the AoI of each ST, and adjusting the channel scheduling and energy allocation of the STs according to the situation of the AoI. Therefore, for the above cognitive radio network with T as a frame period, the main task of the Agent is to be responsible for the channel scheduling and energy allocation of multiple STs according to the state information of the current time frame, including the channel states, channel gains of multiple PTs, the energies and AoIs of multiple STs, so as to optimize the long-term throughput of the secondary network under the AoI constraint.

[0030] Since the number of channels is less than the number of STs, and the busy and idle states of different channels may be different, the STs need to perform different behaviors according to whether they are assigned channels, the channel occupancy situation, and the channel state. Among them, denotes that the i th ST (ST i ) is not assigned a channel, denotes that the channel assigned to ST i is k -1, . When , ST i will perform an energy harvesting operation to harvest energy from all the currently busy channels; when , if channel k -1 is in a busy state, that is, the channel is occupied, ST i will perform ambient backscatter (AB) mode transmission; when , if channel k -1 is in an idle state, ST i will perform overlay mode transmission according to the consumable energy allocated by the Agent for it.

[0031] This application provides a method for optimizing the long-term throughput of multi-channel, multi-user secondary networks in cognitive radio networks under AoI constraints. This method is primarily based on an improved Soft Actor-Critic (SAC) algorithm, specifically the Conditional-SAC (C-SAC) algorithm. Figure 2 As shown, the main steps include: Step S1: Define the state space, action space, and reward function required by the C-SAC algorithm; Step S2: Design the neural network required for the C-SAC algorithm, initialize it, and then deploy it on the Agent; Step S3: Train the reinforcement learning network model according to the SAC algorithm mode, and finally output the channel and energy management strategy. Step S4: According to the channel and energy management strategy, channel allocation and energy allocation are performed on multiple STs. The STs perform different actions according to the allocation results, obtain the throughput, and update the AoI. Step S5: Format and store the obtained throughput and AoI, and conduct phased training in specified rounds to ensure the continuous improvement of secondary network throughput and AoI performance.

[0032] Each of the above steps is explained in detail below.

[0033] Step S1: Define the state space, action space, and reward function required for the C-SAC algorithm. The C-SAC algorithm extends the SAC algorithm, enabling it to adapt to different action variations and improving its stability and convergence speed. The reinforcement learning network model uses two fully connected neural networks as Actor policy networks. The first Actor network is the channel policy network, used for channel allocation, and the second Actor network is the energy policy network, used for energy allocation. The output of the channel policy network is also used as input, along with other state parameters, into the energy policy network. Finally, the outputs of the two Actor networks are used as the final channel and energy allocation results. Four fully connected neural networks are used as the critic action value network to evaluate the quality of the channel and energy allocation policies. Two neural networks serve as prediction critic networks, and two serve as target critic networks.

[0034] In this application, the state space, action space, and reward function required by the C-SAC algorithm are all related to the scenario, and generally include the following steps: Step S1.1: Define the state space. The state space includes: Energy state space: ,in Indicates the first i ST (ST)i ) in the t The initial energy of the frame. E Indicates the battery's maximum capacity; Channel occupied state space: ,in Indicates channel i busy, Indicates channel i idle; Channel gain state space: ,in, , ,in, Indicates from the first j PT (PT) j ) to ST i Channel gain, Indicates from ST i The channel gain to the Agent; to accelerate convergence, this application redefines the channel gain state space. .according to , Conversion, ( These represent the flow from ST to Agent and from PT, respectively. j Short-term channel attenuation to ST, d i ST i Distance to Agent D i,j Indicates PT j To ST i distance, S represents the path loss exponent. gain It can be updated to represent as ; AoI state space: ,in Indicates regarding ST i AoI, This represents the maximum AoI that the Agent can tolerate; Therefore, the state space is .

[0035] Step S1.2: Define the action space. The action space is primarily designed based on the channel and energy allocation strategies used to achieve the algorithm's optimization objective. The optimization objective is to optimize the long-term throughput of the secondary network while satisfying AoI constraints under multi-channel conditions by improving channel and energy allocation strategies. The action space includes: Channel allocation action space: ,in, ST i No channel was assigned. ST i The assigned channel number is k -1; Allocate energy action space: ,in ST i In the t The energy that a frame can consume is less than [amount missing]. This means that it is not allowed to allocate more energy than the initial available energy.

[0036] Therefore, the action space is .

[0037] Step S1.3: Define the reward function. The reward function required for training the neural network model is designed based on the optimization objective. The optimization objective is to improve the long-term throughput of the secondary network under multi-channel conditions by improving channel and energy allocation strategies, while satisfying the AoI constraint. Therefore, the reward function mainly consists of the throughput function and the AoI function, as defined in formula (1): (1) in, Indicates the first t Total frame throughput This represents the average total throughput. N This represents the total number of STs. Indicating that ST i The penalty term is mainly used to control the AoI of the secondary network to satisfy the AoI constraint.

[0038] about It should be noted that the actions performed by the ST differ depending on the channel state, resulting in different throughputs. Therefore, this application considers both scenarios simultaneously through parameter settings. That is, the channel k- 1 is occupied, ST i AB mode transmission will be performed, ST i The resulting throughput is shown in formula (2).

[0039] (2) Where W is the transmission bandwidth. P The transmit power of PT The reflection coefficient, Channel noise power, ST i Whether a channel has been allocated at time t, when ST i When a channel is assigned, ,otherwise, .

[0040] like That is, the channel k- 1. Not occupied, ST i Overlay mode transmission will be performed, ST i The resulting throughput is shown in formula (3).

[0041] (3) Total throughput per frame For all STs in the first t The sum of the throughput generated in the frame is shown in formula (4).

[0042] (4) Regarding penalties It should be noted that a significant penalty will be imposed when the AoI exceeds the threshold. and In cases where an idle channel is allocated but the throughput is very small, the penalty term also considers the allocated energy; the more energy allocated, the greater the penalty.

[0043] Step S2: Design the neural network required for the C-SAC algorithm, initialize it, and deploy it on the Agent. This generally includes the following steps: Step S2.1: Initialize the training rounds TL, the number of frames in each round TS, the difference in training times between the actor channel policy network and the actor energy policy network TC, the target entropy H_, the temperature parameter TP, and the learning rates LR of the actor policy network and the critic action value network. A and LR C Discount factor required for long-term optimization Soft update factor Small batch size BS, clear and replay memory pool .

[0044] Step S2.2: Randomly initialize the actor channel policy network, actor energy policy network, and the weights of prediction critic network 1 and prediction critic network 2. The weights of target critic network 1 and target critic network 2 are initialized to [values ​​to be filled in]. .

[0045] Step S3: Train the neural network according to the SAC algorithm mode, and finally output the channel and energy management strategy. This generally includes the following steps: Step S3.1: Randomly select a state s0 from the state space, and let s t =s0.

[0046] Step S3.2, based on the current 1 and s t Actor channel policy network outputs channel allocation actions .

[0047] Step S3.3, based on the current 2, s t And the channel allocation action output by the Actor channel policy network. The Actor energy strategy network outputs energy allocation actions. In order to ensure that the energy consumed by ST does not exceed the existing energy value, The percentage of energy consumed in the lesson, combined and To obtain the complete channel and energy allocation actions .

[0048] Step S3.4: Broadcast the channel allocation result and energy allocation result to ST. ST performs energy capture, overlay mode transmission, and AB mode transmission according to the allocation result, and obtains the total throughput generated by the current frame according to formula (4). Then, the corresponding penalty value is obtained according to formula (1). and rewards .

[0049] It should be noted that in formula (1), The throughput can be limited to below 10, which facilitates training. And when... If the AoI exceeds the constraints, a large penalty will be imposed. In This ensures that an overflow occurs when the AoI exceeds the constraints. The penalty value, plus This is because if the AoI exceeds the constraint, allocating more energy when an idle channel is allocated is a waste, and therefore the penalty value is larger.

[0050] Step S3.5: Detect the new channel state and channel gain state, update the energy state and AoI state, and merge them to obtain the state s of the next frame. t+1 and tuple Stored in the memory bank.

[0051] Step S3.6: Train and update the weights of the neural network according to the SAC algorithm mode. Since the Actor policy network is divided into a channel policy network and an energy policy network, and the output of the channel policy network is also the input of the energy policy network, in order to ensure the stability of training, this patent adopts an alternating training method, that is, after each TS training of the energy policy network, the channel policy network is trained once.

[0052] Step S3.7: Repeat steps S3.1-S3.6 until the training termination condition is met, and finally output the channel and energy management strategy.

[0053] It should be noted that using the SAC algorithm to train the neural network model is a relatively mature technology, and will not be elaborated on here.

[0054] Step S4: Allocate channels and energy to multiple STs according to the channel and energy management strategy. The STs perform different actions based on the allocation results, obtain the throughput, and update the AoI. This generally includes the following steps: Step S4.1: Initialize scene-related parameters, such as the number of PTs M, the number of STs N, the distance between STs and PTs D, the distance between STs and the Gateway d, the battery capacity of STs E, and the maximum AoI. Backscattering coefficient η wait.

[0055] Step S4.2: Deploy the two trained Actor policy network models on the secondary receiver Agent, then input the corresponding detected state value to obtain the channel allocation and energy allocation results. According to the allocation results, ST performs energy capture, overlay mode transmission, and AB mode transmission to obtain the cumulative throughput and AoI.

[0056] It should be noted that the obtained channel allocation and energy allocation results still need to be processed according to the requirements of the action space. For example, the channel allocation result should be 0~M, and 1~M cannot be repeated, and the energy allocation result range is [0,1], such as ST. i If the allocated energy percentage is 0.1, then its available energy is the existing energy value multiplied by 0.1. For unallocated channels, and for allocated channels that are busy STs, energy allocation will be ignored.

[0057] Step S5 involves formatting and storing the obtained throughput and AoI, and then performing phased training in specified rounds to ensure continuous improvement in the secondary network throughput and AoI performance. This generally includes the following steps: Step S5.1: Combine the throughput and AoI obtained in step S4 according to... The format is stored in the memory.

[0058] Step S5.2: Once the set update round T is reached, phased training is performed according to the SAC algorithm mode. A fixed amount of data is extracted from the memory bank to perform several training cycles, and the Actor channel policy network and Actor energy policy network are updated to optimize the policy network, improve the throughput of the secondary network, reduce AoI, and ensure data freshness.

[0059] It should be noted that if the update rate is lower than the preset threshold, the update round T will be extended, which can reduce training time and frequency.

[0060] To demonstrate the throughput and AoI performance of the scenario, the signal source (PT) in this application experiment is a TV signal tower, and the scenario parameter list is shown in Table 1.

[0061] Table 1. List of Scene Parameters

[0062] The list of algorithm parameters is shown in Table 2.

[0063] Table 2 List of Algorithm Hyperparameters

[0064] The schematic diagram of the neural network implemented by the C-SAC algorithm is shown below. Figure 3 As shown, where Indicates the current input state. This indicates a new input state. The actions output by the Actor channel policy network and the Actor energy policy network. The action output under the new state condition is predicted by the smaller value between the predicted action value network 1 and the predicted action value network 2, which is used to evaluate the value of the action. The target action value is the smaller value between the target action value network 1 and the target action value network 2, which is used to train the predicted action value network 1 and the predicted action value network 2.

[0065] The experimental results of the C-SAC algorithm in a multi-channel, multi-user cognitive radio network are shown in the figure below. Figure 4 As shown in the figure, the first series (red) represents the average throughput considering both throughput metrics and AoI constraints, the second series (blue) represents the average throughput considering only throughput metrics, and the third series (green) represents the average throughput considering only AoI metrics. The figure shows that while considering both throughput metrics and AoI constraints results in a lower throughput than considering only throughput metrics, it is close to the throughput value considering only throughput metrics and significantly higher than the throughput considering only AoI metrics.

[0066] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for throughput optimization in a multi-channel cognitive radio network under an AoI constraint, characterized in that, The method comprises the following steps: Step 1, constructing a cognitive radio network comprising a primary network and a secondary network; Step 2, defining a state space, an action space and a reward function of an improved soft actor-critic C-SAC algorithm based on the cognitive radio network; Step 3, constructing a reinforcement learning network model according to the state space, the action space and the reward function of the C-SAC algorithm, and deploying the model on a receiver to train according to the SAC algorithm; Step 4, interacting the trained reinforcement learning network model with an environment, and training periodically to continuously improve the long-term total throughput of the secondary network.

2. The method for throughput optimization in multi-channel cognitive radio networks under AoI constraint according to claim 1, characterized in that, The cognitive radio network includes two sub-networks: a primary network and a secondary network; wherein the primary network is composed of M signal sources PT, each PT occupies a channel, that is, there are M available channels; there are N secondary transmitters ST and a secondary receiver Agent in the secondary network, M < N; each ST is equipped with a limited capacity E battery and a single antenna, and transmits data to the Agent on different channels; the Agent is equipped with multiple antennas and a wired energy supply, simultaneously receives data transmitted on different channels, and trains and optimizes the channel scheduling and energy allocation tasks of the ST according to the environmental state; the Agent is responsible for recording and updating the information age AoI of each ST, and adjusting the channel scheduling and energy allocation of the ST according to the AoI; the task of the Agent is to schedule and allocate the channels and energy of multiple STs according to the current state information, including the channel state, channel gain of multiple PTs, energy and AoI of multiple STs, to realize the long-term throughput optimization of the secondary network under the AoI constraint.

3. The method for throughput optimization in multi-channel cognitive radio networks under AoI constraint according to claim 2, characterized in that, The state space comprises an energy state space, a channel occupancy state space, a channel gain state space and an AoI state space, wherein: Energy state space: where denotes the i th secondary transmitter ST i The initial energy of the t th frame, E denotes the maximum capacity of the battery; Channel occupancy state space: wherein represents that the channel t is busy at the i frame represents that the channel i is idle; Channel gain state space: ; , wherein, denotes the channel gain from the j th signal source PT j to ST i , and denotes the channel gain from ST i to the Agent; according to , scaling, denotes the short-term channel fading from ST to the Agent, respectively, and from PT j to ST, d i denotes the distance of ST i to the Agent, D i,j denotes the distance of PT j to ST i , and denotes the path loss exponent, S gain is updated as ; AoI state space: where denotes the AoI with respect to the ST i , denotes the maximum AoI that the Agent can tolerate; The state space is .

4. The method for throughput optimization in multi-channel cognitive radio networks under AoI constraint according to claim 3, characterized in that, The action space is designed according to a channel and energy allocation strategy for achieving an algorithm optimization target; The optimization target is to achieve long-term throughput optimization of the secondary network and satisfy the AoI constraint by improving the channel and energy allocation strategy under the condition of multiple channels, and the action space is defined as follows: Channel allocation action space: wherein, ST i not allocated a channel, ST i allocated channel number k -1; Energy allocation action space: wherein ST i In the first t frame consumed energy, the value is lower than i.e. not allowing to allocate values over the initial available energy; Action space is .

5. The method for throughput optimization in multi-channel cognitive radio networks under AoI constraint according to claim 4, characterized in that, The reward function is designed according to the optimization target, and the optimization target is to achieve long-term throughput optimization of the secondary network and satisfy the AoI constraint by improving the channel and energy allocation strategy under the condition of multiple channels, and the reward function is composed of a total throughput function and an AoI function.

6. The method for throughput optimization in multi-channel cognitive radio networks under AoI constraint according to claim 5, characterized in that, The total throughput function is implemented as follows: Channel model: The channel adopts block fading model, the change of each channel gain follows Rayleigh distribution, the noise adopts additive white Gaussian noise, and the noise variance is ; Energy model: When ST i is not assigned a channel, ST i will perform an energy capture operation, capturing energy from all channels that are currently busy; the captured energy is shown below: wherein, P j represents the transmission power of PT j , represents the transmission power of ST i , the energy conversion coefficient, T is the frame length; it is assumed that the transmission power of each PT is the same, all P , that is P j = P , the energy conversion coefficient of each ST is the same, all , that is , the frame length T = 1; therefore, the above formula is converted to: ; When and ST i is assigned a channel, and the assigned channel is free, ST i will consume the assigned energy perform overlay mode transmission; Thus, the energy change follows the formula: When , , otherwise ; Throughput model: if the channel k- 1 is occupied, ST i will perform AB mode transmission, according to Shannon formula, ST i generated throughput , wherein W is the transmission bandwidth, is the reflection coefficient; If , the channel k- 1 is not occupied, ST i The overlay mode transmission will be performed, according to the Shannon formula, ST i The throughput generated ; Thus, the total throughput function for each frame .

7. The method for throughput optimization in multi-channel cognitive radio networks under AoI constraint according to claim 6, characterized in that, The AoI function is implemented as follows: AoI model: AoI function update mode is: , R min For the minimum throughput constraint, when ST i The throughput of each frame satisfies the minimum throughput constraint, which means that the data is fully received, and the corresponding AoI value is 0 Fall to 1, otherwise superimposed 1, until reaching the maximum AoI threshold .

8. The method for throughput optimization in multi-channel cognitive radio networks under AoI constraint according to claim 7, characterized in that, The reward function The design takes into account both the long-term throughput and the AoI constraint of the secondary network and is defined as follows: wherein denotes the average total throughput, N denotes the total number of STs, denotes a penalty term for ST i , mainly used to control the AoI of the secondary network to satisfy the AoI constraint.

9. The method for throughput optimization in multi-channel cognitive radio networks under AoI constraint according to claim 8, characterized in that, The reinforcement learning network model comprises: two fully connected neural networks as Actor policy networks, a first Actor network being a channel policy network for performing channel allocation, and a second Actor network being an energy policy network for performing energy allocation, the output of the channel policy network also being input as an input together with other state parameters to the energy policy network, and the outputs of the two Actor networks being finally taken as the final channel and energy allocation results; four fully connected neural networks as critic action value networks for evaluating the channel and energy allocation strategy, wherein two neural networks are used as prediction critic networks, and two neural networks are used as target critic networks.

10. The method of claim 1 to 9, wherein, The step 4 is implemented as follows: Training the reinforcement learning network model according to the SAC algorithm to output a channel and energy management strategy; Allocating channels and energy to multiple STs according to the channel and energy management strategy, the STs performing different behaviors according to the allocation results, obtaining throughputs, and updating the AoI; Storing the obtained throughputs and AoI, and training periodically at a specified round to ensure continuous improvement of the throughput and AoI performance of the secondary network.

Images