A resource optimization method for multi-IRS-assisted broadband CR system based on deep reinforcement learning

Abstract

The present invention discloses a multi-IRS-assisted broadband CR system resource optimization method based on deep reinforcement learning, the method comprising step 1: broadband spectrum perception. Step 2: secondary network signal transmission. Step 3: establish a secondary network achievable rate maximization problem framework, and list the objectives and constraints to be optimized. Step 4: establish a Markov process according to the proposed secondary network achievable rate maximization problem framework. Step 5: use the SAC algorithm to optimize the model until the model converges. Step 6: output the result to obtain the total achievable rate of the secondary network. The present invention overcomes the problems of resource waste caused by fading and spectrum idleness generated during wireless signal propagation, and has significant improvements in broadband spectrum perception performance, spectrum utilization and achievable rate of secondary networks. Compared with existing traditional algorithms, the present invention reduces computational complexity and improves convergence speed.

Description

Technical Field

[0001] The present invention relates to a multi-IRS-assisted broadband CR system resource optimization method based on deep reinforcement learning, and belongs to the technical field of wireless communication. Background Art

[0002] Spectrum resources are the core resources for the development of mobile communications. Limited spectrum resources cannot meet people's needs for ultra-high data transmission rates and massive access in future wireless networks. Improving spectrum efficiency is an urgent problem to be solved in the development of the communications industry. Cognitive Radio (CR) enables secondary networks to use the frequency band of the primary network for communication, and is one of the key technologies to improve spectrum efficiency. At present, the research on CR mainly focuses on spectrum sensing, spectrum sharing and power control. Among them, spectrum sensing can adopt matched filter detection method, energy detection method and static cycle feature detection method. The energy detection method does not require prior knowledge of the primary network, and has the characteristics of simple implementation and accurate detection. It is the most commonly used method for spectrum sensing. Opportunistic Spectrum Access (OSA) is a new technology that uses radio spectrum, also known as "spectrum sharing technology". It improves spectrum utilization by transmitting data on unused spectrum. Its specific working method is that the secondary network detects the idle spectrum in the primary network, and the secondary network uses spectrum holes for signal transmission. When it detects that the primary user begins to reuse the idle spectrum, the secondary network immediately avoids it to avoid interfering with the primary user. However, in the CR system, the perception and transmission process often has problems such as low spectrum utilization efficiency and narrow signal coverage, which seriously affects the system's perception performance and the achievable rate of the secondary network. Therefore, in order to improve the performance of the CR system, a technology is urgently needed to solve the above problems.

[0003] Intelligent Reflective Surface (IRS) is an effective technology that can effectively solve the above problems. Specifically, IRS can enhance signal transmission by reflecting and focusing signals, thereby improving network coverage and transmission speed; at the same time, IRS can also reflect signals to a suitable location, thereby increasing signal reception strength and signal-to-noise ratio, and improving spectrum utilization efficiency. Therefore, the introduction of IRS in the CR system as an emerging technical means can effectively enhance the signal transmission and perception performance of the CR network, thereby improving the performance of the entire system. Nowadays, the results of applying IRS to CR systems have emerged one after another. In the paper "Secure and Energy Efficient Transmission for IRS-Assisted Cognitive Radio Networks" (IEEE Trans. Cognitive Commun. and Network, vol. 8, no. 1, pp. 170-185, March 2022) published by Xuewen Wu, Jingxiao Ma, et al., the performance trade-off between energy consumption and secure transmission in IRS-assisted CR networks is studied. The iterative alternating optimization technique is used to jointly optimize the beam profile of the cognitive base station and the reflection coefficient of the IRS, and the second-order cone programming approximation method is used to reduce the computational complexity. Simulation results show that IRS can significantly improve the spectrum efficiency and physical layer security of the CR network. However, the authors only considered a single primary user and a single secondary user, and did not consider the general scenario of multiple users in reality. Wei Wu, Zi Wang, Yuhang Wu, Fuhui Zhou et al. published a paper titled "Joint Sensing and Transmission Optimization for IRS-Assisted Cognitive Radio Networks" (IEEE Trans. Wireless Commun, 2023) that proposed that the IRS-assisted CR network uses a block coordinate descent algorithm to jointly optimize the sensing time, the beam profile of the secondary base station, and the reflection coefficient of the IRS to maximize the average achievable rate of the secondary network. Simulation results show that the scheme proposed in this paper is significantly better than the baseline scheme in terms of achievable rate and sensing performance, and IRS can greatly improve the spectrum efficiency of cognitive radio networks.

[0004] The above problems are all optimized by using convex optimization technology. When the dimensions of the action space and state space are too large, the computational complexity of convex optimization will increase significantly, and convex optimization technology is difficult to cope with large-scale communication scenarios. Based on the above situation, the deep reinforcement learning (DRL) algorithm has begun to attract widespread attention in the industry, which can well deal with the problems brought by the above convex optimization technology. "Intelligent Resource Allocation for IRS-Enhanced OFDM Communication Systems: A Hybrid Deep Reinforcement Learning Approach" (IEEE Trans. Wireless Commun, 2022) published by Wei Wu, Fengchun Yang, Fuhui Zhou et al. studied two application scenarios of IRS-enhanced downlink orthogonal frequency division multiplexing communication and spectrum sharing IRS-enhanced orthogonal frequency division multiplexing communication. The two scenarios require joint optimization of subcarrier allocation, beam coverage of base stations and IRS reflection coefficient to maximize the system and rate, and joint optimization of channel allocation, beam coverage of secondary base stations and reflection coefficient of IRS to maximize the secondary user and rate. The DQN-DDPG algorithm and D3QN-TD3 algorithm are used to optimize the above two system models respectively. The simulation results show that the proposed scheme can significantly improve the transmission rate compared with the baseline scheme. However, the spectrum sharing technology used in this paper can easily cause excessive interference to the primary user because the secondary user and the primary user share the same channel. Summary of the invention

[0005] The purpose of the present invention is to address the defects and shortcomings of the above-mentioned prior art and propose a multi-IRS-assisted broadband CR system resource optimization method based on deep reinforcement learning. This method solves the spectrum resource utilization efficiency problem and broadband spectrum perception and sub-channel allocation problem in the prior art implementation by proposing a multi-IRS-assisted broadband CR system resource optimization method based on deep reinforcement learning. At the same time, compared with the existing traditional algorithms, the present invention reduces the computational complexity and improves the convergence speed.

[0006] The technical solution adopted by the present invention to solve the technical problem is: a multi-IRS-assisted broadband CR system resource optimization method based on deep reinforcement learning, the method comprising the following steps:

[0007] Step 1: Wideband spectrum sensing.

[0008] The multi-IRS-assisted broadband CR system of the present invention divides the frequency band into C sub-bands, senses each sub-band respectively, and the sensed results are divided into two cases, namely:

[0009] Step 1-1: Sub-band c is in an idle state.

[0010] The secondary base station can only receive noise signals, that is, the secondary base station receives signals where n c (t) means the mean is 0 and the variance is At this time, the secondary network can use sub-band c for signal transmission.

[0011] Step 1-2: Sub-band c is in a busy state.

[0012] The secondary base station receives a joint fading signal composed of the direct signal, the reflected signal and the additive Gaussian white noise after channel fading, that is, in They represent the channels from the primary base station to the secondary base station, from the primary base station to the rth IRS, and from the rth IRS to the secondary base station in the sub-spectrum c, respectively. Q is the number of primary users, and R is the number of IRSs. represents the reflection coefficient of the rth IRS in the perception stage, and Respectively represent the mth in the rth IRS r The amplitude and phase shift of the reflected element, where Assume that the reflection amplitude in the present invention is in an ideal state, that is, represents the state of sub-spectrum c, Indicates that sub-spectrum c is in idle state, Indicates that sub-spectrum c is in a busy state. q represents the transmission power of the primary base station, and x(t) represents the complex baseband signal sent by the primary base station. At this time, the secondary network can no longer use the sub-spectrum c for signal transmission.

[0013] Step 2: Secondary network signal transmission.

[0014] When an idle sub-band is detected, the secondary network uses it for information transmission and determines the achievable rate of information transmission in the band. Subsequently, the total achievable rate of information transmission by the secondary network using all available idle sub-bands is calculated.

[0015] Step 3: Establish the framework of the secondary network achievable rate maximization problem and list the objectives and constraints that need to be optimized.

[0016] Step 4: Establish a Markov process based on the proposed secondary network achievable rate maximization problem framework.

[0017] Step 5: Use the SAC algorithm to optimize the model until the model converges.

[0018] Step 6: Output the result and obtain the total achievable rate of the secondary network.

[0019] Beneficial effects:

[0020] 1. The present invention overcomes the problems of resource waste caused by fading and spectrum idleness during wireless signal propagation, and has significant improvements in broadband spectrum perception performance, spectrum utilization, and achievable rate of secondary networks.

[0021] 2. The present invention uses the SAC algorithm to jointly process the action space composed of the phase shift of IRS in the perception stage and the transmission stage, the beamforming of the secondary base station, channel allocation, IRS selection and the perception time. Compared with the traditional alternating optimization algorithm, the present invention solves the problem that multiple optimization variables are difficult to optimize at the same time, greatly reduces the computational complexity and improves the convergence speed. In addition, the present invention also solves the problem of the simultaneous existence of continuous actions and discrete actions in the action space. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a system model diagram of the present invention.

[0023] Figure 2 This is a convergence diagram of the present invention using a deep reinforcement learning algorithm to optimize a broadband cognitive radio system assisted by multiple intelligent reflectors.

[0024] Figure 3 This is how the total achievable rate of the secondary network changes with the transmission power of the secondary base station in different scenarios of the present invention.

[0025] Figure 4 The figure shows how the secondary network achievable rate of the present invention changes with the sensing time in different scenarios.

[0026] Figure 5 Flowchart of the SAC algorithm method. DETAILED DESCRIPTION

[0027] The invention is further described in detail with reference to the accompanying drawings.

[0028] like Figure 1 As shown, the present invention provides a multi-IRS assisted broadband CR system resource optimization method based on deep reinforcement learning, the method comprising the following steps:

[0029] Step 1: Wideband spectrum sensing.

[0030] The secondary base station performs spectrum sensing on each sub-band to determine the current state of the sub-band. The specific calculation method of the detection probability and false alarm probability of sub-band c is as follows:

[0031]

[0032]

[0033] Among them, η c is the detection threshold, N is the number of secondary base station antennas, τ is the sensing time, f s is the sampling frequency, γ c is the received signal-to-noise ratio of the signal sent by the primary base station to the secondary base station in sub-band c, is the target detection probability, Q(·) is the right tail function of the standard normal distribution,

[0034] By using IRS, the present invention can almost ignore the influence of the signal strength of the primary base station on the system detection threshold, and its detection threshold is:

[0035]

[0036] in, is the maximum tolerable false alarm probability of the system, and δ is a constant close to zero, which is used to ensure that the false alarm probability does not exceed

[0037] Step 2: Information transmission.

[0038] When it is detected that the sub-band is in an idle state, the secondary network can use the sub-band for signal transmission. Assuming that the sub-band c is in an idle state, the received signal transmitted by the secondary base station using the sub-band c to the secondary user k is:

[0039]

[0040] in, They represent the channels from the secondary base station to the kth secondary user, from the rth IRS to the kth user, and from the secondary base station to the rth IRS in sub-band c, respectively. represents the reflection coefficient of the rth IRS during the transmission phase, is the beamforming vector for the kth secondary user in subband c, is the complex baseband transmission signal sent by the secondary base station to the kth secondary user in sub-band c, n k The mean is 0 and the variance is Additive white Gaussian noise. Indicates the selection of IRS, indicates that the rth IRS is used when transmitting the signal to the kth secondary user, This means that the rth IRS is not used when transmitting signals to the kth secondary user.

[0041] In addition, there is a situation where the primary network is using sub-band c but the secondary base station misses detection. In this case, the received signal transmitted by the secondary base station to the secondary user k using sub-band c is:

[0042]

[0043] in, is the interference of the primary network to the secondary user k, and its interference signal is:

[0044]

[0045] in, They represent the channels from the primary base station to the kth secondary user and from the primary base station to r IRSs respectively.

[0046] According to the above two situations, their signal-to-interference-noise ratios are:

[0047]

[0048]

[0049] The probabilities of the above two situations occurring are:

[0050]

[0051]

[0052] So in subband c, the achievable rate of the kth secondary user is:

[0053]

[0054] Among them, T is the total time occupied by the perception stage and the transmission stage, τ is the time occupied by the perception stage, represent the probability that sub-band c is in idle state and busy state respectively.

[0055] Therefore, the total achievable rate for all secondary users is

[0056]

[0057] Wherein, K is the number of secondary users.

[0058] Step 3: Establish the framework of the secondary network achievable rate maximization problem and list the objectives and constraints that need to be optimized.

[0059] The goal of the present invention is to maximize the achievable rate of the secondary network while ensuring the maximum interference power constraint of the primary user and the service quality requirements of the secondary network. To achieve this goal, the present invention jointly optimizes channel allocation, IRS selection, sensing time, beamforming of the secondary base station and phase shift of the IRS. The problem framework is:

[0060]

[0061]

[0062]

[0063]

[0064]

[0065]

[0066] C6:0≤τ≤T,

[0067]

[0068]

[0069]

[0070]

[0071]

[0072]

[0073] in, is the minimum detection probability, is the maximum tolerable false alarm probability, P max is the maximum transmit power of the secondary base station, is the minimum achievable rate of secondary user k, is the tolerable maximum interference power of the primary user q, C3 is the maximum transmission power constraint of the secondary base station at subcarrier frequency c, C4 is the minimum rate requirement of the secondary user, C5 is the maximum tolerable interference of the primary user, C6 is the perception time constraint, C7 and C8 are non-convex unit modulus constraints, C9 represents the channel occupancy, C10 indicates that each user can occupy at least one channel for signal transmission, C11 is the use of the intelligent reflective surface, and C12 indicates that at least one IRS is used when sending information from the secondary base station to the secondary user.

[0074] Step 4: Establish a Markov process based on the proposed secondary network achievable rate maximization problem framework.

[0075] This discrete-continuous optimization problem is regarded as a Markov decision process problem, the multi-IRS-assisted broadband CR system of the present invention is taken as the environment, and the central controller of the IRS is taken as the agent.

[0076] In addition to the above process, a Markov decision process problem also includes state space, action space, transition probability and reward function.

[0077] State space: The state space should include all parameters that determine the action and the action in the previous time slot, that is, the channel state information of all communication links at time slot t, and the action in time slot t-1. Therefore, the state space s in time slot t (t) ={a (t-1) ,H (t)}.

[0078] Action space: The action space a at time slot t (t) It consists of two parts, namely the discrete action space consisting of channel allocation, IRS selection and perception time. and the continuous action space consisting of the secondary base station beam profile and the reflection coefficient of the IRS

[0079] Transition probability: Pr(s (t+1) |s (t) ,a (t) ) is the state transition probability, which represents the state in state s (t) Take action a (t) , the state changes to s (t+1) probability.

[0080] Reward function: The goal of the present invention is to maximize the total reachability of secondary users while ensuring the minimum reachability requirement of secondary users, while ensuring that the interference to the primary user is less than the maximum tolerable interference, so the reward function is r = μ1R s +μ2δ k +μ3δ i , where μ1, μ2, μ3 are constants.

[0081]

[0082]

[0083]

[0084] Since on-policy requires more computing resources and is more expensive, off-policy is more suitable in the present invention. The action space of the present invention includes both discrete actions (τ, ρ, γ) and continuous actions (W, θ t ,θs ), so a random strategy should be used.

[0085] In summary, the reinforcement learning method adopted by the present invention is the soft action-criteria (SAC) algorithm.

[0086] Step 5: Use the SAC algorithm to optimize the model until the model converges. The specific process of the SAC algorithm is as follows: Figure 5 shown.

[0087] Step 6: Output the result to obtain the total achievable rate of the secondary network.

[0088] The effects of the present invention are further described in detail below in conjunction with simulation experiments, specifically including:

[0089] 1) Simulation conditions and parameter settings

[0090] The simulation experiment of the present invention is carried out on the simulation platform of Python3.6 and tensorflow1.12. The computer model is Intel Core i5, equipped with NVIDIA GEFORCE RTX 3050 independent graphics card.

[0091] In the present invention, the channel type from the base station to the user is a Rice channel, and the channels from the base station to the IRS and from the IRS to the user are Rayleigh channels. The path loss caused by the distance d is in, is the path loss index, PL0 is the path loss at the reference distance D0 = 1, and the path loss index from the primary base station to the IRS, from the secondary base station to the IRS, from the IRS to the primary user, and from the IRS to the secondary user Channel loss index from the primary or secondary base station to the primary or secondary user and from the primary base station to the secondary base station

[0092] In the present invention, the number of antennas of the secondary base station is M=6, the frame duration is T=100ms, and the sampling frequency is f s =6MHz, the noise variance received by the secondary base station Target detection probability Maximum tolerable false alarm probability The probability that subband c is idle In addition, the noise variance received by secondary user k is The transmission power P of the primary base station to the primary user q q =15dbm.

[0093] 2) Simulation content

[0094] Figure 2This is a convergence diagram of the deep reinforcement learning algorithm optimized in the case of a multi-intelligent reflector-assisted broadband cognitive radio system. The deep reinforcement learning was iterated 200 times. As shown in the figure, the reward of the present invention gradually increases during the iteration process and finally converges after 25 iterations. The results show that the algorithm proposed in the present invention has good convergence performance.

[0095] Figure 3 It shows how the total achievable rate of the secondary network changes with the transmission power of the secondary base station in different scenarios. The present invention compares the number of different intelligent reflective surfaces, the number of channels, whether there is spectrum sensing, and whether the phase shift of the two stages of sensing and transmission is optimized respectively. As can be seen from the figure, as the transmission power of the secondary base station increases, the achievable rate of the secondary network will also increase. In addition, by comparing the scheme of the present invention with the benchmark scheme, we can find that the intelligent reflective surface, broadband communication and opportunistic spectrum access technology have a significant effect on improving the achievable rate of the secondary network. First, with the increase in the number of intelligent reflective surfaces, the controllability of its signal transmission is enhanced. The use of multiple intelligent reflective surfaces can flexibly control the signal transmission path and direction, so as to better cope with complex signal environments and transmission requirements. Secondly, broadband communication divides a broadband signal into multiple narrowband sub-channels for communication. Each sub-channel is orthogonal and independent, eliminating mutual interference between users and improving the spectrum utilization efficiency of the system. Finally, OSA technology allows users to communicate on unused spectrum, avoiding the waste of communication resources.

[0096] Figure 4 The present invention studies the change of the secondary network achievable rate with the perception time in different scenarios. The present invention compares the impact of different numbers of intelligent reflectors on the secondary network achievable rate. As can be seen from the figure, as the perception time increases, the secondary network achievable rate first increases and then decreases. The reason is that as the perception time increases, the false alarm probability of the system gradually decreases, and the spectrum utilization efficiency gradually increases. However, when the perception performance increases to a critical value, the system's perception performance will no longer change. At this time, increasing the perception time will occupy the time for the secondary network to transmit information, thereby causing the secondary network achievable rate to decrease. Therefore, in the design of the CR system, it is necessary to reasonably balance the perception and transmission time to maximize the achievable rate of the secondary network.

[0097] In summary, the present invention is a multi-IRS assisted broadband CR system resource optimization method based on the SAC algorithm. By deploying multiple intelligent reflection surfaces around the base station or user to directionally reflect the transmission signal, the present invention significantly improves the perception performance and the achievable rate of the secondary network.

[0098] In this field, the above is only a preferred example of the present invention and is not intended to limit the present invention. Any modification, replacement, improvement and polishing made by any technician in this field within the spirit and concept of the present invention should be covered within the protection scope of the present invention.

Claims

1. A multi-IRS-assisted broadband CR system resource optimization method based on deep reinforcement learning, characterized in that: The method comprises the following steps: Step 1: Wideband spectrum sensing, including: The secondary base station performs spectrum sensing on each sub-band to determine the current state of the sub-band. The specific calculation method of the detection probability and false alarm probability of sub-band c is: Among them, η c is the detection threshold, N is the number of secondary base station antennas, τ is the sensing time, f s is the sampling frequency, γ c is the received signal-to-noise ratio of the signal sent by the primary base station to the secondary base station in sub-band c, is the target detection probability, Q(·) is the right tail function of the standard normal distribution, By using IRS, the signal strength of the primary base station has negligible impact on the system detection threshold, and its detection threshold is: in, is the maximum tolerable false alarm probability of the system, and δ is a constant close to zero, which is used to ensure that the false alarm probability does not exceed The multi-IRS-assisted broadband CR system divides the frequency band into C sub-bands and senses each sub-band separately. The sensing results are divided into two cases, namely: Step 1-1: sub-band c is in an idle state; The secondary base station can only receive noise signals, i.e. where n c (t) means the mean is 0 and the variance is σ c 2 Additive Gaussian white noise, at this time, the secondary network can use sub-band c for signal transmission; Step 1-2: Sub-band c is in a busy state; The secondary base station receives a joint fading signal composed of the direct signal, the reflected signal and the additive Gaussian white noise after channel fading, that is, in denote the channels from the primary base station to the secondary base station, from the primary base station to the r-th IRS, and from the r-th IRS to the secondary base station in the sub-spectrum c, respectively. represents the reflection coefficient of the rth IRS in the perception stage, and Respectively represent the mth in the rth IRS r The amplitude and phase shift of the reflected element, where Assume that the reflection amplitude is in an ideal state, that is, represents the state of sub-spectrum c, Indicates that sub-spectrum c is in idle state, Indicates that sub-spectrum c is in a busy state, P q represents the transmission power of the primary base station, and x(t) represents the complex baseband signal sent by the primary base station. At this time, the secondary network can no longer use the sub-spectrum c for signal transmission; Step 2: Secondary network signal transmission; When an idle sub-band is detected, the secondary network uses it for information transmission and determines the achievable rate of information transmission in the band. Subsequently, the total achievable rate of information transmission by the secondary network using all available idle sub-bands is calculated, including: When it is detected that the sub-band is in an idle state, the secondary network can use the sub-band for signal transmission. Assuming that the sub-band c is in an idle state, the received signal transmitted by the secondary base station using the sub-band c to the secondary user k is: in, denote the channels from the secondary base station to the kth secondary user, from the rth IRS to the kth user, and from the secondary base station to the rth IRS in sub-band c, respectively. represents the reflection coefficient of the rth IRS during the transmission phase, is the beamforming vector for the kth secondary user in subband c, is the complex baseband transmission signal sent by the secondary base station to the kth secondary user in sub-band c, n k The mean is 0 and the variance is Additive Gaussian white noise, Indicates the selection of IRS, indicates that the rth IRS is used when transmitting the signal to the kth secondary user, This means that the rth IRS is not used when transmitting signals to the kth secondary user; In addition, there is a situation where the primary network is using sub-band c but the secondary base station misses detection. In this case, the received signal transmitted by the secondary base station to the secondary user k using sub-band c is: in, is the interference of the primary network to the secondary user k, and its interference signal is: in, denote the channels from the primary base station to the kth secondary user and from the primary base station to the rth IRS, respectively. According to the above two situations, their signal-to-interference-noise ratios are: The probabilities of the above two situations occurring are: So in subband c, the achievable rate of the kth secondary user is: Among them, T is the total time occupied by the perception stage and the transmission stage, τ is the time occupied by the perception stage, denote the probability that sub-band c is in idle state and busy state respectively, The total achievable rate for all secondary users is: Step 3: Establish the framework of the secondary network achievable rate maximization problem and list the objectives and constraints that need to be optimized, including: The channel allocation, IRS selection, sensing time, secondary base station beamforming and IRS phase shift are jointly optimized. The problem framework is: C6:0≤τ≤T, in, is the minimum detection probability, is the maximum tolerable false alarm probability, P max is the maximum transmit power of the secondary base station, is the minimum achievable rate of secondary user k, is the tolerable maximum interference power of the primary user q, C3 is the maximum transmission power constraint of the secondary base station at subcarrier frequency c, C4 is the minimum rate requirement of the secondary user, C5 is the maximum tolerable interference of the primary user, C6 is the perception time constraint, C7 and C8 are non-convex unit modulus constraints, C9 represents the channel occupancy, C10 indicates that each user can occupy at least one channel for signal transmission, C11 is the use of the intelligent reflective surface, and C12 indicates that at least one IRS is used when sending information from the secondary base station to the secondary user; Step 4: Establish a Markov process based on the proposed secondary network achievable rate maximization problem framework; Step 5: Use the SAC algorithm to optimize the model until the model converges; Step 6: Output the result and get the total achievable rate of the secondary network.

2. According to the method of claim 1, a multi-IRS-assisted broadband CR system resource optimization method based on deep reinforcement learning is characterized in that: The step 4 comprises: This discrete-continuous optimization problem is considered as a Markov decision process problem, with the multi-IRS-assisted broadband CR system as the environment and the central controller of the IRS as the agent; A Markov decision process problem also includes state space, action space, transition probability and reward function; State space: The state space should include all parameters that determine the action and the action in the previous time slot, that is, the channel state information of all communication links at time slot t, and the action in time slot t-1. Therefore, the state space s at time slot t (t) ={a (t-1) ,H (t) }; Action space: The action space a at time slot t (t) It consists of two parts, namely the discrete action space consisting of channel allocation, IRS selection and perception time. and the continuous action space consisting of the secondary base station beam profile and the reflection coefficient of the IRS Transition probability: Pr(s (t+1) |s (t) ,a (t) ) is the state transition probability, which represents the state in state s (t) Take action a (t) , the state changes to s (t+1) probability; Reward function: maximize the total reachability of secondary users while ensuring the minimum reachability of secondary users, and ensure that the interference to the primary user is less than the maximum tolerable interference, so the reward function is r = μ1R s +μ2δ k +μ3δ i , where μ1, μ2, μ3 are constants; Among them, off-policy is used, and the action space includes both discrete actions (τ, ρ, γ) and continuous actions (W, θ t ,θ s ), so a random strategy should be used; The reinforcement learning method adopted is the soft action-criteria (SAC) algorithm.

Images