IRS-assisted multi-user MISO cognitive communication energy efficiency optimization method based on continuous control

Abstract

The invention relates to an IRS-assisted multi-user MISO cognitive communication energy efficiency optimization method based on continuous control, and belongs to the field of cognitive communication energy efficiency optimization. The method comprises the following steps: (1) modeling an energy harvesting multi-user MISO cognitive communication system based on IRS assistance; and (2) proposing a PER-SD3 deep reinforcement learning algorithm, constructing an interaction process between the intelligent agent and the environment, and solving the problem of cognitive communication energy efficiency optimization in the IRS-assisted energy harvesting multi-user MISO cognitive communication system by the intelligent agent through continuous learning training. According to the method, the problem of cognitive communication energy efficiency optimization can be well solved.

Description

Technical Field

[0001] The present invention belongs to the field of cognitive communication energy efficiency optimization, and in particular relates to an IRS-assisted multi-user MISO cognitive communication energy efficiency optimization method based on continuous control. Background Art

[0002] The scarcity of spectrum resources caused by the increase in the number of wireless communication devices is considered to be one of the key challenges in wireless communications. CR (cognitive radio) technology is expected to be an effective solution to improve the spectrum efficiency of wireless communications, which allows PU to share spectrum with SU. With the maturity of theories and technologies related to digital metamaterials, IRS equipped with digital metamaterials can achieve multi-dimensional manipulation of electromagnetic waves, which is considered to be one of the key technologies for 5G-A and 6G. Cognitive communication devices use PU spectrum resources in Underlay mode. With the increase in the number of communication devices, the performance of the communication system depends not only on throughput, but also on energy efficiency indicators. Energy consumption is an important issue that needs to be considered in the communication system. This application considers the multi-antenna ST to collect RF energy from PT to assist in data transmission between multiple SRs, and studies the problem of maximizing the energy efficiency of cognitive communication systems by jointly controlling the transmit beamforming at the ST and the phase shift matrix at the IRS.

[0003] For the above optimization problems, there are complex couplings between optimization variables. It is difficult for traditional optimization algorithms to achieve global optimal solutions for such high-dimensional complex problems. Deep reinforcement learning has become an important technology for solving IRS-assisted wireless communications in recent years. Due to the continuous dynamic nature of the environment, the proposed communication model requires real-time continuous control of the ST beamforming matrix and the IRS phase shift matrix. Summary of the invention

[0004] The purpose of the present invention is to use a deep reinforcement learning algorithm to continuously interact with the environment to solve the optimization problem and achieve higher flexibility. Therefore, a continuously controlled IRS (Intelligent Reflecting Surface) assisted multi-user MISO (Multiple-Input Single-Output) cognitive communication energy efficiency optimization method is provided.

[0005] To achieve the above object, the technical solution of the present invention is: an IRS-assisted multi-user MISO cognitive communication energy efficiency optimization method based on continuous control, comprising:

[0006] (1) Modeling an IRS-assisted energy harvesting multi-user MISO cognitive communication system;

[0007] (2) A PER-SD3 deep reinforcement learning algorithm is proposed to construct the interaction process between the intelligent agent and the environment. Through continuous learning and training, the intelligent agent solves the problem of cognitive communication energy efficiency optimization in the IRS-assisted energy harvesting multi-user MISO cognitive communication system.

[0008] In one embodiment of the present invention, in step (1), the system model of the IRS-assisted energy harvesting multi-user MISO cognitive communication system is specifically constructed as follows:

[0009] The energy harvesting multi-user MISO cognitive communication system based on IRS assistance is composed of a primary user communication system and a cognitive communication system. The primary user communication system includes a primary user PU, which is composed of a primary user transmitter PT and a primary user receiver PR. The cognitive communication system is composed of a secondary user transmitter ST equipped with M transmitting antennas and K single antennas SR. The energy harvesting multi-user MISO cognitive communication system based on IRS assistance adopts an Underlay spectrum sharing mode, that is, the secondary user SU is allowed to share the PU spectrum resources without affecting the PU. Considering that there are obstacles blocking the ST to the K SRs, the direct signal transmission from the ST to each SR is assumed to be negligible. In order to overcome the scenario of severe signal blocking between the transmitter and the receiver, an intelligent reflector IRS is carried in the cognitive communication system to enhance the required signal power through the IRS. L passive reflective elements and a microcontroller are deployed on the IRS.

[0010] At ST, the transmitted signal for the kth SR is s T,k ,s T,k is an independent random variable that follows a Gaussian distribution and is determined by the transmit beamforming vector w T,k Therefore, the total transmitted signal X is expressed as

[0011]

[0012] Among them, w T,k ∈C M×1 W T The kth column vector, W T is the transmit beamforming matrix at ST, expressed as

[0013]

[0014] The maximum transmit power constraint at ST is expressed as

[0015]

[0016] The existence of IRS affects the PR receiving signal and K SRs. The signal transmitted from ST needs to be transmitted to IRS first, and then reflected to the corresponding user after being processed by IRS. The IRS reflection coefficient matrix is ​​used to model IRS. The IRS reflection coefficient matrix is ​​expressed as

[0017]

[0018] in, β l (θ l )∈(0,1) represents the amplitude coefficient of the lth reflection element of IRS, θ l ∈[0,2π) represents the phase shift coefficient of the lth reflection element of IRS; considering the IRS reflection amplitude coefficient model that is closer to reality, it is suitable for various semiconductor devices that realize IRS, β l (θ l ) is expressed as

[0019]

[0020] Among them, β min ∈[0,1], h≥0, and d≥0 are constants related to the IRS hardware implementation circuit.

[0021] In one embodiment of the present invention, in step (1), the channel model and signal model of the IRS-assisted energy harvesting multi-user MISO cognitive communication system are specifically constructed as follows:

[0022] The acquisition of CSI (Channel State Information) between IRS and users is a prerequisite for using IRS. Considering that part of the purpose of IRS design is to reduce the demand for channel information and reduce the design complexity, it is difficult to estimate the CSI between IRS and PR, ST, and K SRs. Regarding the acquisition of CSI, the passive pilot method is used, that is, the base station is used to estimate the CSI between IRS and the wireless communication link. It is assumed that the channel coefficients between ST and IRS, between ST and PR, between IRS and K SRs, between IRS and PR, and between PT and PR are

[0023] In practice, IRS is deployed close to ST and SR to mitigate the impact of buildings on communication by assisting communication transmission; therefore, assuming that there is a LoS path in the reflection link, G T,R 、h R,k 、u RP 、u SP For the Rice channel model, the channel coefficient G between ST and IRS is T,R , expressed as

[0024]

[0025] Among them, β rician Expressed as the Ricean factor of the ST-IRS link, Represents the LoS component, which is location-dependent and remains unchanged over a relatively long time range. represents the NLoS component, Obey the Rayleigh fading model;

[0026] Assume that the ST is equipped with a uniform linear array antenna along the direction (1, 0, 0) or the x-axis, and the IRS is a uniform linear array antenna placed along the direction (0, 1, 0) or the y-axis; the LoS channel between the ST and the IRS is expressed as

[0027]

[0028] The superscript H represents the conjugate transpose, v IRS With v ST They represent the steering vectors of IRS and ST, respectively.

[0029]

[0030]

[0031] α AoA With α AoD denote the signal arrival azimuth at IRS and the departure angle from IRS to ST respectively; the direction cosine of the arrival azimuth cos(α AoA ), direction cosine of the departure angle cos(α AoD ) is given by the following two formulas

[0032] cos(α AoA )=[0,1,0]e SR

[0033] cos(α AoD )=[1,0,0]e SR

[0034] Direction vector e SR The calculation method and Consistent, e SR The calculation of P requires obtaining the position coordinates of ST and IRS respectively, where P SR With P IRS They represent ST position and IRS position respectively;

[0035] Assume that there are obstacles between PT and SR, and between ST and SR, and the interference is assumed to be Gaussian noise; therefore, in the cognitive communication system, the signal received by the kth SR is expressed as

[0036]

[0037] in, represents the additive white Gaussian noise at the kth SR, and the noise power is In addition, for the kth SR, the signal-to-interference-noise ratio of its received signal is expressed as

[0038]

[0039] In the Underlay spectrum sharing mode, the interference temperature IT imposed by ST on PR is expressed as

[0040]

[0041] Therefore, the signal-to-interference-noise ratio at PR is expressed as

[0042]

[0043] Among them, p p is the transmission power. To ensure that the PU is not affected by the SU, the signal-to-interference-to-noise ratio received by the PR must meet the QoS requirements of the normal communication of the PU, that is, γ P ≥γ P,min , γ P,min It is the minimum signal to interference and noise ratio threshold of PU.

[0044] In one embodiment of the present invention, in step (1), the energy harvesting and energy efficiency modeling of the IRS-assisted energy harvesting multi-user MISO cognitive communication system is specifically constructed as follows:

[0045] In order to improve the energy utilization rate of cognitive communication systems and reduce the energy consumption of batteries and power grids, it is considered that ST is equipped with an energy harvesting device to collect the RF energy radiated by PT and use an energy conversion device to convert it into electrical energy to support data transmission between ST and K SRs.

[0046] Each time slot is divided into two stages: energy collection stage and data transmission stage. In the energy collection stage, ST collects RF energy from PT and converts it into electrical energy. After the energy collection stage, ST uses the collected energy to assist in data transmission in the corresponding time slot. In a time slot with a time length of τ(t), the energy collected by ST is expressed as

[0047] E eh =ατ(t)ηp P g PS

[0048] Among them, α represents the energy collection time factor, η is the energy conversion efficiency, g PS represents the energy collection gain. The collected energy is used to compensate for part of the energy consumption of the cognitive communication system. In a time slot, the energy consumption of the cognitive communication system consists of the energy consumption of the ST transmission power, the power consumption of K SRs and IRSs, and the energy consumption of the cognitive communication system hardware heating. The energy consumed by SU sensing PU spectrum resources is ignored. The energy consumption of the cognitive communication system is expressed as

[0049] P total =P ST +P IRS +P SR +P c -E eh

[0050] Among them, P ST Indicates the energy consumed by ST transmission power; P IRS It represents the power consumption of IRS, which depends on the type and resolution of IRS reflective elements. The total power consumption of L reflective elements on IRS is Lp l (b), p l (b) represents the power consumption of each reflective element with b-bit resolution; P SR represents the total power consumption of K SRs; P c represents the energy consumption of the hardware heating of the cognitive communication system; therefore, the total power consumption of the cognitive communication system is given by

[0051]

[0052] K SRs can achieve a system communication rate of

[0053]

[0054] The energy efficiency of the system is defined as the ratio of the total achievable system communication rate to the total power consumption. Therefore, the energy efficiency of the considered IRS-assisted energy harvesting multi-user MISO cognitive communication system with K SRs is formulated as

[0055]

[0056] Assuming that in the energy harvesting stage, the energy collected by RF energy harvesting can meet the circuit consumption of the cognitive communication system itself, there are the following energy constraints:

[0057] E eh =ατ(t)ηp P g PS ≥p c τ(t)

[0058] From the above formula, we can conclude that

[0059]

[0060] Assuming that during the entire time slot, the energy collected by energy harvesting cannot fully compensate for the energy consumption of the entire cognitive communication system, there are the following energy constraints

[0061]

[0062] From the above formula, we can conclude that

[0063]

[0064] Therefore, considering the energy harvesting condition, the constraints that the energy harvesting factor must satisfy are:

[0065]

[0066] In order to consider the impact of energy harvesting on system energy efficiency, the first-order derivative of the energy harvesting time factor in the energy efficiency formula is obtained as follows:

[0067]

[0068] From the above formula, we can get ηp P g PS -p c ≥0, that is The numerator of is non-negative, so the energy efficiency is an increasing function of the energy collection time factor α.

[0069] In one embodiment of the present invention, in step (2), the goal of the cognitive communication energy efficiency optimization problem in the IRS-assisted energy harvesting multi-user MISO cognitive communication system is to jointly control the beamforming vector w at ST. T,k and the reflection coefficient matrix Φ at the IRS to maximize the energy efficiency of the cognitive communication system while ensuring the normal communication QoS requirements of the PU. The optimization problem is formulated as

[0070]

[0071]

[0072] Where constraint C1 represents the maximum transmit power p of beamforming at ST max Constraints; Constraint C2 represents the value restriction of the energy collection time factor α of ST in the time slot; Constraint C3 represents the normal communication QoS requirement of PU. In the Underlay spectrum sharing model, the signal to noise ratio at PU must be higher than the minimum threshold γ P,min; Constraint C4 represents the phase shift range that the L reflective elements on the IRS can provide; The above formula is a non-convex optimization problem.

[0073] In one embodiment of the present invention, in step (2), the PER-SD3 deep reinforcement learning algorithm consists of the SD3 algorithm and the PER algorithm.

[0074] In one embodiment of the present invention, the SD3 algorithm is an improvement of the deep deterministic policy gradient DDPG algorithm. The SD3 algorithm integrates 8 neural networks, including two actor networks, two critic networks, two target actor networks and two target critic networks; when two actor networks are applied, the agent selects the action with the largest Q-value function among the two output actions; the two actor networks and the two critic networks need to be trained with parameters, while the two target networks are soft-updated by copying the weights of the two actor networks and the two critic networks.

[0075] In one embodiment of the present invention, during the training phase of the PER-SD3 deep reinforcement learning algorithm, the PER algorithm uses a priority queue to store experience samples, adopts a SumTree-based data structure, and selects samples based on priority.

[0076] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0077] Compared with the prior art, the present invention has the following beneficial effects:

[0078] (1) Modeling a multi-user MISO cognitive communication system based on IRS-assisted energy harvesting. Considering the design cost, SRs are single antennas. IRS expands the communication coverage by adjusting and enhancing the signal propagation path. ST harvests the RF energy radiated by PT and provides a convenient solution for powering wireless devices.

[0079] (2) To address the energy efficiency optimization problem of cognitive communication, we innovatively proposed the PER-SD3 deep reinforcement learning algorithm. This algorithm constructs the interaction process between the agent and the environment. Through continuous learning and training, the agent can eventually solve the energy efficiency optimization problem well, and the effect is the best compared with other related algorithms.

[0080] (3) Simulation results show that the model proposed in this chapter has important application prospects. The introduction of IRS and RF energy harvesting technology can effectively improve the energy efficiency of cognitive communication systems; the proposed PER-SD3 algorithm shows superiority compared with other methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1Schematic diagram of the IRS-assisted energy harvesting multi-user MISO cognitive communication system model of the present invention.

[0082] Figure 2 It is a time slot model.

[0083] Figure 3 It is the interaction process between the intelligent agent and the environment.

[0084] Figure 4 This is a schematic diagram of the SumTree data structure.

[0085] Figure 5 It is the PER-SD3 algorithm framework.

[0086] Figure 6 Setup for a multi-user MISO cognitive communication system for IRS-assisted energy harvesting.

[0087] Figure 7 The energy efficiency curves of different algorithms vary with time steps.

[0088] Figure 8 This is the curve of energy efficiency changing with energy collection time factor under different algorithms.

[0089] Fig. 9 The energy efficiency curves under different algorithms vary with PT transmission power.

[0090] Fig.10 The energy efficiency curves under different algorithms vary with PU signal-to-interference-noise ratio.

[0091] Fig.11 This is the impact curve of the number of IRS reflective elements on energy efficiency under different algorithms.

[0092] Fig.12 The system throughput under different algorithms varies with the ST maximum transmit power constraint p max Change curve.

[0093] Fig.13 The system energy efficiency under different algorithms varies with the ST maximum transmit power constraint p max Change curve.

[0094] Fig.14 The effect of different learning rates of the agent on the average reward.

[0095] Fig.15 The effect of different decay rates on the average reward of the agent. DETAILED DESCRIPTION

[0096] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0097] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0098] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0099] The present invention provides an IRS-assisted multi-user MISO cognitive communication energy efficiency optimization method based on continuous control, comprising:

[0100] (1) Modeling an IRS-assisted energy harvesting multi-user MISO cognitive communication system;

[0101] (2) A PER-SD3 deep reinforcement learning algorithm is proposed to construct the interaction process between the intelligent agent and the environment. Through continuous learning and training, the intelligent agent solves the problem of cognitive communication energy efficiency optimization in the IRS-assisted energy harvesting multi-user MISO cognitive communication system.

[0102] The following is the specific implementation process of the present invention.

[0103] 1. IRS-assisted multi-user MISO cognitive communication system model based on continuous control algorithm

[0104] 1.1 System Model

[0105] like Figure 1 As shown in the figure, an energy harvesting multi-user MISO cognitive communication system assisted by an IRS is considered. The PU consists of a PT and a PR; in the cognitive communication system, it consists of a ST equipped with M transmitting antennas and K single-antenna SRs. The Underlay spectrum sharing mode is adopted. Specifically, the SU is allowed to share the PU spectrum resources without affecting the PU. Considering that there are obstacles blocking the ST to the K SRs, the direct signal transmission from the ST to the SR is assumed to be negligible. In order to overcome the scenario of severe signal blocking between the transmitter and the receiver, an IRS is carried in the cognitive communication system to enhance the required signal power through the IRS. L passive reflective elements and a microcontroller are deployed on the IRS.

[0106] At the multi-antenna ST, the transmitted signal for the kth SR is s T,k ,s T,kis an independent random variable that follows a Gaussian distribution. It is determined by the transmit beamforming vector w T,k Therefore, the total transmitted signal X can be expressed as

[0107]

[0108] in, W T The kth column vector, W T is the transmit beamforming matrix at ST, expressed as

[0109]

[0110] The maximum transmit power constraint at ST is expressed as

[0111]

[0112] The existence of IRS affects the PR received signal and the K SRs. The signal transmitted from the ST needs to be transmitted to the IRS first, and then reflected to the corresponding user after being processed by the IRS. The IRS reflection coefficient matrix is ​​used to model the IRS. The IRS reflection coefficient matrix is ​​expressed as

[0113]

[0114] in, β l (θ l )∈(0,1) represents the amplitude coefficient of the lth reflection element of IRS, θ l ∈[0,2π) represents the phase shift coefficient of the lth reflection element of IRS. Different from other literatures that directly consider the research of IRS under ideal amplitude coefficient, the present invention considers a more practical IRS reflection amplitude coefficient model, which is generally applicable to various semiconductor devices for realizing IRS. Specifically, β l (θ l ) is expressed as

[0115]

[0116] Among them, β min ∈[0,1], h≥0, and d≥0 are constants related to the IRS hardware implementation circuit. In practice, after the IRS hardware circuit design is completed, the parameters can be easily found by standard curve fitting tools. It is assumed that the hardware circuit of each reflector element on the IRS is consistent. Therefore, the above constants are applicable to each reflector element of the IRS.

[0117] 1.2 Channel Model and Signal Model

[0118] The acquisition of CSI between IRS and users is a prerequisite for using IRS. Considering that part of the purpose of IRS design is to reduce the demand for channel information and reduce the design complexity. Therefore, the CSI between IRS and PR, ST and K SRs is difficult to estimate. Regarding the acquisition of CSI, existing research includes using the base station to estimate the CSI between IRS and the wireless communication link through a passive pilot method. Assume that the channel coefficients between ST and IRS, between ST and PR, between IRS and K SRs, between IRS and PR, and between PT and PR are

[0119] In fact, the IRS is deployed close to the ST and SR to mitigate the impact of the building on communication by assisting communication transmission. Therefore, it can be assumed that there is a LoS path in the reflection link. Assuming G T,R 、h R,k 、u RP 、u SP is the Rice channel model, with the channel G between ST and IRS T,R For example, it is expressed as

[0120]

[0121] Among them, β rician Expressed as the Ricean factor of the ST-IRS link, Represents the LoS component, which is location-dependent and remains unchanged over a relatively long time range. represents the NLoS component, It obeys the Rayleigh fading model.

[0122] Assume that the ST is equipped with a uniform linear array antenna along the direction (1, 0, 0) (i.e., the x-axis), and the IRS is a uniform linear array antenna placed along the direction (0, 1, 0) (i.e., the y-axis). The LoS channel between the ST and the IRS is expressed as

[0123]

[0124] Among them, v IRS With v ST They represent the steering vectors of IRS and ST, respectively.

[0125]

[0126]

[0127] The relevant parameter settings are the same as those in Chapter 3. AoA With α AoDThe direction cosine of the arrival azimuth of the signal at the IRS and the angle of departure (AoD) from the IRS to the ST are cos(α AoA ), direction cosine of the departure angle cos(α AoD ) is given by the following two formulas

[0128] cos(α AoA )=[0,1,0]e SR

[0129] cos(α AoD )=[1,0,0]e SR

[0130] Direction vector e SR The calculation method and Consistent, e SR The calculation of P requires obtaining the position coordinates of ST and IRS respectively, where P SR With P IRS They represent ST position and IRS position respectively;

[0131] Assume that there are obstacles between PT and SR, and between ST and SR, and the interference is assumed to be Gaussian noise. Therefore, in the cognitive communication system, the signal received by the kth SR is expressed as

[0132]

[0133] in, represents the additive white Gaussian noise at the kth SR, and the noise power is In addition, for the kth SR, the signal-to-interference-noise ratio of its received signal is expressed as

[0134]

[0135] In the Underlay spectrum sharing mode, the interference temperature (IT) imposed by ST on PR is expressed as

[0136]

[0137] Therefore, the signal-to-interference-noise ratio at PR is expressed as

[0138]

[0139] Among them, p p is the transmission power. To ensure that the PU is not affected by the SU, the signal-to-interference-noise ratio received by the PR must meet the normal communication QoS requirements of the PU, that is, γ P ≥γ P,min , γ P,min It is the minimum signal to interference and noise ratio threshold of PU.

[0140] 1.3 Energy harvesting and energy efficiency modeling

[0141] In order to improve the energy utilization of cognitive communication systems and reduce the energy consumption of batteries and power grids, it is considered that ST is equipped with an energy harvesting device. By collecting the RF energy radiated by PT, an energy conversion device is used to convert it into electrical energy to support data transmission from ST to K SRs.

[0142] like Figure 2 As shown in Figure 1, each time slot can be divided into two phases: energy collection phase and data transmission phase. The present invention considers the collection-use model. In the energy collection phase, the ST collects the RF energy of the PT and converts it into electrical energy. After the energy collection phase, the ST uses the collected energy to assist in the data transmission of the time slot. In this work, a linear energy collection model is adopted. In a time slot with a time length of τ(t), the energy collected by the ST is expressed as

[0143] E eh =ατ(t)ηp P g PS

[0144] Among them, α represents the energy collection time factor, η is the energy conversion efficiency, g PS Represents the energy harvesting gain. The harvested energy is used to compensate for part of the energy consumption of the cognitive communication system. In a time slot, the energy consumption of the cognitive communication system is composed of the energy consumption of the ST transmission power, the power consumption of K SRs and IRSs, the heat generation of the cognitive communication system hardware, etc., and the energy consumed by the SU sensing the PU spectrum resources is ignored. It should be emphasized that the electronic components used in the IRS are usually designed with low power consumption, and the IRS is a passive device. Compared with traditional wireless devices such as base stations or mobile devices, the IRS does not need to transmit and receive RF signals. Considering the above factors, the energy consumption of the cognitive communication system is expressed as

[0145] P total =P ST +P IRS +P SR +P c -E eh

[0146] Among them, P ST Indicates the energy consumed by ST transmission power, P IRS represents the power consumption of the IRS, which depends on the type and resolution of the IRS reflective elements that modulate the transmitted electromagnetic signal. For 3-bit, 4-bit, 5-bit and 6-bit resolution phase shift, the typical power consumption of each element is 1.5, 4.5, 6 and 7.8 mW respectively. Therefore, the total power consumption of the L reflective elements on the IRS is Lpl (b), p l (b) represents the power consumption of each reflective element with b-bit resolution. SR P represents the total power consumption of K SRs. c represents the physical consumption of the cognitive communication system such as hardware heating. Therefore, the total power consumption of the cognitive communication system can be given by the following formula:

[0147]

[0148] The signal-to-interference-noise ratio γ at the kth SR is given above k Therefore, K SRs can achieve a system communication rate of

[0149]

[0150] The energy efficiency of the system is defined as the ratio of the total achievable rate to the total power consumption. Therefore, the energy efficiency of the considered IRS-assisted energy harvesting multi-user MISO cognitive communication system with K SRs is formulated as

[0151]

[0152] Assuming that in the energy harvesting stage, the energy collected by RF energy harvesting can meet the circuit consumption of the cognitive communication system itself, the following energy constraints apply:

[0153] E eh =ατ(t)ηp P g PS ≥p c τ(t)

[0154] From the above formula, we can conclude that

[0155]

[0156] Assuming that during the entire time slot, the energy collected by energy harvesting cannot fully compensate for the energy consumption of the entire cognitive communication system, the following energy constraints are imposed:

[0157]

[0158] From the above formula, we can conclude that

[0159]

[0160] Therefore, considering the energy harvesting condition, the constraints that the energy harvesting factor must satisfy are:

[0161]

[0162] To consider the impact of energy harvesting on system energy efficiency, the following derivation is made. Taking the first-order derivative of the energy harvesting time factor in the above energy efficiency formula, we get the following formula:

[0163]

[0164] From the above formula, we can get ηp P g PS -p c ≥0, that is The numerator of is non-negative, so the energy efficiency is an increasing function of the energy collection time factor α.

[0165] 1.4. Optimization Problem Formulation

[0166] The present invention aims to jointly control the beamforming vector w at ST T,k and the reflection coefficient matrix Φ at the IRS to maximize the energy efficiency of the cognitive communication system while ensuring the normal communication QoS requirements of the PU. The complete optimization problem is formulated as

[0167]

[0168]

[0169] Where constraint C1 represents the maximum transmit power p of beamforming at ST max Constraints; Constraint C2 represents the value restriction of the energy collection time factor α of ST in the time slot; Constraint C3 represents the normal communication QoS requirement of PU. In the Underlay spectrum sharing model, the signal to noise ratio at PU must be higher than the minimum threshold γ P,min ; Constraint C4 represents the phase shift range that can be provided by the L reflective elements on the IRS.

[0170] It can be seen that the above formula is a non-convex optimization problem. Since the control variable w T,k and Φ are complexly coupled, which means it is difficult to solve effectively using standard convex optimization methods. To this end, an algorithm based on deep reinforcement learning is proposed to deal with this problem.

[0171] 2. Energy efficiency optimization of IRS-assisted multi-user MISO cognitive communication system based on deep reinforcement learning

[0172] 2.1. Optimization Problem Transformation Based on Deep Reinforcement Learning

[0173] Mathematically modeling the interaction between the environment and the agent is a prerequisite for using deep reinforcement learning. Figure 3is the interaction process between the agent and the environment. Deep reinforcement learning can be described by an MDP defined as a five-tuple (S, A, R, P, γ). Among them, S represents the state space of the environment, A represents the action space of the agent, R represents the transition probability between states of the reward function, P represents the transition probability between states, and γ represents the discount factor. The following analyzes and designs the state space, action space, reward function, and state transition probability, which determine the characteristics of deep reinforcement learning problems and the effectiveness of solutions.

[0174] State space: state space s t ∈S defines the set of environmental states that the agent can observe. The environmental states include the CSI, PU, ​​SU and IRS deployment positions, the phase shifts of all IRS reflective elements, the PT transmit power, and the ST transmit beamforming matrix. Therefore, the observed state at any time step is expressed as

[0175] s t = {G T,R ,h R,k ,u RP ,u SP ,W T ,Φ}

[0176] In the above formula, the state will be input into the neural network of the deep reinforcement learning algorithm. For complex input scenarios, the real part and the imaginary part need to be split and used as input respectively. T,R 、h R,k The number of state variables is 2LM and 2LK respectively, and the cognitive communication device interferes with link u RP with u SP The total number of state variables is 2(M+L), and the sum of the number of states of the IRS reflection phase shift and ST transmit beamforming matrix is ​​2MK+L. In summary, the dimension of the state space S is expressed as 2LM+2LK+2(M+L)+2MK+L.

[0177] Action space: Action space a t ∈A represents the set of actions that the agent can perform. The agent selects an action based on the current state to affect the state transition of the environment. At the tth time step, the agent action space consists of two components, including the transmit beamforming matrix W of the ST T and the phase Φ of all reflective elements of the IRS. Note in particular that the action chosen by the agent on the IRS is the phase shift of each reflective element.

[0178]

[0179] Reward function: Maximizing the energy efficiency of the IRS-assisted energy harvesting multi-user MISO cognitive communication system requires considering all constraints and paying attention to the minimum signal-to-interference-to-noise ratio requirement of PU communication defined in constraint C3. Based on the above analysis, the reward function is set as

[0180]

[0181] in

[0182]

[0183] The reward function consists of two parts. The first term represents the system energy efficiency; the second term is the cost function, which is defined as the transmission cost of violating the minimum communication QoS of the PU. λ is the corresponding coefficient. The reward function imposes a level of protection on the PU communication. When the PU is guaranteed to communicate normally, the cost is zero (i.e., SINR out =0); otherwise, SINR out =1.

[0184] The goal of the agent is to iteratively interact with the environment to maximize the cumulative reward according to the defined task and reward function.

[0185] The cumulative reward is expressed as

[0186]

[0187] Among them, γ is the discount factor. By introducing the discount factor, the agent can weigh the relationship between short-term behavior and long-term goals.

[0188] State transition probability: The state transition probability describes the probability of change of the environment state after the agent performs an action; the state transition probability function of the agent after performing an action and the reward function obtained under the corresponding action are related to the specific action, describing the probability of transitioning from one state to another and the immediate reward obtained; by defining a strategy in the MDP, the agent's actions in a certain state are guided; the strategy can be deterministic, that is, a specific action is specified for each state; it can also be random, that is, an action is selected according to the probability distribution in each state.

[0189] 2.2、Design based on PER-SD3 algorithm framework

[0190] In actual situations, the environment is in continuous change, and the actions performed by the agent are difficult to be discretely divided. In order to solve this problem, the present invention innovatively proposes a deep reinforcement learning algorithm based on PER-SD3 to achieve continuous control of ST transmit beamforming and IRS reflector element phase shift. The algorithm consists of two parts: SD3 and PER algorithms.

[0191] The SD3 algorithm is an improvement of the Deep Deterministic Policy Gradient (DDPG) algorithm. The characteristic of the DDPG algorithm is that it directly outputs the action value without matching the corresponding action set through the Q value. This enables the DDPG algorithm to perform fine control and adjustment in the continuous action space. DDPG includes four neural networks, namely the actor network, the critic network, the target actor network, and the target critic network. The agent performs a t Then get the new state s t+1 And get reward t , the DDPG algorithm uses the experience replay buffer to store and reuse previous experience samples. At each time step, the algorithm will replay the current sample (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool. During the training process, a batch of experience samples are randomly sampled to update the parameters of the policy and critic network. Therefore, it is necessary to use the critic network and the target critic network to estimate the actual target value. The actual target Q value is as follows

[0192] y t =r t +γQ'(s t+1 ,a t+1 ,w c2 )

[0193] Q'(s t+1 ,a t+1 ,w c2 ) indicates that the agent is in state s t+1 Next select action a t+1 The optimal Q value obtained. The process of updating the critic network parameters is expressed as

[0194]

[0195] Among them, μ c is the learning rate of the critic network, and the critic network parameter w c1 The gradient value can be expressed as

[0196]

[0197] Among them, N b Represents the size of the training batch. The goal of training the critic network is to accurately estimate the value function of the state-action pair by continuously adjusting the network parameters to guide the selection and optimization of the strategy. The training critic network is updated according to the value function, while the training policy network is updated as follows

[0198]

[0199] Among them, μ a and w a1 They are the learning rate and weight parameters of the policy network. The update of the policy network is achieved by reducing the policy gradient of the agent. The expression is as follows

[0200]

[0201] The above formula includes the target critic network relative to the target action a t The gradient of the training decision network with respect to the parameter w a1 The parameters of the target network are updated by soft updating, so that the parameters of the target network gradually approach the parameters of the current network at each update, thereby improving the stability and convergence of the algorithm. The expression is as follows

[0202]

[0203]

[0204] in, is the soft update factor, which is set between 0 and 1. The soft update method slows down the change of the target network, ensures slow parameter updates, stabilizes the learning process of the Q value network, and speeds up the convergence of the algorithm.

[0205] The DDPG algorithm has certain advantages in continuous control, but it overestimates the Q value and has a negative impact on the function approximation of the neural network. Therefore, the present invention adopts the more advanced SD3 algorithm.

[0206] The algorithm framework of SD3 integrates 8 neural networks, including two actor networks, two critic networks, two target actor networks and two target critic networks. When two actor networks are applied, the agent selects the action with the largest Q value function among the two output actions. The actor and critic networks need to be trained with parameters, while the target network is soft-updated by copying the weights of the actor and critic networks. To address the overestimation problem, the SD3 algorithm uses a clipped dual Q learning method to obtain the minimum estimate between the two target critic networks, as shown below

[0207] y1,y2=r t +γmin i＝1,2 Q′ i (s t+1 ,a t+1 ,w c2i )

[0208] Among them, Q′ i (s t+1 ,a t+1 ,w c2i) represents the optimal Q value of the two target critic networks. SD3 uses the Softmax operator to reduce any overestimation and underestimation bias in continuous control. The Softmax operator is defined as follows

[0209]

[0210] Where β is the parameter of the Softmax operator. In the continuous action space, it is difficult to use the Softmax operator to process, because the Softmax operator involves integral operations. To solve this problem, the Q function of the Softmax operation is expressed as the expected form through Importance Sampling (IS), and an unbiased estimate is obtained by the following method, which is expressed as

[0211]

[0212] In the above formula, P(a t+1 ) represents the probability density function that obeys the Gaussian distribution.

[0213]

[0214] In the above formula, Q′1(s t+1 ,a t+1 ,w c21 ) and Q′2(s t+1 ,a t+1 ,w c22 ) represents a value with weight w c21 and w c22 The output Q value of the target critic network. N t represents the noise that follows the normal distribution, which is clipped to the interval [-c, c] so that the sampling action a t+1 The sampled actions are limited to a certain range to ensure that the action is close to the original action.

[0215] a t+1 =[-c+π(s t+1 ,w a2i ),c+π(s t+1 ,w a2i )]

[0216] Among them, w a2i is the weight of the target actor network. For the critic network, SD3 uses cropped double Q learning combined with the Softmax operator to estimate the Q value. That is, the estimated value of the critic network is expressed as follows

[0217] y i =r t +γΓ SD3 (s t+1 )

[0218] Among them, Γ SD3 (s t+1 ) represents the Softmax operator of SD3 in the continuous action space, expressed as

[0219]

[0220] In deep reinforcement learning, experience samples are generated as the agent interacts with the environment sequentially. These samples are not independent and identically distributed. In order to break the association between samples, we use a size of D b The experience replay pool stores experience samples (s t ,a t ,r t ,s t+1 ), and uniformly sample mini-batch (size is N b ) updates the network parameters of the actor and critic at each time step. Traditional deep reinforcement learning trains neural network parameters by randomly sampling batches of data from the experience replay pool to train the parameters. The temporal difference error (TD-error) in deep reinforcement learning can be defined as

[0221] δ j =r t +γQ'(s j+1 ,a j+1 ,w c2 )-Q(s j ,a j ,w c1 )

[0222] The goal of deep reinforcement learning training is to minimize the expected TD-error. Experience that increases TD-error is of high importance to the training network. To this end, priority experience replay introduces a priority mechanism to solve this problem. Each experience sample is assigned a priority. The priority can be calculated based on the importance index of the error size of the experience sample. Therefore, TD-error is used to measure the value of experience.

[0223] To alleviate the overfitting problem in training and ensure that even experiences with TD-error of 0 can be effectively extracted, we can use probabilistic experience sampling. The sampling probability of each experience is defined as

[0224]

[0225] Among them, σ is used to adjust the priority. Note that when σ = 0, it degenerates into random sampling. The empirical priority is defined as

[0226] P j =|δj |+ε

[0227] Among them, ε is a constant used to prevent the probability from being 0 and ensure that even samples with lower priority have a certain chance to be selected and learned.

[0228] In order to avoid the need to sort the priorities P(j) of all samples each time sampling, this chapter adopts a sorting algorithm based on the SumTree data structure, such as Figure 4 As shown in the figure, each leaf node stores the priority of a sample, while non-leaf nodes store the sum of the values ​​of two child nodes. Therefore, the root node of SumTree is the sum of all priorities.

[0229] Using SumTree enables efficient sampling without sorting the priority P(j) of all samples. However, the high frequency experience replay of TD-error experience inevitably has a negative impact on the network training process, and the critic network parameter w c1 The gradient value formula is differentiated as follows

[0230]

[0231] From the above formula, we can see that the frequent replay of high TD-error experience causes the parameters of the critic network to change in the direction of the maximum gradient, making the network training process prone to oscillation or even divergence. To eliminate the impact of such changes, we set the critic network parameter w c1 IS weight is added to the calculation of

[0232]

[0233] Among them, N b represents the size of the experience replay pool, P(j) is the probability of sampling experience j, and the parameter ξ∈[0,1] controls the degree of correction and determines the impact of PER on the convergence of the training process. So we have

[0234]

[0235] w c1 The parameter update is expressed as

[0236]

[0237] In the PER-SD3 algorithm, gradient ascent is used to update the parameters of the two actor networks, gradient descent is used to update the parameters of the two critic networks, and the target actor network and target critic network are updated by soft update. The parameter update is similar to DDPG and will not be repeated here. Figure 5 It is the PER-SD3 algorithm framework. The specific algorithm flow is shown in Table 1.

[0238] Table 1 IRS-assisted energy harvesting multi-user MISO cognitive communication energy efficiency optimization

[0239]

[0240]

[0241]

[0242] 2.3 Algorithm Complexity Analysis

[0243] In the training phase, the PER algorithm uses a priority queue to store experience samples, adopts a SumTree-based data structure, and selects samples according to their priority. The time complexity of building a priority queue is O(|D b |log2(|D b |)), where D b is the size of the experience replay pool. The SD3 algorithm consists of a finite number of actor networks and critic networks. Each network can be regarded as a component of a DNN. Assume that L n and n i are the number of DNN layers and the number of neurons in the i-th layer. For one of the DNNs, the computational complexity can be expressed as

[0244]

[0245] Therefore, the computational complexity of the PER-SD3 algorithm training process can be expressed as

[0246]

[0247] Among them, N ep With T s Represent the number of algorithm iterations and the total time step of each iteration respectively. After the algorithm training converges, using the trained model parameters, only the actor network needs to be executed to realize the intelligent agent function. The computational complexity can be reduced to

[0248]

[0249] 3. Simulation results and discussion

[0250] Use simulation to evaluate the performance of deep reinforcement learning solutions. Figure 6As shown in the figure, the proposed system is intuitively represented by a two-dimensional coordinate system. In practice, the system is in a three-dimensional coordinate system, and the unit is set to meters (m). The positions of ST and IRS are (0, 0, 5) and (20, 30, 5), respectively. SRs includes K users, which are randomly distributed inside the right rectangular area of ​​10m×10m with a height of 1.5m. The position coordinates of PT and PR are (-20, 5, 10) and (-15, 20, 1.5). IRS hardware constant β min , h, and d are set to 0.6, 0, and 1.5 respectively. The maximum transmit power p at ST max The power varies from 5W to 20W, and the number of IRS reflective elements varies from 4 to 50. The simulation results are implemented with Python 3.9 and Anaconda 2021 distribution, using a learning framework based on torch 1.7.1.

[0251] Each neural network in the PER-SD3 algorithm framework is designed with four layers: input layer, two hidden layers, and output layer. The dimensions of the input and output layers correspond to their respective input and output dimensions. The model in this chapter sets the number of hidden layer neurons of the DNN to be the same. Adam is used as the optimizer for the actor network and the critic network, and tanh is used as the activation function. In addition, other simulation parameters are given in Table 2. Each iteration contains 20,000 time steps, and the simulation is the average result under 100 iterations.

[0252] Table 2 IRS-assisted energy harvesting multi-user MISO cognitive communication system simulation parameters

[0253]

[0254] The present invention will be considered for comparison with the following benchmark algorithm schemes.

[0255] (a) DDPG algorithm: This algorithm is the most commonly used continuous control method. DDPG uses the actor-critic network as the basic framework and has achieved good performance in many continuous control tasks.

[0256] (b) Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm: The TD3 algorithm introduces two sets of networks to estimate Q values ​​in the critic and target critic networks, which is an improved version of the DDPG algorithm. In addition, the TD3 algorithm also uses a delayed update strategy, that is, by delaying the update of the target critic network and the actor network, the convergence speed is accelerated and the stability of the algorithm is improved.

[0257] Figure 7 The energy efficiency changes of the three algorithms at different time steps are studied. Figure 7It can be seen that the energy efficiency optimization method based on the PER-SD3 algorithm keeps increasing in energy efficiency in the first 5000 time steps, and gradually converges and stabilizes at around 7500 steps, with the energy efficiency stabilized at around 0.7. The TD3 algorithm maintains a similar growth rate as the proposed algorithm in the first 4000 time steps, but does not maintain stable convergence after 5000 steps, and the convergence process is accompanied by a certain degree of fluctuation. The DDPG algorithm is significantly behind the above two. This is because the PER-SD3 and TD3 algorithms are improved and optimized based on the DDPG, and DDPG cannot reflect better performance and convergence. Figure 7 It can be calculated that the energy efficiency of the PER-SD3 algorithm is 20.7% higher than that of TD3 and 35.3% higher than that of DDPG. The simulation results show that the proposed energy efficiency optimization method can improve the energy efficiency of cognitive communication systems and is superior to other deep reinforcement learning algorithms.

[0258] Figure 8 The following is the change curve of the energy collection time factor on energy efficiency under different algorithms. It can be observed from the figure that the system energy efficiency increases with the increase of the energy collection time factor α. When α increases, the energy efficiency increases monotonically. A larger energy collection time factor allows the ST to collect the RF energy radiated by the PT for a longer time, reducing the dependence on the power supply equipment, thereby improving the energy efficiency of the SU. Fig. 9 The SU energy efficiency curve under different PT transmit powers shows that when the energy harvesting factor is fixed, the SU energy efficiency increases with the increase of PT transmit power. In addition, as the PT transmit power increases, the PU signal-to-interference-to-noise ratio increases, and the influence of the PU minimum signal-to-interference-to-noise ratio constraint on the ST's transmit beamforming and IRS reflection phase shift will be reduced, making the optimization of energy efficiency more flexible. Figure 8 , 9 It can be observed that the benchmark algorithms TD3 and DDPG have the same change trend as the PER-SD3 algorithm, but the energy efficiency optimization effect lags behind the proposed algorithm significantly.

[0259] Fig.10 The curves of energy efficiency changing with PU signal-to-noise ratio under different algorithms. It can be observed from the figure that energy efficiency gradually decreases with the increase of PU minimum signal-to-noise ratio threshold. Due to the increase of PU signal-to-noise ratio, the demand for communication QoS becomes more prominent, making the interference constraint control of SU on PU more stringent. When the PU interference constraint is small, the agent still controls the ST transmit beamforming and IRS reflection phase shift to maintain the optimal energy efficiency. However, when the PU interference constraint is large, the interference link from ST through IRS and then reflected to PR needs to be flexibly adjusted by the agent to balance the energy efficiency optimization and interference constraint of SU. It can also be seen from the figure that the proposed PER-SD3 algorithm significantly improves the protection level of energy efficiency under PU signal-to-noise ratio compared with other deep reinforcement learning algorithms.

[0260] Fig.11 The effect of the number of IRS reflective elements on energy efficiency under different algorithms was studied. It can be seen that the energy efficiency under the three algorithms increases with the increase of the number of IRS reflective elements L, which means that appropriately increasing the number of IRS reflective elements can improve the wireless communication environment. This performance improvement stems from the fact that by increasing the number of reflective elements L, higher power can be achieved at the IRS, and higher reflection beamforming gain can be achieved to design the IRS phase shift to improve the received desired signal and reduce signal interference to the PU. When the number of IRS reflective elements is above 40, the effect of increasing the number of reflective elements on energy efficiency gradually tends to stabilize, indicating that there is a certain upper limit to the improvement of energy efficiency by IRS. In addition, increasing the number of IRS reflective elements comes at the cost of increasing the complexity of the IRS. The more the number, the more difficult it is to train. Therefore, in practical applications, it is necessary to comprehensively consider various factors to find the optimal solution for the number of IRS reflective elements.

[0261] Fig.12 , 13 The ST maximum transmission power constraint p under different algorithms is studied respectively. max The impact on the throughput and energy efficiency of cognitive communication systems. Fig.12 It can be observed that the maximum transmission power of ST is p max As the maximum transmit power p increases, the throughput of the cognitive communication system will first increase, but as the maximum transmit power p max After increasing to 16W, the throughput growth rate slows down and tends to be stable. This can be explained as follows: although increasing the transmit power of the SU can improve the throughput, the increase in transmit power will interfere with the PR, causing the signal-to-interference-to-noise ratio of the PU to decrease. In the Underlay spectrum sharing mode, the PU's normal communication QoS requirements must be met, otherwise the deep reinforcement learning algorithm reward will be penalized by the cost function. Fig.13 It can be observed that the increase of ST maximum transmit power will lead to a decrease in energy efficiency. Although the increase of ST maximum transmit power will improve SU signal-to-interference-noise ratio and improve throughput within a certain range, higher transmit power will lead to an increase in energy consumption, thereby reducing energy efficiency. In summary, when designing and optimizing cognitive communication systems, it is necessary to comprehensively consider the balance between throughput and energy efficiency to achieve the best system performance.

[0262] The performance of deep reinforcement learning algorithms is closely related to the hyperparameter settings of the neural network, which need to be defined under a given system. In the PER-SD3 algorithm, the critic network and the actor network use a constant learning rate and decay rate. Fig.14The effects of different learning rates of the agent on the average reward are compared. It can be observed that different learning rates have a significant impact on the performance of the algorithm. The algorithm performs best when the learning rate is 0.001. When the learning rate is too small, the convergence speed of the algorithm becomes slow and the ideal performance cannot be achieved; when the learning rate is too large, the parameter update is too large, causing the model to oscillate or become unstable during training. Fig.15 The relationship between average reward and time step length under different decay rates is compared. It can be observed that the decay rate has little effect on the performance and convergence speed of the algorithm. Therefore, proper hyperparameter adjustment can help improve the performance of the algorithm.

[0263] The present invention aims to solve the energy efficiency optimization problem of IRS-assisted energy harvesting multi-user MISO cognitive communication. Taking into account the complexity and variability of the wireless communication environment, a deep reinforcement learning algorithm is adopted to enable the system to obtain good performance in a complex environment. In order to solve the energy efficiency optimization problem in a complex environment, the energy efficiency optimization problem is first modeled as an MDP, and the innovative PER-SD3 deep reinforcement learning algorithm is proposed to jointly control the transmit beamforming and IRS phase shift matrix of the ST. The intelligent agent continuously interacts with the environment and learns complex strategies to achieve the optimal system energy efficiency. The results show that compared with other benchmark algorithm schemes, the proposed algorithm is 20.7% higher than the TD3 algorithm and 35.3% higher than the DDPG algorithm in energy efficiency optimization. In addition, the performance sensitivity of the proposed algorithm model is analyzed through the simulation part to verify the optimal value of the hyperparameters.

[0264] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0265] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0266] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0267] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0268] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0269] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. An IRS-assisted multi-user MISO cognitive communication energy efficiency optimization method based on continuous control, characterized in that: include: (1) Modeling an IRS-assisted energy harvesting multi-user MISO cognitive communication system; (2) A PER-SD3 deep reinforcement learning algorithm is proposed to construct the interaction process between the intelligent agent and the environment. Through continuous learning and training, the intelligent agent solves the problem of cognitive communication energy efficiency optimization in the IRS-assisted energy harvesting multi-user MISO cognitive communication system.

2. According to claim 1, the IRS-assisted multi-user MISO cognitive communication energy efficiency optimization method based on continuous control is characterized in that: In step (1), the system model of the IRS-assisted energy harvesting multi-user MISO cognitive communication system is specifically constructed as follows: The energy harvesting multi-user MISO cognitive communication system based on IRS assistance is composed of a primary user communication system and a cognitive communication system. The primary user communication system includes a primary user PU, which is composed of a primary user transmitter PT and a primary user receiver PR. The cognitive communication system is composed of a secondary user transmitter ST equipped with M transmitting antennas and K single antennas SR. The energy harvesting multi-user MISO cognitive communication system based on IRS assistance adopts an Underlay spectrum sharing mode, that is, the secondary user SU is allowed to share the PU spectrum resources without affecting the PU. Considering that there are obstacles blocking the ST to the K SRs, the direct signal transmission from the ST to each SR is assumed to be negligible. In order to overcome the scenario of severe signal blocking between the transmitter and the receiver, an intelligent reflector IRS is carried in the cognitive communication system to enhance the required signal power through the IRS. L passive reflective elements and a microcontroller are deployed on the IRS. At ST, the transmitted signal for the kth SR is s T,k ,s T,k is an independent random variable that follows a Gaussian distribution and is determined by the transmit beamforming vector w T,k Therefore, the total transmitted signal X is expressed as Among them, w T,k ∈C M×1 W T The kth column vector, W T is the transmit beamforming matrix at ST, expressed as The maximum transmit power constraint at ST is expressed as The existence of IRS affects the PR receiving signal and K SRs. The signal transmitted from ST needs to be transmitted to IRS first, and then reflected to the corresponding user after being processed by IRS. The IRS reflection coefficient matrix is ​​used to model IRS. The IRS reflection coefficient matrix is ​​expressed as in, represents the amplitude coefficient of the lth reflection element of IRS, θ l ∈[0,2π) represents the phase shift coefficient of the lth reflection element of IRS; considering the IRS reflection amplitude coefficient model that is closer to reality, it is suitable for various semiconductor devices that realize IRS, β l (θ l ) is expressed as Among them, β min ∈[0,1], h≥0, and d≥0 are constants related to the IRS hardware implementation circuit.

3. The method for optimizing energy efficiency of IRS-assisted multi-user MISO cognitive communication based on continuous control according to claim 2, characterized in that: In step (1), the channel model and signal model of the IRS-assisted energy harvesting multi-user MISO cognitive communication system are specifically constructed as follows: The acquisition of channel state information (CSI) between IRS and users is a prerequisite for using IRS. Considering that part of the purpose of IRS design is to reduce the demand for channel information and reduce design complexity; Therefore, it is difficult to estimate the CSI between IRS and PR, ST, and K SRs. Regarding the acquisition of CSI, the passive pilot method is used, that is, the base station is used to estimate the CSI between IRS and the wireless communication link. It is assumed that the channel coefficients between ST and IRS, between ST and PR, between IRS and K SRs, between IRS and PR, and between PT and PR are In practice, IRS is deployed close to ST and SR to mitigate the impact of buildings on communication by assisting communication transmission; therefore, assuming that there is a LoS path in the reflection link, G T,R 、h R,k 、u RP 、u SP For the Rice channel model, the channel coefficient G between ST and IRS is T,R , expressed as Among them, β rician Expressed as the Ricean factor of the ST-IRS link, Represents the LoS component, which is location-dependent and remains unchanged over a relatively long time range. represents the NLoS component, Obey the Rayleigh fading model; Assume that the ST is equipped with a uniform linear array antenna along the direction (1, 0, 0) or the x-axis, and the IRS is a uniform linear array antenna placed along the direction (0, 1, 0) or the y-axis; the LoS channel between the ST and the IRS is expressed as The superscript H represents the conjugate transpose, v IRS With v ST They represent the steering vectors of IRS and ST, respectively. α AoA With α AoD denote the signal arrival azimuth at IRS and the departure angle from IRS to ST respectively; the direction cosine of the arrival azimuth cos(α AoA ), direction cosine of the departure angle cos(α AoD ) is given by the following two formulas cos(a AoA )=[0,1,0]e SR cos(a AoD )=[1,0,0]e SR Direction vector e SR The calculation method and Consistent, e SR The calculation of P requires obtaining the position coordinates of ST and IRS respectively, where P SR With P IRS They represent ST position and IRS position respectively; Assume that there are obstacles between PT and SR, and between ST and SR, and the interference is assumed to be Gaussian noise; therefore, in the cognitive communication system, the signal received by the kth SR is expressed as in, represents the additive white Gaussian noise at the kth SR, and the noise power is In addition, for the kth SR, the signal-to-interference-noise ratio of its received signal is expressed as In the Underlay spectrum sharing mode, the interference temperature IT imposed by ST on PR is expressed as Therefore, the signal-to-interference-noise ratio at PR is expressed as Among them, p p is the transmission power. To ensure that the PU is not affected by the SU, the signal-to-interference-to-noise ratio received by the PR must meet the QoS requirements of the normal communication of the PU, that is, γ P ≥γ P,min , γ P,min It is the minimum signal to interference and noise ratio threshold of PU.

4. The method for optimizing energy efficiency of IRS-assisted multi-user MISO cognitive communication based on continuous control according to claim 3 is characterized in that: In step (1), the energy harvesting and energy efficiency modeling of the IRS-assisted energy harvesting multi-user MISO cognitive communication system is specifically constructed as follows: In order to improve the energy utilization rate of cognitive communication systems and reduce the energy consumption of batteries and power grids, it is considered that ST is equipped with an energy harvesting device to collect the RF energy radiated by PT and use an energy conversion device to convert it into electrical energy to support data transmission between ST and K SRs. Each time slot is divided into two stages: energy collection stage and data transmission stage. In the energy collection stage, ST collects RF energy from PT and converts it into electrical energy. After the energy collection stage, ST uses the collected energy to assist in data transmission in the corresponding time slot. In a time slot with a time length of τ(t), the energy collected by ST is expressed as E eh =at(t)ηp P g PS Among them, α represents the energy collection time factor, η is the energy conversion efficiency, g PS represents the energy collection gain. The collected energy is used to compensate for part of the energy consumption of the cognitive communication system. In a time slot, the energy consumption of the cognitive communication system consists of the energy consumption of the ST transmission power, the power consumption of K SRs and IRSs, and the energy consumption of the cognitive communication system hardware heating. The energy consumed by SU sensing PU spectrum resources is ignored. The energy consumption of the cognitive communication system is expressed as P total =P ST +P IRS +P SR +P c -E eh Among them, P ST Indicates the energy consumed by ST transmission power; P IRS It represents the power consumption of IRS, which depends on the type and resolution of IRS reflective elements. The total power consumption of L reflective elements on IRS is Lp l (b), p l (b) represents the power consumption of each reflective element with b-bit resolution; P SR represents the total power consumption of K SRs; P c represents the energy consumption of the hardware heating of the cognitive communication system; therefore, the total power consumption of the cognitive communication system is given by K SRs can achieve a system communication rate of The energy efficiency of the system is defined as the ratio of the total achievable system communication rate to the total power consumption. Therefore, the energy efficiency of the considered IRS-assisted energy harvesting multi-user MISO cognitive communication system with K SRs is formulated as Assuming that in the energy harvesting stage, the energy collected by RF energy harvesting can meet the circuit consumption of the cognitive communication system itself, there are the following energy constraints: E eh =at(t)ηp P g PS ≥p c τ(t) From the above formula, we can conclude that Assuming that during the entire time slot, the energy collected by energy harvesting cannot fully compensate for the energy consumption of the entire cognitive communication system, there are the following energy constraints From the above formula, we can conclude that Therefore, considering the energy harvesting condition, the constraints that the energy harvesting factor must satisfy are: In order to consider the impact of energy harvesting on system energy efficiency, the first-order derivative of the energy harvesting time factor in the energy efficiency formula is obtained as follows: From the above formula, we can get ηp P g PS -p c ≥0, that is The numerator of is non-negative, so the energy efficiency is an increasing function of the energy collection time factor α.

5. The method for optimizing energy efficiency of IRS-assisted multi-user MISO cognitive communication based on continuous control according to claim 4 is characterized in that: In step (2), the goal of the cognitive communication energy efficiency optimization problem in the IRS-assisted energy harvesting multi-user MISO cognitive communication system is to jointly control the beamforming vector w at the ST T,k and the reflection coefficient matrix Φ at the IRS to maximize the energy efficiency of the cognitive communication system while ensuring the normal communication QoS requirements of the PU. The optimization problem is formulated as C3:c P ≥c P,min Where constraint C1 represents the maximum transmit power p of beamforming at ST max Constraints; Constraint C2 represents the value restriction of the energy collection time factor α of ST in the time slot; Constraint C3 represents the normal communication QoS requirement of PU. In the Underlay spectrum sharing model, the signal to noise ratio at PU must be higher than the minimum threshold γ P,min ; Constraint C4 represents the phase shift range that the L reflective elements on the IRS can provide; The above formula is a non-convex optimization problem.

6. The method for optimizing energy efficiency of IRS-assisted multi-user MISO cognitive communication based on continuous control according to claim 5, characterized in that: The energy efficiency optimization problem of cognitive communication in multi-user MISO cognitive communication system based on IRS-assisted energy harvesting is transformed as follows: Mathematical modeling of the interaction process between the environment and the agent is a prerequisite for using deep reinforcement learning. Deep reinforcement learning is described by a Markov decision process MDP defined as a five-tuple (S, A, R, P, γ), where S represents the state space of the environment, A represents the action space of the agent, R represents the transition probability between reward function states, P represents the transition probability between states, and γ represents the discount factor; specifically, State space: state space s t ∈S defines the set of environmental states that the agent can observe; the environmental states include the deployment positions of CSI, PU, ​​SU and IRS, the phase shifts of all IRS reflective elements, PT transmit power, and ST transmit beamforming matrix; therefore, the observed state at any time step is expressed as s t ={G T,R ,h R,k ,u RP ,u SP ,W T ,Φ} In the above formula, the state will be input into the neural network in the deep reinforcement learning algorithm. For complex input scenarios, the real part and the imaginary part need to be split and used as input respectively, G T,R 、h R,k The number of state variables is 2LM and 2LK respectively, and the cognitive communication device interferes with link u RP with u SP The total number of state variables is 2(M+L), the sum of the number of IRS reflection phase shift and ST transmit beamforming matrix states is 2MK+L; the dimension of the state space S is expressed as 2LM+2LK+2(M+L)+2MK+L; Action space: Action space a t ∈A represents the set of actions that the agent can perform; the agent selects an action based on the current state to affect the state transition of the environment. At the tth time step, the agent action space consists of two components, including the transmit beamforming matrix W of the ST T and the phase Φ of all reflective elements of the IRS, the action chosen by the agent on the IRS is the phase shift of each reflective element; Reward function: Maximizing the energy efficiency of the IRS-assisted energy harvesting multi-user MISO cognitive communication system requires considering all constraints and taking into account the minimum signal-to-interference-to-noise ratio requirement of PU communication defined in constraint C3. The reward function is set as in The reward function consists of two parts. The first term represents the system energy efficiency. The second term is the cost function, which is defined as the transmission cost of violating the minimum communication QoS of the PU. λ is the corresponding coefficient. The reward function imposes a protection level on the PU communication. When the PU is guaranteed to communicate normally, the cost is zero, that is, SINR out =0; otherwise, SINR out =1; The goal of the agent is to iteratively interact with the environment to maximize the cumulative reward according to the defined task and reward function. The cumulative reward is expressed as Among them, γ is the discount factor. By introducing the discount factor, the agent can weigh the relationship between short-term behavior and long-term goals; State transition probability: The state transition probability describes the probability of change of the environment state after the agent performs an action; the state transition probability function of the agent after performing an action and the reward function obtained under the corresponding action are related to the specific action, describing the probability of transitioning from one state to another and the immediate reward obtained; by defining a strategy in the MDP, the agent's actions in a certain state are guided; the strategy can be deterministic, that is, a specific action is specified for each state; it can also be random, that is, an action is selected according to the probability distribution in each state.

7. The method for optimizing energy efficiency of IRS-assisted multi-user MISO cognitive communication based on continuous control according to claim 6, characterized in that: In step (2), the PER-SD3 deep reinforcement learning algorithm consists of the SD3 algorithm and the PER algorithm.

8. The method for optimizing energy efficiency of IRS-assisted multi-user MISO cognitive communication based on continuous control according to claim 7, characterized in that: The SD3 algorithm is an improvement on the deep deterministic policy gradient DDPG algorithm. The SD3 algorithm integrates 8 neural networks, including two actor networks, two critic networks, two target actor networks and two target critic networks. When two actor networks are applied, the agent selects the action with the largest Q-value function among the two output actions. The two actor networks and the two critic networks need to be trained with parameters, while the two target networks are soft-updated by copying the weights of the two actor networks and the two critic networks.

9. The method for optimizing energy efficiency of IRS-assisted multi-user MISO cognitive communication based on continuous control according to claim 7, characterized in that: During the training phase of the PER-SD3 deep reinforcement learning algorithm, the PER algorithm uses a priority queue to store experience samples, adopts a SumTree-based data structure, and selects samples based on priority.

10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 9 can be implemented.

Images