Inter-device energy-aware spectrum management in optical wireless communications based on deep reinforcement learning

Abstract

The present application relates to an energy-aware spectrum management method between devices in optical wireless communication based on deep reinforcement learning, and belongs to the field of optical wireless communication. The present application proposes a multi-agent reinforcement learning framework to optimize the performance of D2D (Device to Device) selection spectrum. By comparing the equivalent signal-to-interference-and-noise ratio values of D2D multiplexing visible light or wireless communication spectrum, a spectrum multiplexing correlation matrix of D2D is constructed. The residual energy of D2D nodes and the system energy efficiency are designed as the reward function value of D2D multiplexing optical wireless communication spectrum. The expected value of the cumulative discount reward obtained by maximizing the D2D multiplexing spectrum is taken as the optimization target. By introducing a D2D multi-agent actor-critic deep reinforcement learning algorithm based on cooperative game, the expected value of the cumulative discount reward of D2D spectrum multiplexing is improved. The method described in the present application can improve the problems of uneven load between indoor optical wireless communication cells and low communication quality of edge devices, and improve the communication ability of edge devices and the energy efficiency of the system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of optical wireless communication and relates to a spectrum management method for energy perception between devices in optical wireless communication based on deep reinforcement learning. BACKGROUND

[0002] In recent years, visible light communication (VLC) technology based on LED (Light-emitting Diode) has attracted extensive attention of researchers due to its low power consumption, high security, no electromagnetic interference and other advantages, and can provide about 400 THz of unlicensed spectrum. Existing research shows that VLC can use visible light waves with a wavelength of 380-780 nm to control the on-off of LEDs through coded data information, so as to realize light transmission of data information. At the receiving end, the on-off of high-speed flashing LEDs is captured by a photodetector configured by a receiving device, and after a series of signal processing, the original data information transmitted by the sending end can be obtained.

[0003] However, VLC communication faces many problems, such as: its data transmission depends on the LoS (Line of Sight) link, the LoS link is easy to be blocked, the coverage range of the VLC AP (Access Point) is small, and the dense deployment of multiple VLC APs can easily cause serious inter-cell communication interference. In order to overcome the above-mentioned shortcomings of VLC, in recent years, some scholars have proposed to combine VLC and traditional RF (Radio Frequency) communication technology, and this idea of VLC-RF heterogeneous networking has attracted the interest of many scholars in related fields. On the one hand, VLC AP can provide high-speed communication services, and on the other hand, RF AP has a large coverage range and anti-LoS blocking characteristics, which can ensure the stability of low-speed transmission links.

[0004] However, with the rapid development of smart city and Internet of Everything application requirements, when there are too many user devices indoors, a large number of device-to-device communication through VLC AP or RF AP switching will greatly increase the burden of AP and consume a large amount of limited spectrum and power resources of VLC-RF heterogeneous network. D2D (Device to Device) communication is a communication technology in which devices communicate directly without switching through AP. In recent years, it has been applied in many fields, such as V2V (Vehicle-to-Vehicle) communication. For the new generation of Internet of Everything scenarios, D2D communication technology will greatly reduce communication overhead and improve the communication capability of edge devices in the network, and increase the reachable rate of the network.

[0005] However, the spectrum resource used by D2D forms some interference to the cellular communication users of the indoor VLC-RF heterogeneous network. When the number of D2D requesting spectrum allocation is large, the VLC-RF heterogeneous network needs to coordinate the contradiction between the reachable rate of network communication and inter-cell interference, and the spectrum resource utilization rate of D2D multiplexing cellular users. The spectrum allocation of multiple D2D will increase the complexity of spectrum multiplexing and interference management of each VLC AP and RF AP. In addition, the energy storage of D2D devices is limited, which limits the communication endurance of the device. Therefore, in this application, we propose to use D2D communication technology as an auxiliary communication means of VLC-RF, provide a way for devices to communicate with other devices without connecting to the base station, multiplex VLC-RF cellular user spectrum for D2D communication, and use a deep reinforcement learning algorithm to solve the spectrum selection and interference management problem of D2D multiplexing VLC-RF cellular user spectrum, and improve the energy efficiency performance of D2D multiplexing spectrum. SUMMARY

[0006] Therefore, the core of the present application is to provide an energy-aware spectrum management between devices in optical wireless communication based on deep reinforcement learning, which improves the energy efficiency of the system under the premise of guaranteeing the QoE of cellular users and the residual energy of D2D nodes.

[0007] To achieve the above purpose, the present application provides the following technical scheme:

[0008] S1: input D2D device set N, VLC AP set, transmit power P of each VLC AP V and bandwidth B V ; transmit power P of RF AP R and total bandwidth B R , cellular user set M using RF communication; the spectrum of users accessing VLC and RF uses orthogonal frequency division multiplexing, and D2D devices need to multiplex RF frequency bands or multiplex VLC carrier resources; according to the Lambert radiation model, the channel gain and reachable rate value between the VLC-D2D transmitter and receiver using VLC carrier are calculated, and according to the path loss model, the channel gain and reachable rate value between RF-D2D using RF frequency band are calculated;

[0009] S2: calculate the energy collected by D2D devices in the idle time of communication, construct the spectrum multiplexing association matrix of D2D multiplexing CUE spectrum according to the equivalent SINR of VLC-D2D and the SINR value of RF-D2D;

[0010] S3: Taking the residual energy of the D2D node and the system energy efficiency as the reward function value of the D2D multiplexing CUE spectrum, a Markov decision model of VLC-D2D and RF-D2D spectrum allocation is established, a state space, an action strategy and a reward function of an intelligent agent selecting a multiplexing spectrum from a spectrum multiplexing correlation matrix are constructed, and an optimization objective function of maximizing the expected value of the cumulative discount reward of the D2D multiplexing CUE spectrum is established;

[0011] S4: Using an Actor-Critic (Actor-Critic) deep reinforcement learning (Deep Reinforcement Learning, DRL) algorithm to optimize the spectrum selection of the D2D multi-agent, and improve the expected value of the cumulative discount reward obtained by the D2D multi-agent in the VLC-RF network.

[0012] The S1 specific method is:

[0013] S101: In the indoor VLC-RF system, there are M cellular user equipments using RF communication, which are saved in set M; there are N D2D pairs in the indoor, which are saved in set N; when the D2D user equipment (D2D User Equipment, DUE) multiplexes the spectrum of the RF cellular user equipment (Cellular User Equpment, CUE) to transmit data, the D2D is referred to as RF-D2D; when the DUE multiplexes the carrier communication of the VLC, the D2D is referred to as VLC-D2D; the total bandwidth B of the RF R is equally divided into M cellular users, and the center frequency of each bandwidth is the center frequency of the modulated carrier; the VLC mainly improves the downlink communication of the M users in the indoor, and the total bandwidth of all VLC APs adopts orthogonal frequency division multiple access, and the number of subcarriers is greater than M, so that the downlink optical line transmission can avoid interference; each RF-D2D can only multiplex one subcarrier of the RF in each time slot, and the subcarrier of the RF can be multiplexed by multiple RF-D2D in each time slot, and signal interference can occur between the D2D using the same spectrum and between the CUE and the RF-D2D multiplexing the carrier communication thereof; the VLC-D2D multiplexes the photon carrier communication of the VLC;

[0014] S102: According to the Lambert radiation model, the channel gain of the VLC-D2D communication is calculated;

[0015] According to the Lambert radiation model, the channel gain between the nth pair of VLC-D2D is:

[0016]

[0017] In the above formula (1), represents the Lambert radiation coefficient, and φ 1 / 2represents the half-power beamwidth of the D2D transmitter using VLC, and φ represents the beamwidth of the D2D transmitter; A r represents the receiving area of a photo detector (PD) of the D2D receiver; represents the straight-line distance between the transmitting end and the receiving end of the D2D device; ψ n,n is the incident angle of the D2D pair receiver; ψ c is the field of view angle of the DUE receiver; T s (ψ n,n ) and g(ψ n,n ) respectively represent the gain of the optical filter and the optical concentrator when the DUE receives the optical signal at ψ n,n ;

[0018] S103: Calculate the SINR (Signal to Interference plus Noise Ratio) value of the VLC-D2D received signal;

[0019] wherein the SINR of the D2D received signal of the nth pair using VLC communication can be represented as:

[0020]

[0021] In the above formula (2), P opt is the average transmission optical power value of the DUE transmitter, which is proportional to the direct current bias of the LED (Light Emitting Diode); η represents the photoelectric conversion coefficient; represents the channel gain between the nth VLC-D2D transmitter and the corresponding receiver PD, represents the channel gain between the ith pair of VLC-D2D transmitter and the corresponding receiver PD; is the subcarrier bandwidth of the nth pair of VLC-D2D communication, and the VLC frequency band of the present application adopts fixed carrier bandwidth allocation, so that is a constant; N V represents the Gaussian white noise power spectral density of the DUE receiving optical signal;

[0022] S104: Calculate the achievable rate value of the data signal received by the VLC-D2D receiver;

[0023] wherein the achievable rate of the data signal of the nth pair of VLC-D2D can be represented as:

[0024]

[0025] S105: Calculate the channel gain value of RF-D2D according to the path loss model;

[0026] wherein the channel gain of the n-th pair of multiplexed RF frequency band for D2D communication can be expressed as:

[0027]

[0028] wherein K represents the path loss constant; β n,n represents the fast fading gain caused by multipath effect, subject to exponential distribution; λ n,n represents the slow fading gain caused by shadow fading, subject to lognormal distribution; d n,n represents the distance between the transmitter and the receiver of the n-th pair of D2D devices; and α is the path loss factor;

[0029] S106: calculate the SINR value of the RF-D2D received signal;

[0030] wherein the SINR value of the n-th pair of RF-D2D is calculated by:

[0031]

[0032] wherein α n,m is a binary variable, α n,m = 1 when the RF-D2D n multiplexes the RF frequency spectrum of the cellular user m, otherwise α n,m = 0; similarly, α i,m = 1 when the RF-D2D i multiplexes the RF frequency spectrum of the cellular user m, otherwise α i,m = 0; are the transmit powers of the cellular user m and the D2D n, respectively; and are the transmit powers of the n-th pair and the i-th pair of D2D, respectively; is the channel gain value of the cellular user m and the transmitting DUE of D2D, which can be calculated by Rayleigh fading model; and are the channel gain values of the n-th pair of multiplexed RF frequency communication of D2D, DUE i and DUE n; is the subcarrier bandwidth of the n-th pair of RF-D2D communication; N R represents the Gaussian white noise power spectral density of the DUE received RF signal;

[0033] S107: calculate the achievable rate value of the multiplexed RF carrier communication of D2D according to Shannon formula;

[0034] wherein the achievable rate value of the n-th pair of multiplexed RF carrier communication of D2D is calculated by:

[0035]

[0036] In the above formula, B n,m Let be the subcarrier bandwidth for the nth pair of RF-D2D communication using multiplexed RF channel m.

[0037] The specific method of S2 is as follows:

[0038] S201: Calculate the equivalent SINR value for VLC-D2D;

[0039] The formula for calculating the equivalent SINR value of the nth VLC-D2D pair is as follows:

[0040]

[0041] In the above formula, Γ RF =3dB and Γ VLC =5.4dB represents the channel coding loss factor for RF network and VLC network, respectively;

[0042] S202: Add all D2Ds to the set F of the alliance V alliance. VLC In the middle, judge the alliance F in turn. VLC The equivalent SINR value of the D2D pair, if its equivalent SINR value is less than the set threshold. Then add the D2D pair to the set F of the R alliance. RF Otherwise, the D2D alliance information remains unchanged; in this application, we set

[0043] This invention employs game theory, dividing all D2D games into two coalitions, referred to as the V-coalition and the R-coalition, denoted by the symbol F. VLC and F RF The D2D device is located in F VLC Communication is performed using VLC spectrum, located in F RF D2D devices in the system communicate by reusing the uplink spectrum of cellular users.

[0044] S203: Check F sequentially VLC In D2D, if the equivalent SINR value of the VLC spectrum used by the D2D is less than the SINR value of the RF spectrum multiplexed by the D2D, then the D2D is added to the RF spectrum. RF Similarly, check F in sequence. RF In D2D, if the SINR value of the multiplexed RF spectrum of the D2D is less than the equivalent SINR value of the VLC spectrum used by the D2D, then the D2D is added to the RF spectrum. VLC middle;

[0045] S206: For F RFWhen the RF-D2D is close to the CUE which reuses the RF spectrum, the RF-D2D has great co-frequency interference with the CUE using the same spectrum, and the RF-D2D cannot reuse the spectrum of the adjacent CUE. Therefore, a series of authorized cells are set according to the geographical position information of the CUE and the RF-D2D, and M CUEs are arranged in ascending order according to the transmission power of the uplink access RF of the CUE: In the set M, the strong interference radius of each CUE is determined according to the transmission power of the CUE, and the RF-D2D which reuses the spectrum of the CUE and is in the strong interference range of the CUE is found and saved in the set of RF-D2D in the strong interference range of the CUE;

[0046] Wherein, the position information of the CUE m is sent to the base station by the uplink, and R m is the strong interference radius of the CUE m, which is positively correlated with the uplink transmission power of the CUE m, that is,

[0047] Wherein, the set of RF-D2D in the strong interference range of the CUE m Can be expressed as:

[0048]

[0049] In the above formula, Dn represents the nth pair of D2D, (x m ,y m ) represents the position of the CUE m, (x n ,y n ) is the position of the DUE which reuses the spectrum of the CUE m and is far away from the CUE m, and R m is the strong interference radius of the CUE m;

[0050] S207: Determine the authorized RF-D2D device set which reuses the spectrum of the CUE, and determine the authorized CUE set of each RF-D2D;

[0051] Wherein, the authorized RF-D2D device set of the CUE m and F m-D Can be expressed as:

[0052]

[0053] Wherein, the authorized CUE set F n-C of the RF-D2D n can be expressed as:

[0054]

[0055] In the above formula, C m represents the CUE m, and C m is the far distance value between the RF-D2D and the CUE m.m CUEs outside the interference radius of CUE m become one of the candidate CUEs for the RF-D2D spectrum multiplexing;

[0056] S208: Constructing the spectrum multiplexing correlation matrix of D2D multiplexing CUE spectrum;

[0057] Wherein, the spectrum multiplexing correlation matrix of all D2D multiplexing CUE spectrum is defined as:

[0058]

[0059] In the matrix C B , b m,n is a binary variable, when the uplink spectrum of CUE m can be authorized to RF-D2D n multiplexing, then b m,n is 1, otherwise, b m,n is 0, m∈M, n∈N; in the matrix C B , b 0,n indicates whether D2D n is authorized to use VLC spectrum for communication, if authorized, then b 0,n is 1, otherwise, b 0,n is 0.

[0060] Wherein, the S3 specific method is:

[0061] S301: Abstracting each D2D of indoor VLC-RF as an intelligent agent, and marking the spectrum block resource state of VLC-RF system as c t =[s t ,v,F t ,D t ,I t ], wherein, C represents the state set of each resource block in each time slot; s t represents the CUE spectrum block index number available for D2D in t time slot, which is obtained by searching the spectrum multiplexing correlation matrix C B obtained in S208; v represents whether the spectrum block is VLC spectrum, if s t is VLC spectrum, v=1, otherwise, v=0; F t represents whether s t spectrum block is occupied by CUE uplink communication; D t represents whether s t spectrum block is occupied by other D2D; I t represents whether s t spectrum block allocation to the current D2D forms interference to CUE; the distance vector information of the intelligent agent of D2D to each CUE in VLC-RF system is d t =[d1,...,dM The state of the agent in the VLC-RF system during time slot t is represented as: E t =[c t ,d t-1 ], where d t-1 This represents the distance vector information between D2D and each CUE in the previous time slot;

[0062] S302: The action space of the CUE using D2D multiplexing VLC-RF is represented as: a t =[i t ,k t ], i t Let i be a binary variable. t =1 indicates that D2D needs to transmit a signal in time slot t; k t This indicates the index number of the CUE spectrum block used by D2D in time slot t, where k t ∈{0,1,...,M},k t =0 indicates that D2D uses VLC spectrum communication;

[0063] S303: Calculate the reward function value for D2D multiplexing CUE spectrum;

[0064] The formula for calculating the reward function value for D2D n-multiplexed CUE m spectrum blocks in communication is as follows:

[0065]

[0066] In the above formula, Re n R represents the reward value obtained by D2D n-times multiplexing CUE spectrum. n This represents the achievable rate value obtained by D2D n-multiplexing VLC or RF spectrum. β represents the weighting factor between the device's remaining energy and the achievable data rate via VLC-RF access; r neg This represents the penalty value for the reduction in achievable rate or remaining energy of the CUE caused by interference introduced by the agent reusing the CUE spectrum block. In this application, let r neg =-1; This represents the achievable rate value of CUE. This represents the communication rate threshold of the CUE. P represents the energy threshold for D2D device communication; total The total transmit power value for D2D device communication in a VLC-RF system is determined by the combined transmit power of the communication devices connected to the VLC AP, the transmit power of the CUE devices connected to the RF AP, and the D2D transmit power. This represents the average remaining energy value of all connected VLC-D2D and RF-D2D devices;

[0067] S304: modeling the decision of D2D multiplexing VLC or RF spectrum as a Markov Decision Process (MDP), the MDP tuple of D2D is represented as: (E t t n t t t+1 n

[0068] S306: calculating the cumulative discounted reward value and the expected value of the cumulative discounted reward of the agent after selecting the action;

[0069]

[0070]

[0071]

[0072]

[0073]

[0074] n

[0075]

[0076] S401: when D2D sends a communication request for spectrum allocation to the VLC-RF system, the control center of the VLC-RF system abstracts D2D as an agent, downloads historical trajectory data in the experience storage pool from the cloud, and trains the Actor-Critic strategy of the deep learning network; ​​​​​​​​​​​​​​​​​

[0077] S402: input the request information of the agent D2D, and search the spectrum reuse correlation matrix C according to the search spectrum B to obtain information of whether each spectrum block is available, and count the state information E of each spectrum block in the VLC-RF system t and the action space a of D2D t , construct the MDP information (E t , a t , r, p, g) of D2D;

[0078] S403: use the Q function method of deep learning to calculate the expected value of the cumulative discount reward of the spectrum block selected by the agent:

[0079] wherein the expected value of the cumulative discount reward calculated by the Q function of the resource state-action policy (E t , a t ) of the agent at time t is:

[0080]

[0081] In the above formula, Q π (E t , a t ) represents the expected value of the potential long-term discount reward that the agent can obtain by using action a t in state E t ; E t+1 represents the resource-action policy at time t+1; E represents the combined set of the c t state of each resource block and the distance vector d t-1 of the agent; P(E t+1 |E t , a t ) represents the transition probability function value of the agent using action a t to transition from the current resource state E t to the next resource state E t+1 , P(E t+1 |E t , a t ) e p; g e g; Q π (E t+1 , a t ) represents the expected value of the potential long-term discount reward that the agent can obtain by using action a t+1 in resource state E t ;

[0082] S404: use the Bellman optimal equation method to find the best state-action policy of the agent;

[0083] Wherein, the optimal state-action policy of the agent n is represented as:

[0084]

[0085] In the above formula, π * represents the optimal policy (E * , a * ) that satisfies

[0086] Wherein, the Bellman optimal equation for finding the optimal state-action policy can be described as:

[0087]

[0088] S405: A trajectory generated by the interaction of the agent with the VLC-RF resource environment is adopted The Q function value is updated;

[0089] Wherein, the trajectory data The formula for updating the Q function value of the nth agent is:

[0090]

[0091] In the above formula, η represents the learning rate, and η ∈ [0, 1];

[0092] S406: In deep reinforcement learning, the Actor-Critic global policy of all D2D multi-agent selecting VLC-RF communication spectrum is represented as π = {π1,..., π n ,..., π N}, and the corresponding network parameters are represented as θ = {θ1,..., θ N}; The output of the Critic network of the nth agent is The Q function value represents the cumulative discounted reward expectation value of the Actor-Critic policy of the nth agent; at the same time, the Q function value reflects the evaluation value obtained by the agent after adopting the Actor action on the Critic network; the deep neural network is used as the nonlinear function fitter of the Q function, and the continuous Q function of the Critic network is obtained;

[0093] S407: The network parameters θ of the Q function value of the agent when minimizing the loss function are calculated, and the DRL network weight parameters θ are updated;

[0094] Wherein, the updating formula of the minimum loss function of the Q function value of the agent n is:

[0095]

[0096] In the above formula, This represents the cumulative discounted reward Q-function estimate of agent n using the online policy μ′, where μ′={μ′1,...,μ′} N} is the policy set of the target Actor network of DRL, and its parameters are copied from the online network. The online network copies its parameters to the target Actor network of DRL every certain number of steps. The purpose is to avoid the Actor network overestimating the Q function value of the agent. It is the action experience storage pool where the training data of the deep reinforcement learning network is stored, (a1, a2, ..., a N () represents a set of historical action data for N agents in an observable experience storage pool. This represents the set of state-action policies for the L-1 observations prior to time t, with element O. t =[c t ,a t [E] represents the state-action policy observed by the agent in time slot t. t ,a t ], where L is the number of historical observations, and x = (o1,...,o2) N () represents an observable set of state-action policies for N agents.

[0097] S408: The Deep Deterministic Policies Gradient (DDPG) method is used to find the lower objective gradient value of all agents under the condition of historical state-action policy, which maximizes the expected value of the cumulative discounted reward of the Q function for a given action; then, by comparison, the action of the agent that minimizes the lower objective gradient value is selected.

[0098] The main idea of ​​DDPG is to adopt The gradient descent directional iterative method adjusts the parameters θ of the trained Critic policy network to maximize the Q function value, thus overcoming the drawback of high variance gradient estimation often exhibited by gradient policy methods in multi-agent environments.

[0099] DDPG solves for the action 'a' taken by the nth agent. t The method for representing the lower objective gradient value of the maximum Q-function value is as follows:

[0100]

[0101] In the above formula, θ n μ represents the state of agent n when the network parameters are θ. n The network state is represented by θ. n The continuous policy of agent n at time θ∈θ; J(μ) n) Use a continuous policy μ for agent n n The expected value of cumulative discount rewards over time. This represents solving for the agent's state θ. n J(μ) n Gradient value; The agent n represents a state-action policy O based on history. t Use action a t Solving for the agent's state θ n μ n Gradient value; Represents an agent n based on the set of states-action policies and action sets (a1, a2, ..., a) of N agents x. N ) continuity strategy μ n Q function value, This represents solving for the agent's action a. t strategy μ n The gradient value of the Q-function;

[0102] S409: Output the optimal set of actions for all agents (a1, a2, ..., a...). N Then, the optimal action set (a1, a2, ..., a) for all agents to obtain the maximum Q-function value is determined. N ) is represented as a t The state c of the agent before optimization t and VLC-RF environment resource status d t Marked as E t The intelligent agent uses optimized action a t The set of states after that is denoted as E. t+1 The computational agent adopts E t+1 Re reward function value Re n The set is denoted as r t+1 Generate a trajectory data (E) t ,a t ,r t+1 E t+1 ), stored in the experience storage pool of DRL In the middle, when the experience storage pool When the trajectory data is full, the trajectory data that has been stored for a long time in the experience storage pool will be replaced with new trajectory data. Attached Figure Description

[0103] To make the objectives, technical solutions, and advantages of the present invention clearer, the invention will be further described below in conjunction with the accompanying drawings:

[0104] Figure 1 Schematic diagram of D2D spectrum allocation in indoor VLC-RF heterogeneous network;

[0105] Figure 2 Interference diagram of indoor D2D spectrum allocation;

[0106] Figure 3 D2D spectrum allocation model diagram based on multi-agent reinforcement learning of cooperative game; Specific embodiments

[0107] The present application is described below by way of specific embodiments, and other advantages and effects of the present application can be easily understood by those skilled in the art from the description herein. The present application can also be implemented or applied in other different embodiments, and various modifications or changes can be made to the details herein based on different views and applications without departing from the spirit of the present application. It should be noted that the drawings provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0108] The drawings are only used for exemplary illustration, and the representation is only a schematic diagram, not a physical diagram, and should not be understood as a limitation of the present application; in order to better illustrate the embodiments of the present application, some components in the drawings may be omitted, enlarged or reduced, and do not represent the actual product size; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0109] Attached Figure 1 D2D spectrum management diagram for indoor VLC-RF heterogeneous network. In the attached Figure 1 , there are 1 RF AP and multiple VLC APs using LED light, M cellular user equipment (CUE), represented as M = {1,..., M}; there are N D2D pairs, represented as N = {1,..., N}. When the D2D user equipment (DUE) multiplexes the uplink spectrum of the CUE for data transmission, it is called RF-D2D; when the DUE uses VLC spectrum for communication, it is called VLC-D2D; the transmission power P V and bandwidth B V of each VLC AP; the transmission power P R and total bandwidth B RThe number of uplink RF sub-channels is the same as the number of CUEs in the network. Through the use of Orthogonal Frequency Division Multiple Access (OFDMA), there is no interference between CUEs. A CUE's spectrum can be multiplexed by multiple RF-D2D devices in each time slot, and each RF-D2D device can only multiplex the spectrum of one CUE in each time slot. Assume all D2D devices are within the coverage area of ​​the RF AP. When a CUE's spectrum is multiplexed by RF-D2D devices, the RF-D2D devices may be subject to interference from CUE, and CUE may also be subject to interference from RF-D2D devices. When different D2D pairs use the same spectrum, interference from co-frequency signals may also occur between these co-frequency DUEs.

[0110] Appendix Figure 2 This diagram illustrates interference in indoor D2D spectrum allocation within a VLC-RF heterogeneous network, including VLC-D2D with multiple pairs of multiplexed VLC spectrum communications and RF-D2D with multiple pairs of multiplexed RF spectrum communications. The coverage radius of the RF AP's transmitted signal is R, while the coverage cell radius of each VLC AP's optical signal is R0. m To address co-channel interference between RF-based CUEs and RF-D2Ds, as well as between different D2Ds reusing the same CUE spectrum, the interference magnitude is estimated based on the distance between the CUE and the RF-D2D, and the SINR value is calculated to avoid co-channel interference between RF-D2Ds reusing the CUE spectrum and the CUE. Similarly, the interference magnitude is estimated based on the distance between different D2Ds, and the SINR value is calculated to prevent nearby D2D pairs from reusing the same CUE spectrum. Furthermore, a similar method is employed to prevent nearby VLC-D2Ds and VLC CUEs from using the same spectrum.

[0111] Appendix Figure 3 The diagram illustrates a D2D spectrum allocation model for multi-agent reinforcement learning based on cooperative game theory in an indoor environment. Each D2D pair in the VLC-RF heterogeneous network is abstracted as an agent in a deep reinforcement learning network. When selecting CUE spectrum communication to reuse in the VLC-RF heterogeneous network, each agent adopts an Actor-Critic strategy. First, it downloads historical trajectory data from the cloud to train the parameter set θ of the deep reinforcement learning network. Then, each agent observes the resource status in the VLC-RF heterogeneous network in time slot t to obtain the state information c of each spectrum block. t =[s t ,v,F t D t ,I t Check the position information between CUEs. t =[d1,...,d M Determine the state information E of each agent in the spectrum block of the VLC-RF system in time slot t. t =[c t ,d t-1The Actor attempts to allocate spectrum blocks to agents, and the Critic evaluates the effectiveness of the spectrum allocation actions. This application evaluates the effectiveness by calculating the expected cumulative discount reward for each agent's reuse of spectrum blocks. This value reflects the agent's achievable rate and energy cost, and reflects the energy efficiency contribution of the agent to the VLC-RF system. A deep deterministic gradient policy gradient method is used to find the spectrum block selection action that maximizes the expected cumulative discount reward for each agent; this action represents the spectrum block selected by the agent. Then, the resource state information E of all spectrum blocks in the system before the agent selects the spectrum block with the maximum expected cumulative discount reward is collected. t Select the action information a of the spectrum block t Spectrum block resource status information E after spectrum block selection t+1 The expected value r of the cumulative discount reward obtained after selecting a spectrum block. t+1 This forms a set of trajectory data (E t a t r t+1 E t+1 The trajectory is uploaded to the experience storage pool of trajectory data, which serves as the training data for the Actor-Critic strategy of requesting spectrum allocation for the agent in the next time slot. When the buffer in the experience storage pool is full, the new trajectory data replaces the trajectory data that has been stored in the experience storage pool for a long time.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A device-to-device energy-sensing spectrum management method based on deep reinforcement learning in optical wireless communication, characterized by: The method includes the following steps: S1: Input D2D (Device to Device) device set N, VLC (Visible Light Communication) access point (AP) set, and the transmit power P of each VLC AP. V and bandwidth B V The transmit power P of an RF (Radio Frequency) AP R and total bandwidth B R M is the set of cellular users using RF communication; the spectrum of users accessing VLC and RF uses orthogonal frequency division multiplexing, and communication between D2D devices requires multiplexing the RF frequency band or multiplexing the VLC carrier resources; according to the Lambert radiation model, calculate the channel gain and achievable rate between VLC-D2D transmitters and receivers using VLC carriers, and according to the path loss model, calculate the channel gain and achievable rate between RF-D2D devices using the RF frequency band; The steps for calculating the achievable rates of VLC-D2D and RF-D2D are as follows: S101: In an indoor VLC-RF system, there are M cellular user equipments using RF communication, stored in set M; there are N D2D pairs indoors, stored in set N; when a D2D user equipment (DUE) multiplexes the spectrum of an RF cellular user equipment (CUE) to transmit data, D2D is called RF-D2D; when a DUE multiplexes the carrier communication of VLC, D2D is called VLC-D2D; the total bandwidth of RF is B. R The bandwidth is evenly distributed among M cellular users, with the center frequency of each portion being the center frequency of the modulation carrier. VLC primarily enhances downlink communication for the M indoor users. The total bandwidth of all VLC APs utilizes Orthogonal Frequency Division Multiple Access (OFDMA), and its number of subcarriers is greater than M. Therefore, downlink optical transmission can avoid interference. Each RF-D2D can only multiplex one RF subcarrier in each time slot. RF subcarriers can be multiplexed by multiple RF-D2Ds in each time slot. Signal interference may occur between D2Ds using the same spectrum, and between CUEs and RF-D2Ds that multiplex their carriers for communication. VLC-D2Ds multiplex the optical carriers of the VLC for communication. S102: Calculate the channel gain of VLC-D2D communication based on the Lambert radiation model; According to the Lambert radiation model, the channel gain between the nth VLC-D2D pairs is: In the above formula, θ represents the Lambert radiation coefficient, and θ = -ln2 / ln(cosφ) 1 / 2 );φ 1 / 2 A represents the half-power radiation angle of a D2D transmitter using VLC, where φ represents the radiation angle of the D2D transmitter; r This indicates the receiving area of ​​the photodetector (PD) of the D2D receiver; ψ represents the straight-line distance between the transmitter and receiver of a D2D device; n,n Let ψ be the incident angle of the D2D receiver. c T is the field of view of the DUE receiver; s (ψ n,n ) and g(ψ n,n ) respectively represent DUE with ψ n,n The gain of the optical filter and optical condenser when receiving light signals; S103: Calculate the SINR (Signal to Interference plus Noise Ratio) value of the VLC-D2D received signal; The SINR of the nth pair of D2D received signals using VLC communication can be expressed as: In the above formula, P opt η is the average emitted optical power of the DUE transmitter, which is proportional to the DC bias of the LED (Light Emitting Diode); η represents the photoelectric conversion coefficient. This represents the channel gain between the nth VLC-D2D transmitter and the corresponding receiver's PD. This represents the channel gain between the PD of the i-th VLC-D2D transmitter and the corresponding receiver; Given the subcarrier bandwidth of the nth VLC-D2D communication pair, this application uses a fixed carrier bandwidth allocation for the VLC band. N is a constant; V This represents the Gaussian white noise power spectral density of the optical signal received by the DUE. S104: Calculate the achievable data rate of the VLC-D2D receiver; The achievable data rate of the nth VLC-D2D received data signal can be expressed as: S105: Calculate the channel gain value of RF-D2D based on the path loss model; The channel gain for D2D communication in the nth multiplexed RF band can be expressed as: In the above formula, K represents the path loss constant; β n,n λ represents the rapidly decaying gain due to multipath effects, which follows an exponential distribution; n,n The d represents the slow decay gain caused by shadow decay, which follows a log-normal distribution; n,n The distance between the transmitter and receiver of the nth pair of D2D devices is represented by α; α is the path loss factor. S106: Calculate the SINR value of the RF-D2D received signal; The formula for calculating the SINR value of the nth RF-D2D pair is as follows: In the above formula, α n,m It is a binary variable, α, when RF-D2D n multiplexes the RF spectrum of cellular user m. n,m =1, otherwise, α n,m =0; similarly, when RF-D2D i multiplexes the RF spectrum of cellular user m, α i,m =1, otherwise, α i,m =0; These are the transmit powers of cellular user m and D2D n, respectively; and Let represent the transmit power of the nth and ith D2D pairs, respectively; The channel gain value between cellular user m and the transmitting end DUE in D2D can be calculated using the Rayleigh fading model; and These represent the channel gain values ​​for D2D multiplexed RF frequency communication formed by the nth pair and D2D, DUE i, and DUE n, respectively. N represents the subcarrier bandwidth of the nth RF-D2D communication pair; R This represents the Gaussian white noise power spectral density of the RF signal received by the DUE. S107: Calculate the achievable rate value of D2D in multiplexed RF carrier communication according to Shannon's formula; The formula for calculating the achievable rate of the nth D2D pair in multiplexed RF carrier communication is as follows: In the above formula, B n,m The subcarrier bandwidth for the nth pair of RF-D2D communication using multiplexed RF channel m; S2: Calculate the energy collected by the D2D device during the idle time of communication, and construct the spectrum reuse correlation matrix of the D2D multiplexed CUE spectrum based on the equivalent SINR of VLC-D2D and the SINR value of RF-D2D. S3: Using the remaining energy of D2D nodes and system energy efficiency as the reward function value for D2D CUE spectrum reuse, establish Markov decision models for VLC-D2D and RF-D2D spectrum allocation, construct the state space, action policy and reward function of the agent to select reused spectrum from the spectrum reuse correlation matrix, and establish the optimization objective function to maximize the expected value of the cumulative discount reward for D2D CUE spectrum reuse; The specific steps of S3 are as follows: S301: Abstract each D2D of the indoor VLC-RF system into an intelligent agent, and mark the spectrum block resource status of the VLC-RF system as c. t =[s t ,v,F t D t ,I t ],in, C represents the set of states for each resource block and each time slot; s t This represents the CUE spectrum block index number available in time slot D2D, whose value is obtained by retrieving the spectrum reusability correlation matrix C obtained in step S208. B We obtain; v indicates whether the spectrum block is a VLC spectrum, if s t For VLC spectrum, v = 1; otherwise, v = 0; F t s t Is the spectrum block occupied by CUE uplink communication? (D) t s t Is the spectrum block occupied by other D2D? t s t Does the allocation of a spectrum block to the current D2D system interfere with the CUE? The distance vector information of the D2D agent from each CUE in the VLC-RF system is d. t =[d1,...,d M The state of the agent in the VLC-RF system during time slot t is represented as: E t =[c t ,d t-1 ], where d t-1 This represents the distance vector information between D2D and each CUE in the previous time slot; S302: The action space of the CUE using D2D multiplexing VLC-RF is represented as: a t =[i t ,k t ], i t Let i be a binary variable. t =1 indicates that D2D needs to transmit a signal in time slot t; k t This indicates the index number of the CUE spectrum block used by D2D in time slot t, where k t ∈{0,1,...,M},k t =0 indicates that D2D uses VLC spectrum communication; S303: Calculate the reward function value for D2D multiplexing CUE spectrum; The formula for calculating the reward function value for D2D n-multiplexed CUE m spectrum blocks in communication is as follows: In the above formula, Re n R represents the reward value obtained by D2D n-times multiplexing CUE spectrum. n This represents the achievable rate value obtained by D2D n-multiplexing VLC or RF spectrum. β represents the weighting factor between the device's remaining energy and the achievable data rate via VLC-RF access; r neg This represents the penalty value for the reduction in achievable rate or remaining energy of the CUE caused by interference introduced by the agent reusing the CUE spectrum block. In this application, let r neg =-1; This represents the achievable rate value of CUE. This represents the communication rate threshold of the CUE. P represents the energy threshold for D2D device communication; total The total transmit power value for D2D device communication in a VLC-RF system is determined by the combined transmit power of the communication devices connected to the VLC AP, the transmit power of the CUE devices connected to the RF AP, and the D2D transmit power. This represents the average remaining energy value of all connected VLC-D2D and RF-D2D devices; S304: Model the decision-making process of D2D multiplexing VLC or RF spectrum as a Markov Decision Process (MDP). The MDP tuple representation of D2D is: (E t ,a t ,r,p,γ), where r is the reward function value Re. n Set, n∈N; p represents the D2D agent starting from state E t Perform action a t Transition status to E t+1 The probability set whose elements p n The transition probability of agent n is expressed as: This represents the resource state of agent n in time slot t. This represents the resource state of agent n in time slot t+1. Let γ represent the action taken by agent n in time slot t; γ represents the set of reward discount factors, with element values ​​γ∈[0,1]. If γ→0, it means that the agent is concerned with the immediate reward at the current moment. As γ increases, the agent will be more concerned with the future reward value. S306: Calculate the cumulative discounted reward value and the expected value of the cumulative discounted reward after the agent selects an action; The formula for calculating the cumulative discounted reward of agent n in time slot t is as follows: In the above formula, T represents the observation time slot period value; Let be the instantaneous reward value of agent n in time slot t+j, and The formula for calculating the expected value of the cumulative discounted reward for agent n is as follows: In the above formula, J n It is the expected value of the cumulative discounted reward for agent n. For mathematical expectation operations; S4: Use the Actor-Critic (DRL) deep reinforcement learning algorithm to optimize the spectrum selection of D2D multi-agents and improve the expected value of the cumulative discounted reward obtained by D2D multi-agents in the VLC-RF network.

2. The energy-aware spectrum management method for inter-device optical wireless communication based on deep reinforcement learning as described in claim 1, characterized in that: The specific method of S2 is as follows: S201: Calculate the equivalent SINR value for VLC-D2D; The formula for calculating the equivalent SINR value of the nth VLC-D2D pair is as follows: In the above formula, Γ RF =3dB and Γ VLC =5.4dB represents the channel coding loss factor for RF network and VLC network, respectively; S202: Add all D2Ds to the set F of the alliance V alliance. VLC In the middle, judge the alliance F in turn. VLC The equivalent SINR value of the D2D pair, if its equivalent SINR value is less than the set threshold. Then add the D2D pair to the set F of the R alliance. RF Otherwise, the D2D alliance information remains unchanged; in this application, we set S203: Check F sequentially VLC In D2D, if the equivalent SINR value of the VLC spectrum used by the D2D is less than the SINR value of the RF spectrum multiplexed by the D2D, then the D2D is added to the RF spectrum. RF Similarly, check F in sequence. RF In D2D, if the SINR value of the multiplexed RF spectrum of the D2D is less than the equivalent SINR value of the VLC spectrum used by the D2D, then the D2D is added to the RF spectrum. VLC middle; S206: For F RF In D2D, based on the transmit power of the CUEs accessing the uplink RF, the M CUEs are arranged in ascending order as follows: In set M, the strong interference radius of each CUE is determined according to the CUE transmit power. RF-D2Ds that reuse the CUE spectrum and are within the strong interference range of the CUE are found and stored in the RF-D2D set within the strong interference range of the CUE. Among them, the RF-D2D set located within the strong interference range of CUE m It can be represented as: In the above formula, D n Describes the nth pair of D2D, (x m ,y m ) represents the position of CUE m, (x n ,y n R represents the location of the DUE that reuses the spectrum of CUE m and is relatively far from CUE m. m The strong interference radius of CUE m; S207: Determine the set of licensed RF-D2D devices for reusing CUE spectrum, and determine the set of licensed CUEs for each RF-D2D device; Among them, CUE m's licensed RF-D2D device set and F m-D It can be represented as: Among them, the authorized CUE set F of RF-D2D n n-C It can be represented as: In the above formula, C m Let CUE m be the value when C... m Compared to RF-D2D, the larger distance value in C m CUEs located outside the interference radius become one of the candidate CUEs for licensable RF-D2D multiplexing spectrum; S208: Construct the spectrum reusability correlation matrix for D2D multiplexed CUE spectrum; The spectrum reusability correlation matrix for all D2D multiplexed CUE spectrum is defined as follows: In matrix C B In the middle, b m,n This represents a binary variable, b, when the uplink spectrum of CUE m can be licensed to RF-D2Dn for multiplexing. m,n The value is 1, otherwise, b m,n For 0, m∈M, n∈N; in matrix C B In the middle, b 0,n This indicates whether D2D n is authorized to use VLC spectrum for communication; if authorized, then b 0,n If it is 1, then b 0,n It is 0.

3. The energy-aware spectrum management method for inter-device optical wireless communication based on deep reinforcement learning as described in claim 1, characterized in that: The specific method of S4 is as follows: S401: When D2D sends a communication request for spectrum allocation to the VLC-RF system, the control center of the VLC-RF system abstracts D2D as an intelligent agent. The control center downloads historical trajectory data from the experience storage pool in the cloud and trains the Actor-Critic strategy of the deep learning network. S402: Input the request information from the D2D agent, and use the reusable correlation matrix C based on the retrieved spectrum. B Obtain information on the availability of each spectrum block, and statistically analyze the status information E of each spectrum block in the VLC-RF system. t and D2D motion space a t Constructing D2D MDP information (E t ,a t ,r,p,γ); S403: Using the Q-function method of deep learning, calculate the expected value of the cumulative discounted reward for the agent's selection of a spectrum block: Wherein, the agent's resource state-action policy (E) at time t t a t The expected value of the cumulative discount reward calculated using the Q function is: In the above formula, Q π (E t ,a t ) indicates that the agent is in E t Use action a in the state t The expected value of potential long-term discount rewards that can be obtained; E t+1 This represents the resource-action strategy for time slot t+1; E represents the c of each resource block. t The distance vector d between the state and the agent t-1 The set of combinations; P(E) t+1 |E t ,a t ) indicates that the agent uses action a t From the current resource state E t Transition to the next resource state E t+1 The transition probability function value, P(E) t+1 |E t ,a t )∈p; γ∈γ;Q π (E t+1 ,a t ) indicates that the agent is in E t+1 Use action a under resource status t The expected value of potential long-term discount rewards that can be obtained; S404: Use the Bellman optimal equation method to find the agent's optimal state-action policy; The optimal state-action policy of agent n is represented as: In the above formula, π * Describing the optimal strategy (E) * a * ),satisfy The Bellman optimality equation for finding the best state-action policy can be described as follows: S405: A trajectory generated by the interaction between an agent and the VLC-RF resource environment. Update the Q function value; Among them, trajectory data The formula for updating the Q-function value of the nth agent is: In the above formula, η represents the learning rate, η∈[0,1]; S406: In deep reinforcement learning, the global policy of the Actor-Critic for selecting the VLC-RF communication spectrum for all D2D multi-agents is represented as π={π1,...,π n ,...,π N The corresponding network parameters are represented as θ = {θ1,...,θ}. N The output of the Critic network for the nth agent is... The Q-function value represents the expected cumulative discounted reward of the Actor-Critic policy of the nth agent; at the same time, the Q-function value reflects the evaluation value obtained by the agent on the Critic network after taking the Actor action; using a deep neural network as a nonlinear function fitter for the Q-function, a continuous Q-function of the Critic network is obtained. S407: Calculate the network parameters θ when the agent's Q-function value minimizes the loss function, and update the DRL network weight parameters θ; The update formula for the loss function that minimizes the Q-function value of agent n is: In the above formula, This represents the cumulative discounted reward Q-function estimate of agent n using the online policy μ′, where μ′={μ1′,...,μ′} N } is the set of policies for the target Actor network of DRL, and its parameters are copied from the online network; It is the action experience storage pool where the training data of the deep reinforcement learning network is stored, (a1, a2, ..., a N () represents a set of historical action data for N agents in an observable experience storage pool. This represents the set of state-action policies for the L-1 observations prior to time t, with element O. t =[c t ,a t [E] represents the state-action policy observed by the agent in time slot t. t ,a t ], where L is the number of historical observations, and x = (o1,...,o2) N () represents an observable set of state-action policies for N agents. S408: The Deep Deterministic Policies Gradient (DDPG) method is used to find the lower objective gradient value of all agents under the condition of historical state-action policy, which maximizes the expected value of the cumulative discounted reward of the Q function for a given action; then, by comparison, the action of the agent that minimizes the lower objective gradient value is selected. DDPG solves for the action 'a' taken by the nth agent. t The method for representing the lower objective gradient value of the maximum Q-function value is as follows: In the above formula, θ n μ represents the state of agent n when the network parameters are θ. n The network state is represented by θ. n The continuous policy of agent n at time θ∈θ; J(μ) n ) Use a continuous policy μ for agent n n The expected value of cumulative discount rewards over time. This represents solving for the agent's state θ. n J(μ) n Gradient value; The agent n represents a state-action policy O based on history. t Use action a t Solving for the agent's state θ n μ n Gradient value; Represents an agent n based on the set of states-action policies and action sets (a1, a2, ..., a) of N agents x. N ) continuity strategy μ n Q function value, This represents solving for the agent's action a. t strategy μ n The gradient value of the Q-function; S409: Output the optimal set of actions for all agents (a1, a2, ..., a...). N Then, the optimal action set (a1, a2, ..., a) for all agents to obtain the maximum Q-function value is determined. N ) is represented as a t The state c of the agent before optimization t and VLC-RF environment resource status d t Marked as E t The intelligent agent uses optimized action a t The set of states after that is denoted as E. t+1 The computational agent adopts E t+1 Re reward function value Re n The set is denoted as r t+1 Generate a trajectory data (E) t ,a t ,r t+1 E t+1 ), stored in the experience storage pool of DRL In the middle, when the experience storage pool When the trajectory data is full, the trajectory data that has been stored for a long time in the experience storage pool will be replaced with new trajectory data.

Images