Inter-device energy sensing spectrum management in optical wireless communication based on deep reinforcement learning

Abstract

The invention relates to an inter-device energy sensing spectrum management method in optical wireless communication based on deep reinforcement learning, and belongs to the field of optical wireless communication. According to the method, a multi-agent reinforcement learning framework is provided to optimize the performance of D2D (Device to Device) frequency spectrum selection, a D2D frequency spectrum reusable incidence matrix is constructed by comparing equivalent signal to interference noise ratio values of D2D multiplexing visible light or wireless communication frequency spectrums, and the D2D frequency spectrum reusable incidence matrix can be used for realizing D2D frequency spectrum multiplexing. According to the design, residual energy and system energy efficiency of D2D nodes are used as reward function values of a D2D multiplexing optical wireless communication spectrum, and an expected value of an accumulated discount reward obtained by maximizing the D2D multiplexing spectrum is used as an optimization target; by introducing a D2D multi-agent actor-commentator deep reinforcement learning algorithm based on a cooperative game, an expected value of an accumulated discount reward of D2D frequency spectrum multiplexing is improved. According to the method provided by the invention, the problems of load imbalance between cells and low communication quality of edge equipment in indoor optical wireless communication can be improved, and the communication capability of the edge equipment and the energy efficiency of the system are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical wireless communications and relates to a spectrum management method for energy perception between devices in optical wireless communications based on deep reinforcement learning. Background Art

[0002] In recent years, the visible light communication (VLC) technology based on LED (Light-emitting Diode) has attracted extensive attention from researchers due to its advantages of low power consumption, high security, no electromagnetic interference, and the ability to provide approximately 400THz of unlicensed spectrum. Existing research shows that VLC can use visible light waves with a wavelength of 380nm-780nm to control the on and off of LEDs through encoded data information to achieve optical transmission of data information. At the receiving end, the photodetector configured in the receiving device is used to capture the on and off of the high-speed flashing LED, and after a series of signal processing, the original data information transmitted by the sending end can be obtained.

[0003] However, VLC communication faces many problems, such as: its data transmission relies on line of sight (LoS) links, LoS links are easily blocked, VLC access points (AP) have a small coverage range, and dense deployment of multiple VLC APs can easily cause serious interference in inter-cell communication. In order to overcome the above-mentioned shortcomings of VLC, in recent years, some scholars have proposed combining VLC with traditional RF (Radio Frequency) communication technology. The idea of ​​VLC-RF heterogeneous networking has attracted the interest of many scholars in related fields. On the one hand, VLC APs can provide high-speed communication services, and on the other hand, RF APs have a large coverage range and anti-line of sight link blocking characteristics, which can ensure the stability of low-speed transmission links.

[0004] However, with the rapid development of smart cities and the demand for smart applications of all things, when there are too many user devices indoors, the communication between a large number of devices is forwarded through VLC AP or RF AP, which will greatly increase the burden on the AP and consume a lot of limited spectrum and power resources of the VLC-RF heterogeneous network. D2D (Device to Device) communication is a communication technology that allows devices to communicate directly without AP forwarding. In recent years, it has been applied in many fields, such as vehicle-to-vehicle (V2V). For the new generation of Internet scenarios with the intelligent connection of all things, D2D communication technology will greatly reduce communication overhead and improve the communication capabilities of edge devices in the network, increasing the achievable rate of the network.

[0005] However, the spectrum resources used by D2D cause some interference to the cellular communication users of the indoor VLC-RF heterogeneous network. When the number of D2Ds requesting spectrum allocation is large, the VLC-RF heterogeneous network needs to coordinate the contradiction between the spectrum of D2D reused cellular users on the reachable rate of network communication and the interference between cells and the utilization of spectrum resources. Multi-D2D spectrum allocation will increase the complexity of spectrum reuse and interference management of each VLC AP and RF AP. In addition, the energy storage of D2D devices is limited, which limits the endurance communication capability of the device. Therefore, the energy consumption of the device needs to be considered in D2D communication. To this end, in this application, we propose to use D2D communication technology as an auxiliary communication means of VLC-RF, provide D2D communication technology to allow devices to communicate with other devices without connecting to the base station, and perform D2D communication by reusing the VLC-RF cellular user spectrum. Deep reinforcement learning algorithms are used to solve the spectrum selection and interference management problems of D2D reused VLC-RF cellular user spectrum, and improve the energy efficiency performance of D2D reused spectrum. Summary of the invention

[0006] In view of this, the core of the present invention is to provide an energy-aware spectrum management between devices in optical wireless communication based on deep reinforcement learning, so as to improve the energy efficiency of the system while ensuring the QoE of cellular users and the residual energy of D2D nodes.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] S1: Input D2D device set N, VLC AP set, and the transmission power P of each VLC AP V and bandwidth B V ; RF AP transmission power P R and the total bandwidth B R , a set of cellular users M using RF communication; the spectrum of users accessing VLC and RF uses orthogonal frequency division multiplexing, and communication between D2D devices needs to reuse the RF frequency band or reuse VLC carrier resources; according to the Lambertian radiation model, the channel gain and achievable rate value between the VLC-D2D transmitter and receiver using the VLC carrier are calculated, and according to the path loss model, the channel gain and achievable rate value between RF-D2D using the RF frequency band are calculated;

[0009] S2: Calculate the energy collected by the D2D device during the idle time of communication, and construct the spectrum reuse association matrix of the D2D multiplexing CUE spectrum according to the equivalent SINR of VLC-D2D and the SINR value of RF-D2D;

[0010] S3: Taking the residual energy of D2D nodes and system energy efficiency as the reward function value of D2D reused CUE spectrum, a Markov decision model for VLC-D2D and RF-D2D spectrum allocation is established, the state space, action strategy and reward function of the intelligent agent selecting the reused spectrum from the spectrum reuse association matrix are constructed, and the optimization objective function for maximizing the expected value of the cumulative discounted reward of D2D reused CUE spectrum is established;

[0011] S4: Use the Actor-Critic Deep Reinforcement Learning (DRL) algorithm to optimize the spectrum selection of D2D multi-agents and improve the expected value of the cumulative discounted rewards obtained by D2D multi-agents in the VLC-RF network.

[0012] Among them, the specific method of S1 is:

[0013] S101: In an indoor VLC-RF system, there are M cellular user equipments using RF communication, which are stored in a set M; there are N D2D pairs indoors, which are stored in a set N; when a D2D user equipment (DUE) reuses the spectrum of a cellular user equipment (CUE) of RF to transmit data, D2D is called RF-D2D; when a DUE reuses the carrier of VLC for communication, D2D is called VLC-D2D; the total bandwidth of RF is B R It is equally divided among M cellular users, and the center frequency of each bandwidth is the center frequency of the modulated carrier. VLC mainly improves the downlink communication of M users indoors. The total bandwidth of all VLC APs adopts orthogonal frequency division multiple access, and the number of its subcarriers is greater than M. Therefore, downlink optical transmission can avoid interference. Each RF-D2D can only reuse one RF subcarrier in each time slot. The RF subcarrier can be reused by multiple RF-D2Ds in each time slot. Signal interference may occur between D2Ds using the same spectrum, and between CUE and RF-D2D that reuses its carrier for communication. VLC-D2D reuses VLC's optical subcarrier for communication.

[0014] S102: Calculate the channel gain of VLC-D2D communication according to the Lambertian radiation model;

[0015] According to the Lambertian radiation model, the channel gain between the nth pair of VLC-D2D is:

[0016]

[0017] In the above formula (1), represents the Lambertian emissivity, and φ 1 / 2represents the half-power radiation angle of the D2D transmitter using VLC, φ represents the radiation angle of the D2D transmitter; A r Represents the receiving area of ​​the photodetector (PD) of the D2D receiver; Indicates the straight-line distance between the D2D device transmitter and the device receiver; ψ n,n is the incident angle of D2D to the receiver, ψ c is the field of view of the DUE receiver; T s (ψ n,n ) and g(ψ n,n ) represent DUE with ψ n,n The gain of the optical filter and optical concentrator when receiving light signals;

[0018] S103: Calculate the SINR (Signal to Interference plus Noise Ratio) value of the VLC-D2D received signal;

[0019] Among them, the SINR of the nth pair of D2D received signals using VLC communication can be expressed as:

[0020]

[0021] In the above formula (2), P opt It is the average optical power of the DUE transmitter, which is proportional to the DC bias of the LED (Lighting Emitting Diode); η represents the photoelectric conversion coefficient; represents the channel gain between the nth VLC-D2D transmitter and the PD of the corresponding receiver, represents the channel gain between the PD of the i-th pair of VLC-D2D transmitters and the corresponding receiver; is the subcarrier bandwidth of the nth pair of VLC-D2D communication. The VLC frequency band of this application adopts fixed carrier bandwidth allocation. is a constant; N V represents the Gaussian white noise power spectral density of the optical signal received by the DUE;

[0022] S104: Calculate a achievable rate value of a data signal received by the VLC-D2D receiver;

[0023] The achievable rate of the nth pair of VLC-D2D received data signals can be expressed as:

[0024]

[0025] S105: Calculate the channel gain value of RF-D2D according to the path loss model;

[0026] Among them, the channel gain of the nth pair of D2D communication of the reused RF frequency band can be expressed as:

[0027]

[0028] In the above formula, K represents the path loss constant; β n,n represents the fast fading gain caused by multipath effect, which obeys exponential distribution; n,n represents the slow decay gain caused by shadow decay, which obeys the log-normal distribution; d n,n represents the distance between the transmitter and receiver of the nth pair of D2D devices; α is the path loss factor;

[0029] S106: Calculate the SINR value of the RF-D2D received signal;

[0030] The calculation formula of the SINR value of the nth RF-D2D pair is:

[0031]

[0032] In the above formula, α n,m is a binary variable. When RF-D2D n reuses the RF spectrum of cellular user m, α n,m =1, otherwise, α n,m =0; Similarly, when RF-D2D i reuses the RF spectrum of cellular user m, α i,m =1, otherwise, α i,m =0; are the transmit powers of cellular user m and D2D n respectively; and denote the transmission power of the nth and ith D2D pairs respectively; is the channel gain value between the cellular user m and the D2D transmitting end DUE, which can be calculated by the Rayleigh fading model; and Respectively represent the channel gain values ​​of the D2D multiplexed RF frequency communication composed of the nth pair and D2D, DUE i and DUE n; N is the subcarrier bandwidth of the nth pair of RF-D2D communications; R represents the Gaussian white noise power spectrum density of the RF signal received by the DUE;

[0033] S107: Calculate the achievable rate value of the D2D using the multiplexed RF carrier communication according to the Shannon formula;

[0034] The calculation formula for the achievable rate value of the nth pair of D2Ds using multiplexed RF carrier communication is:

[0035]

[0036] In the above formula, B n,m is the subcarrier bandwidth of the nth pair of RF-D2D communication multiplexed RF channel m.

[0037] Among them, the specific method of S2 is:

[0038] S201: Calculate the equivalent SINR value of VLC-D2D;

[0039] The calculation formula of the equivalent SINR value of the nth VLC-D2D pair is:

[0040]

[0041] In the above formula, Γ RF =3dB and Γ VLC =5.4dB represents the channel coding loss factor of RF network and VLC network respectively;

[0042] S202: Add all D2Ds to the alliance V and the set F of the alliance VLC In the above example, we determine the alliance F VLC The equivalent SINR value of the D2D pair in the D2D network is less than the set threshold. Then the D2D pair is added to the set F of the R alliance RF Otherwise, the D2D alliance information remains unchanged; in this application, we set

[0043] The present invention adopts the idea of ​​game theory and divides all D2D into two alliances, namely V alliance and R alliance, which are respectively represented by F VLC and F RF , the D2D device is located at F VLC Use VLC spectrum for communication, located at F RF The D2D devices in the network communicate by reusing the cellular user uplink spectrum.

[0044] S203: Check F in sequence VLC If the equivalent SINR value of the D2D using the VLC spectrum is less than the SINR value of the D2D multiplexing the RF spectrum, the D2D is added to the F RF Similarly, check F RF If the SINR value of the D2D multiplexing RF spectrum is less than the equivalent SINR value of the D2D using the VLC spectrum, the D2D is added to the F VLC middle;

[0045] S206: F RFIn D2D, when RF-D2D is close to the CUE that reuses the RF spectrum, there is a large co-channel interference between RF-D2D and the CUE that uses the same spectrum, so the RF-D2D cannot reuse the spectrum of its neighboring CUE. To this end, we set up a series of authorized cells according to the geographic location information of CUE and RF-D2D, and arrange M CUEs in ascending order according to the transmit power of the CUE that accesses RF in the uplink: In the set M, determine the strong interference radius of each CUE according to the CUE transmit power, find the RF-D2D that reuses the CUE spectrum and is within the strong interference range of the CUE, and save it in the RF-D2D set within the strong interference range of the set CUE;

[0046] Among them, the location information of CUE m will be sent to the base station by uplink. Let R m is the strong interference radius of CUE m, which is positively correlated with the uplink transmission power of CUE m, that is,

[0047] Among them, the RF-D2D set within the strong interference range of CUE m It can be expressed as:

[0048]

[0049] In the above formula, Dn represents the nth pair of D2D, (x m ,y m ) represents the position of CUE m, (x n ,y n ) is the position of DUE that reuses the spectrum of CUE m and is far away from CUE m, R m is the strong interference radius of CUE m;

[0050] S207: Determine a set of authorized RF-D2D devices that reuse CUE spectrum, and determine an authorized CUE set for each RF-D2D;

[0051] Among them, the authorized RF-D2D device set of CUE m and F m-D It can be expressed as:

[0052]

[0053] Among them, the authorized CUE set F of RF-D2D n n-C It can be expressed as:

[0054]

[0055] In the above formula, C m Indicates CUE m, when C m The maximum distance value with RF-D2D is Cm CUEs outside the interference radius become one of the candidate CUEs that can be authorized to use RF-D2D multiplexing spectrum;

[0056] S208: constructing a spectrum reusability association matrix for D2D multiplexing CUE spectrum;

[0057] Among them, the spectrum reusability association matrix of all D2D multiplexing CUE spectrum is defined as:

[0058]

[0059] In the matrix C B Middle, b m,n represents a binary variable. When the uplink spectrum of CUE m can be authorized to be reused by RF-D2D n, then b m,n The value is 1, otherwise, b m,n is 0, m∈M, n∈N; in the matrix C B Middle, b 0,n Indicates whether D2D n is authorized to use the VLC spectrum for communication. If authorized, then b 0,n is 1, otherwise, b 0,n is 0.

[0060] Among them, the specific method of S3 is:

[0061] S301: Abstract each D2D of the indoor VLC-RF as an intelligent agent, and mark the spectrum block resource status of the VLC-RF system as c t =[s t ,v,F t ,D t ,I t ],in, C represents the state set of each time slot of each resource block; s t The index number of the CUE spectrum block available for D2D in time slot t is obtained by retrieving the spectrum reusable association matrix C obtained in step S208. B Get; v indicates whether the spectrum block is a VLC spectrum, if s t is the VLC spectrum, v=1, otherwise, v=0; F t Indicates t Whether the spectrum block is occupied by CUE uplink communication; D t Indicates t Whether the spectrum block is occupied by other D2D; I t Indicates t Whether the spectrum block allocated to the current D2D causes interference to the CUE; the distance vector information of the D2D agent from each CUE in the VLC-RF system is d t =[d1,...,dM ]; then the state of the agent in the VLC-RF system at time slot t is expressed as: E t =[c t ,d t-1 ], where d t-1 Indicates the distance vector information of D2D from each CUE in the previous time slot;

[0062] S302: The action space of the CUE of D2D multiplexing VLC-RF is expressed as: a t =[i t ,k t ],i t is a binary variable, i t =1 means that D2D needs to transmit signals in time slot t; k t Indicates the index number of the CUE spectrum block used by D2D in time slot t, where k t ∈{0,1,...,M},k t =0 means D2D uses the spectrum of VLC for communication;

[0063] S303: Calculate the reward function value of D2D multiplexing CUE spectrum;

[0064] The calculation formula of the reward function value for D2D n multiplexing CUE m spectrum block for communication is:

[0065]

[0066] In the above formula, Re n represents the reward value obtained by D2D n multiplexing CUE spectrum, R n It indicates the achievable rate value obtained by D2D n multiplexing VLC or RF spectrum. β represents the weight coefficient between the remaining energy of the device and the achievable rate obtained by accessing VLC-RF; r neg represents the penalty value of the interference introduced by the agent's reuse of the CUE spectrum block, which leads to a reduction in the achievable rate of the CUE or a reduction in the residual energy. In this application, let r neg = -1; Indicates the achievable rate value of CUE. Indicates the communication rate threshold of CUE; P represents the energy threshold of D2D device communication; total It is the total transmission power value of D2D device communication in the VLC-RF system, which is determined by the transmission power of the communication device connected to the VLC AP, the transmission power of the CUE device connected to the RF AP, and the D2D transmission power. Indicates the average remaining energy value of all connected VLC-D2D and RF-D2D;

[0067] S304: Model the decision of D2D to reuse VLC or RF spectrum as a Markov decision process (MDP). The MDP tuple of D2D is represented as: t ,a t ,r,p,γ), where r is the reward function value Re n Set, n∈N; p represents the D2D agent from state E t Execute action a t Transfer status to E t+1 The probability set whose element p n represents the transition probability of agent n, expressed as: represents the resource status of agent n in time slot t, represents the resource status of agent n in time slot t+1, represents the action taken by agent n at time slot t; γ represents the set of discount factors for rewards, and its element value γ∈[0,1]. If γ→0, it means that the agent cares about the immediate reward at the current moment. As γ increases, the agent will care more about the future reward value.

[0068] S306: Calculate the cumulative discounted reward value and the expected value of the cumulative discounted reward after the agent selects the action;

[0069] The calculation formula for the cumulative discounted reward of agent n in the t time slot is:

[0070]

[0071] In the above formula, T represents the observation time slot period value; is the instantaneous reward value of agent n at time slot t+j, and

[0072] The calculation formula for the expected value of the cumulative discounted reward of agent n is:

[0073]

[0074] In the above formula, J n is the expected value of the cumulative discounted reward of agent n of the agent, which is also the optimization goal of this application;

[0075] Wherein, the specific method of S4 is:

[0076] S401: When D2D sends a communication request for spectrum allocation to the VLC-RF system, the control center of the VLC-RF system abstracts D2D as an intelligent agent, downloads historical trajectory data from the experience storage pool from the cloud, and trains the Actor-Critic strategy of the deep learning network;

[0077] S402: Input the request information of the intelligent agent D2D, and retrieve the spectrum reusable association matrix C B Get the information of whether each spectrum block is available, and count the status information of each spectrum block in the VLC-RF system t and the action space a of D2D t , construct the D2D MDP information (E t ,a t ,r,p,γ);

[0078] S403: Using the Q function method of deep learning, calculate the expected value of the cumulative discounted reward for the agent's selection of spectrum blocks:

[0079] Among them, the agent’s resource state-action strategy (E t , a t )’s Q function calculates the expected value of the cumulative discounted reward as:

[0080]

[0081] In the above formula, Q π (E t ,a t ) indicates that the agent is in E t In state a t The expected value of the potential long-term discounted reward that can be obtained; E t+1 represents the resource-action strategy for time slot t+1; E represents the c of each resource block t The distance vector d between the state and the agent t-1 The combination set of P(E t+1 |E t ,a t ) indicates that the agent takes action a t , from the current resource state E t Transition to next resource state E t+1 The transition probability function value, P(E t+1 |E t ,a t )∈p; γ∈γ;Q π (E t+1 ,a t ) indicates that the agent is in E t+1 Take action a in resource state t the expected value of potential long-term discounted rewards that can be obtained;

[0082] S404: Use Bellman optimal equation method to find the best state-action strategy of the intelligent agent;

[0083] Among them, the optimal state-action strategy of agent n is expressed as:

[0084]

[0085] In the above formula, π * represents the best strategy (E * , a * ),satisfy

[0086] Among them, the Bellman optimal equation for finding the best state-action strategy can be described as:

[0087]

[0088] S405: A trajectory generated by the interaction between the agent and the VLC-RF resource environment Update the Q function value;

[0089] Among them, trajectory data The formula for updating the Q function value of the nth agent is:

[0090]

[0091] In the above formula, η represents the learning rate, η∈[0,1];

[0092] S406: In deep reinforcement learning, the Actor-Critic global strategy for all D2D multi-agents to select the VLC-RF communication spectrum is expressed as π = {π1, ..., π n ,...,π N}, the corresponding network parameters are expressed as θ={θ1,...,θ N}; The output of the Critic network of the nth agent is The Q function value represents the expected value of the cumulative discounted reward of the Actor-Critic strategy of the nth agent; at the same time, the Q function value reflects the evaluation value obtained by the agent on the Critic network after taking the Actor action; using a deep neural network as a nonlinear function fitter of the Q function, the continuous Q function of the Critic network is obtained;

[0093] S407: Calculate the network parameter θ of the agent's Q function value when minimizing the loss function, and update the DRL network weight parameter θ;

[0094] Among them, the update formula of the minimization loss function of the Q function value of agent n is:

[0095]

[0096] In the above formula, represents the estimated value of the cumulative discounted reward Q function of agent n using the online strategy μ′, where μ′={μ′1,...,μ′ N} is the strategy set of the target Actor network of DRL, and its parameters are copied from the online network. The online network copies its parameters to the Actor target network of DRL every certain number of steps to prevent the Actor network from overestimating the Q function value of the agent. It is the action experience storage pool where the training data of the deep reinforcement learning network is stored, (a1, a2, ..., a N ) is a set of historical action data of N agents in the observable experience storage pool, represents the state-action strategy set of L-1 observations before time t, whose element O t =[c t ,a t ] is the state-action strategy observed by the agent at time slot t [E t ,a t ], L is the number of historical observations, x=(o1,...,o N ) represents a set of observable state-action strategies for N agents,

[0097] S408: Using the Deep Deterministic Policies Gradient (DDPG) method, find the lower target gradient value of the maximum cumulative discounted reward expectation value of the Q function for a given action of all agents under the condition of the state-action strategy based on the history; then, by comparison, select the action of the agent that minimizes the small target gradient value;

[0098] The main idea of ​​DDPG is to use The directional iteration method of gradient descent adjusts the parameter θ of the trained Critic policy network to maximize the Q function value, in order to overcome the disadvantage of high variance gradient estimation often shown by the gradient policy method in multi-agent environments;

[0099] Among them, DDPG solves the action a taken by the nth agent t The lower target gradient value of the maximum Q function value is expressed as:

[0100]

[0101] In the above formula, θ n represents the state of agent n when the network parameter is θ, μ n Indicates that the network state is θ n The continuous strategy of agent n at the time, θ∈θ; J(μ n) uses the continuous strategy μ for agent n n The expected value of the cumulative discounted reward when Represents the solution to the agent state θ n J(μ n ) gradient value; Represents the state-action strategy O of agent n based on history t Take action a t Solve for the agent state θ n μ n Gradient value; represents the set of x state-action strategies and actions (a1, a2, ..., a N )'s continuity strategy μ n The Q function value of Represents the solution of the agent action a t Strategy μ n The gradient value of the Q function;

[0102] S409: Output the optimal action set of all agents (a1, a2, ..., a N ); Then, the optimal action set (a1, a2, ..., a N ) is represented by a t , the state c before the agent optimization t and VLC-RF environment resource status d t Marked as E t , the agent takes the optimized action a t The state set after is recorded as E t+1 , the computational agent adopts E t+1 The reward function value Re n The set is denoted by r t+1 , generate a trajectory data (E t ,a t ,r t+1 ,E t+1 ), stored in the DRL experience pool In the experience pool When the trajectory data in the experience storage pool is full, the new trajectory data will replace the trajectory data with the longest storage time in the experience storage pool. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] In order to make the purpose, technical solution and advantages of the present invention clearer, the invention will be further described below with reference to the accompanying drawings:

[0104] Figure 1 Schematic diagram of D2D spectrum allocation in indoor VLC-RF heterogeneous networks;

[0105] Figure 2 Interference diagram of indoor D2D spectrum allocation;

[0106] Figure 3 Diagram of D2D spectrum allocation model based on cooperative game-based multi-agent reinforcement learning; Specific implementation plan

[0107] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents described in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and the features in the embodiments can be combined with each other without conflict.

[0108] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0109] Attached Figure 1 Schematic diagram of D2D spectrum management in indoor VLC-RF heterogeneous networks. Figure 1 In the figure, there is one RF AP and multiple VLC APs using LEDs for lighting. There are M cellular user equipments (CUEs), represented by M = {1, ..., M}; there are N D2D pairs, represented by N = {1, ..., N}. When the D2D user equipment (DUE) reuses the uplink spectrum of the CUE for data transmission, it is called RF-D2D; when the DUE uses the VLC spectrum for communication, it is called VLC-D2D; the transmission power of each VLC AP is P V and bandwidth B V ; RF (Radio Frequency) AP's transmission power P R and the total bandwidth B R, the number of RF subchannels in the uplink is the same as the number of CUEs in the network. By using orthogonal frequency division multiple access, there is no interference between CUEs. The spectrum of CUE can be multiplexed by multiple RF-D2Ds in each time slot, while each RF-D2D can only multiplex the spectrum of one CUE in each time slot. It is assumed that all D2D devices are within the coverage of the RF AP. When the spectrum of CUEm is multiplexed by RF-D2D, RF-D2D may be interfered with by CUEm, and CUEm may also be interfered with by RF-D2D. When different D2D pairs use the same spectrum, interference of the same-frequency signals may also occur between these same-frequency DUEs.

[0110] Attached Figure 2 Figure 1 is a schematic diagram of interference in indoor D2D spectrum allocation under VLC-RF heterogeneous network, including multiple pairs of VLC-D2D communicating using VLC spectrum and multiple pairs of RF-D2D communicating using RF spectrum. The coverage radius of the RF AP transmission signal is R, and the coverage radius of each VLC AP optical signal is R. m ; In order to solve the co-channel interference between RF CUE and RF-D2D and between different D2Ds that reuse the same CUE spectrum, the interference size is estimated according to the distance between CUE and RF-D2D, and the SINR value is calculated to avoid co-channel interference between RF-D2D and CUE that reuse the CUE spectrum; the interference size is estimated according to the distance between different D2Ds, and the SINR value is calculated to avoid different D2D pairs that are close to each other from reuse the same CUE spectrum. In addition, a similar method as above is used to avoid using the same spectrum between VLC-D2D and VLC CUE that are close to each other.

[0111] Attached Figure 3 Figure 2 is a D2D spectrum allocation model based on cooperative game-based multi-agent reinforcement learning for indoor use. Each pair of D2D in the VLC-RF heterogeneous network is abstracted as an agent in the deep reinforcement learning network. When each agent selects to reuse the CUE spectrum of the VLC-RF heterogeneous network for communication, it adopts the Actor-Critic strategy and first downloads historical trajectory data from the cloud to train the parameter set θ of the deep reinforcement learning network. Each agent observes the resource status in the VLC-RF heterogeneous network at time slot t and obtains the status information c of each spectrum block. t =[s t ,v,F t ,D t ,I t ], check the position information between CUEs t =[d1,...,d M ]; determine the state information E of each agent in the spectrum block in the VLC-RF system at time slot t t =[c t ,d t-1], Actor tries to allocate spectrum blocks to the agent, and the effect of the spectrum allocation action is evaluated by Critic. The evaluation effect of this application is to calculate the expected value of the cumulative discounted reward of each agent for reusing the spectrum block. This value reflects the achievable rate and energy consumption cost of the agent, and can reflect the energy efficiency of the agent's contribution to the VLC-RF system. Through the deep deterministic gradient policy gradient method, the spectrum block selection action that maximizes the expected value of the cumulative discounted reward of each agent is found. This action represents the spectrum block selected by the agent. Then, the resource status information E of all spectrum blocks in the system before the agent selects the spectrum block with the maximum expected value of the cumulative discounted reward is obtained. t , select the action information a of the spectrum block t , spectrum block resource status information E after spectrum block selection t+1 , the expected value of the cumulative discounted reward obtained after selecting the spectrum block r t+1 , forming a set of trajectory data (E t , a t , r t+1 , E t+1 ), upload the trajectory to the experience storage pool of trajectory data as the training data for the Actor-Critic strategy of the agent requesting spectrum allocation in the next time slot. When the buffer in the experience storage pool is full of data, the new trajectory data will replace the trajectory data stored in the experience storage pool for a long time.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. Energy-aware spectrum management between devices in optical wireless communications based on deep reinforcement learning, characterized by: The method comprises the following steps: S1: Input D2D (Device to Device) device set N, VLC (Visible Light Communication) access point (AP) set, and the transmission power P of each VLC AP V and bandwidth B V ; RF (Radio Frequency) AP's transmission power P R and the total bandwidth B R , a set of cellular users M using RF communication; the spectrum of users accessing VLC and RF uses orthogonal frequency division multiplexing, and communication between D2D devices needs to reuse the RF frequency band or reuse VLC carrier resources; according to the Lambertian radiation model, the channel gain and achievable rate value between the VLC-D2D transmitter and receiver using the VLC carrier are calculated, and according to the path loss model, the channel gain and achievable rate value between RF-D2D using the RF frequency band are calculated; The steps for calculating the achievable rates of VLC-D2D and RF-D2D are as follows: S101: In an indoor VLC-RF system, there are M cellular user equipments using RF communication, which are stored in a set M; there are N D2D pairs indoors, which are stored in a set N; when a D2D user equipment (DUE) reuses the spectrum of a cellular user equipment (CUE) of RF to transmit data, D2D is called RF-D2D; when a DUE reuses the carrier of VLC for communication, D2D is called VLC-D2D; the total bandwidth of RF is B R It is equally divided among M cellular users, and the center frequency of each bandwidth is the center frequency of the modulated carrier. VLC mainly improves the downlink communication of M users indoors. The total bandwidth of all VLC APs adopts orthogonal frequency division multiple access, and the number of its subcarriers is greater than M. Therefore, downlink optical transmission can avoid interference. Each RF-D2D can only reuse one RF subcarrier in each time slot. The RF subcarrier can be reused by multiple RF-D2Ds in each time slot. Signal interference may occur between D2Ds using the same spectrum, and between CUE and RF-D2D that reuses its carrier for communication. VLC-D2D reuses VLC's optical subcarrier for communication. S102: Calculate the channel gain of VLC-D2D communication according to the Lambertian radiation model; According to the Lambertian radiation model, the channel gain between the nth pair of VLC-D2D is: In the above formula, θ represents the Lambertian radiation coefficient, and θ = -ln2 / ln(cosφ 1 / 2 ), φ1 / 2 represents the half-power radiation angle of the D2D transmitter using VLC, and φ represents the radiation angle of the D2D transmitter; A r Represents the receiving area of ​​the photodetector (PD) of the D2D receiver; Indicates the straight-line distance between the D2D device transmitter and the device receiver; ψ n,n is the incident angle of D2D to the receiver, ψ c is the field of view of the DUE receiver; T s (ψ n,n ) and g(ψ n,n ) represent DUE with ψ n,n The gain of the optical filter and optical concentrator when receiving light signals; S103: Calculate the SINR (Signal to Interference plus Noise Ratio) value of the VLC-D2D received signal; Among them, the SINR of the nth pair of D2D received signals using VLC communication can be expressed as: In the above formula, P opt It is the average optical power of the DUE transmitter, which is proportional to the DC bias of the LED (Lighting Emitting Diode); η represents the photoelectric conversion coefficient; represents the channel gain between the nth VLC-D2D transmitter and the PD of the corresponding receiver, represents the channel gain between the PD of the i-th pair of VLC-D2D transmitters and the corresponding receiver; is the subcarrier bandwidth of the nth pair of VLC-D2D communication. The VLC frequency band of this application adopts fixed carrier bandwidth allocation. is a constant; N V represents the Gaussian white noise power spectral density of the optical signal received by the DUE; S104: Calculate a achievable rate value of a data signal received by the VLC-D2D receiver; The achievable rate of the nth pair of VLC-D2D received data signals can be expressed as: S105: Calculate the channel gain value of RF-D2D according to the path loss model; Among them, the channel gain of the nth pair of D2D communication of the reused RF frequency band can be expressed as: In the above formula, K represents the path loss constant; β n,n represents the fast fading gain caused by multipath effect, which obeys exponential distribution; n,n represents the slow decay gain caused by shadow decay, which obeys the log-normal distribution; d n,n represents the distance between the transmitter and receiver of the nth pair of D2D devices; α is the path loss factor; S106: Calculate the SINR value of the RF-D2D received signal; The calculation formula of the SINR value of the nth RF-D2D pair is: In the above formula, α n,m is a binary variable. When RF-D2D n reuses the RF spectrum of cellular user m, α n,m =1, otherwise, α n,m =0; Similarly, when RF-D2D i reuses the RF spectrum of cellular user m, α i,m =1, otherwise, α i,m =0; are the transmit powers of cellular user m and D2D n respectively; and denote the transmission power of the nth and ith D2D pairs respectively; is the channel gain value between the cellular user m and the D2D transmitting end DUE, which can be calculated by the Rayleigh fading model; and Respectively represent the channel gain values ​​of the D2D multiplexed RF frequency communication composed of the nth pair and D2D, DUE i and DUE n; N is the subcarrier bandwidth of the nth pair of RF-D2D communications; R represents the Gaussian white noise power spectrum density of the RF signal received by the DUE; S107: Calculate the achievable rate value of the D2D using the multiplexed RF carrier communication according to the Shannon formula; The calculation formula for the achievable rate value of the nth pair of D2Ds using multiplexed RF carrier communication is: In the above formula, B n,m is the subcarrier bandwidth of the nth pair of RF-D2D communication multiplexing RF channel m; S2: adopt the method of claim 2 to calculate the energy collected by the D2D device during the idle time of communication, and construct a spectrum reusability association matrix of the D2D multiplexing CUE spectrum according to the equivalent SINR of VLC-D2D and the SINR value of RF-D2D; S3: adopt the method of claim 3, take the residual energy of the D2D node and the system energy efficiency as the reward function value of the D2D reused CUE spectrum, establish a Markov decision model for VLC-D2D and RF-D2D spectrum allocation, construct the state space, action strategy and reward function of the intelligent agent to select the reused spectrum from the spectrum reuse association matrix, and establish an optimization objective function for maximizing the expected value of the cumulative discounted reward of the D2D reused CUE spectrum; S4: The method of claim 4 is adopted, and the Actor-Critic Deep Reinforcement Learning (DRL) algorithm is used to optimize the spectrum selection of D2D multi-agents and improve the expected value of the cumulative discounted rewards obtained by D2D multi-agents in the VLC-RF network.

2. The energy-aware spectrum management between devices in optical wireless communication based on deep reinforcement learning as claimed in claim 1, characterized in that: The specific method of S2 is: S201: Calculate the equivalent SINR value of VLC-D2D; The calculation formula of the equivalent SINR value of the nth VLC-D2D pair is: In the above formula, Γ RF =3dB and Γ VLC =5.4dB represents the channel coding loss factor of RF network and VLC network respectively; S202: Add all D2Ds to the alliance V and the set F of the alliance VLC In the above example, we determine the alliance F VLC The equivalent SINR value of the D2D pair in the D2D network is less than the set threshold. Then the D2D pair is added to the set F of the R alliance RF Otherwise, the D2D alliance information remains unchanged; in this application, we set S203: Check F in sequence VLC If the equivalent SINR value of the D2D using the VLC spectrum is less than the SINR value of the D2D multiplexing the RF spectrum, the D2D is added to the F RF Similarly, check F RF If the SINR value of the D2D multiplexing RF spectrum is less than the equivalent SINR value of the D2D using the VLC spectrum, the D2D is added to the F VLC middle; S206: F RF In D2D, according to the transmit power of the CUE accessing RF in the uplink, the M CUEs are arranged in ascending order as follows: In the set M, determine the strong interference radius of each CUE according to the CUE transmit power, find the RF-D2D that reuses the CUE spectrum and is within the strong interference range of the CUE, and save it in the RF-D2D set within the strong interference range of the set CUE; Among them, the RF-D2D set within the strong interference range of CUE m It can be expressed as: In the above formula, D n represents the nth pair of D2D, (x m ,y m ) represents the position of CUE m, (x n ,y n ) is the position of DUE that reuses the spectrum of CUE m and is far away from CUE m, R m is the strong interference radius of CUE m; S207: Determine a set of authorized RF-D2D devices that reuse CUE spectrum, and determine an authorized CUE set for each RF-D2D; Among them, the authorized RF-D2D device set of CUE m and F m-D It can be expressed as: Among them, the authorized CUE set F of RF-D2D n n-C It can be expressed as: In the above formula, C m Indicates CUE m, when C m The maximum distance value with RF-D2D is C m CUEs outside the interference radius become one of the candidate CUEs that can be authorized to use RF-D2D multiplexing spectrum; S208: constructing a spectrum reusability association matrix for D2D multiplexing CUE spectrum; Among them, the spectrum reusability association matrix of all D2D multiplexing CUE spectrum is defined as: In the matrix C B Middle, b m,n represents a binary variable. When the uplink spectrum of CUE m can be authorized to be reused by RF-D2Dn, then b m,n The value is 1, otherwise, b m,n is 0, m∈M, n∈N; in the matrix C B Middle, b 0,n Indicates whether D2D n is authorized to use the VLC spectrum for communication. If authorized, b 0,n is 1, otherwise, b 0,n is 0.

3. The energy-aware spectrum management between devices in optical wireless communication based on deep reinforcement learning as claimed in claim 1, characterized in that: The S3 specific method is: S301: Abstract each D2D of the indoor VLC-RF as an intelligent agent, and mark the spectrum block resource status of the VLC-RF system as c t =[s t ,v,F t ,D t ,I t ],in, C represents the state set of each time slot of each resource block; s t The index number of the CUE spectrum block available for D2D in time slot t is obtained by retrieving the spectrum reusable association matrix C obtained in step S208. B Get; v indicates whether the spectrum block is a VLC spectrum, if s t is the VLC spectrum, v=1, otherwise, v=0; F t Indicates t Whether the spectrum block is occupied by CUE uplink communication; D t Indicates t Whether the spectrum block is occupied by other D2D; I t Indicates t Whether the spectrum block allocated to the current D2D causes interference to the CUE; the distance vector information of the D2D agent from each CUE in the VLC-RF system is d t =[d1,...,d M ]; then the state of the agent in the VLC-RF system at time slot t is expressed as: E t =[c t ,d t-1 ], where d t-1 Indicates the distance vector information of D2D from each CUE in the previous time slot; S302: The action space of the CUE of D2D multiplexing VLC-RF is expressed as: a t =[i t ,k t ],i t is a binary variable, i t =1 means that D2D needs to transmit signals in time slot t; k t Indicates the index number of the CUE spectrum block used by D2D in time slot t, where k t ∈{0,1,...,M},k t =0 means D2D uses the spectrum of VLC for communication; S303: Calculate the reward function value of D2D multiplexing CUE spectrum; The calculation formula of the reward function value for D2D n multiplexing CUE m spectrum block for communication is: In the above formula, Re n represents the reward value obtained by D2D n multiplexing CUE spectrum, R n It indicates the achievable rate value obtained by D2D n multiplexing VLC or RF spectrum. β represents the weight coefficient between the remaining energy of the device and the achievable rate obtained by accessing VLC-RF; r neg represents the penalty value of the interference introduced by the agent's reuse of the CUE spectrum block, which leads to a reduction in the achievable rate of the CUE or a reduction in the residual energy. In this application, let r neg = -1; Indicates the achievable rate value of CUE. Indicates the communication rate threshold of CUE; P represents the energy threshold of D2D device communication; total It is the total transmission power value of D2D device communication in the VLC-RF system, which is determined by the transmission power of the communication device connected to the VLC AP, the transmission power of the CUE device connected to the RF AP, and the D2D transmission power. Indicates the average remaining energy value of all connected VLC-D2D and RF-D2D; S304: Model the decision of D2D to reuse VLC or RF spectrum as a Markov decision process (MDP). The MDP tuple of D2D is represented as: t ,a t ,r,p,γ), where r is the reward function value Re n Set, n∈N; p represents the D2D agent from state E t Execute action a t Transfer status to E t+1 The probability set whose element p n represents the transition probability of agent n, expressed as: represents the resource status of agent n in time slot t, represents the resource status of agent n in time slot t+1, represents the action taken by agent n at time slot t; γ represents the set of discount factors for rewards, and its element value γ∈[0,1]. If γ→0, it means that the agent cares about the immediate reward at the current moment. As γ increases, the agent will care more about the future reward value. S306: Calculate the cumulative discounted reward value and the expected value of the cumulative discounted reward after the agent selects the action; The calculation formula for the cumulative discounted reward of agent n in the t time slot is: In the above formula, T represents the observation time slot period value; is the instantaneous reward value of agent n at time slot t+j, and The calculation formula for the expected value of the cumulative discounted reward of agent n is: In the above formula, J n is the expected value of the cumulative discounted reward for agent n of the agents.

4. The energy-aware spectrum management between devices in optical wireless communication based on deep reinforcement learning as claimed in claim 1, characterized in that: The specific method of S4 is: S401: When D2D sends a communication request for spectrum allocation to the VLC-RF system, the control center of the VLC-RF system abstracts D2D as an intelligent agent, downloads historical trajectory data from the experience storage pool from the cloud, and trains the Actor-Critic strategy of the deep learning network; S402: Input the request information of the intelligent agent D2D, and retrieve the spectrum reusable association matrix C B Get the information of whether each spectrum block is available, and count the status information of each spectrum block in the VLC-RF system t and the action space a of D2D t , construct the D2D MDP information (E t ,a t ,r,p,γ); S403: Using the Q function method of deep learning, calculate the expected value of the cumulative discounted reward for the agent's selection of spectrum blocks: Among them, the agent’s resource state-action strategy (E t , a t )’s Q function calculates the expected value of the cumulative discounted reward as: In the above formula, Q π (E t ,a t ) indicates that the agent is in E t In state a, take action t The expected value of the potential long-term discounted reward that can be obtained; E t+1 represents the resource-action strategy for time slot t+1; E represents the c of each resource block t The distance vector d between the state and the agent t-1 The combination set of P(E t+1 |E t ,a t ) indicates that the agent takes action a t , from the current resource state E t Transition to next resource state E t+1 The transition probability function value, P(E t+1 |E t ,a t )∈p; R n t ∈r;γ∈γ;Q π (E t+1 ,a t ) indicates that the agent is in E t+1 Take action a in resource state t the expected value of potential long-term discounted rewards that can be obtained; S404: Use Bellman optimal equation method to find the best state-action strategy of the intelligent agent; Among them, the optimal state-action strategy of agent n is expressed as: In the above formula, π * represents the best strategy (E * , a * ),satisfy Among them, the Bellman optimal equation for finding the best state-action strategy can be described as: S405: A trajectory generated by the interaction between the agent and the VLC-RF resource environment Update the Q function value; Among them, trajectory data The formula for updating the Q function value of the nth agent is: In the above formula, η represents the learning rate, η∈[0,1]; S406: In deep reinforcement learning, the Actor-Critic global strategy for all D2D multi-agents to select the VLC-RF communication spectrum is expressed as π = {π1, ..., π n ,...,π N }, the corresponding network parameters are expressed as θ={θ1,...,θ N }; The output of the Critic network of the nth agent is The Q function value represents the expected value of the cumulative discounted reward of the Actor-Critic strategy of the nth agent; at the same time, the Q function value reflects the evaluation value obtained by the agent on the Critic network after taking the Actor action; using a deep neural network as a nonlinear function fitter of the Q function, the continuous Q function of the Critic network is obtained; S407: Calculate the network parameter θ of the agent's Q function value when minimizing the loss function, and update the DRL network weight parameter θ; Among them, the update formula of the minimization loss function of the Q function value of agent n is: In the above formula, represents the estimated value of the cumulative discounted reward Q function of agent n using the online strategy μ′, where μ′={μ1′,...,μ′ N } is the strategy set of the target Actor network of DRL, and its parameters are copied from the online network; It is the action experience storage pool where the training data of the deep reinforcement learning network is stored, (a1, a2, ..., a N ) is a set of historical action data of N agents in the observable experience storage pool, represents the state-action strategy set of L-1 observations before time t, whose element O t =[c t ,a t ] is the state-action strategy observed by the agent at time slot t [E t ,a t ], L is the number of historical observations, x=(o1,...,o N ) represents a set of observable state-action strategies for N agents, S408: Using the Deep Deterministic Policies Gradient (DDPG) method, find the lower target gradient value of the maximum cumulative discounted reward expectation value of the Q function for a given action of all agents under the condition of the state-action strategy based on the history; then, by comparison, select the action of the agent that minimizes the small target gradient value; Among them, DDPG solves the action a taken by the nth agent t The lower target gradient value of the maximum Q function value is expressed as: In the above formula, θ n represents the state of agent n when the network parameter is θ, μ n Indicates that the network state is θ n The continuous strategy of agent n at the time, θ∈θ; J(μ n ) uses the continuous strategy μ for agent n n The expected value of the cumulative discounted reward when Represents the solution to the agent state θ n J(μ n ) gradient value; Represents the state-action strategy O of agent n based on history t Take action a t Solve for the agent state θ n μ n Gradient value; represents the set of x state-action strategies and actions (a1, a2, ..., a N )'s continuity strategy μ n The Q function value of Represents solving the agent action a t Strategy μ n The gradient value of the Q function; S409: Output the optimal action set of all agents (a1, a2, ..., a N ); Then, the optimal action set (a1, a2, ..., a N ) is represented by a t , the state c before the agent optimization t and VLC-RF environment resource status d t Marked as E t , the agent takes the optimized action a t The state set after is recorded as E t+1 , the computational agent adopts E t+1 The reward function value Re n The set is denoted by r t+1 , generate a trajectory data (E t ,a t ,r t+1 ,E t+1 ), stored in the DRL experience pool In the experience pool When the trajectory data in the experience storage pool is full, the new trajectory data will replace the trajectory data with the longest storage time in the experience storage pool.

Images