A multi-agent deep Q network optimization indoor VLC-RF resource energy efficiency allocation method

Abstract

The present application relates to a kind of multi-agent deep Q network optimization indoor VLC-RF resource energy efficiency allocation method, belong to optical wireless communication technical field.The present application is to improve the system energy efficiency of VLC-RF heterogeneous network, user is abstracted as the agent of deep Q network, user selects the action of VLC-RF access point, power distribution and subcarrier resource allocation is abstracted as the action of agent, the product of the proportional fair satisfaction degree value that considers user to the communication rate alliance satisfaction or to the channel reliability alliance satisfaction and system energy consumption profit value is designed as the reward Q value of deep Q network, using a variable greed factor control agent's alliance conversion, obtains the access AP channel state of user side emphasis;Then, using deep Q network selects the action of maximum Q value for user, optimizes multiple user association access point in VLC-RF system, access point transmission power and subcarrier resource allocation.The proposed method can improve the system energy efficiency of VLC-RF while increasing user satisfaction.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of optical wireless communication, and relates to a multi-agent deep Q network optimization indoor VLC-RF resource energy efficiency allocation method. BACKGROUND

[0002] In recent years, visible light communication (VLC) technology based on LED (Light Emitting Diode) has attracted extensive attention of researchers because it has the advantages of low power consumption, high security, no electromagnetic interference communication and can provide about 400THz of unlicensed spectrum while meeting indoor lighting. VLC can utilize visible light waves with a wavelength of 380nm-780nm to control the on and off of LEDs through coded information, and can realize high-speed transmission of data information. At the receiving end, the on and off of high-speed flashing LEDs is captured by a photodetector, and after a series of signal processing, the original information transmitted by the sending end can be decoded.

[0003] However, VLC also has its limitations, such as data transmission relying on line of sight (LoS) link, small coverage of VLC access point (AP), and intensive deployment of multiple VLC APs causing serious interference. In order to overcome the above-mentioned shortcomings of VLC, some scholars have proposed to combine VLC and traditional RF (Radio Frequency) communication technology in recent years, which has attracted the interest of many scholars in related fields. On the one hand, VLC AP can provide high-speed data communication services, and on the other hand, RF AP can further ensure the stability and reliability of the link. This is powerful for improving the satisfaction experience of users with different satisfaction requirements, including high-speed requirement users or high reliability requirement users. In the VLC-RF (Visible Light Communication-Radio Frequency) heterogeneous network, users can adopt different access and resource allocation schemes according to the link state to obtain an improved satisfaction access strategy, thereby forming a user classification of high-speed satisfaction users or high-reliability satisfaction users. In the future indoor VLC-RF heterogeneous network, a satisfactory resource allocation scheme is essential.

[0004] For VLC-RF heterogeneous network, traditional resource allocation methods, such as convex optimization, linear programming method, etc. need to obtain channel state information in advance, which is often more difficult in the case of indoor mobile user equipment. Deep learning algorithm has higher universality and stronger robustness than convex optimization method when dealing with optimization problems. The main principle is to represent the user equipment as an agent in an unknown channel state environment. The agent interacts with the current environment by maximizing the long-term reward. Similar to the mechanism of seeking benefits and avoiding harm, the agent obtains the optimal strategy corresponding to the maximum reward value of the user equipment accessing a certain VLC AP or RF AP, controlling the transmission power value of the AP and selecting the subcarrier channel of the AP. This optimal strategy will guide the agent to adjust the access AP, adjust the AP transmission power and adjust the subcarrier allocation action. The channel state and resource state of VLC-RF network change over time, and the strategy of the agent also needs to be adjusted constantly. The agent can use iterative algorithms to constantly explore and try the action selection to obtain the optimal access AP, control AP transmission power and allocate subcarrier action. Therefore, in the VLC-RF network, the dilemma faced by the heuristic algorithm and the convex optimization algorithm, such as the user and the central controller cannot obtain the global channel state information and resource availability due to the dynamic change of the VLC-RF network resources, the complexity of the algorithm for user access AP and resource allocation is too high when the number of indoor devices is large, the deep learning algorithm can better solve the above problems. SUMMARY

[0005] Therefore, the core of the present application is to provide a multi-agent deep Q network optimization indoor VLC-RF resource energy efficiency allocation method, which can improve the energy efficiency of the system while ensuring the fairness of the users.

[0006] In order to achieve the above purpose, the present application provides the following technical scheme:

[0007] S1: input user equipment set M, each user request rate value, 1 RF AP (Radio Frequency Access Point, radio frequency access point) and N VLC APs are saved in AP set N, the RF AP is in set N The serial number is 0, and the subchannel set of the AP is K, and the transmission power of the VLC AP is P V , the transmission power of the RF AP is P R ; set the observation time slot number T, the SINR threshold, the discount factor γ, the profit value ζ of the unit communication reachable rate, the generation value σ required by the unit transmission power, and the weight factor κ of the communication rate demand satisfied by the system; calculate the channel gain and the achievable reachable rate value of the user equipment accessing the VLC AP and the RF AP, and calculate the total reachable rate value of the system;

[0008] S2: According to the achievable rate value, the average achievable rate and average power consumption of each AP per unit scheduling period are calculated to construct the energy consumption profit function of the AP; the satisfaction degree proportional fairness function of the user request to access different APs is constructed, the probability optimization method is used to optimize the user alliance conversion according to the link state and satisfaction demand by using the method of coalition game, and the optimization objective function reflecting the system energy efficiency and user satisfaction is constructed;

[0009] S3: The VLC-RF access user and the allocated resource are represented by a Markov decision process, the user is abstracted as an agent, the state space, action space, state transition probability matrix of the agent, and the Q function value of the agent are constructed as the state-action strategy reward of the user selecting AP access, AP transmit power and allocating subcarrier resources according to the link state information; the designed reward function is the system optimization objective function; the Markov decision process is optimized by DQN (Deep Q-Learning Network, deep Q learning network), a variable greedy factor is used to control the agent to adopt an adjustable probability exploration strategy in different resource allocation stages, the Q function value of the user association VLC-AP access point, the transmit power of the access point and the subcarrier resource allocation in the system is optimized, and the optimized resource allocation scheme of each user is obtained.

[0010] Specifically, a multi-agent deep Q network optimization indoor VLC-RF resource energy efficiency allocation method is provided, and the specific steps of S1 are as follows:

[0011] S101: According to the Lambert radiation model, the channel gain of the user accessing the VLC AP is calculated;

[0012] Wherein, the channel gain of user m accessing VLC AP n at time t is:

[0013]

[0014] In formula (1), represents the Lambert radiation coefficient, and φ 1 / 2 represents the half-power radiation angle of the VLC AP, φ represents the radiation angle of the VLC AP, A r represents the receiving area of the receiver photo detector (PD) of the UE, represents the straight-line distance between UE m and AP n, ψ m,n is the incident angle of UE m receiving the VLC AP n signal, ψ c is the field of view angle of the UE receiver, T s (ψ m,n ) and g(ψ m,n ) respectively represent the Lambert radiation coefficient and the Lambert radiation coefficient of UE m at ψm,n The gain of the optical filter and optical concentrator when receiving the optical signal;

[0015] S102: Calculate the signal-to-interference-plus-noise ratio value of the user accessing the VLC AP;

[0016] According to the Shannon formula, the SINR (Signal to Interference plus Noise Ratio) value of the user m accessing the subchannel k of the VLC AP n is:

[0017]

[0018] In formula (2), respectively, the kth subcarrier optical power value allocated by the nth andn'th VLC AP when connecting UE m at the tth time slot; η is the PD photoelectric conversion coefficient of the UE; N V , respectively, the power spectral density of the VLC noise and the channel bandwidth of the kth subchannel of the VLC AP, wherein B V is the total modulation bandwidth that can be communicated by each VLC AP, and the total number of subchannels of each VLC AP is K; N\0 represents the set of all VLC APs except the RF AP, wherein 0 is the index symbol of the RF AP in the set N;

[0019] S103: Calculate the achievable rate value of the user accessing the VLC AP;

[0020] Wherein, at the tth time slot, the achievable rate value obtained by the user m accessing the subchannel k of the VLC AP n is:

[0021]

[0022] S104: Calculate the achievable rate value of the VLC AP;

[0023] Wherein, at the tth time slot, the achievable rate value of the nth VLC AP can be calculated by the following formula:

[0024]

[0025] In formula (4), is a binary variable, if the mth UE is associated with the nth VLC AP at the tth time slot and uses the kth subchannel of the VLC AP n, otherwise,

[0026] S105: Calculate the channel gain value of the user accessing the RF AP;

[0027] wherein the link channel gain between the RF AP and the mth UE at the tth time slot can be expressed as:

[0028]

[0029] In equation (5), d is an exponential distribution random variable with mean value of 2.46 dB; L(d (m,0) (t)) is the path loss function of the RF link, which is defined as:

[0030]

[0031] In equation (6), d (m,0) (t) is the distance between the RF AP and the mth UE at the tth time slot; B1, B2 are constants of the gain model, which are 35.3 and 21.3, respectively; f c is the center frequency of the carrier, in GHz; X is a Gaussian random variable with mean value of 0 and variance of 1.8 dB;

[0032] S106: calculating the SINR value of the user accessing the RF AP;

[0033] wherein the SINR value of the signal received by the UE m on the kth subcarrier of the RF AP at the tth time slot is calculated as:

[0034]

[0035] In equation (7), d is the power value of the kth subcarrier allocated by the RF AP to the connected UE m at the tth time slot; is the average noise power of the RF system;

[0036] S107: calculating the achievable rate value of the user accessing the RF AP;

[0037] wherein the achievable rate of the UE m associated with the RF AP at the tth time slot is calculated according to the Shannon formula as:

[0038]

[0039] In equation (8), d is the bandwidth of the kth RF AP subchannel;

[0040] S108: calculating the achievable rate value of the RF AP;

[0041] wherein the achievable rate of the RF AP at the tth time slot is calculated as:

[0042]

[0043] In formula (9), is a binary variable, when user m associates the kth subchannel of the RF AP at the tth time slot, Otherwise,

[0044] S109: Calculate the total achievable rate value of VLC-RF;

[0045] Wherein, in the tth time slot, the total achievable rate of the VLC-RF system is:

[0046]

[0047] Specifically, a multi-agent deep Q network is provided to optimize the energy efficiency allocation of indoor VLC-RF resources. The S2 specific steps are:

[0048] S201: Calculate the historical average power consumption value of each AP;

[0049] Wherein, the power calculation method of the average power consumption value of the AP is:

[0050]

[0051] In the above formula, T is the number of time slots in a scheduling period, is the total transmission power of the VLC AP n at the tth time slot, is the total transmission power value of the RF AP at the tth time slot, Θ is the weight value of the equivalent transmission power of the VLC AP to the transmission power of the RF AP, Θ = 1;

[0052] S202: Calculate the historical average achievable rate value of each AP;

[0053] Wherein, the calculation formula of the average achievable rate of each AP is:

[0054]

[0055] S203: Calculate the average energy efficiency profit value of each AP in the VLC-RF system;

[0056] Wherein, the calculation formula of the average energy efficiency profit value of each AP is:

[0057]

[0058] In the above formula, ζ is the profit obtained by the system per unit of communication achievable rate, and σ represents the cost required by the unit transmission power of the AP;

[0059] S204: Statistics of the average probability value of the high-rate requirement of the user request being satisfied by the system;

[0060] wherein the average probability of the high-rate requirement of the user equipment m being satisfied by the VLC-RF system in a round of scheduling period T is:

[0061]

[0062] In the above formula, μ m (t) is the probability of the high-rate requirement of the user equipment m being satisfied by the system at the tth time slot, and the value is: μ m (t) = pro{R ser (t) ≥ R req (t)}, pro(·) is the probability function value satisfying the condition in the bracket; R req (t) represents the communication rate value requested by the user m at the tth time slot, R ser (t) is the reachable rate value obtained by the user m from the AP at the tth time slot, and

[0063] S205: Statistics of the average probability value of the high-reliability requirement of the user equipment m being satisfied by the system;

[0064] wherein the high-reliability requirement is represented by the connection uninterrupted probability, and the average probability of the high-reliability requirement of the user equipment m being satisfied by the VLC-RF system in a round of scheduling period T is:

[0065]

[0066] In the above formula, λ m (t) is the probability of the SINR value of the high-reliability requirement of the user equipment m being satisfied at the tth time slot, λ m (t) = pro{γ m (t) ≥ γ th}, γ th is the SINR threshold value for uninterrupted communication, γ m (t) is the SINR value of the user m at the tth time slot,

[0067] S206: In order to improve the probability of the rate requirement or the reliability connection of the user being satisfied, improve the QoS (Quality of Service) of the user, model the selection of the user equipment selecting which VLC AP or RF AP to access to obtain a higher QoS as a coalition game, and calculate the utility function value of the coalition game; wherein the system initializes to establish the starting moment of the coalition game, and let all users in the system first join the coalition F1;

[0068] Wherein, the user access AP coalition game model is:

[0069] G = (M, V, F) (16)

[0070] In the formula, F is the coalition set, F = {F1, F2}, wherein, F1 and F2 represent high speed coalition user set and high reliability coalition user set respectively, and F1∪F2=M;V is the utility function of the coalition;

[0071] Wherein, the utility function V(F) of the coalition F can be expressed as:

[0072]

[0073] S207: select a user from the set M without trying coalition conversion, calculate the utility function value after coalition conversion, if the coalition utility value of the user after coalition conversion is greater than the utility value before conversion, go to step S208; otherwise, the coalition conversion of the user is invalid, and the coalitions F1 and F2 remain unchanged, go to step S209;

[0074] Wherein, the user m changes from the coalition F1 to the coalition F2, and the utility function after conversion is:

[0075]

[0076] S208: for the user requesting coalition conversion, randomly generate a probability conversion value p, if the probability value is greater than the variable greedy factor value, then convert the user from one coalition to another coalition according to the probability value p; otherwise, the user converts from one coalition to another coalition with the variable greedy factor value;

[0077] Wherein, the calculation formula of the variable greedy factor P(L t ) of the user m requesting coalition conversion is:

[0078]

[0079] In the formula, L t = log2 t / L0, wherein L0 represents a constant, L0 = T, t ∈ T;

[0080] S209: if all users in the set M have tried coalition conversion, determine the users in F1 and F2, go to step S210; otherwise, return to step S208;

[0081] S210: calculate the proportional fairness function value of the different needs of the users in the system being satisfied by the system;

[0082] Wherein, the calculation formula of the proportional fairness function value of the different needs of the users in the system being satisfied by the system is:

[0083]

[0084] In the above formula, κ is a weight factor of the high-rate communication requirement of the user satisfied by the system, κ ∈ [0, 1], and 1-κ is a weight factor of the reliability requirement of the user satisfied by the system;

[0085] S211: An optimization objective function model of energy efficiency-user satisfaction maximization of the VLC-RF system is established:

[0086] The optimization objective function and the constraint condition of the energy efficiency-user satisfaction maximization of the VLC-RF system are as follows:

[0087]

[0088] The formula (21) is an optimization objective of the system, ρ represents a set of subcarrier association relationships of each UE and the AP, and the element value of ρ is a Boolean variable, for example, ρ m,n,k = 1, indicating that the UE m is associated with the subcarrier k of the AP n; P A represents a set of transmission power values obtained by each UE from the AP, for example, the element value p m,n = P1, indicating that the UE m obtains the P1 transmission power value provided by the AP n; B represents a set of subcarrier bandwidth values obtained by each UE from the AP, for example, the element value B m,n,k = B1, indicating that the UE m obtains the B1 bandwidth value provided by the subcarrier k of the AP n; the optimization objective of the formula (21) needs an algorithm to realize the association of each user to the AP, the power value obtained by the user from the AP, and the allocated subcarrier within a scheduling period;

[0089] In the formula (22), and respectively represent the maximum number of UEs that can be connected by a single VLC AP and a single RF AP, and are respectively set to 5 and 10; in the constraint condition, C1 and C2 respectively represent that the bandwidth allocated to the UE by each VLC AP and RF AP cannot exceed the total bandwidth value allowed by the AP; C3 represents that the average power transmitted by each AP cannot exceed the maximum transmission power of the AP, P max is the maximum power value transmitted by the AP; C4 represents the maximum number of UEs that can be associated with a single VLC AP; C5 represents the maximum number of UEs that can be associated with the RF AP; C6 represents that in order to guarantee the QoS of the user, the SINR of the signal received by each user must exceed the SINR threshold value, γ th is the SINR threshold value; C7 represents that each UE is connected to at most one subcarrier of one AP within the same time slot, and t is the time slot number observed;

[0090] Specifically, the provided method for optimizing indoor VLC-RF resource energy efficiency allocation using a multi-agent deep Q-network includes the following steps in S3:

[0091] S301: To achieve the optimization objective of the multi-agent deep Q network optimization method for indoor VLC-RF resource energy efficiency allocation, each UE in the VLC-RF system is abstracted as an agent, and a state space S is constructed for each agent. S represents the set of channel state information of the link between each agent and its associated AP, and S is expressed as: S={s1,...,s m ,...,s M}, where s m ={0,1}, s m =1 indicates that the quality of the signal received by UE m meets the QoS requirements; specifically, for users with high-speed requirements, when R ser (t)≥R req When (t), s m =1, otherwise, s m =0; For users with high reliability requirements, when γ m (t)≥γ th At that time, s m =1, otherwise, s m =0; This shows that the state space size is 2. M M is the number of users, and the size of the state space grows exponentially with the increase of the number of users;

[0092] S302: Construct the action space A of the agent, where A is the set of actions taken by each agent. The action space A is abstracted by which AP the user associates with, what transmit power is selected, and which sub-channel is accessed. A is then represented as: A = {a1(t),...,a...} m (t),...,a M (t)}, where a m (t)={ρ m B m ,P m}, and ρ m ={ρ0,...,ρ n ,...,ρ N}, where ρ n ={0,1}, if UE m is associated with AP n, then ρ n =1, otherwise, ρ n =0; B m ={B1,...,B K If UE m obtains subchannel k of AP, then B k =1, otherwise, B k =0; P m{P1, P2, P3} are three different power levels; it can be seen that the size of the action space of each user in the tth time slot is 3 x N x K; when N or K is very large, the action space is very large, when user m selects action a m (t) after, the action space of the remaining users can be expressed as

[0093] S303: modeling the link state and the action selection strategy of the agent decision as a Markov decision process (S, A, R, P), R represents the average reward value of the agent after the agent adopts A in the VLC-RF system; P represents the set of probabilities that the agent transfers from the current state to the next state after the agent performs action A; let t = 0;

[0094] S304: if t = T-1, the average energy consumption profit value and the proportional fairness function value of the system are calculated, then the average reward value of each agent is calculated, let t = 0, and step S305 is turned; otherwise, step S305 is directly turned;

[0095] wherein, in the previous scheduling period T, the average reward value obtained by each agent is:

[0096]

[0097] In the above formula, (s, a) represents the average EEP·PF value in the previous scheduling period; since the average reward values of the agents are all the same, the value is EEP·PF / M, and the optimization goal of the system is to maximize EEP·PF, therefore, as long as the number of users M in the system is unchanged, the optimization goal of the individual agent is the same as the optimization goal of the system;

[0098] S305: calculating the instantaneous reward value of the agent and the discount reward value of the system after the agent adjusts the action strategy;

[0099] wherein, in the tth time slot of the current scheduling period, the instantaneous reward value of the agent group after adopting the strategy (s t ,a t ) is expressed as:

[0100]

[0101] In the above formula,

[0102] wherein, in a scheduling period T, the discount reward value of the VLC-RF system is:

[0103]

[0104] In the above formula, γ is a decay discount factor of the reward value, and γ ∈ [0, 1];

[0105] S306: In order to find the strategies adopted by each agent to achieve the system optimization goal, the system designs the Q function as the R function of the Markov decision process (S,A,R,P). The Q function is determined by the expected value of the discounted reward of the agent in the state space S-action space A to determine the strategy.

[0106] The Q function of the VLC-RF system is defined as follows:

[0107]

[0108] In the above formula, s represents the mathematical expectation function; t and a t These represent the link state set s selected by all users in user set M at time t. t and action set a t The cumulative discount reward value R of the strategy t (s t ,a t );

[0109] S307: The agents interact with the channel state-policy and, using the method described in step S207, perform coalition transformation on each agent, randomly generating probability transformation values ​​p, p∈P, and calculating the variable greedy factor P(L) of the coalition transformation. t ), update the channel state space of each agent alliance from s t Change to s t+1 The action space starts from a t Change to a t+1 Calculate the Q-function value after the state space is updated;

[0110] The Q-function value for updating the state space can be calculated using the following formula:

[0111]

[0112] In the above formula, θ represents the set of learning rates of each agent; a′ represents the learning rate in state space s t+1 The action subspace of A that can be used at that time;

[0113] S308: Fit the Q-function using DQN (Deep Q-Learning Network), i.e.: use the best (s,a) values ​​from the DQN experience replay buffer to train the DQN network; calculate the DQN expected objective function and (s... t+1 ,a t+1 The loss function values ​​between )

[0114] Wherein, the expected objective function of DQN and (s t+1 ,a t+1The loss function value between () is defined as:

[0115]

[0116] In the above formula, y t Let $\mathbf{a}$ represent the expected objective function of DQN, and $\mathbf{a}$.

[0117] S309: Solve for the set of θ parameter values ​​that are minimized when the loss function is minimized by using an iterative update method;

[0118] The iterative update equation for the θ parameter set can be expressed as:

[0119]

[0120] In the above formula, Indicates rounding up to the nearest integer; This is the gradient iteration step size;

[0121] S310: Substitute θ into the function Q(s) t+1 ,a t+1 ), recalculate Q(s) t+1 ,a t+1 )value;

[0122] S311: If t = T-1, by comparing the Q-function values ​​of the state-policy space (s,a), find the optimal policy in the state-action space corresponding to the largest Q-function value, and then assign the state space, action space, and reward function Q to the optimal policy. * (s * ,a * The optimal policy of the VLC-RF system is stored in the experience replay buffer of DQN as a training sample for the DQN network, and the optimal policy of the VLC-RF system is output. Proceed to step S312; otherwise, let t = t + 1 and proceed to step S304.

[0123] Among them, the optimal action space s that maximizes the Q function value * and strategy space a * for:

[0124]

[0125] In the above formula, Q * (s,a * (s, a) represents the optimal Q-function value and the optimal policy a. * .

[0126] S312: If the algorithm runs out of time, output the optimal strategy of the VLC-RF system (s * ,a * Otherwise, let t = 0 and go to step S305. BRIEF DESCRIPTION OF DRAWINGS

[0127] In order to make the objects, technical solutions and points of the present application clearer, the following will further describe the present application with reference to the drawings:

[0128] Figure 1 Indoor VLC-RF heterogeneous network model schematic diagram;

[0129] Figure 2 VLC-RF network resource allocation model diagram based on DQN;

[0130] Figure 3 User alliance conversion flowchart based on system energy efficiency and user satisfaction of alliance game;

[0131] Figure 4 Flowchart of indoor VLC-RF resource energy efficiency allocation method based on multi-agent deep Q network. DETAILED DESCRIPTION

[0132] The present application can be implemented or applied in other different specific embodiments, and various modifications or changes can be made to the details in the specification based on different views and applications without departing from the spirit of the present application. It should be noted that the diagrams provided in the following examples only illustrate the basic concept of the present application in a schematic manner, and the following examples and features in the examples can be combined with each other without conflict.

[0133] The drawings are only used for illustrative explanation, and the representation is only a schematic diagram, not a physical diagram, and should not be understood as a limitation of the present application; in order to better illustrate the embodiments of the present application, some components in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0134] The drawings are only used for illustrative explanation, and the representation is only a schematic diagram, not a physical diagram, and should not be understood as a limitation of the present application; in order to better illustrate the embodiments of the present application, some components in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted. Figure 1 The drawing is a schematic diagram of the heterogeneous network model of the indoor VLC-RF heterogeneous network. In a room of 10m x 10m x 3m, 16 VLC APs are evenly distributed in a rectangular shape on the ceiling. The VLC AP relies on LED for optical communication, which is used for downlink optical link transmission of communication signals. The width of the LED itself is ignored, and the interval between each LED is 2.5m. A RF AP is deployed in the center of the room for uplink and downlink communication. The RF AP and the VLC AP are connected to the central controller of the indoor wireless access point through the power line, and the central controller is connected to the Internet network through the power line. A plurality of user equipment is randomly placed at a height of hPD On the workbench, the specific ceiling height is h.

[0135] Appendix Figure 2 for Figure 2 A resource allocation model diagram for VLC-RF networks based on DQN is shown. In a VLC-RF network, each user is abstracted as an agent. The reachable rate or link outage probability of communication between the user and the access point (AP) is abstracted as the link state S. The subcarrier of the AP that the user accesses is abstracted as action A. The target value of system optimization obtained after the user takes an action is abstracted as the reward function R. In a DQN network, the Q function value is equal to the R value that determines an action. (See attached diagram.) Figure 2 In this process, firstly, at the beginning of each iteration, the central controller calculates the alliance utility value based on the user's service requests from the previous round using an alliance game approach, and classifies the alliance to which the user's service belongs, i.e., determining whether the user's access to the AP prioritizes communication rate or link reliability. Then, the user is classified into different alliances, and the user's channel state information S is updated and input into the DQN target network. The DQN calculates the user's choice of different actions A, i.e., choosing different APs, AP transmit power values, and subcarriers, calculating the reward function value and Q value for each action. The action with the highest Q value is output and fed back to the user as the optimal action corresponding to the user's maximum reward value. Simultaneously, this optimal action is stored as trajectory data in the DQN experience pool. The DQN online network optimizes its internal parameters by periodically using historical trajectory data from the experience pool to minimize the loss function and update the DQN parameters θ. The optimized parameters are then input into the DQN target network for timely action selection in handling user requests.

[0136] Appendix Figure 3 The flowchart for user alliance conversion based on system energy efficiency and user satisfaction in alliance game theory mainly consists of the following steps:

[0137] Input: User equipment set M, requested rate values ​​for each user, AP set N, RF AP with index 0 in set N, VLCAP with indices 1, 2, ..., N in set N, AP sub-channel set K, VLC AP transmit power P V The transmit power of the RF AP is P. R Set the number of observation slots T, SINR threshold, slot t value, and weighting factor κ for the system to satisfy communication rate requirements. Assume coalition F1 = M.

[0138] Output: The alliance to which each user belongs after the coalition game, and the fairness function value of the proportion of different user needs being met by the system;

[0139] Step 1: Calculate the channel gain of user accessing VLC AP at time slot t according to Lambert radiation model from formula (1)

[0140] Step 2: Calculate the SINR value of user accessing VLC AP at time slot t from formula (2)

[0141] Step 3: Calculate the achievable rate value of user accessing VLC AP at time slot t from formula (3)

[0142] Step 4: Calculate the achievable rate value of VLC AP at time slot t from formula (4)

[0143] Step 5: Calculate the channel gain value of user accessing RF AP at time slot t from formula (5)

[0144] S106: Calculate the SINR value of user accessing RF AP at time slot t from formula (7)

[0145] Step 7: Calculate the achievable rate value of user accessing RF AP at time slot t from formula (8)

[0146] Step 8: Calculate the achievable rate value of RF AP at time slot t from formula (9)

[0147] Step 9: Calculate the average probability value μ of user's high rate requirement being satisfied by the system within scheduling period T from formula (14) m , m ∈ M;

[0148] Step 10: Calculate the average probability value λ of user device m's high reliability requirement being satisfied by the system within scheduling period T from formula (15) m , m ∈ M;

[0149] Step 11: Establish the coalition game model G = (M, V, F) of user accessing AP from formula (16), F is the coalition set, F = {F1, F2}, wherein F1 and F2 respectively represent the high rate coalition user set and the high reliability coalition user set, and F1 ∪ F2 = M;

[0150] Step 12: Calculate the utility value V(F) of the coalition from formula (17)

[0151] Step 13: Select a user from set M who has not tried to convert the coalition, calculate the variable greedy factor value by formula (19); for the user who requests coalition conversion, randomly generate a probability conversion value p, if the probability value is greater than the variable greedy factor value, then the user tries to convert from the equivalent coalition to another coalition according to the probability value p; otherwise, the user converts from one coalition to another coalition with the variable greedy factor value;

[0152] Step 14: Calculate the utility function value after coalition conversion by formula (18), if the user's coalition utility value after coalition conversion is greater than the utility value before conversion, perform coalition conversion on the user; otherwise, the user's attempted coalition conversion is invalid, and the coalitions F1 and F2 remain unchanged, go to step 15;

[0153] Wherein, the user m changes from coalition F1 to coalition F2, and the utility function after conversion is:

[0154]

[0155] Step 15: If all users in set M have tried coalition conversion, determine the users in F1 and F2, go to step 16; otherwise, return to step 13;

[0156] Step 16: Calculate the proportionally fair function PF value of the different needs of users in the system being satisfied by the system by formula (20);

[0157] Step 17: Output the F1 and F2 sets after coalition game, and the PF value.

[0158] Appendix Figure 4 The flow chart of the method for optimizing indoor VLC-RF resource energy efficiency based on multi-agent deep Q network mainly includes the following steps:

[0159] Input: User equipment set M, user request rate value, AP set N, RF AP in set N with sequence number 0, VLC AP in set N with sequence number 1, 2, …, N, AP subchannel set K, VLC AP transmission power P V , RF AP transmission power P R ; User set F1 and F2 after coalition game and PF value; Set observation time slot number T, SINR threshold, weight factor κ of communication rate demand satisfied by the system, discount factor γ, unit communication reachable rate profit value ζ, unit transmission power required cost value σ, set t = 0;

[0160] Output: User associated AP, AP transmission power and subcarrier allocated to user;

[0161] Step 1: Calculate the reachable rate value of each VLC AP at time t by formula (4)

[0162] Step 2: Calculate the achievable rate value of the RF AP at the tth time slot by formula (9)

[0163] Step 3: Calculate the total achievable rate value R of the VLC-RF system at the tth time slot by formula (10) sum (t) ;

[0164] Step 4: Calculate the historical average power consumption value P of each AP in the VLC-RF system by formula (11) ave ;

[0165] Step 5: Calculate the historical average achievable rate value of each AP by formula (12)

[0166] Step 6: Calculate the average energy efficiency profit value EEP of each AP in the VLC-RF system by formula (13)

[0167] Step 7: Establish the optimization objective function model of the energy efficiency-user satisfaction maximization of the VLC-RF system by formula (21) and (22), that is, maximize the value of EEPxPF;

[0168] Step 8: Abstract each UE in the VLC-RF system as an agent, construct the state space S of each agent, the action space A of the agent, model the link state and the action selection strategy of the agent decision as a Markov decision process (S, A, R, P); R represents the average reward value of the agent obtained after the agent adopts A; P represents the set of probabilities of the agent transferring from the current state to the next state after the agent performs the action A;

[0169] Step 9: If t = T-1, calculate the average reward value of each agent Let t = 0;

[0170] Step 10: Calculate the instantaneous reward value r of the agent by formula (24) t (s t ,a t ), calculate the discount reward value of the system according to formula (25)

[0171] Step 11: Set the Q function of the VLC-RF system as:

[0172] Step 12: Perform the coalition game shown in FIG. 10 Figure 3 , obtain the coalition game results F1 and F2 of the users; update the channel state space of each agent coalition from s t to st+1 , the action space from a t to a t+1 , the Q function value of the updated state space Q(s t+1 , a t+1 ) is calculated;

[0173] Wherein, the Q function value of the updated state space can be calculated by the following formula:

[0174]

[0175] Step 13: the excellent (s, a) of the DQN experience replay buffer is used to train the DQN network, and the DQN fitted Q function is obtained; the loss function value between the DQN expected target function and (s t+1 , a t+1 ) is calculated;

[0176] Wherein, the loss function value between the DQN expected target function and (s t , a t ) is defined as:

[0177]

[0178] Wherein, y t represents the DQN expected target function, and

[0179] Step 14: the θ parameter value set corresponding to the minimum loss function value is solved by iterative updating method;

[0180] Wherein, the iterative updating equation of the θ parameter value set can be expressed as:

[0181]

[0182] Step 15: the θ is substituted into the function Q(s t+1 , a t+1 ), and the Q(s t+1 , a t+1 ) value is recalculated;

[0183] Step 16: if t = T-1, the Q function value of the state-policy space (s, a) is compared, and the optimal policy of the state-action space corresponding to the maximum Q function value is found, the optimal policy corresponding to the state space and the action space and the reward function Q * (s * , a * ) are saved in the experience replay buffer of the DQN, as the training sample of the DQN network, and step 17 is turned to; otherwise, let t = t + 1, and turn to step 9;

[0184] Wherein, the optimal action space s* and policy space a * is:

[0185]

[0186] In the above formula, Q * (s,a * ) represents the optimal Q function value and the optimal policy a * corresponding to the state-policy space (s,a)

[0187] Step 17: If the algorithm running time ends, output the optimal policy Q * (s * ,a * ) adopted by each user in the VLC-RF system, determine the AP accessed by the user in the current scheduling period, the transmission power of the AP and the subcarriers allocated to the user; otherwise, let t=0, and go to step 10.

[0188] Finally, it should be pointed out that the above preferred embodiments are only used to illustrate the technical solutions of the present application and not to limit it, and although the present application has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present application.

Claims

1. A method for optimizing indoor VLC-RF resource energy efficiency allocation by using multi-agent deep Q network, characterized in that: The method comprises the following steps: S1: input user equipment set M, each user request rate value, 1 RF AP (Radio Frequency Access Point, radio frequency access point) and N VLC AP are stored in AP (Access Point, access point) set N, the RF AP is 0 in set N, the subchannel set of AP is K, the transmission power of VLC (Visible Lighting Communication, visible light communication) AP is P V , the transmission power of RF AP is P R ; set observation time slot number T, SINR threshold value, discount factor γ, profit value ζ of unit communication reachable rate, required generation value σ of unit transmission power, weight factor κ of communication rate demand satisfied by the system; respectively calculate the channel gain and achievable reachable rate value of user equipment accessing VLC AP and RF AP, calculate the total reachable rate value of the system, t∈T; Wherein, according to Lambert radiation model, the channel gain of user m accessing VLC AP n at the t time slot is: In the above formula, represents Lambertian radiation coefficient, and φ 1 / 2 represents the half-power radiation angle of the VLC AP, φ represents the radiation angle of the VLC AP, A r represents the receiving area of the receiver photo detector (PD) of the UE, represents the straight-line distance between the UE m and the AP n, ψ m,n is the incident angle when the UE m receives the VLC AP n signal, ψ c is the field of view angle of the UE receiver, T s (ψ m,n ) and g(ψ m,n ) respectively represent the gain of the optical filter and the optical concentrator when the UE m receives the light signal at ψ m,n . Wherein, according to Shannon formula, the SINR (Signal to Interference plus Noise Ratio) value of user m accessing subchannel k of VLC AP n is: In the above formula, respectively, the kth subcarrier optical power value allocated by the nth andn'th VLC APs to the UE m in the tth time slot; denotes the channel gain value of user m accessing the VLC AP n'; η is the PD photoelectric conversion coefficient of the UE; N V , respectively, the power spectral density of the VLC noise and the channel bandwidth of the kth subchannel of the VLC AP, wherein, B V is the total modulation bandwidth that can be communicated by each VLC AP, and the total number of subchannels of each VLC AP is K; N\0 represents the set of all VLC APs except the RF AP, wherein 0 is the index symbol of the RF AP in the set N; Wherein, the achievable rate value of user m accessing subchannel k of VLC AP n at the t time slot is: Wherein, the achievable rate value of the n VLC AP at the t time slot can be calculated by the following formula: In the above formula, is a binary variable, if the mth UE is associated with the nth VLC AP using its kth sub-channel at time slot t, Otherwise, Wherein, the link channel gain between RF AP and the m UE at the t time slot can be expressed as: In the above equation, An exponential distribution random variable with mean 2.46 dB is used to describe the small-scale fading of the RF; L(d (m ,0) (t)) is the path loss function of the RF link, which is defined as In the above formula, d (m,0) (t) is the distance between the RF AP and the mth UE in the t time slot; B1 and B2 are constants of the gain model, with values of 35.3 and 21.3, respectively; f c is the center frequency of the carrier, with a unit of GHz; X is a Gaussian random variable with a mean of 0 and a variance of 1.8 dB; Wherein, the SINR value of the signal received by UE m on the k subcarrier of RF AP at the t time slot is calculated by the following formula: In the above formula, Pm(k, t) is the power value assigned by the RF AP at time slot t to the kth subcarrier of the connected UE m; Pnoise is the average noise power of the RF system; Wherein, according to Shannon formula, the achievable rate of UE m associated with RF AP at the t time slot is: In the above formula, is the bandwidth of the kth RF AP subchannel. Wherein, the calculation formula of the achievable rate of RF AP at the t time slot is: In the above formula, is a binary variable, when user m associates with the kth subchannel of the RF AP at the tth time slot, Otherwise, Wherein, the total achievable rate of the VLC-RF system at the t time slot is: S2: According to the achievable rate value, the average achievable rate and average power consumption of each AP per scheduling period are calculated, and the energy consumption profit function of the AP is constructed; The satisfaction proportionally fair function of the user request to access different APs is constructed, the probability optimization method is used to optimize the user's alliance conversion according to the link state and satisfaction demand by using the method of coalition game, and the optimization objective function reflecting the system energy efficiency and user satisfaction is constructed; Specifically, the specific steps of S2 are: S201: Calculate the historical average power consumption value of each AP; Wherein, the power calculation method of the average power consumption value of the AP is: In the above formula, T is the number of time slots in a scheduling period, Ptotal(t) is the total transmission power of the VLC AP n at time slot t, Ptotal(t) is the total transmission power of the VLC AP n at time slot t, Θ is the weight value of the equivalent transmission power of the VLC AP to the transmission power of the RF AP. S202: Calculate the historical average achievable rate value of each AP; Wherein, the calculation formula of the average achievable rate of each AP is: S203: Calculate the average energy efficiency profit value of each AP in the system; Wherein, the calculation formula of the average energy efficiency profit value of the AP is: In the above formula, ζ is the profit obtained by the system per unit communication achievable rate, and σ represents the cost required by the unit transmission power of the AP; S204: Calculate the average probability value of the high rate demand of user equipment m being satisfied by the system; Wherein, the average probability of the high rate demand of user equipment m being satisfied by the VLC-RF system within a scheduling period T is: In the above formula, μ m (t) is the probability that the high-rate requirement of user equipment m is satisfied by the system in the tth time slot, and its value is: m (t) = pro{R ser (t) ≥ R req (t)}, pro(·) is the probability function value satisfying the condition in the brackets; R req (t) represents the communication rate value requested by user m in the tth time slot, R ser (t) is the reachable rate value obtained by user m from the AP in the tth time slot, S205: Calculate the average probability value of the high reliability demand of user equipment m being satisfied by the system; Wherein, the high reliability demand is represented by the connection uninterrupted probability, and the average probability of the high reliability demand of user equipment m being satisfied by the VLC-RF system within a scheduling period T is: In the above formula, λ m (t) is the probability that the SINR value of the high-reliability requirement of user equipment m in the tth time slot communication is satisfied, λ m (t) = pro{γ m (t) ≥ γ th}, γ th is the SINR threshold value for uninterrupted communication, γ m (t) is the SINR value of user m in time slot t, S206: In order to improve the rate requirement or the probability of reliability connection of the user being satisfied, the QoS (Quality of Service) of the user is improved, the selection of the user equipment selecting which VLC AP or RF AP to access to obtain higher QoS is modeled as a coalition game, and a utility function value of the coalition game is calculated; wherein, the system initialization establishes a coalition game starting time, and all users are first added to the coalition F1; The coalition game model of the user accessing the AP is: G=(M,V,F) In the above formula, F is a coalition set, F = {F1, F2}, wherein F1 and F2 respectively represent a high-rate coalition user set and a high-reliability coalition user set, and F1∪F2=M; V is a utility function of the coalition; The utility function V(F) of the coalition F can be expressed as: S207: A user who has not tried coalition conversion is selected from the set M, the utility function value after coalition conversion is calculated, if the coalition utility value after the user coalition conversion is greater than the coalition utility value before the conversion, step S208 is converted; otherwise, the coalition conversion tried by the user is invalid, the coalitions F1 and F2 remain unchanged, and step S209 is converted; Wherein, the user m changes from the coalition F1 to the coalition F2, and the utility function after conversion is: S208: For the user requesting coalition conversion, a probability conversion value p is randomly generated, if the probability value is greater than the variable greedy factor value, the user is converted from one coalition to another coalition according to the probability value p; otherwise, the user is converted from one coalition to another coalition with the variable greedy factor value; where the variable greed factor P(L t ) for user m requesting coalition conversion is calculated as: In the above formula, L t = log2 t / L0, where L0represents a constant, L0= T, t ∈ T; S209: If all users in the set M have tried coalition conversion, the users in F1 and F2 are determined, and step S210 is converted; otherwise, step S208 is returned; S210: The proportional fairness function value of the different requirements of the users in the system being satisfied by the system is calculated; The proportional fairness function value of the different requirements of the users being satisfied by the system is calculated as: In the above formula, κ is a weight factor of the high-rate communication requirement of the user being satisfied by the system, κ∈[0,1], and 1-κ is a weight factor of the reliability requirement of the user being satisfied by the system; S211: An optimization objective function of maximizing the energy efficiency-user satisfaction of the VLC-RF system is established: The optimization objective function of maximizing the energy efficiency-user satisfaction of the VLC-RF system is: max{EEP·PF} In the above formula, the association of each user accessing the AP, the power value obtained by the user from the AP and the allocated subcarrier need to be adjusted within the scheduling period. S3: The VLC-RF access users and the allocated resources are represented by a Markov decision process, the users are abstracted as agents, the state space, the action space, the state transition probability matrix of the agents, and the Q function value of the agents are constructed as the state-action strategy reward of the agents according to the link state information, the decision action is: the user selects the AP access, the AP transmission power and the allocated subcarrier resource of the AP; the designed reward function is the system optimization objective function; the Markov decision process is optimized through the DQN (Deep Q-Learning Network, deep Q learning network), a variable greedy factor is used to control the adjustable probability exploration strategy of the agents in different resource allocation stages, the Q function value of the user association VLC-AP access point, the access point transmission power and the subcarrier resource allocation in the system is optimized, and the optimized resource allocation scheme of each user is obtained.

2. The method of claim 1, wherein the multi-agent deep Q network is used to optimize the VLC-RF resource energy efficiency allocation in an indoor environment. The specific method of S3 is: S301: To achieve the optimization goal of the multi-agent deep Q network optimization indoor VLC-RF resource energy efficiency allocation method, each UE in the VLC-RF system is abstracted as an agent, and the state space S of each agent is constructed, S represents the channel state information set of the link between each agent and the AP associated with it, S is represented as: S = {s1,...,s m ,...,s M}, wherein s m ={0,1}, s m =1 indicates that the quality of the signal received by the UE m meets the QoS requirement; specifically, for high-rate demand users, when R ser (t)≥R req (t), s m =1, otherwise, s m =0; for high-reliability demand users, when γ m (t)≥γ th , s m =1, otherwise, s m =0; S302: Construct the action space A of the intelligent agent, A is the action set taken by each intelligent agent, and the association of the user with which AP and the selection of the transmission power and the access to which subchannel are abstracted as the action space of A, and A is represented as: A = {a1(t),...,a m (t),...,a M (t)} where a m (t) = {p m ,B m ,P m} and p m = {p0,...,p n ,...,p N} where p n = {0,1}, if UE m is associated with AP n, then p n = 1, otherwise, p n = 0; B m = {B1,...,B K}, if UE m obtains the subchannel k of the AP, then B k = 1, otherwise, B k = 0; P m = {P1, P2, P3} is three different power levels; S303: The link state and the selected action space strategy of the agent decision are modeled as a Markov decision process (S, A, R, P), R represents the average reward value of the agent obtained by the agent after adopting A; P represents the set of probabilities that the agent transfers from the current state to the next state after the agent performs the action A; let t = 0; S304: If t = T-1, the average energy consumption profit value and the proportional fairness function value of the system are calculated, then the average reward value of each agent is calculated, let t = 0, and step S305 is turned; otherwise, step S305 is turned. In the above formula, (s, a) represents the average EEP·PF value in the previous scheduling period; since the average reward value of each agent is the same, the value is EEP·PF / M, the optimization objective of the system is to maximize EEP·PF, therefore, as long as the number of users M in the system is unchanged, the optimization objective of the individual agent is the same as the optimization objective of the system; S305: The instantaneous reward value of the agent and the discount reward value of the system are calculated after the agent adjusts the action strategy; In the above formula, γ is a decay discount factor of the reward value, γ ∈ [0, 1]; wherein, at the tth time slot of the current scheduling period, the instantaneous reward value after the agent group adopts the strategy (s t ,a t ) is expressed as: In the above formulae, S306: In order to find the strategy adopted by each agent to achieve the optimization objective of the system, the system designs the Q function as the R function of the Markov decision process (S, A, R, P), the Q function is determined by the expected value of the discount reward value of the strategy determined by the agent in the state space S-action space A; In the above formula, the updated Q function value of the state space can be calculated by the following formula: S309: The θ parameter value set corresponding to the minimum loss function value is solved through the iterative updating method; In the above formula, the iterative updating equation of the θ parameter set can be represented as: In the above formula, s represents the mathematical expectation function; t and a t These represent the link state set s selected by all users in user set M at time t. t and action set a t The cumulative discount reward value R of the strategy t (s t ,a t ); S307: The intelligent agent converts the alliance of each intelligent agent by interacting with the channel state-strategy, randomly generates a probability conversion value p, p∈P, calculates the variable greedy factor P(L t ) of the alliance conversion, updates the channel state space of the alliance of each intelligent agent from s t to s t+1 , updates the action space from a t to a t+1 , and calculates the Q function value after updating the state space; ​ In the above equation, θ represents a set of learning rates for each agent; a' represents a subspace of actions of A that can be taken at state space s t+1 ; and a represents an action taken by A at state space s S308: fitting the Q function using DQN (Deep Q-Learning Network), that is, using the excellent (s, a) in the DQN experience replay buffer to train the DQN network; calculating the loss function value between the DQN expected target function and (s t+1 ,a t+1 ) wherein the loss function value between the DQN expected target function and (s t+1 ,a t+1 ) is defined as: In the above formula, y t represents the DQN expected target function, and ​ ​ In the above formulae, denotes rounding up to the nearest integer; is the gradient iteration step size; S310: Substitute θ into the function Q(s t+1 ,a t+1 ), and recalculate the value of Q(s t+1 ,a t+1 ); S311: If t=T-1, each finds the optimal policy of the state space-action space corresponding to the maximum Q function value of the state-policy space (s, a) by comparing the Q function values, saves the state space and action space and the reward function Q * (s * ,a * ) corresponding to the optimal policy in the experience replay buffer of the DQN as the training sample of the DQN network, and goes to step S312; otherwise, let t=t+1, and goes to step S304. wherein the optimal action space s that maximizes the Q function value * and policy space a * is: In the above formula, Q * (s, a * ) represents the optimal Q function value and optimal policy a corresponding to the state-policy space (s, a) * ; S312: If the algorithm running time ends, output the optimal strategy (s * ,a * ) of the VLC-RF system; otherwise, let t = 0, and go to step S305.

Images