Multi-agent deep Q network optimization indoor VLC-RF resource energy efficiency allocation method

Abstract

The invention relates to a multi-agent deep Q network optimization indoor VLC-RF resource energy efficiency distribution method, and belongs to the technical field of optical wireless communication. In order to improve the system energy efficiency of the VLC-RF heterogeneous network, a user is abstracted as an intelligent agent of a deep Q network, the user selects a VLC-RF access point, and power allocation and subcarrier resource allocation are abstracted as the action of the intelligent agent. A product of a proportional fairness satisfaction value and a system energy consumption profit value considering satisfaction of a user to a communication rate alliance or satisfaction of the user to a channel reliability alliance is designed to serve as a reward Q value of a deep Q network, alliance conversion of an intelligent agent is controlled by using a variable greedy factor, and an access AP channel state emphasized by the user is obtained; and then, selecting an action with the maximum Q value for the user by adopting a deep Q network, and optimizing a plurality of user associated access points, access point transmitting power and subcarrier resource allocation in the VLC-RF system. According to the method, the user satisfaction is increased, and the system energy efficiency of the VLC-RF can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical wireless communications and relates to a method for optimizing indoor VLC-RF resource energy efficiency allocation using a multi-agent deep Q network. Background Art

[0002] In recent years, the visible light communication (VLC) technology based on LED (Lighting Emitting Diode) has attracted extensive attention from researchers because of its advantages of low power consumption, high security, and electromagnetic interference-free communication while meeting the needs of indoor lighting. It can also provide about 400THz of unlicensed spectrum. VLC can use visible light waves with a wavelength of 380nm-780nm to control the on and off of LEDs through encoded information, and can achieve high-speed transmission of data information. At the receiving end, a photodetector is used to capture the on and off of a high-speed flashing LED, and after a series of signal processing, it can decode the original information transmitted by the sending end.

[0003] However, VLC also has its limitations, such as data transmission relying on line of sight (LoS), small coverage of VLC access points (AP), and the intensive deployment of multiple VLC APs that can easily cause serious interference. In order to overcome the above-mentioned shortcomings of VLC, some scholars have proposed combining VLC with traditional RF (Radio Frequency) communication technology in recent years, and this idea has attracted the interest of many scholars in related fields. On the one hand, VLC AP can provide high-speed data communication services, and on the other hand, RF AP can further ensure the stability of the connection and link reliability. This is powerful for improving the satisfaction experience of users with different satisfaction requirements, where the satisfaction requirements include: high-speed demand users or high-reliability demand users. In VLC-RF (Visible Light Communication-Radio Frequency) heterogeneous networks, users can adopt different access and resource allocation schemes according to the link status to obtain access strategies that improve satisfaction, thereby forming a user classification of high-speed satisfied users or high-reliability satisfied users. In future indoor VLC-RF heterogeneous networks, a satisfactory resource allocation scheme is essential.

[0004] For VLC-RF heterogeneous networks, traditional resource allocation methods, such as convex optimization and linear programming methods, require channel state information to be obtained in advance, which is often difficult when user devices move indoors. When dealing with optimization problems, deep learning algorithms are more universal and robust than convex optimization methods. The main principle is to place the representative user device as an intelligent agent in an environment with unknown channel states. The intelligent agent interacts with the current environment by maximizing the principle of long-term rewards, similar to the mechanism of seeking benefits and avoiding harm, to obtain the optimal strategy corresponding to the maximum reward value of a user device accessing a VLCAP or RF AP, controlling the AP's transmit power value, and selecting the AP's subcarrier channel. This optimal strategy will guide the intelligent agent to adjust the accessed AP, adjust the AP transmit power, and adjust the subcarrier allocation. The channel state and resource state of the VLC-RF network change over time, and the strategy of the intelligent agent also needs to be continuously adjusted. The intelligent agent can use iterative algorithms to continuously explore and trial and error the selection of actions to obtain the optimal actions of accessing APs, controlling AP transmit power, and allocating subcarriers. Therefore, in VLC-RF networks, the difficulties faced by heuristic algorithms and convex optimization algorithms, such as the inability of users and central controllers to obtain global channel status information and resource availability due to the dynamic changes in VLC-RF network resources, and the high complexity of the algorithm for user access to APs and resource allocation when there are a large number of indoor devices, can be better solved by deep learning algorithms. Summary of the invention

[0005] In view of this, the core of the present invention is to provide a multi-agent deep Q network optimization indoor VLC-RF resource energy efficiency allocation method to improve the energy efficiency of the system while ensuring user fairness.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] S1: Input user device set M, each user's requested rate value, 1 RF AP (Radio Frequency Access Point) and N VLC APs are stored in AP set N, the RF AP is numbered 0 in set N, the AP's subchannel set is K, and the VLC AP's transmit power P V , the transmission power of RF AP is P R ; Set the number of observation time slots T, SINR threshold, discount factor γ, profit value ζ per unit communication achievable rate, cost value σ per unit transmission power, and weight factor κ for the communication rate requirement to be met by the system; calculate the channel gain and achievable achievable rate value of the user equipment accessing the VLC AP and RF AP respectively, and calculate the total achievable rate value of the system;

[0008] S2: According to the achievable rate value, the average achievable rate and average power consumption per unit scheduling period of each AP are calculated, and the energy consumption profit function of the AP is constructed; a fair function is constructed to evaluate the satisfaction ratio of users requesting to access different APs, and the alliance game method is adopted. According to the link status and satisfaction requirements, the probability optimization method is used to optimize the user's alliance conversion, and the optimization objective function that reflects the system energy efficiency and user satisfaction is constructed;

[0009] S3: The VLC-RF access users and resource allocation are represented by a Markov decision process. The users are abstracted as intelligent agents. The state space, action space, state transition probability matrix, and Q function value of the intelligent agent are constructed as rewards for the state-action strategy of the intelligent agent based on link state information. The decision action is: the user selects AP access, AP transmit power, and allocates AP subcarrier resources. The designed reward function is the system optimization objective function. The Markov decision process is optimized by DQN (Deep Q-Learning Network). A variable greedy factor is used to control the intelligent agent to adopt an adjustable probability exploration strategy in different resource allocation stages. The Q function values ​​of the user-associated VLC-AP access points, access point transmit power, and subcarrier resource allocation in the system are optimized to obtain the optimized resource allocation plan for each user.

[0010] Specifically, the present application provides a multi-agent deep Q network optimization indoor VLC-RF resource energy efficiency allocation method, the specific steps of S1 are:

[0011] S101: Calculate the channel gain of the user accessing the VLC AP according to the Lambertian radiation model;

[0012] Among them, in the tth time slot, the channel gain of user m accessing VLC AP n is:

[0013]

[0014] In formula (1), represents the Lambertian emissivity, and φ 1 / 2 represents the half-power radiation angle of the VLC AP, φ represents the radiation angle of the VLC AP, and A r Indicates the receiving area of ​​the UE's receiver photodetector (PD), represents the straight-line distance between UE m and AP n, ψ m,n is the incident angle when UE m receives the signal from VLC AP n, ψ c is the field of view of the UE receiver, T s (ψ m,n ) and g(ψ m,n ) represent UE m with ψm,n The gain of the optical filter and optical concentrator when receiving light signals;

[0015] S102: Calculate the signal-to-interference-and-noise ratio value of the user accessing the VLC AP;

[0016] According to Shannon's formula, the SINR (Signal to Interference plus Noise Ratio) of subchannel k of user m accessing VLC AP n is:

[0017]

[0018] In formula (2), are the kth subcarrier optical power values ​​allocated by the nth and n'th VLC APs when they connect to UE m in the tth time slot; η is the PD photoelectric conversion coefficient of the UE; N V , are the power spectral density of VLC noise and the channel bandwidth of the kth subchannel of VLC AP, respectively, where B V is the total modulation bandwidth that each VLC AP can communicate with, The total number of subchannels of each VLC AP is K; N\0 represents a set of all VLC APs except RF APs, where 0 is the index symbol of RF AP in set N;

[0019] S103: Calculate the achievable rate value that the user can obtain by accessing the VLC AP;

[0020] Among them, in the tth time slot, the achievable rate value obtained by user m accessing subchannel k of VLC AP n is:

[0021]

[0022] S104: Calculate the achievable rate value of the VLC AP;

[0023] In the tth time slot, the achievable rate value that the nth VLC AP can achieve can be calculated by the following formula:

[0024]

[0025] In formula (4), is a binary variable. If the mth UE is associated with the nth VLC AP in the tth time slot and uses the kth subchannel of VLC AP n, then otherwise,

[0026] S105: Calculate the channel gain value of the user accessing the RF AP;

[0027] In which, in the tth time slot, the link channel gain between the RF AP and the mth UE can be expressed as:

[0028]

[0029] In formula (5), is a random variable that follows an exponential distribution and describes the RF small-scale fading, with a mean of 2.46 dB; L(d (m,0) (t)) is the path loss function of the RF link, which is defined as;

[0030]

[0031] In formula (6), d (m,0) (t) is the distance between the RF AP and the mth UE in time slot t; B 1 , B 2 are the constants of the gain model, and their values ​​are 35.3 and 21.3; f c is the center frequency of the carrier, in GHz; X is a Gaussian random variable with mean 0 and variance 1.8 dB;

[0032] S106: Calculate the SINR value of the user accessing the RF AP;

[0033] Among them, in the tth time slot, the calculation formula of the SINR value of the signal received by UE m on the kth subcarrier of the RF AP is:

[0034]

[0035] In formula (7), is the power value of the kth subcarrier allocated by the RF AP to the connected UE m at time slot t; is the average noise power of the RF system;

[0036] S107: Calculate the achievable rate value of the user accessing the RF AP;

[0037] According to Shannon's formula, in the tth time slot, the achievable rate obtained by UE m associated with the RF AP is:

[0038]

[0039] In formula (8), is the bandwidth of the kth RF AP subchannel;

[0040] S108: Calculate the achievable rate value of the RF AP;

[0041] The calculation formula for the achievable rate that the RF AP can achieve in the tth time slot is:

[0042]

[0043] In formula (9), is a binary variable. When user m associates the kth subchannel of the RF AP in the tth time slot, otherwise,

[0044] S109: Calculate the total achievable rate value of VLC-RF;

[0045] Among them, in the tth time slot, the total achievable rate that the VLC-RF system can achieve is:

[0046]

[0047] Specifically, the present application provides a multi-agent deep Q network optimization indoor VLC-RF resource energy efficiency allocation method, the specific steps of S2 are:

[0048] S201: Calculate the historical average power consumption value of each AP;

[0049] The average power consumption of an AP is calculated as follows:

[0050]

[0051] In the above formula, T is the number of time slots in a scheduling cycle, is the total transmit power of VLC AP n in time slot t, is the total transmission power value of RF AP in time slot t, Θ is the weight of the VLC AP transmit power equivalent to the RF AP transmit power. In this application, Θ=1;

[0052] S202: Calculate the historical average achievable rate value of each AP;

[0053] The calculation formula for the average achievable rate of each AP is:

[0054]

[0055] S203: Calculate the average energy efficiency profit value of each AP in the VLC-RF system;

[0056] The calculation formula for the average energy efficiency profit value of each AP is:

[0057]

[0058] In the above formula, ζ is the profit of the system to obtain the unit communication rate, and σ represents the cost required for the AP's unit transmission power;

[0059] S204: Counting the average probability value of the high rate demand requested by the user being satisfied by the system;

[0060] Among them, the average probability that the high rate demand of user equipment m is met by the VLC-RF system within a scheduling period T is:

[0061]

[0062] In the above formula, μ m (t) is the probability that the high-rate demand of user equipment m in time slot t is satisfied by the system, and its value is: μ m (t) = pro{R ser (t)≥R req (t)}, pro(·) is the probability function value that satisfies the conditions in brackets; R req (t) represents the communication rate value requested by user m in the tth time slot, R ser (t) is the achievable rate value obtained by user m from the AP in the tth time slot, and

[0063] S205: Counting the average probability value of the high reliability requirement of the user equipment m being met by the system;

[0064] The high reliability requirement is expressed by the probability of uninterrupted connection. The average probability that the high reliability requirement of user equipment m is met by the VLC-RF system within the scheduling period T is:

[0065]

[0066] In the above formula, λ m (t) is the probability that the high reliability requirement of user equipment m is met in the SINR value of the communication in the tth time slot, λ m (t) = pro{γ m (t)≥γ th}, γ th is the SINR threshold for uninterrupted communication, γ m (t) is the SINR value of user m in time slot t,

[0067] S206: In order to improve the user's rate requirement or the probability of reliable connection being met and improve the user's QoS (Quality of Service), the user device's choice of which VLC AP or RF AP to access to obtain a higher QoS is modeled as a coalition game, and the utility function value of the coalition game is calculated; wherein, at the start time of the system initialization establishment of the coalition game, all users in the system are allowed to join the coalition F 1 ;

[0068] Among them, the alliance game model of users accessing AP is:

[0069] G=(M,V,F) (16)

[0070] In the above formula, F is the alliance set, F = {F 1 ,F 2}, where F 1 and F 2 denote the high-rate alliance user set and the high-reliability alliance user set respectively, and F 1 ∪F 2 =M; V is the utility function of the alliance;

[0071] Among them, the utility function V(F) of alliance F can be expressed as:

[0072]

[0073] S207: Select a user from the set M who has not attempted alliance conversion, and calculate the utility function value after the alliance conversion. If the alliance utility value of the user after the alliance conversion is greater than the alliance utility value before the conversion, go to step S208; otherwise, the alliance conversion attempted by the user is invalid, and the alliance F 1 and F 2 Remain unchanged and go to step S209;

[0074] Among them, user m is from alliance F 1 Switch to Alliance F 2 , the transformed alliance utility function is:

[0075]

[0076] S208: For the user who requests alliance switching, a probability conversion value p is randomly generated. If the probability value is greater than the variable greed factor value, the user is switched from one alliance to another alliance according to the probability value p; otherwise, the user is switched from one alliance to another alliance according to the variable greed factor value;

[0077] Among them, the variable greed factor P(L t ) is calculated as:

[0078]

[0079] In the above formula, L t =log 2 t / L 0 , where L 0 represents a constant. In this application, L 0 =T, t∈T;

[0080] S209: If all users in set M have tried alliance conversion, F is determined 1 and F 2 If the user is in, go to step S210; otherwise, return to step S208;

[0081] S210: Calculating a fairness function value of the proportion of different demands of users in the system being satisfied by the system;

[0082] The calculation formula of the fairness function value of the proportion of different needs of users being met by the system is:

[0083]

[0084] In the above formula, κ is the weight factor of the user's high-speed communication demand being met by the system, κ∈[0,1], 1-κ is the weight factor of the user's reliability demand being met by the system;

[0085] S211: Establish an optimization objective function model for maximizing energy efficiency and user satisfaction of VLC-RF system:

[0086] Among them, the optimization objective function and constraints of the energy efficiency-user satisfaction maximization of the VLC-RF system of this application are:

[0087]

[0088]

[0089] Formula (21) is the optimization goal of the system, ρ represents the subcarrier association relationship set between each UE and AP, and its element value is a Boolean variable, such as: m,n,k =1, indicating that UE m is associated with subcarrier k of AP n; A Represents a set of transmit power values ​​obtained by each UE from the AP, for example: its element value p m,n =P 1 , indicating that UE m obtains the P provided by AP n 1 Transmit power value; B represents the set of subcarrier bandwidth values ​​obtained by each UE from the AP, for example: its element value B m,n,k =B 1, indicating that UE m obtains B provided by subcarrier k of AP n 1 Bandwidth value; The optimization goal of formula (21) requires the algorithm to be achieved by adjusting the association of each user accessing the AP, the power value obtained by the user from the AP, and the allocated subcarriers within the scheduling period;

[0090] In formula (22), and Respectively represent the maximum number of UEs that can be connected to a single VLC AP and RF AP, which are set to 5 and 10 respectively in this application; in the constraint condition, C 1 and C 2 The bandwidth allocated to the UE by each VLC AP and RF AP cannot exceed the total bandwidth allowed by the AP; C 3 Indicates that the average power transmitted by each AP cannot exceed the maximum transmit power of the AP, P max is the maximum power value transmitted by the AP; C 4 Indicates the maximum number of UEs that can be associated with a single VLC AP; C 5 Indicates the maximum number of UEs that can be associated with the RF AP; C 6 In order to ensure user QoS, the SINR of the signal received by each user must exceed the SINR threshold, γ th is the SINR threshold; C 7 Indicates that each UE can connect to at most one subcarrier of one AP in the same time slot, and t is the observed time slot number;

[0091] Specifically, the present application provides a multi-agent deep Q network optimization indoor VLC-RF resource energy efficiency allocation method, the specific steps of S3 are:

[0092] S301: To achieve the optimization goal of this application, each UE in the VLC-RF system is abstracted as an agent, and a state space S of each agent is constructed. S represents the channel state information set of the link between each agent and its associated AP. S is expressed as: S = {s 1 ,...,s m ,...,s M}, where s m ={0,1},s m =1 indicates that the quality of the signal received by UE m meets the QoS requirement; specifically, for users with high rate requirements, when R ser (t)≥R req (t), s m =1, otherwise, s m =0; for users with high reliability requirements, when γ m (t)≥γ th When m=1, otherwise, s m =0; we can see that the state space size is 2 M , M is the number of users, and the size of the state space grows exponentially with the number of users;

[0093] S302: Construct an action space A of the agent, where A is a set of actions taken by each agent. The AP to which the user is associated, the transmit power selected, and the subchannel to which the user is connected are abstracted as the action space of A. Then A is expressed as: A = {a 1 (t),...,a m (t),...,a M (t)}, where a m (t) = {ρ m ,B m ,P m}, and ρ m ={ρ 0 ,...,ρ n ,...,ρ N}, where ρ n ={0,1}, if UE m is associated with AP n, then ρ n =1, otherwise, ρ n =0; B m = {B 1 ,...,B K}, if UE m obtains AP's subchannel k, then B k =1, otherwise, B k =0;P m = {P 1 ,P 2 ,P 3} are three different power levels; it can be seen that the action space size of each user in the t time slot is 3×N×K; when N or K is very large, the action space is very large. When user m selects action a in the t time slot m (t), the action space of the remaining users can be expressed as

[0094] S303: Model the link state of the agent's decision and the strategy of selecting the action space as a Markov decision process (S, A, R, P), where R represents the average reward value of the agent obtained after the agent adopts A in the VLC-RF system; P represents the set of probabilities of the agent transferring from the current state to the next state after the agent performs action A; let t = 0;

[0095] S304: If t=T-1, calculate the average energy consumption profit value and proportional fairness function value of the system, and then calculate the average reward value of each agent, set t=0, and go to step S305; otherwise, directly go to step S305;

[0096] Among them, in the previous round of scheduling period T, the average reward value obtained by each agent is:

[0097]

[0098] In the above formula, (s, a) represents the average EEP·PF value in the previous scheduling period; since the average reward value of each agent is the same, which is EEP·PF / M, the optimization goal of the system is to maximize EEP·PF. Therefore, as long as the number of users M in the system remains unchanged, the optimization goal of the individual agent is the same as the optimization goal of the system.

[0099] S305: Calculate the instantaneous reward value of the agent and the discounted reward value of the system after the agent adjusts its action strategy;

[0100] Among them, in the tth time slot of the current scheduling cycle, the agent group adopts the strategy (s t ,a t ) is expressed as:

[0101]

[0102] In the above formula,

[0103] Among them, within a scheduling period T, the discount reward value of the VLC-RF system is:

[0104]

[0105] In the above formula, γ is the attenuation discount factor of the reward value, γ∈[0,1];

[0106] S306: In order to find the strategies adopted by each agent to achieve the system optimization goal, the system designs a Q function as the R function of the Markov decision process (S, A, R, P). The Q function is determined by the expected value of the discounted reward value of the agent's strategy in the state space S-action space A;

[0107] Among them, the Q function of the VLC-RF system is defined as:

[0108]

[0109] In the above formula, represents the mathematical expectation function; s t and a t They represent the link state set s selected by all users in the user set M at time t. t and action set a t The cumulative discounted reward value R of the strategy t (s t ,a t );

[0110] S307: The agent interacts with the channel state-strategy and uses the method described in step S207 to perform alliance conversion on each agent, randomly generate a probability conversion value p, p∈P, and calculate the variable greed factor P(L t ), update the channel state space of each agent alliance from s t becomes s t+1 , the action space changes from a t becomes a t+1 , calculate the Q function value after the state space is updated;

[0111] Among them, the Q function value of the updated state space can be calculated by the following formula:

[0112]

[0113] In the above formula, θ represents the learning rate set of each agent; a′ represents the learning rate of each agent in the state space s t+1 The action subspace of A that can be adopted when ;

[0114] S308: Use DQN (Deep Q-Learning Network) to fit the Q function, that is, use the excellent (s, a) in the DQN experience replay buffer to train the DQN network; calculate the DQN expected objective function and (s t+1 ,a t+1 ) between the loss function values;

[0115] Among them, DQN expects the objective function and (s t+1 ,a t+1 ) is defined as:

[0116]

[0117] In the above formula, y t represents the DQN expected objective function, and

[0118] S309: solving the set of θ parameter values ​​corresponding to the minimum loss function value through an iterative update method;

[0119] Among them, the iterative update equation of the θ parameter set can be expressed as:

[0120]

[0121] In the above formula, Indicates rounding up to an integer; is the gradient iteration step size;

[0122] S310: Substitute θ into the function Q(st+1 ,a t+1 ), recalculate Q(s t+1 ,a t+1 )value;

[0123] S311: If t = T-1, by comparing the Q function values ​​of the state-strategy space (s, a), each finds the optimal strategy of the state space-action space corresponding to the maximum Q function value, and the state space and action space and reward function Q corresponding to the optimal strategy are * (s * ,a * ) is saved in the DQN experience playback buffer as a training sample of the DQN network, and the optimal strategy of the VLC-RF system is output, and the process goes to step S312; otherwise, let t = t + 1, and go to step S304;

[0124] Among them, the optimal action space s that maximizes the Q function value * and strategy space a * for:

[0125]

[0126] In the above formula, Q * (s,a * ) indicates that (s,a) corresponds to the optimal Q function value and the optimal strategy a * .

[0127] S312: If the algorithm running time ends, output the optimal strategy (s * ,a * ); otherwise, set t=0 and go to step S305. BRIEF DESCRIPTION OF THE DRAWINGS

[0128] In order to make the purpose, technical solution and advantages of the present invention clearer, the invention will be further described below with reference to the accompanying drawings:

[0129] Figure 1 Schematic diagram of indoor VLC-RF heterogeneous network model;

[0130] Figure 2 Resource allocation model diagram of VLC-RF network based on DQN;

[0131] Figure 3 User alliance conversion flow chart based on system energy efficiency and user satisfaction in alliance game;

[0132] Figure 4 Flowchart of the method for optimizing indoor VLC-RF resource energy efficiency allocation based on multi-agent deep Q network. Specific implementation plan

[0133] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents described in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and the features in the embodiments can be combined with each other without conflict.

[0134] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0135] Attached Figure 1 The schematic diagram of the heterogeneous network model of the indoor VLC-RF heterogeneous network is shown in Figure 1. In a room of 10m×10m×3m, 16 VLC APs are evenly distributed in a rectangular shape on the ceiling. VLC APs rely on LEDs for optical communication, which are used for downlink optical link transmission of communication signals. Ignoring the width of the LED lights themselves, the interval between each LED is 2.5m. An RF AP is deployed in the center of the room for uplink and downlink communications. The RF AP and the VLC AP are connected to the central controller of the indoor wireless access point through power lines, and the central controller is connected to the Internet through power lines. Multiple user devices are randomly placed at a height of h from the ground. PD On the workbench, the specific ceiling height of the workbench is h.

[0136] Attached Figure 2 for Figure 2 The resource allocation model diagram of the VLC-RF network based on DQN. In the VLC-RF network, each user is an abstract intelligent agent. The achievable rate of the link transmission communication information between the user and the AP or the attribute information of the link interruption probability is abstracted as the link state S. The subcarrier of which AP the user accesses is abstracted as action A. The target value of the system optimization obtained by the user after taking the action is abstracted as the reward function as R. In the DQN network, the Q function value is equal to the R value of an action. Figure 2In the process, first, at the beginning of each iteration, the central controller uses the alliance game method to calculate the alliance utility value through the service request of the user in the previous round, and classifies the attribute alliance to which the user's service belongs, that is, to determine whether the user's access to the AP focuses on the communication rate value or the link reliability; then, the user is classified into different alliances, the channel state information S of the user is updated, and input into the DQN target network, and the DQN calculates the user's selection of different actions A, that is, the selection of access to different APs, AP transmit power values ​​and subcarriers, calculates the reward function value and Q value of each action, and feeds back the action output with the largest Q value to the user as the best action corresponding to the maximum reward value of the user; at the same time, the best action is stored in the experience pool of the DQN as trajectory data, and the internal parameters of the user's DQN online network are optimized. The DQN online network periodically uses the historical trajectory data in the experience pool to minimize the loss function to update the parameters θ of the DQN, and then the results of the DQN online network optimization parameters are input into the DQN target network for processing the timely action selection of user requests.

[0137] Attached Figure 3 This is a user alliance conversion flow chart based on system energy efficiency and user satisfaction of alliance game, which is mainly divided into the following steps:

[0138] Input: user device set M, each user's requested rate value, AP set N, RF AP is numbered 0 in set N, VLCAP is numbered 1, 2, ..., N in set N, AP subchannel set is K, VLC AP's transmit power P V , the transmission power of RF AP is P R ; Set the number of observation time slots T, SINR threshold, time slot t value, and the weight factor κ of the communication rate requirement being met by the system. Let alliance F 1 =M, Alliance

[0139] Output: the alliance to which each user belongs after the alliance game, and the fairness function value of the proportion of different needs of users being met by the system;

[0140] Step 1: According to formula (1), based on the Lambertian radiation model, at the tth time slot, calculate the channel gain of the user accessing the VLC AP: m∈M,n∈N\0,

[0141] Step 2: According to formula (2), the signal-to-interference-noise ratio of each user accessing the VLC AP in the tth time slot is m∈M,n∈N\0,k∈K;

[0142] Step 3: According to formula (3), calculate the achievable rate value that the user can obtain by accessing the VLC AP in the tth time slot: m∈M,n∈N\0,k∈K;

[0143] Step 4: Calculate the achievable rate value of the VLC AP at the tth time slot using formula (4):

[0144] Step 5: According to formula (5), in the tth time slot, calculate the channel gain value of the user accessing the RF AP

[0145] S106: According to formula (7), in the tth time slot, calculate the SINR value of the user accessing the RF AP

[0146] Step 7: According to formula (8), calculate the achievable rate value of the user accessing the RF AP in the tth time slot:

[0147] Step 8: Calculate the achievable rate value of the RF AP at the tth time slot using formula (9):

[0148] Step 9: According to formula (14), the average probability μ of the high-rate demand requested by the user being met by the system within the scheduling period T is calculated m , m∈M;

[0149] Step 10: According to formula (15), the average probability value λ of the high reliability requirement of user equipment m being met by the system within the scheduling period T is calculated m , m∈M;

[0150] Step 11: Based on formula (16), establish the alliance game model G = (M, V, F) for users to access AP, where F is the alliance set, and F = {F 1 ,F 2}, where F 1 and F 2 denote the high-rate alliance user set and the high-reliability alliance user set respectively, and F 1 ∪F 2 =M;

[0151] Step 12: Calculate the utility value V(F) of the alliance using formula (17);

[0152] Step 13: Select a user from the set M who has not attempted alliance switching, and calculate the variable greed factor value according to formula (19); for the user who requests alliance switching, randomly generate a probability conversion value p. If the probability value is greater than the variable greed factor value, the user will try to switch from the equivalent alliance to another alliance according to the probability value p; otherwise, the user will switch from one alliance to another alliance with the variable greed factor value;

[0153] Step 14: Calculate the utility function value after alliance conversion according to formula (18). If the alliance utility value of the user after alliance conversion is greater than the alliance utility value before conversion, the user is converted to an alliance; otherwise, the alliance conversion attempted by the user is invalid, and the alliance F 1 and F 2 Keep unchanged and go to step 15;

[0154] Among them, user m is from alliance F 1 Switch to Alliance F 2 , the transformed alliance utility function is:

[0155]

[0156] Step 15: If all users in set M have tried the alliance conversion, F is determined 1 and F 2 If the user is in, go to step 16; otherwise, return to step 13;

[0157] Step 16: Using formula (20), calculate the value of the fairness function PF of the proportion of different needs of users in the system being satisfied by the system;

[0158] Step 17: Output the coalition game F 1 and F 2 Set, PF value.

[0159] Attached Figure 4 The flowchart of the method for optimizing indoor VLC-RF resource energy efficiency allocation based on multi-agent deep Q network is mainly divided into the following steps:

[0160] Input: user device set M, each user's requested rate value, AP set N, RF AP is numbered 0 in set N, VLCAP is numbered 1, 2, ..., N in set N, AP subchannel set is K, VLC AP's transmit power P V , the transmission power of RF AP is P R ; After user alliance game F 1 and F 2 The user set and PF value are set; the number of observation time slots T, SINR threshold, weight factor κ of the communication rate requirement being met by the system, discount factor γ, profit value ζ per unit communication achievable rate, cost value σ per unit transmission power are set, and t = 0;

[0161] Output: AP associated with the user, AP transmit power, and subcarriers allocated to the user;

[0162] Step 1: Calculate the achievable rate value of each VLC AP at the tth time slot using formula (4): n∈N\0;

[0163] Step 2: Calculate the achievable rate value of the RF AP at the tth time slot using formula (9):

[0164] Step 3: Calculate the total achievable rate value R of the VLC-RF system at the tth time slot using formula (10): sum (t);

[0165] Step 4: Calculate the historical average power consumption value P of each AP in the VLC-RF system using formula (11): ave ;

[0166] Step 5: Calculate the historical average achievable rate of each AP using formula (12):

[0167] Step 6: Calculate the average energy efficiency profit value EEP of each AP in the VLC-RF system using formula (13);

[0168] Step 7: Based on formulas (21) and (22), establish the optimization objective function model of the VLC-RF system's energy efficiency-user satisfaction maximization, that is, maximize the EEP×PF value;

[0169] Step 8: Abstract each UE in the VLC-RF system as an agent, construct the state space S of each agent, the action space A of the agent, and model the link state decided by the agent and the strategy of selecting the action space as a Markov decision process (S, A, R, P); R represents the average reward value of the agent obtained after the agent adopts A in the VLC-RF system; P represents the set of probabilities of the agent transferring from the current state to the next state after the agent performs action A;

[0170] Step 9: If t = T-1, calculate the average reward value of each agent Let t = 0;

[0171] Step 10: Calculate the agent’s instantaneous reward value r by formula (24): t (s t ,a t ), according to formula (25), calculate the system's discount reward value R t (s t ,a t ),

[0172] Step 11: Set the Q function of the VLC-RF system to:

[0173] Step 12: Execute the attached Figure 3 The alliance game shown in the figure obtains the user's alliance game result F 1 and F 2; Update the channel state space of each agent alliance from s t becomes s t+1 , the action space changes from a t becomes a t+1 , calculate the Q function value Q(s) after the state space is updated t+1 ,a t+1 );

[0174] Among them, the Q function value of the updated state space can be calculated by the following formula:

[0175]

[0176] Step 13: Use the excellent (s, a) in the DQN experience replay buffer to train the DQN network and obtain the DQN fitting Q function; calculate the DQN expected objective function and (s t+1 ,a t+1 ) between the loss function values;

[0177] Among them, DQN expects the objective function and (s t ,a t ) is defined as:

[0178]

[0179] Among them, y t represents the DQN expected objective function, and

[0180] Step 14: Use the iterative update method to find the set of θ parameter values ​​corresponding to the minimum loss function value;

[0181] Among them, the iterative update equation of the θ parameter value set can be expressed as:

[0182]

[0183] Step 15: Substitute θ into the function Q(s t+1 ,a t+1 ), recalculate Q(s t+1 ,a t+1 )value;

[0184] Step 16: If t = T-1, by comparing the Q function values ​​of the state-strategy space (s, a), find the optimal strategy of the state space-action space corresponding to the maximum Q function value, and convert the state space and action space and reward function Q corresponding to the optimal strategy into * (s * ,a * ) is saved in the DQN experience replay buffer as a training sample for the DQN network, and then go to step 17; otherwise, let t = t + 1 and go to step 9;

[0185] Among them, the optimal action space s that maximizes the Q function value * and strategy space a * for:

[0186]

[0187] In the above formula, Q * (s,a * ) represents the optimal Q function value and optimal strategy a corresponding to the state-strategy space (s, a) * ;

[0188] Step 17: If the algorithm running time ends, output the optimal strategy Q adopted by each user in the VLC-RF system * (s * ,a * ), determine the AP accessed by the user in the current scheduling period, the transmit power of the AP, and the subcarriers allocated to the user; otherwise, set t=0 and go to step 10.

[0189] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A multi-agent deep Q network optimization method for indoor VLC-RF resource energy efficiency allocation, characterized by: The method comprises the following steps: S1: Input user device set M, each user's requested rate value, 1 RF AP (Radio Frequency Access Point) and N VLC APs are stored in AP (Access Point) set N, RF AP is numbered 0 in set N, AP's subchannel set is K, VLC (Visible Lighting Communication) AP's transmit power P V , the transmission power of RF AP is P R ; Set the number of observation time slots T, SINR threshold, discount factor γ, profit value ζ per unit communication achievable rate, cost value σ per unit transmission power, and weight factor κ for the communication rate requirement to be met by the system; calculate the channel gain and achievable achievable rate value of the user equipment accessing VLC AP and RF AP respectively, and calculate the total achievable rate value of the system, t∈T; According to the Lambertian radiation model, in the tth time slot, the channel gain of user m accessing VLC AP n is: In the above formula, θ represents the Lambertian radiation coefficient, and θ = -ln2 / ln(cosφ 1 / 2 ), φ1 / 2 represents the half-power radiation angle of the VLC AP, φ represents the radiation angle of the VLC AP, A r Indicates the receiving area of ​​the UE's receiver photodetector (PD), represents the straight-line distance between UE m and AP n, ψ m,n is the incident angle when UE m receives the signal from VLC AP n, ψ c is the field of view of the UE receiver, T s (ψ m,n ) and g(ψ m,n ) represent the gains of the optical filter and the optical concentrator when UE m receives the light signal with ψm,n; According to the Shannon formula, the SINR (Signal to Interference plus Noise Ratio) value of the subchannel k of the user m accessing the VLC AP n is: In the above formula, are the kth subcarrier optical power values ​​allocated by the nth and n'th VLC APs when they connect to UE m in the tth time slot; η is the PD photoelectric conversion coefficient of the UE; N V , are the power spectral density of VLC noise and the channel bandwidth of the kth subchannel of VLCAP, respectively, where B V is the total modulation bandwidth that each VLC AP can communicate with, The total number of subchannels of each VLC AP is K; N\0 represents a set of all VLC APs except RF APs, where 0 is the index symbol of RF AP in set N; Among them, in the tth time slot, the achievable rate value that user m can obtain by accessing subchannel k of VLC AP n is: In the tth time slot, the achievable rate value that the nth VLC AP can achieve can be calculated by the following formula: In the above formula, is a binary variable. If the mth UE is associated with the nth VLC AP and uses its kth subchannel in the tth time slot, then otherwise, In which, in the tth time slot, the link channel gain between the RF AP and the mth UE can be expressed as: In the above formula, is a random variable that follows an exponential distribution and describes the RF small-scale fading, with a mean of 2.46 dB; L(d (m ,0) (t)) is the path loss function of the RF link, which is defined as; In the above formula, d (m,0) (t) is the distance between the RF AP and the mth UE in time slot t; B1 and B2 are constants of the gain model, with values ​​of 35.3 and 21.3 respectively; f c is the center frequency of the carrier, in GHz; X is a Gaussian random variable with mean 0 and variance 1.8 dB; Among them, in the tth time slot, the calculation formula of the SINR value of the signal received by UE m on the kth subcarrier of the RF AP is: In the above formula, is the power value of the kth subcarrier allocated by the RF AP to the connected UE m at time slot t; is the average noise power of the RF system; According to Shannon's formula, in the tth time slot, the achievable rate obtained by UE m associated with the RF AP is: In the above formula, is the bandwidth of the kth RF AP subchannel; The calculation formula for the achievable rate that the RF AP can achieve in the tth time slot is: In the above formula, is a binary variable. When user m associates the kth subchannel of the RF AP in the tth time slot, otherwise, Among them, in the tth time slot, the total achievable rate that the VLC-RF system can achieve is: S2: adopt the method described in claim 2, calculate the average achievable rate and average power consumption of each AP per scheduling period according to the achievable rate value, and construct the energy consumption profit function of the AP; construct a fair function for evaluating the satisfaction ratio of user requests to access different APs, adopt the alliance game method, adopt the probability optimization method to optimize the user's alliance conversion according to the link status and satisfaction requirements, and construct an optimization objective function that reflects the system energy efficiency and user satisfaction; S3: The method described in claim 3 is adopted to represent VLC-RF access users and resource allocation using a Markov decision process, and the user is abstracted as an intelligent agent. The state space, action space, state transition probability matrix, and Q function value of the intelligent agent are constructed as rewards for the state-action strategy of the intelligent agent based on link state information, and the decision action is: the user selects AP access, AP transmit power, and allocates AP subcarrier resources. The designed reward function is the system optimization objective function; the Markov decision process is optimized by DQN (Deep Q-Learning Network), and a variable greedy factor is used to control the intelligent agent to adopt an adjustable probability exploration strategy in different resource allocation stages, optimize the Q function value of the user-associated VLC-AP access point, access point transmit power, and subcarrier resource allocation in the system, and obtain an optimized resource allocation plan for each user.

2. A method for optimizing indoor VLC-RF resource energy efficiency allocation using a multi-agent deep Q network as claimed in claim 1, characterized in that: The specific method of S2 is: S201: Calculate the historical average power consumption value of each AP; The average power consumption of an AP is calculated as follows: In the above formula, T is the number of time slots in a scheduling cycle, is the total transmit power of VLC AP n in time slot t, is the total transmission power value of RF AP in time slot t, Θ is the weight of VLC AP transmit power equivalent to RF AP transmit power; S202: Calculate the historical average achievable rate value of each AP; The calculation formula for the average achievable rate of each AP is: S203: Calculate the average energy efficiency profit value of each AP in the system; The calculation formula of the average energy efficiency profit value of AP is: In the above formula, ζ is the profit of the system to obtain the unit communication rate, and σ represents the cost required for the AP's unit transmission power; S204: Counting the average probability value of the high rate demand of the user equipment m being satisfied by the system; Among them, the average probability that the high rate demand of user equipment m is met by the VLC-RF system within a scheduling period T is: In the above formula, μ m (t) is the probability that the high rate demand of user equipment m is met by the system in the tth time slot, and its value is: μ m (t) = pro{R ser (t)≥R req (t)}, pro(·) is the probability function value that satisfies the conditions in brackets; R req (t) represents the communication rate value requested by user m in the tth time slot, R ser (t) is the achievable rate value obtained by user m from the AP in the tth time slot, S205: Counting the average probability value of the high reliability requirement of the user equipment m being met by the system; The high reliability requirement is expressed by the probability of uninterrupted connection. The average probability that the high reliability requirement of user equipment m is met by the VLC-RF system within the scheduling period T is: In the above formula, λ m (t) is the probability that the high reliability requirement of user equipment m is met in the SINR value of the communication in the tth time slot, λ m (t) = pro{γ m (t)≥γ th }, γ th is the SINR threshold for uninterrupted communication, γ m (t) is the SINR value of user m in time slot t, S206: In order to increase the probability of satisfying the user's rate requirement or reliable connection, and improve the user's QoS (Quality of Service), the user device's choice of which VLC AP or RF AP to access to obtain a higher QoS is modeled as a coalition game, and the utility function value of the coalition game is calculated; wherein, the system initializes the coalition game start time, and allows all users to join the coalition F1 first; Among them, the alliance game model of users accessing AP is: G=(M,V,F) In the above formula, F is the alliance set, F = {F1, F2}, where F1 and F2 represent the high-rate alliance user set and the high-reliability alliance user set respectively, and F1∪F2=M; V is the utility function of the alliance; Among them, the utility function V(F) of alliance F can be expressed as: S207: Select a user from the set M who has not attempted alliance conversion, and calculate the utility function value after the alliance conversion. If the alliance utility value of the user after the alliance conversion is greater than the alliance utility value before the conversion, go to step S208; otherwise, the alliance conversion attempted by the user is invalid, and the alliances F1 and F2 remain unchanged, and go to step S209; Among them, user m changes from alliance F1 to alliance F2, and the utility function of the converted alliance is: S208: For the user who requests alliance switching, a probability conversion value p is randomly generated. If the probability value is greater than the variable greed factor value, the user is switched from one alliance to another alliance according to the probability value p; otherwise, the user is switched from one alliance to another alliance according to the variable greed factor value; Among them, the variable greed factor P(L t ) is calculated as: In the above formula, L t =log2 t / L0, where L0 represents a constant. In this application, L0=T, t∈T; S209: If all users in the set M have tried the alliance conversion, the users in F1 and F2 are determined, go to step S210; otherwise, return to step S208; S210: Calculating a fairness function value of the proportion of different demands of users in the system being satisfied by the system; The calculation formula of the fairness function value of the proportion of different needs of users being met by the system is: In the above formula, κ is the weight factor of the user's high-speed communication demand being met by the system, κ∈[0,1], 1-κ is the weight factor of the user's reliability demand being met by the system; S211: Establish the optimization objective function of VLC-RF system energy efficiency-user satisfaction maximization: Among them, the optimization objective function of maximizing energy efficiency-user satisfaction of the VLC-RF system of this application is: max{EEP·PF} In the above formula, it is necessary to adjust the association of each user accessing the AP, the power value obtained by the user from the AP, and the allocated subcarrier within the scheduling period.

3. A method for optimizing indoor VLC-RF resource energy efficiency allocation using a multi-agent deep Q network as claimed in claim 1, characterized in that: The S3 specific method is: S301: To achieve the optimization goal of this application, each UE in the VLC-RF system is abstracted as an agent, and the state space S of each agent is constructed. S represents the channel state information set of the link between each agent and its associated AP. S is expressed as: S = {s1, ..., s m ,...,s M }, where s m ={0,1},s m =1 indicates that the quality of the signal received by UE m meets the QoS requirement; specifically, for users with high rate requirements, when R ser (t)≥R req (t), s m =1, otherwise, s m =0; for users with high reliability requirements, when γ m (t)≥γ th When m =1, otherwise, s m =0; S302: Construct the action space A of the agent, where A is the set of actions taken by each agent. The AP to which the user is associated, the transmit power selected, and the subchannel to which the user is connected are abstracted as the action space of A. Then A is expressed as: A = {a1(t), ..., a m (t),...,a M (t)}, where a m (t) = {ρ m ,B m ,P m }, and ρ m ={ρ0,...,ρ n ,...,ρ N }, where ρ n ={0,1}, if UE m is associated with AP n, then ρ n =1, otherwise, ρ n =0; B m ={B1,...,B K }, if UE m obtains AP's subchannel k, then B k =1, otherwise, B k =0;P m ={P1, P2, P3} are three different power levels; S303: Model the link state of the agent's decision and the strategy of selecting the action space as a Markov decision process (S, A, R, P), where R represents the average reward value of the agent obtained after the agent adopts A in the VLC-RF system; P represents the set of probabilities of the agent transferring from the current state to the next state after the agent performs action A; let t = 0; S304: If t=T-1, calculate the average energy consumption profit value and proportional fairness function value of the system, and then calculate the average reward value of each agent, set t=0, and go to step S305; otherwise, go to step S305; Among them, in the previous round of scheduling period T, the average reward value obtained by each agent is: In the above formula, (s, a) represents the average EEP·PF value in the previous scheduling period; since the average reward value of each agent is the same, which is EEP·PF / M, the optimization goal of the system is to maximize EEP·PF. Therefore, as long as the number of users M in the system remains unchanged, the optimization goal of the individual agent is the same as the optimization goal of the system. S305: Calculate the instantaneous reward value of the agent and the discounted reward value of the system after the agent adjusts its action strategy; Among them, in the tth time slot of the current scheduling cycle, the agent group adopts the strategy (s t ,a t ) is expressed as: In the above formula, Among them, within a scheduling period T, the discount reward value of the VLC-RF system is: In the above formula, γ is the attenuation discount factor of the reward value, γ∈[0,1]; S306: In order to find the strategies adopted by each agent to achieve the system optimization goal, the system designs a Q function as the R function of the Markov decision process (S, A, R, P). The Q function is determined by the expected value of the discounted reward value of the agent's strategy in the state space S-action space A; Among them, the Q function of the VLC-RF system is defined as: In the above formula, represents the mathematical expectation function; s t and a t They represent the link state set s selected by all users in the user set M at time t. t and action set a t The cumulative discounted reward value R of the strategy t (s t ,a t ); S307: The agent interacts with the channel state-strategy and adopts the method of claim 2 to perform alliance conversion on each agent, randomly generates a probability conversion value p, p∈P, and calculates the variable greed factor P(L t ), update the channel state space of each agent alliance from s t becomes s t+1 , the action space changes from a t becomes a t+1 , calculate the Q function value after the state space is updated; Among them, the Q function value of the updated state space can be calculated by the following formula: In the above formula, θ represents the learning rate set of each agent; a′ represents the learning rate of each agent in the state space s t+1 The action subspace of A that can be adopted when ; S308: Use DQN (Deep Q-Learning Network) to fit the Q function, that is, use the excellent (s, a) in the DQN experience replay buffer to train the DQN network; calculate the DQN expected objective function and (s t+1 ,a t+1 ) between the loss function values; Among them, DQN expects the objective function and (s t+1 ,a t+1 ) is defined as: In the above formula, y t represents the DQN expected objective function, and S309: solving the set of θ parameter values ​​corresponding to the minimum loss function value through an iterative update method; Among them, the iterative update equation of the θ parameter set can be expressed as: In the above formula, Indicates rounding up to an integer; is the gradient iteration step size; S310: Substitute θ into the function Q(s t+1 ,a t+1 ), recalculate Q(s t+1 ,a t+1 )value; S311: If t = T-1, by comparing the Q function values ​​of the state-strategy space (s, a), each finds the optimal strategy of the state space-action space corresponding to the maximum Q function value, and the state space and action space and reward function Q corresponding to the optimal strategy are * (s * ,a * ) is saved in the DQN experience playback buffer as a training sample for the DQN network, and the process goes to step S312; otherwise, let t = t + 1, and the process goes to step S304; Among them, the optimal action space s that maximizes the Q function value * and strategy space a * for: In the above formula, Q*(s,a*) represents the optimal Q function value and optimal strategy a corresponding to the state-strategy space (s,a) * ; S312: If the algorithm running time ends, output the optimal strategy (s * ,a * ); otherwise, set t=0 and go to step S305.

Images