A reinforcement learning-based resource allocation method in user-centric wireless optical communication networks

Abstract

The present invention discloses a resource allocation method based on reinforcement learning in a user-centric wireless optical communication network, including: calculating the channel gain between the user terminal and the access point; clustering the users by using the K-means clustering algorithm; associating the access point with the user to construct a user-centric cell; expressing the optimization problem of determining the modulation mode allocation and power allocation strategy under the constraints of clipping distortion and power to achieve the maximum total throughput; corresponding the optimization problem to the reinforcement learning problem, solving the reinforcement learning problem by using the Q learning algorithm, and obtaining the allocation strategy. The throughput and interruption probability that can be achieved by the present invention are superior to those of the traditional network-centric wireless optical communication system.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless optical communication, and in particular relates to a resource allocation method based on reinforcement learning in a user-centric wireless optical communication network. Background Art

[0002] In recent years, the demand for wireless data services has grown rapidly, and wireless radio frequency spectrum resources have become increasingly scarce, posing several challenges to existing wireless communication technologies. Wireless optical communication has become a research hotspot that has attracted much attention as a feasible solution to alleviate the strong demand for mobile data, due to its advantages such as rich spectrum resources, no electromagnetic radiation, and good confidentiality.

[0003] Wireless optical communication has the dual functions of lighting and communication. It uses energy-efficient light-emitting diodes (LEDs) as access points (APs) to transmit information to user terminals (UEs) equipped with photodetectors (PDs). Currently, in order to increase the achievable data rate, point-to-point transmission in wireless optical communication has been widely studied, from simple on-off keying (OOK) and pulse position modulation (PPM) to more complex optical orthogonal frequency division modulation (OOFDM).

[0004] The achievements at the link level have also led to the development of system-level research on wireless optical communication networks. Traditional cells designed with a network-centric (NC) design will cause serious inter-cell interference (ICI), which will reduce system performance. In contrast, non-standard cells designed with a user-centric (UC) design have been proven to effectively improve system performance, and multi-user interference within the cell can be eliminated by using precoding technology. However, current research is usually based on idealized assumptions, and rarely considers the clipping distortion caused by the nonlinear characteristics of LEDs, nor the actual achievable throughput. Therefore, considering clipping distortion, it is very important to select the optimal modulation mode allocation strategy under the transmission power constraint to maximize the system throughput. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a resource allocation method based on reinforcement learning in a user-centric wireless optical communication network in view of the deficiencies of the above-mentioned prior art.

[0006] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0007] A resource allocation method based on reinforcement learning in a user-centric wireless optical communication network, comprising:

[0008] Step 1, for a user-centric wireless optical communication network, calculating a channel gain between a user terminal and an access point;

[0009] Step 2: According to the location information of the user terminal, the user is clustered using the K-means clustering algorithm;

[0010] Step 3: Based on the channel gain obtained in step 1 and the user clustering result obtained in step 2, the access point is associated with the user to build a user-centered cell;

[0011] Step 4: for the user-centric cell constructed in step 3, the modulation mode allocation and power allocation strategy are determined under the constraints of clipping distortion and power to achieve the optimization problem of maximizing the total throughput;

[0012] Step 5: Correspond the optimization problem to a reinforcement learning problem, use the Q learning algorithm to solve the reinforcement learning problem, and obtain the optimal allocation strategy.

[0013] To optimize the above technical solutions, the specific measures taken also include:

[0014] The above step 1 includes:

[0015] According to the direct path channel gain between user k and access point q and the channel gain of the first reflection path The total channel gain between user k and access point q is obtained: Then we get the channel gain matrix Among them, N u Represents the total number of users in the system; N a Represents the total number of access points in the system; the element [H] located in the kth row and qth column of the matrix H (k,q) :=h k,q .

[0016] The direct path channel gain between user k and access point q is for:

[0017]

[0018] Where m = ln2 / ln[cos(φ 1 / 2 )] is the Lambertian emission order, φ 1 / 2 represents the half-power angle; A represents the physical area of ​​the photodetector; D k,q represents the distance between user k and access point q; φ represents the irradiance angle; ψ represents the incident angle; T s (ψ) represents the gain of the optical filter; g(ψ) = n 2 / sin 2 (ψ FoV ) is the gain of the optical concentrator, n is the refractive index; ψ FoV Indicates the angular width of the receiver's field of view.

[0019] The channel gain of the first reflection path mentioned above is for:

[0020]

[0021] in, This represents the reflection term arising from a small reflection point:

[0022]

[0023] Among them, D k,r is the distance between user k and the reflection point; D r,q is the distance between the reflection point and the access point q; dA wall represents the reflection area; ρ represents the reflection coefficient; α and β are the irradiance angles to the reflection point and the user, respectively.

[0024] The above step 2 introduces the clustering validity function E and determines the optimal number of clusters k opt :

[0025]

[0026] in, Represents the intra-class distance, that is, the sum of the distances from the data samples contained in each cluster to the centers of each cluster; represents the distance between clusters, that is, the sum of the distances from all cluster centers to the global center; C i represents the i-th cluster; x is the sample in the i-th cluster; represents the center of the i-th cluster; Represents the center of all samples, N u Represents the total number of users in the system; k represents the number of clusters.

[0027] The above step three includes:

[0028] (1) Based on the channel gain matrix H, each user is assigned a strongest access point that provides the strongest channel gain;

[0029] (2) For each user cluster obtained in step 2, the access point that is within a distance threshold d from the cluster center is used as the extended access point of the cluster;

[0030] (3) The strongest access point and the extended access point are stored in the access point set to form a user-centered cell: For the i-th non-standard cell C i , there is |A i | Access points in set A i In the set U i |U i | users provide services. Among them |A i |,|Ui |Respectively set A i With set U i The number of elements in .

[0031] In (1) above, if a user has the same channel gain with respect to multiple access points, the user is randomly assigned to one of the access points; if an access point faces q k and q f If the association request conflicts, the channel gain matrix H is searched again to find the next priority access point q′ k and q′ f ,if Higher than Then the access point is associated with user k; as well as They represent the kth row and qth row in the channel matrix H. k Column, row f, q′ f Column, kth row, q′th k Column and row f, q f Elements of a column.

[0032] The above step 4 describes the optimization problem of determining the modulation mode allocation and power allocation strategy under the constraints of clipping distortion and power to achieve the maximum total throughput, which specifically includes:

[0033] (1) Zero-forcing precoding technology is used to reduce multi-user interference in the cell. Then the i-th non-standard cell C i The multi-user system model in is:

[0034] y i =H i G i P i x i +τ i ;

[0035] in, is the i-th non-standard cell C i Channel gain matrix; is the precoding matrix; P i is the allocated power matrix; is the transmitted signal vector; is the received signal vector; N awgn The variance is Additive Gaussian white noise; The variance is Clipping distortion; I i,k Indicates interference;

[0036] (2) Variance of clipping distortion generated by a single LED in DC biased OFDM:

[0037]

[0038] Among them, P led Represents the electrical power of each light-emitting diode; F = Q (λ b )-Q(λ t ) is the attenuation factor, Q(·) represents the Gaussian Q function; ε b =P l -P bias , ε t =P h -P bias , P bias is the DC bias, P l ,P h is the lower and upper limits of linear power; φ(x)=exp(-x 2 / 2) / 2π;

[0039] (3) The optimization problem is: according to the transmission power limit, calculate the maximum throughput that can be achieved in the actual modulation mode:

[0040]

[0041] in, represents the power required when user k in the i-th cell is assigned modulation mode m; represents the signal-to-interference-noise ratio; G B Indicates bandwidth utilization; represents additive Gaussian white noise, N0 is the power spectral density; Represents the attenuation of DC bias; represents the interference received by user k in cell i from other cells, Indicates the maximum transmit power limit; x i,k,m =1 indicates that user k is assigned modulation mode m, x i,k,m =0 means that user k is not assigned modulation mode m; if for user k, This means that the user has not been assigned any modulation mode;

[0042] Considering DCO-OFDM, the throughput that can be achieved in actual modulation mode is:

[0043]

[0044] Among them, r i,k,m represents the rate that can be achieved by user k in the i-th cell under modulation mode m; m∈M, M represents the modulation mode set, including |M| modulation modes; Indicates the information carrying bandwidth.

[0045] The above step 5 corresponds to the optimization problem to the reinforcement learning problem, which specifically includes:

[0046] The wireless optical communication system corresponds to the environment; the user corresponds to the intelligent agent; each level of power constraint corresponds to a state, which limits the maximum transmission power Discretize into J levels: The modulation pattern assignment corresponds to the actions taken by the agent, and the achievable throughput corresponds to the reward obtained by the agent.

[0047] In the above step 5, the Q-learning algorithm is used to solve the reinforcement learning problem, the Q-table is continuously trained until the Q-table converges, and the trained Q-table is used to select the optimal allocation strategy to maximize the system throughput;

[0048] During the entire training process, the user selects actions and updates strategies. The ε-greedy strategy is used to select actions, that is, when making decisions, there is a probability of ε to randomly select actions, and a probability of 1-ε to select actions according to the Q function.

[0049] The values ​​in the Q table are initialized to 0, and the Q values ​​are continuously updated through the strategy update function to obtain the maximum long-term cumulative reward:

[0050]

[0051] Among them, a is the action in state s, 0<α≤1 is the learning rate, which indicates the update rate; r is the reward value; 0≤γ≤1 is the discount factor; a′ is the optimal action in the next state s′.

[0052] The present invention has the following beneficial effects:

[0053] The present invention takes maximizing system throughput as the goal under the consideration of clipping distortion and power constraints, corresponds the original optimization problem to the reinforcement learning problem, and adopts the Q learning algorithm to solve the reinforcement learning problem in order to maximize the system throughput by selecting the optimal modulation mode strategy under power constraints, which can maximize the system throughput of the wireless optical communication network under the condition of satisfying the actual power constraints. The simulation results show that under a series of practical considerations, the throughput and interruption probability of the constructed user-centric wireless optical communication system are superior to those of the traditional network-centric wireless optical communication system. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is the work flow chart of the method of the present invention;

[0055] Figure 2 This is the experimental simulation result diagram of the relationship between system throughput and number of users in two types of cells;

[0056] Figure 3 This is the experimental simulation result diagram of the relationship between system throughput and field of view angle in two types of cells;

[0057] Figure 4 This is the experimental simulation result diagram of the relationship between the outage probability and the number of users in two types of cells;

[0058] Figure 5 This is the experimental simulation result diagram of the relationship between the outage probability and the field of view angle in two types of cells. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0060] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.

[0061] A resource allocation method based on reinforcement learning in a user-centric wireless optical communication network of the present invention comprises:

[0062] Step 1: For the user-centric wireless optical communication network, a wireless optical communication channel model is constructed to calculate the channel gain between the user terminal and the access point.

[0063] This step includes:

[0064] (1), the direct path channel gain between user k and access point q:

[0065]

[0066] Where m = ln2 / ln[cos(φ 1 / 2 )] is the Lambertian emission order, φ 1 / 2 represents the half-power angle; A represents the physical area of ​​the photodetector; D k,q represents the distance between user k and access point q; φ represents the irradiance angle; ψ represents the incident angle; T s (ψ) represents the gain of the optical filter; g(ψ) = n 2 / sin 2 (ψ FoV ) is the gain of the optical concentrator, n is the refractive index; ψ FoV Indicates the angular width of the receiver's field of view.

[0067] (2) Channel gain of the first reflection path

[0068] in, This represents the reflection term arising from a small reflection point:

[0069]

[0070] Among them, D k,r is the distance between user k and the reflection point; D r,q is the distance between the reflection point and the access point q; dA wall represents the reflection area; ρ represents the reflection coefficient; α and β are the irradiance angles to the reflection point and the user, respectively.

[0071] (3), since the reflection and scattering components account for a small proportion and can be ignored, the channel model here only considers the direct path and the first reflection path. The total channel gain between user k and access point q is: Get the channel gain matrix Among them, N u Represents the total number of users in the system; N a Represents the total number of access points in the system; the element [H] located in the kth row and qth column of the matrix H (k,q) :=h k,q .

[0072] Step 2: According to the location information of the user terminal, the user is clustered using the K-means clustering algorithm;

[0073] In this step, the traditional K-means clustering algorithm requires the number of clusters K to be given in advance, but the value of K will affect the final clustering effect and the convergence speed of the entire algorithm. In practical applications, the value of K is difficult to determine. When constructing non-standard cells, the distance from users in the cell to the cell center needs to be as small as possible, and the distance between cells is as far as possible, which helps to reduce interference between cells. A clustering effectiveness function is constructed to comprehensively consider the intra-class distance and the inter-class distance to evaluate the quality of the clustering effect. Introduce the clustering effectiveness function E and determine the optimal number of clusters k opt :

[0074]

[0075] in, Represents the intra-class distance, that is, the sum of the distances from the data samples contained in each cluster to the centers of each cluster; represents the distance between clusters, that is, the sum of the distances from all cluster centers to the global center; C i represents the i-th cluster; x is the sample in the i-th cluster; represents the center of the i-th cluster; Represents the center of all samples; the optimal number of clusters k opt The search range adopts the empirical rules used by most scholars: n represents the number of samples; k represents the number of clusters.

[0076] Step 3: Based on the channel gain obtained in step 1 and the user clustering result obtained in step 2, the access point is associated with the user to build a user-centered cell;

[0077] This step includes:

[0078] (1), based on the channel gain matrix H, each user is assigned an access point that provides the strongest channel gain to ensure that these users are at least served by their associated access points.

[0079] If a user has the same channel gain with respect to multiple access points, the user is randomly assigned to one of the access points;

[0080] If an access point faces q k and q f The association request conflicts, and the channel gain matrix H is searched again to find the second priority access point q′ k and q′ f ,if Higher than The access point is associated with user k; as well as They represent the kth row and qth row in the channel matrix H. k Column, row f, q′ f Column, kth row, q′th k Column and row f, q f Elements of a column.

[0081] (2) For each user cluster obtained in step 2, the access point that is within a distance threshold d from the cluster center is used as the extended access point of the cluster.

[0082] (3) The strongest access point and the extended access point are stored in the access point set to form a user-centered cell: For the i-th non-standard cell C i , there is |A i | Access points in set A i In the set U i |U i | users provide services. Among them |A i |,|U i |Respectively set A i With set U i The number of elements in .

[0083] Step 4: for the user-centric cell constructed in step 3, determine the modulation mode allocation and power allocation strategy under the constraints of clipping distortion and power to achieve the optimization problem of maximizing the total throughput (the optimization problem is given in (3) below);

[0084] This step includes:

[0085] (1) Zero-forcing precoding technology is used to reduce multi-user interference in the cell. In each cell, there is multi-user interference in the overlapping area of ​​two or more access points. The use of zero-forcing precoding technology can reduce multi-user interference in the cell; the transmission signal vector is processed by the channel matrix and the precoding matrix to generate a received signal without multi-user interference. The i-th non-standard cell C i The multi-user system model in is:

[0086] y i =H i G i P i x i +τ i ;

[0087] in, is the channel gain matrix; is the precoding matrix; P i is the allocated power matrix, P i =diag(p i ), p i The kth element of is the power required by user k in the i-th cell; is the transmitted signal vector; is the received signal vector; N awgn The variance is Additive Gaussian white noise; The variance is Clipping distortion; I i,k Indicates interference.

[0088] (2) Each transmission link between the access point and the user uses DC bias optical orthogonal frequency division multiplexing (DCO-OFDM). The nonlinear characteristics of LEDs will cause clipping distortion. The variance of clipping distortion generated by a single light-emitting diode in DCO-OFDM is:

[0089]

[0090] Among them, P led Represents the electrical power of each light-emitting diode; F = Q (λ b )-Q(λ t ) is the attenuation factor, Q(·) represents the Gaussian Q function; ε b =P l -P bias , ε t =P h -P bias , P bias is the DC bias, [P l ,P h ] is the linear power range; φ(x)=exp(-x 2 / 2) / 2π.

[0091] (3) According to the transmit power limit, calculate the maximum throughput that can be achieved in the actual modulation mode:

[0092]

[0093] in, represents the power required when user k in the ith cell is assigned modulation mode m; for modulation mode m, a power with 2 2m A specific square QAM scheme for the constellation size, represents the signal-to-interference-noise ratio; G B Indicates bandwidth utilization, which is close to 1 in DCO-OFDM; represents additive Gaussian white noise, N0 is the power spectral density; Represents the attenuation of DC bias; represents the interference received by user k in cell i from other cells, Indicates the maximum transmit power limit; x i,k,m =1 indicates that user k is assigned modulation mode m, x i,k,m =0 means that user k is not assigned modulation mode m; if for user k, This indicates that the user is not assigned any modulation mode; the power required by the user can be determined by determining the modulation mode assignment.

[0094] Considering DCO-OFDM, the throughput that can be achieved in actual modulation mode is:

[0095]

[0096] Among them, r i,k,m represents the rate that can be achieved by user k in the i-th cell under modulation mode m; m∈M, M represents the modulation mode set, including |M| modulation modes; Indicates the information carrying bandwidth.

[0097] Step 5: Correspond the original optimization problem to a reinforcement learning problem, use the Q-learning algorithm to solve the reinforcement learning problem, and obtain the optimal allocation strategy.

[0098] This step includes:

[0099] (1) The goal of the original optimization problem is to determine the modulation mode allocation and power allocation strategy to maximize the total throughput. Since higher-order modulation modes will bring higher rates, but at the same time the power requirements will also be higher, the user's modulation mode allocation is affected by power. The reinforcement learning problem is that the agent outputs an action in a certain state in the environment, and the environment outputs the next state and the reward brought by this action based on the action taken by the agent. The purpose of the agent is to obtain as many rewards as possible from the environment. Therefore, the original problem can be corresponded to the reinforcement learning problem:

[0100] The wireless optical communication system corresponds to the environment; the user corresponds to the intelligent agent; each level of power constraint corresponds to a state. Discretize into J levels: The modulation pattern assignment corresponds to the actions taken by the agent, and the achievable throughput corresponds to the reward obtained by the agent;

[0101] (2) In each constructed cell, the Q-learning algorithm is used to solve the reinforcement learning problem and training is continued until the Q-table converges.

[0102] During the entire training process, users need to select actions and update strategies; use the ε-greedy strategy to select actions, that is, when making decisions, there is a probability of ε to randomly select actions, and a probability of 1-ε to select actions according to the Q function. This allows users to maintain a certain exploration rate during training.

[0103] In the Q-learning process, the long-term cumulative reward is determined by the evaluation function value of the Q function. The evaluation function value Q(s,a) is defined as the discounted reward cumulative value of the user selecting action a in state s and selecting the optimal action in the next state. The rows of the Q table represent states and the columns represent actions. The values ​​in the Q table are initialized to 0, and the Q value is continuously updated through the policy update function to obtain the maximum long-term cumulative reward:

[0104]

[0105] Among them, 0<α≤1 is the learning rate, which indicates the update rate; r is the reward value; 0≤γ≤1 is the discount factor, γ is 0, which means that only the reward at the current moment is valued and no attention is paid to subsequent feedback; γ is 1, which means that the reward at each moment is treated fairly; γ between 0 and 1 means that the rewards at subsequent moments have less and less influence on the value function over time; a′ is the optimal action for the next state.

[0106] (3) During each training session, since there are multiple users in each cell, there are not only power layer transfers but also transfers between users during the training process. A Q table is established for each user. When a user selects a modulation mode under a certain power constraint level, the power constraint is transferred to a lower power level closest to the remaining power, and the agent is transformed into the next user. The training ends when the power level is transferred to the lowest level or to the last user in the cell. The trained Q table is used to select the optimal modulation mode allocation strategy to maximize the system throughput.

[0107] like Figures 2 to 5 As shown, the experimental simulation results of the relationship between system throughput and the number of users, the experimental simulation results of the relationship between system throughput and the size of the field of view, the experimental simulation results of the relationship between the interruption probability and the number of users, and the experimental simulation results of the relationship between the interruption probability and the size of the field of view are respectively represented under the network-centric and user-centric designs. A resource allocation method based on reinforcement learning in a user-centric wireless optical communication network provided by an embodiment of the present invention aims to maximize the system throughput under the consideration of clipping distortion and power constraints, and corresponds the original optimization problem to a reinforcement learning problem. In order to maximize the system throughput by selecting the optimal modulation mode strategy under power constraints, a Q learning algorithm is used to solve the reinforcement learning problem. The present invention can maximize the system throughput of the wireless optical communication network under the condition of satisfying the actual power constraints. The simulation results show that under a series of practical considerations, the throughput and interruption probability achievable by the constructed user-centric wireless optical communication system are superior to those of the traditional network-centric wireless optical communication system.

[0108] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0109] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A resource allocation method based on reinforcement learning in a user-centric wireless optical communication network, characterized in that: include: Step 1, for a user-centric wireless optical communication network, calculating a channel gain between a user terminal and an access point; Step 2: According to the location information of the user terminal, the user is clustered using the K-means clustering algorithm; Step 3: Based on the channel gain obtained in step 1 and the user clustering result obtained in step 2, the access point is associated with the user to build a user-centered cell; Step 4: for the user-centric cell constructed in step 3, the optimization problem of maximizing the total throughput is formulated under the constraints of clipping distortion and power, where the maximization of the total throughput depends on determining the modulation mode allocation and power allocation strategy; Step 5, the optimization problem is matched to a reinforcement learning problem, and the reinforcement learning problem is solved by using a Q learning algorithm to obtain an allocation strategy; Step 4 describes the optimization problem of determining the modulation mode allocation and power allocation strategy under the constraints of clipping distortion and power to achieve maximum total throughput, which specifically includes: (1) Zero-forcing precoding technology is used to reduce multi-user interference in the cell. Then the i-th non-standard cell C i The multi-user system model in is: y i =H i G i P i x i +τ i ; in, is the i-th non-standard cell C i Channel gain matrix, U i Indicates non-standard cell C i The user set in A i Indicates non-standard cell C i The set of access points in |A i |,|U i | respectively set A i With the set U i The number of elements in ; is the precoding matrix; P i is the allocated power matrix; is the transmitted signal vector; is the received signal vector; N awgn The variance is Additive Gaussian white noise; The variance is Clipping distortion; I i,k Indicates interference; (2) Variance of clipping distortion generated by a single LED in DC biased OFDM: Among them, P led Represents the electrical power of each light-emitting diode; F = Q (λ b )-Q(λ t ) is the attenuation factor, Q(·) represents the Gaussian Q function; ε b =P l -P bias , ε t =P h -P bias , P bias is the DC bias, P l ,P h is the lower and upper limits of linear power; φ(x)=exp(-x 2 / 2) / 2π; (3) The optimization problem is: according to the transmission power limit, calculate the maximum throughput that can be achieved in the actual modulation mode: Where q is the access point; represents the power required when user k in the i-th cell is assigned modulation mode m; represents the signal-to-interference-to-noise ratio, erfc -1 () represents the inverse function of the complementary error function, BER represents the bit error rate; G B Indicates bandwidth utilization; represents additive Gaussian white noise, N0 is the power spectral density; represents the attenuation of DC bias, [G i ] (q,k) Represents the precoding matrix G i The element at the qth row and the kth column in ; represents the interference received by user k in cell i from other cells, C represents the set of user-centered cells constructed in step 3, [H l ] (k,q) Denotes the channel gain matrix H of cell l l The element at row k and column q in Indicates the maximum transmit power limit; x i,k,m =1 indicates that user k is assigned modulation mode m, x i,k,m =0 means that user k is not assigned modulation mode m; if for user k, This means that the user has not been assigned any modulation mode; Considering DCO-OFDM, the throughput that can be achieved in actual modulation mode is: Among them, r i,k,m represents the rate that can be achieved by user k in the i-th cell under modulation mode m; m∈M, M represents the modulation mode set, including |M| modulation modes; Indicates information carrying bandwidth; Step 5 describes how to map the optimization problem to a reinforcement learning problem, including: The wireless optical communication system corresponds to the environment; the user corresponds to the intelligent agent; each level of power constraint corresponds to a state, which limits the maximum transmission power Discretize into J levels: The modulation pattern assignment corresponds to the actions taken by the agent, and the achievable throughput corresponds to the reward obtained by the agent; The step 5 adopts the Q learning algorithm to solve the reinforcement learning problem, continuously trains the Q table until the Q table converges, and uses the trained Q table to select the optimal allocation strategy to maximize the system throughput; During the entire training process, the user selects actions and updates strategies. The ε-greedy strategy is used to select actions, that is, when making decisions, there is a probability of ε to randomly select actions, and a probability of 1-ε to select actions according to the Q function. The values ​​in the Q table are initialized to 0, and the Q values ​​are continuously updated through the strategy update function to obtain the maximum long-term cumulative reward: Among them, a is the action in state s, 0<α≤1 is the learning rate, which indicates the update rate; r is the reward value; 0≤γ≤1 is the discount factor; a′ is the optimal action in the next state s′.

2. According to claim 1, a resource allocation method based on reinforcement learning in a user-centric wireless optical communication network is characterized in that: The step one comprises: According to the direct path channel gain between user k and access point q and the channel gain of the first reflection path The total channel gain between user k and access point q is obtained: Then we get the channel gain matrix Among them, N u Represents the total number of users in the system; N a Represents the total number of access points in the system; the element [H] located in the kth row and qth column of the matrix H (k,q) =h k,q .

3. According to claim 2, a resource allocation method based on reinforcement learning in a user-centric wireless optical communication network is characterized in that: The direct path channel gain between user k and access point q for: in, is the Lambert emission order, represents the half-power angle; A represents the physical area of ​​the photodetector; D k,q represents the distance between user k and access point q; represents the irradiance angle; ψ represents the incident angle; T s (ψ) represents the gain of the optical filter; g(ψ) = n 2 / sin 2 (ψ FoV ) is the gain of the optical concentrator, n is the refractive index; ψ FoV Indicates the angular width of the receiver's field of view.

4. The method for resource allocation based on reinforcement learning in a user-centric wireless optical communication network according to claim 3, characterized in that: The channel gain of the first reflection path for: in, represents the reflection term generated by a small reflection point; ∫ walls It means integrating the components of small reflection points on the wall; Among them, D k,r is the distance between user k and the reflection point; D r,q is the distance between the reflection point and the access point q; dA wall represents the reflection area; ρ represents the reflection coefficient; α and β are the irradiance angles to the reflection point and the user, respectively.

5. The method for resource allocation based on reinforcement learning in a user-centric wireless optical communication network according to claim 1, characterized in that: The second step introduces the clustering validity function E and determines the optimal number of clusters k opt : in, Represents the intra-class distance, that is, the sum of the distances from the data samples contained in each cluster to the centers of each cluster; represents the distance between clusters, that is, the sum of the distances from all cluster centers to the global center; C y represents the y-th cluster; x is the sample in the y-th cluster; represents the center of the y-th cluster; Represents the center of all samples, N u Represents the total number of users in the system; v represents the value of the number of clusters.

6. The method for resource allocation based on reinforcement learning in a user-centric wireless optical communication network according to claim 1, characterized in that: The step three comprises: (1) Based on the channel gain matrix H, each user is assigned a strongest access point that provides the strongest channel gain; (2) For each user cluster obtained in step 2, the access point that is within a distance threshold d from the cluster center is used as the extended access point of the cluster; (3) The strongest access point and the extended access point are stored in the access point set to form a user-centered cell: For the i-th non-standard cell C i , there is |A i | Access points in set A i In the set U i |U i |Provide services to users.

7. The method for resource allocation based on reinforcement learning in a user-centric wireless optical communication network according to claim 6, characterized in that: In step (1) of step 3, if a user has the same channel gain with respect to multiple access points, the user is randomly assigned to one of the access points; If user k requests to associate with access point q k Access point q that user f requests to associate with f Conflict, i.e. q k =q f , then search the channel gain matrix H again to find the next priority access point q′ k and q′ f ,if Higher than Then access point q k is associated with user k; as well as They represent the kth row and qth row in the channel matrix H. k Column, row f, q′ f Column, kth row, q′th k Column and row f, q f Elements of a column.

Images