Method for dynamic resource allocation in relay-assisted multi-channel fso network based on mp-dqn

Abstract

The application provides a relay-assisted multi-channel FSO network dynamic resource allocation method based on MP-DQN, comprising: acquiring a training data set and establishing an MP-DQN agent model; setting a state space, an action space, an action parameter and a reward of the MP-DQN agent model to obtain a set MP-DQN agent model; training the MP-DQN agent model according to the training data set to obtain a trained MP-DQN agent model; and using the trained MP-DQN agent model to perform relay-assisted multi-channel FSO network resource allocation; the method fully considers the changes of FSO channel state information caused by factors such as weather conditions and atmospheric turbulence and the remaining energy condition of the relay node, simultaneously uses reinforcement learning to solve the dynamic resource allocation problem, and can solve the FSO network resource allocation problem of a similar joint power allocation and relay selection mixed action space.

Description

Technical Field

[0001] This invention relates to the field of free-space optical networks, and more specifically to a dynamic resource allocation method for relay-assisted multi-channel FSO networks based on MP-DQN. Background Technology

[0002] Free-space optical communication (FSOC) has attracted widespread attention due to its high-speed transmission capability, high capacity, low cost, strong security, and flexible structure. Despite these advantages, FSO transmission through the atmosphere can be severely affected by channel characteristics, such as fading caused by atmospheric turbulence, geometric loss, and pointing errors. Therefore, the transmission distance and system performance of FSO are significantly limited. To overcome these limitations, many FSO resource allocation methods have been proposed, including power allocation, relay selection, relay location, and joint power allocation and relay selection. Among these, power allocation has become a popular cooperative transmission technique, improving power utilization while mitigating channel fading effects and avoiding resource waste. Relay selection can alleviate the adverse effects of link interruptions by dividing long links into several short links. Currently, research on optimization algorithms for FSO network resource allocation focuses on maximizing or minimizing system indicators (such as channel capacity and interruption probability) by optimizing parameters including power allocation, relay selection, and relay location based on Channel State Information (CSI). With the development and application of reinforcement learning (RL), FSO network resource allocation algorithms based on reinforcement learning have become a hot topic. This is because reinforcement learning methods are committed to maximizing cumulative rewards, which has irreplaceable advantages in dynamic optimization problems. The literature (Gao, Z.; Eisen, M.; Ribei Ro, A. Resource Allocation via Model-FRee Deep Learning in FRee Space Optical CoMMunications. IEEE TRans. WiRel. CoMMun. 2022, 70, 920–934) designed a model-free primal-dual deep learning algorithm to improve transmission capacity and applied the policy gradient method to the original update to estimate the necessary original update gradient information. The literature (Su, Y.; Lu, X.; Zhao, Y.; Huang, L.; Du, X. Cooperative Communication with Relay Selection based on Deep Relay Learning in Wireless Communication Networks. IEEE Sens. J. 2019, 19, 9561–9569) models the CSI and relay selection in cooperative communication as a Markov decision process. A deep q-network (DQN) is used to select the optimal relay node from multiple relays to minimize the probability of network outage.The paper (Gao SJ, Li YT, Geng TW. Deep Relay Learning-Based Relay Selection Algorithm in FRee-Space Optical Co-ope Rative Communications. Applied Sciences. 2022) proposes a DQN-RS algorithm based on Deep Q-Network (DQN) to maximize average capacity. Unlike related works, it considers the switching loss between relay nodes.

[0003] However, current research on FSO network resource allocation algorithms faces several challenges: 1. FSO channel characteristics are susceptible to changes due to weather, turbulence, and other factors. Therefore, even at the same geographical location, FSO channel characteristics vary across different time periods. Furthermore, for practical reasons, to prevent FSO network outages caused by repeater battery depletion, the remaining battery power of repeaters in each time period should be considered in FSO network resource allocation scenarios involving power allocation. In summary, the relay-assisted multi-channel FSO network resource allocation problem based on channel state information should be a dynamic resource allocation problem. 2. Existing FSOC channel measurement and modeling work primarily focuses on terrestrial networks, while the channel characteristics of satellite-to-ground and air-to-ground links differ significantly from terrestrial network channels. Currently, no research has been able to construct robust air-based and space-based network channel models. Therefore, a model-free resource allocation algorithm is needed, which can achieve dynamic resource allocation through real-time performance indicator observations without knowing the system model. 3. However, most of the above-mentioned research on FSO network resource allocation based on deep reinforcement learning (DRL) only focuses on a single discrete action space or a single continuous action space, and uses the main algorithms in reinforcement learning (RL) such as DQN, Q-leaning, and DDPG for processing.

[0004] However, these methods can only handle discrete or continuous actions individually, and cannot handle FSO network resource allocation problems with mixed action spaces such as joint power allocation and relay selection. Summary of the Invention

[0005] The purpose of this invention is to provide a dynamic resource allocation method for relay-assisted multi-channel FSO networks based on MP-DQN. By using a parameterized action space, it can not only handle mixed action spaces, but also achieve a higher total channel capacity while considering the remaining power of relay nodes in each time slot.

[0006] Specifically, the technical solution is as follows:

[0007] A dynamic resource allocation method for relay-assisted multi-channel FSO networks based on MP-DQN includes:

[0008] Collect data, obtain the training dataset, and build the MP-DQN proxy model;

[0009] Set the state space, action space, action parameters, and reward of the MP-DQN agent model to obtain the set MP-DQN agent model;

[0010] The MP-DQN proxy model is trained based on the training dataset to obtain the trained MP-DQN proxy model;

[0011] Use the trained MP-DQN proxy model for relay-assisted multi-channel FSO network resource allocation;

[0012] The MP-DQN proxy model includes a Q-value estimation network and a Q-value target network. The Q-value estimation network interacts with the relay-assisted multi-channel FSO network environment and outputs the action value function Q(s, a). The Q-value target network is responsible for calculating the loss gradient, which is used to update the parameters of the Q-value estimation network after the parameters of the Q-value estimation network are updated.

[0013] The state space includes:

[0014] One or more of the following: channel status information h, the remaining power of the relay node b, and the power e collected by the relay node;

[0015] The state of the relay-assisted multichannel FSO network in the kth time slot is defined as S. k =[h k ,b k ,e k-1 ], h k Let b represent the set of channel state information for each channel in the k-th time slot. k Let e ​​represent the set of remaining power of each repeater in the k-th time slot. k-1 This represents the set of electrical charges collected by each repeater in the (k-1)th time slot.

[0016] The action space includes:

[0017] Let the action in the k-th time slot be defined as a. k ={N(h), P(h)}, P(h) = [P T,N(h) ,P N(h),R ], This represents the set of transmit power allocated to L optical carriers in the communication link between the transmitter and the selected repeater. This represents the set of transmit power allocated to L optical carriers in the communication link between the selected repeater and receiver. P(h) is a continuous action, T represents the transmitter, and R represents the receiver. N = {n | n∈0: num_relay-1} represents the selected repeater, which is a discrete action; nuM_Relay represents the number of repeater nodes in the repeater-assisted multichannel FSO network; action parameters The continuous action parameters associated with discrete actions represent the allocation of corresponding transmit power to L optical carriers in the communication links between the transmitter and the repeater, and between the repeater and the receiver, respectively, which finally form the parameterized action A. PA ={(N, P N )|P N ∈A c , N∈A d}, where A c A d These represent the continuous action space and the discrete action space, respectively.

[0018] The rewards include:

[0019] The reward R k Used to evaluate in state S k Next, execute action a k Performance characteristics, including the total power limit P of the transmitter and selected relays. t and the peak power limit P for each carrier s ;

[0020] The reward function corresponding to the reward is defined as the total channel capacity R in each time slot. k ;

[0021]

[0022] in, ω=[ω1,......,ω L [] represents the weights assigned to different optical carriers;

[0023] in, T f B is the frame duration, and T is the bandwidth. f B is the frame duration, and B is the bandwidth. For transmission power, This refers to the channel state information of the Lth optical carrier in the communication link between the transmitter and the selected repeater. The channel state information of the Lth optical carrier in the selected repeater-receiver communication link is given by R, where R is the photodetector sensitivity, e is the charge coefficient, and Δf is the noise equivalent bandwidth.

[0024] The phrase "collecting data, obtaining a training dataset, and establishing an MP-DQN proxy model" includes:

[0025] The training dataset is a set S of each state of the relay-assisted multichannel FSO network; S k =[h k ,b k ,e k-1 ] represents the k-th state in set S; where h k =[h 1 , ..., h 2*num_relay*L ] represents the set of channel state information for each channel in the k-th time slot; b k =[b 1 , ..., b num_relay ] represents the set of remaining power of each repeater in the k-th time slot; e k-1 =[e 1 , ..., e num_relay ] represents the collection of electrical charges collected by each repeater in the (k-1)th time slot;

[0026] The battery charge gained in the next time slot after the Kth time slot should follow the update process as follows:

[0027]

[0028] Channel state information in a relay-assisted multi-channel FSO network environment is generated using a log-normal distribution model. The remaining power of each relay node is initialized with a fixed value, and the power collected by the relay node in each time slot is randomly generated.

[0029] The phrase "training the MP-DQN proxy model based on the training dataset to obtain the trained MP-DQN proxy model" includes:

[0030] The initialized MP-DQN proxy model was interacted with the relay-assisted multichannel FSO network environment to obtain training dataset 1;

[0031] The MP-DQN proxy model is trained based on the obtained training dataset 1, as follows:

[0032] M samples (s) are randomly drawn from the experience pool R. k ,a k ,R k ,s k+1 ), will s k ,a k Input Q-value to estimate the value of the network. The network obtains the value in state s. k Make the following action a k Action value function Q value Q(s) k ,ak ;ω) and reward R k ,;The set of states s k+1 In the policy network of the input Q-value target network, obtain the set of action parameters P of the Q-value target network. k+1 ; Set the action set s k+1 and Q-value target network action parameter set P k+1 By inputting the value network of the Q-value target network, we obtain the discrete actions of the Q-value target network and the corresponding action value function Q(s) of the discrete actions. k+1 ,a k+1 ;ω′)

[0033] Get TD taRget y k The method is as follows:

[0034] y k =r k +γmaxQ(S k+1 a k+1 ;ω′)

[0035] Where γ is the discount factor for the reward;

[0036] The method for obtaining the loss function of the eval value network is as follows:

[0037]

[0038] The method for obtaining the loss function of the eval policy network is as follows:

[0039]

[0040] The loss function L(ω) with respect to parameter ω is differentiated with respect to ω, and the parameter ω is updated using gradient descent. The loss function L(θ) with respect to parameter θ is differentiated with respect to θ, and the parameter θ is updated using gradient ascent. The update formula is as follows:

[0041]

[0042]

[0043] Where α and β are the learning rates;

[0044] The taRget network is updated using soft updates, and after the MP-DQN model agent completes any resource allocation of the FSO network, the current state s is set. k Transfer to s k+1 The training continues until M samples randomly drawn from the experience pool are completed, resulting in the trained MP-DQN proxy model.

[0045] Before "interacting the initialized MP-DQN proxy model with the relay-assisted multichannel FSO network environment", the following is also included:

[0046] The Q-value estimation network parameters and Q-value target network parameters in the MP-DQN proxy model are initialized respectively, wherein the policy network parameters and value network parameters of the Q-value estimation network are represented by θ and ω, respectively, and the policy network parameters and value network parameters of the Q-value target network are represented by θ′ and ω′, respectively, to obtain the initialized MP-DQN proxy model.

[0047] The method for updating the taRget network via soft update is as follows:

[0048] ω′=τ a ω+(1-τ a )ω′

[0049] θ′=τ p θ+(1-τ p )θ′

[0050] Where τ a and τ p These are the soft update coefficients of the value network and the policy network of the Q-value target network, respectively.

[0051] The phrase "interacting the initialized MP-DQN proxy model with the relay-assisted multi-channel FSO network environment to obtain the training dataset" includes:

[0052] Set the state set S under the current Kth time slot. k =[h k ,b k ,e k-1 The input Q-value estimation network is used to obtain the action parameters P corresponding to each discrete action N. N ;

[0053] Set the state set S under the current Kth time slot. k =[h k ,b k ,e k-1 ] and all motion parameters P in the k-th time slot al k l ={P1,P2,…,P N The data are input into the value network of the Q-value estimation network in batches. The value network of the Q-value estimation network outputs the action value function Q value of each discrete action under the corresponding action parameters. The discrete actions and their action parameters corresponding to each action value function Q value are randomly selected according to the probability ε, and the discrete actions corresponding to the maximum action value function Q value are obtained.

[0054] Select the discrete action and its parameters corresponding to the maximum action value function Q with a probability of (1-ε), and then obtain the parameterized action A. PA In FSO network resource allocation for relay-assisted multi-channel communication, a k =A PA ={N,P N}, where N is the relay selection, P N Power allocation; obtain the current time-to-time return R k and the set of states s at the next moment. k+1 ε represents the exploration probability;

[0055] Based on the above relevant data s k ,a k ,R k ,s k+1 , thus obtaining training dataset 1 (s k ,a k ,R k ,s k+1 The training dataset 1 is stored in the experience pool R to obtain the training dataset, which is used for training the neural network.

[0056] The action value function is:

[0057] Where E represents the expectation, S t+k Let a represent the (t+k)th state. t+k R(S) represents the action performed in the (t+k)th state. t+k ,a t+k ) indicates that action a is performed in the (t+k)th state. t+k The reward obtained, γ discount factor, represents the importance of future returns and has a value range of [0,1]; in the Bellman equation, it is used to reduce the value of future returns.

[0058] The present invention has at least the following beneficial effects:

[0059] The method described in this invention fully considers the changes in FSO channel state information caused by weather conditions, atmospheric turbulence, and the remaining energy of relay nodes, while using reinforcement learning to solve the dynamic resource allocation problem. Furthermore, the method described in this invention can achieve dynamic resource allocation through real-time performance indicator observations without knowing the system model, avoiding explicit channel modeling. Moreover, unlike commonly used reinforcement learning algorithms such as Q-learning, DQN, and DDPG, the proposed MP-DQN deep reinforcement learning algorithm can be applied in hybrid action spaces, enabling it to solve FSO network resource allocation problems in hybrid action spaces such as joint power allocation and relay selection. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a schematic diagram of the proxy model in an example of the present invention;

[0062] Figure 2 This is a flowchart of the method described in this invention;

[0063] Figure 3 This is the relay-assisted multi-channel FSO network structure in an example of the present invention;

[0064] Figure 4 This is a schematic diagram of a parametric motion space model;

[0065] Figure 5 This is a schematic diagram of the relay-assisted multi-channel FSO network resource allocation structure in this embodiment;

[0066] Figure 6 This is a flowchart of the MP-DQN algorithm network provided in the example of the present invention. Detailed Implementation

[0067] Those skilled in the art will understand that the modules in the apparatus of the implementation scenario can be distributed within the apparatus of the implementation scenario as described, or they can be located in one or more apparatuses different from this implementation scenario, with corresponding changes. The modules of the above-described implementation scenario can be combined into one module, or they can be further divided into multiple sub-modules.

[0068] To solve the problems of existing technologies, such as Figure 2 The overall solution of this invention is: to collect data, obtain a training dataset, and establish, as follows: Figure 1 The MP-DQN agent model is configured; the state space, action space, action parameters, and reward of the MP-DQN agent model are set to obtain the configured MP-DQN agent model; the MP-DQN agent model is trained according to the training dataset to obtain the trained MP-DQN agent model; the trained MP-DQN agent model is used for relay-assisted multi-channel FSO network resource allocation.

[0069] To illustrate the specific steps of the method described in this invention in more detail, the following embodiments are provided:

[0070] Specific Implementation Example I:

[0071] One embodiment provided by the present invention:

[0072] Referring to the accompanying drawings, the present invention provides a relay-assisted multi-channel FSO network resource allocation method based on MP-DQN, comprising the following steps:

[0073] Step 1: Collect data and obtain the training dataset;

[0074] The training dataset described above is the set S of each state of the relay-assisted multichannel FSO network; where S k =[h k ,b k ,e k-1 ] represents the k-th state; h k =[h 1 , ..., h 2*num_relay*L ] represents the set of channel state information for each channel in the k-th time slot; b k =[b 1 , ..., b num_relay ] represents the set of remaining power of each repeater in the k-th time slot; e k-1 =[e 1 , ..., e num_relay [] represents the collection of electricity collected by each repeater in the (k-1)th time slot; since the repeater's battery capacity is limited, when the battery is fully charged, the overflow will cause power waste. Therefore, the electricity obtained by the battery in the next time slot should follow an update process:

[0075]

[0076] In step 1, channel state information (CSI) in a relay-assisted multichannel FSO network environment is generated using a log-normal distribution model. The structure of the FSO network is as follows: Figure 3 As shown, the remaining power of each relay node is initialized with a fixed value, and the power collected by the relay node in each time slot is randomly generated.

[0077] The relay-assisted multichannel FSO network structure described in this embodiment can include one transmitter, one receiver, and N repeaters. The transmitter communicates with the receiver using repeaters selected from a given distributed parallel repeater group. Each repeater has a LOS communication link with both the transmitter and the receiver. Each repeater is equipped with a power harvesting device that can obtain power from renewable resources such as wind and solar energy, which is stored in the repeater's battery. The transmitter modulates the signal onto multiple optical carriers and simultaneously transmits them to the selected repeaters. The repeaters aggregate the received signals, modulate orthogonal carriers, and transmit them to the destination node. In each time slot (TS), we assume there is no crosstalk between the orthogonal optical carriers and only one repeater can receive the signal from the transmitter, selected by switching the direction of the transmitted beam.

[0078] Step 2, establish the MP-DQN proxy model;

[0079] The MP-DQN proxy model comprises two networks: an estimation network for the action value function Q(s,a) and a target network for the action value function Q(s,a), namely, a Q-value estimation network and a Q-value target network. Both networks contain a policy network and a value network. The Q-value estimation network is responsible for interacting with the relay-assisted multi-channel FSO network environment and outputting the action value function Q(s,a), as shown in formula (1). The Q-value target network is responsible for calculating the loss gradient for updating the network parameters. After the Q-value estimation network completes its parameter update, the Q-value target network softly updates the parameters of the Q-value estimation network.

[0080]

[0081] Step 3, as follows Figure 4 The state space, action space, action parameters, and reward of the MP-DQN agent model are designed to obtain the designed MP-DQN agent model.

[0082] The state space is defined as follows: channel state information h, remaining power b of the relay node, and power e collected by the relay node; the state space S is the set of all states in K time slots; the state of the entire relay-assisted multichannel FSO network in the kth time slot is defined as S. k =[h k ,b k ,e k-1 ], h k Let b represent the set of channel state information for each channel in the k-th time slot. k Let e ​​represent the set of remaining power of each repeater in the k-th time slot. k-1This represents the set of electrical charges collected by each repeater in the (k-1)th time slot; for ease of processing, it is assumed that the duration τ of each time slot is normalized to 1.

[0083] In the aforementioned action space, the action in the k-th time slot is defined as a. k ={N(h), P(h)}, P(h) = [P T,N(h) ,P N(h),R ], This represents the set of transmit power allocated to each of the L optical carriers in the communication link between the transmitter and the selected repeater; This represents the set of transmit power allocated to L optical carriers in the communication link between the selected repeater and receiver; P(h) represents continuous action; T represents the transmitter; R represents the receiver; N = {n|n∈0: num_relay-1} represents the selected repeater number, which is a discrete action; nuM_Relay represents the number of repeater nodes in the repeater-assisted multichannel FSO network; action parameters The continuous action parameters associated with discrete actions represent the allocation of corresponding transmit power to L optical carriers in the communication links between the transmitter and the repeater, and between the repeater and the receiver, respectively; these parameters constitute the parameterized action A. PA ={(N, P N )|P N ∈A C , N∈A d} indicates that A C A d These represent the continuous action space and the discrete action space, respectively.

[0084] The reward R mentioned k Used to evaluate in state S k Next, execute action a k Performance targets are twofold: First, the total power limit P of the transmitter and selected relays. t Second: Peak power limit P for each carrier s The reward function is defined as the total channel capacity in each time slot, as shown in formula (2). It is worth noting that in this embodiment, it is not assumed that these models are known to implement the algorithm, but only to generate instantaneous capacity observations to replace physical measurements.

[0085]

[0086] in, ω=[ω1,......,ω L [] represents the weights assigned to different optical carriers;

[0087] in, T fT is the frame duration, B is the bandwidth, and T is the bandwidth. f For frame duration, For transmission power, This refers to the channel state information of the Lth optical carrier in the communication link between the transmitter and the selected repeater. The channel state information of the Lth optical carrier in the selected repeater-receiver communication link is given by R, where R is the photodetector sensitivity, e is the charge coefficient, Δf is the noise equivalent bandwidth, and ∈ = 1 is a full-duplex relay, while ∈ = 2 is a half-duplex relay.

[0088] Step 4: Train the MP-DQN proxy model designed in Step 3 using the training dataset obtained in Step 1, to obtain the trained MP-DQN proxy model. The specific steps are as follows:

[0089] 4.1 Initialize the Q-value estimation network parameters and Q-value target network parameters in the MP-DQN proxy model respectively. The policy network parameters and value network parameters of the Q-value estimation network are represented by θ and ω respectively, and the policy network parameters and value network parameters of the Q-value target network are represented by θ′ and ω′ respectively, to obtain the initialized MP-DQN proxy model.

[0090] 4.2 The initialized MP-DQN proxy model is interacted with the relay-assisted multi-channel FSO network environment to obtain training dataset 1, as follows:

[0091] like Figure 5 Set the state set S of the current k-th time slot. k =[h k ,b k ,e k-1 The input Q-value estimation network is used to determine the policy network, which then obtains the corresponding values ​​for each discrete action N, such as... Figure 4 The motion parameter P shown N Set the state set S in the current time slot k =[h k ,b k ,e k-1 ] and all motion parameters P in the k-th time slot al k l ={P 1, P2,…,P N The data is input in batches into the value network of the Q-value estimation network. The value network of the Q-value estimation network outputs the action value function Q value of each discrete action under the corresponding action parameters. The discrete actions and their action parameters corresponding to each action value Q value are randomly selected with a probability of ε. The discrete action and its action parameters corresponding to the maximum action value function Q value are selected with a probability of (1-ε), where ε is the exploration probability. The obtained parameterized action A is then...PA This is used for joint relay selection and power allocation to allocate FSO network resources for relay-assisted multichannel systems, where a k =A PA ={N,P N}, where N is the repeater number, P N Power allocation is performed. The current time-to-time return R is then obtained. k and the set of states s at the next moment. k+1 Finally, based on the above relevant data s k ,a k ,R k ,s k+1 , thus obtaining training dataset 1 (s k ,a k ,R k ,s k+1 ), store training dataset 1 into experience pool R to obtain training dataset, which is used for training neural network;

[0092] It should be noted that: In this embodiment, discrete actions are used to describe the network structure, while relay selection is used in specific scenarios. Therefore, discrete action N is the selected repeater number N; the above description is a common method in this field.

[0093] 4.3 The MP-DQN proxy model is trained using the obtained training dataset, resulting in the trained MP-DQN proxy model, as follows: Figure 6 The details are as follows:

[0094] 4.3.1 Randomly select M samples (s) from the experience pool R. k ,a k ,R k ,s k+1 ), will s k ,a k Input Q-value to estimate the value of the network. The network obtains the value in state s. k Make the following action a k Action value function Q value Q(s) k ,a k ;ω) and reward R k ,;The set of states s k+1 In the policy network of the input Q-value target network, obtain the set of action parameters P of the Q-value target network. k+1 ; Set the action set s k+1 and Q-value target network action parameter set P k+1 By inputting the value network of the Q-value target network, we obtain the discrete actions of the Q-value target network and the corresponding action value function Q(s) of the discrete actions. k+1 ,a k+1 ;ω′)

[0095] Calculate TD taRget y k y k The calculation formula is:

[0096] y k =r k +γmaxQ(S k+1 a k+1 ;ω′)

[0097] Where γ is the discount factor for the reward;

[0098] The loss function of the eval value network is calculated using the following formula:

[0099]

[0100] The loss function of the eval policy network is calculated using the following formula:

[0101]

[0102] The loss function L(ω) with respect to parameter ω is differentiated with respect to ω, and the parameter ω is updated using gradient descent. The loss function L(θ) with respect to parameter θ is differentiated with respect to θ, and the parameter θ is updated using gradient ascent. The update formula is as follows:

[0103] Where α and β are the learning rates;

[0104] The taRget network is updated using a soft update, and the update formula is:

[0105] ω′=τ a ω+(1-τ a )ω′

[0106] θ′=τ p θ+(1-τ p )θ′

[0107] Where τ a and τ p These are the soft update coefficients for the value network and policy network of the Q-value target network, respectively.

[0108] 4.3.4 After the MP-DQN proxy model completes one resource allocation of the FSO network, the current state is changed by s. k Transfer to s k+1 Repeat steps 4.3.1 to 4.3.3 until the M samples randomly drawn from the experience pool are trained, and finally the trained MP-DQN proxy model is obtained.

[0109] Step 5: Use the MP-DQN surrogate model trained in Step 4 to allocate resources for the relay-assisted multi-channel FSO network; the specific steps are as follows:

[0110] 5.1 Obtain the current state variable set S k =[h k ,b k ,e k-1 ];

[0111] 5.2, the set of state variables S obtained in step 5.1 k =[h k ,b k ,e k-1 The MP-DQN agent is input after training, and then the MP-DQN model agent outputs parameterized action A. PA ={N,P N}, where N is the relay selection, P N For power allocation, the MP-DQN model proxies the resource allocation of the relay-assisted multichannel FSO network, which is a joint relay selection and power allocation; where relay selection is a discrete action and power allocation is a continuous action;

[0112] 5.3. The obtained relay node sequence number and the transmit power allocated on each channel are applied to the relay-assisted multi-channel FSO network to allocate resources, thereby obtaining the set of state variables S for the next time step. k+1 =[h k+1 ,b k+1 ,e k ];

[0113] 5.4 Repeat steps 5.1 to 5.3 until the resource allocation for the last time slot is completed.

[0114] The above descriptions only cover a few specific embodiments of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention. The above-mentioned serial numbers are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

Claims

1. A dynamic resource allocation method for relay-assisted multi-channel FSO networks based on MP-DQN, characterized in that, include: Collect data, obtain the training dataset, and build the MP-DQN proxy model; The state space, action space, action parameters, and reward of the MP-DQN agent model are set to obtain the set MP-DQN agent model; The MP-DQN proxy model is trained based on the training dataset to obtain the trained MP-DQN proxy model; Use the trained MP-DQN proxy model for relay-assisted multi-channel FSO network resource allocation; The MP-DQN proxy model comprises a Q-value estimation network and a Q-value target network; wherein the Q-value estimation network interacts with the relay-assisted multi-channel FSO network environment and outputs an action value function. The Q-value target network is responsible for calculating the loss gradient, which is used to update the parameters of the Q-value estimation network after the Q-value estimation network parameters have been updated. The action value function is: ; Where E represents the expectation, Indicates the first A state, Indicates the first The action performed in that state Indicates the first Perform an action in each state. The reward obtained, γ discount factor, represents the importance of future rewards; The training dataset is a set of each state of the relay-assisted multichannel FSO network. ; In the The process of updating the remaining battery power in the next time slot is as follows: ； in, Indicates the first The remaining power of each repeater in each time slot; Indicates the first The collection of electrical charges gathered by each repeater in each time slot; Indicates the duration of each time slot; This represents the communication link between the selected repeater and receiver. The transmission power allocated to each optical carrier; Channel state information in a relay-assisted multi-channel FSO network environment is generated using a log-normal distribution model. The remaining power of each relay node is initialized with a fixed value, and the power collected by the relay node in each time slot is randomly generated.

2. The method for dynamic resource allocation in a relay-assisted multi-channel FSO network based on MP-DQN according to claim 1, characterized in that, The state space includes: Channel state information Remaining power of the relay node The electricity collected by the relay node ; The relay-assisted multi-channel FSO network is in the... The state of each time slot is defined as follows: , Indicates the first The set of channel state information for each channel in each time slot. Indicates the first The remaining power of each repeater in each time slot. Indicates the first The collection of electrical charges gathered by each repeater in each time slot.

3. The method for dynamic resource allocation in a relay-assisted multi-channel FSO network based on MP-DQN according to claim 1, characterized in that, The action space includes: The first The action in each time slot is defined as , , This represents the communication link between the transmitter and the selected repeater. The set of transmit powers allocated to each optical carrier. This represents the communication link between the selected repeater and receiver. The set of transmit powers allocated to each optical carrier. For continuous actions, Represents the transmitter. Represents the receiver. Indicates the selected repeater, which is a discrete action; num_relay indicates the number of repeater nodes in the repeater-assisted multichannel FSO network; action parameters For continuous action parameters associated with discrete actions, it represents the allocation of corresponding transmit power to L optical carriers in the communication links between the transmitter and repeater, and between the repeater and receiver, respectively, ultimately forming a parameterized action space. ,in, These represent the continuous action space and the discrete action space, respectively.

4. The method for dynamic resource allocation in a relay-assisted multi-channel FSO network based on MP-DQN according to claim 1, characterized in that: Return Used to evaluate the state Next action Performance characteristics, including total power limits for the transmitter and selected repeaters. and peak power limits for each carrier. ; The return The corresponding reward function is defined as the total channel capacity in each time slot; ； in, , This represents the weights assigned to different optical carriers; ; T f B is the frame duration, and B is the bandwidth. , For transmission power, For the communication link between the transmitter and the selected repeater, the first... Channel state information of each optical carrier, For the selected repeater and receiver communication link, the first... Channel state information of each optical carrier, For the sensitivity of the photodetector, The charge coefficient, This is the noise equivalent bandwidth.

5. The method for dynamic resource allocation in a relay-assisted multi-channel FSO network based on MP-DQN according to claim 1, characterized in that, The step of training the MP-DQN proxy model based on the training dataset to obtain the trained MP-DQN proxy model includes: The initialized MP-DQN proxy model was interacted with the relay-assisted multichannel FSO network environment to obtain training dataset 1; The MP-DQN proxy model is trained based on the obtained training dataset 1, as follows: Randomly drawn from the experience pool One sample ( , , , ),Will , enter The value estimation network obtains the value of the state in the network. Make an action The action value function Q is expressed as: and returns ; set of states enter In the policy network of the target network, obtain Value target network action parameter set ; Set of actions and Value target network action parameter set enter In the value network of the target network, obtain Discrete actions of the target network and the corresponding action value function of the discrete actions. Value: Q( , ; ); Get The method is as follows: ； Where γ is the discount factor for the reward; The method for obtaining the loss function of the eval value network is as follows: ； The method for obtaining the loss function of the eval policy network is as follows: ； parameters loss function right Take the derivative and update the parameters using gradient descent. ,parameter loss function right Differentiate and update parameters using gradient ascent. The updated formula is: ； ； in, These are the learning rates; The target network is updated using soft updates, and the current state is recorded after the MP-DQN proxy model has completed any FSO network resource allocation. Transfer to the next moment Until the training is completed and the randomly selected experience pool is finished. Using samples, we obtain the trained MP-DQN proxy model; Before interacting with the initialized MP-DQN proxy model in a relay-assisted multichannel FSO network environment, the following steps are also included: Initialize the MP-DQN proxy model respectively Value estimation network parameters and Value target network parameters, where, The strategy network parameters and value network parameters of the value estimation network are respectively used as... and express, The policy network parameters and value network parameters of the target network are respectively used as... and This indicates that the initialized MP-DQN proxy model has been obtained.

6. The method for dynamic resource allocation in a relay-assisted multi-channel FSO network based on MP-DQN according to claim 5, characterized in that: The method for updating the target network using soft updates is as follows: ； ； in, and These are the soft update coefficients for the value network and policy network of the Q-value target network, respectively.

7. The method for dynamic resource allocation in a relay-assisted multi-channel FSO network based on MP-DQN according to claim 1, characterized in that, The Q-value estimation network interacts with the relay-assisted multi-channel FSO network environment and outputs the action value function. ,include: The current number State set under each time slot enter The policy network in the value estimation network obtains the action parameters corresponding to each discrete action. ; The current number State set under each time slot and all motion parameters P in the k-th time slot al k l ={P1,P2,…,P N The data is input in batches into the value network of the Q-value estimation network. The value network of the Q-value estimation network outputs the action value function Q-value of each discrete action under the corresponding action parameters. The probability of randomly selecting each action value function The discrete actions and their parameters corresponding to the values ​​are used to obtain the maximum action value function. The discrete actions corresponding to the values; according to The discrete actions and their parameters corresponding to the maximum action value function Q are selected based on probability, and the resulting parameterized action space is obtained. Used for FSO network resource allocation in relay-assisted multi-channel systems. ,in, For the selected repeater, the action is discrete; Allocate power; receive the reward at the current moment. and the set of states at the next moment. ; To explore probability; Based on the above relevant data , , , , thus obtaining training dataset 1 ( , , , The training dataset 1 is stored in the experience pool to obtain the training dataset, which is used for training the neural network.

Images