FSO adaptive routing method based on deep reinforcement learning

Abstract

In order to solve the problems in the prior art, the application provides a FSO adaptive routing method based on deep reinforcement learning, comprising the following steps: building an undirected weighted topology graph according to the connection relationship of FSO network nodes, and constructing an FSO network environment as an environment of deep reinforcement learning according to the atmospheric environment of the current FSO network; a proxy of deep reinforcement learning obtains data samples through interaction with the environment, and continuously saves the data samples into an experience replay pool; when the storage data of the experience replay pool reaches a maximum storage number, the data samples are randomly extracted from the experience replay pool to train an evaluation network in the proxy, and through repeated training, a state is learned to maximize the cumulative reward, so that the FSO adaptive routing is realized; the number of hops, the node energy and the link margin of the network are comprehensively considered, the reliability of network data transmission is improved, and the energy consumption during routing is reduced.

Description

Technical Field

[0001] The present invention relates to the field of free space optical networks, and in particular to an FSO adaptive routing method based on deep reinforcement learning. Background Art

[0002] Free-space optical (FSO) networks are wireless optical networks that use free space as a transmission medium to transmit data signals at high rates. With the advancement of optical and communication equipment, interest in applying FSO technology to wireless access networks is growing. However, FSO links are susceptible to varying attenuation caused by weather conditions (such as rain, fog, and snow), geometric attenuation, and atmospheric turbulence. This limits the reliability of FSO links within a certain transmission range, hindering data transmission in FSO networks and leading to degradation of various network performance characteristics, such as SNR (signal-to-noise ratio), BER (bit error rate), and data delivery rate. Furthermore, if a node dies during data transmission due to failure or power exhaustion, network connectivity can drop dramatically. The dead node can potentially compromise data transmission on other nodes, disrupting data services and resulting in significant data packet loss.

[0003] To solve the above problems, many practical solutions have been proposed in recent years to adapt to changing atmospheric conditions, such as diversity techniques, hybrid RF / FSO systems and multi-beam transmission systems, to maintain reliable connections and support the required Quality of Service (QoS) level for FSO network users, but this cannot completely overcome the impact of adverse atmospheric conditions on FSO channels, nor can it solve the problem of transmission interruption caused by dead nodes. Routing algorithms can calculate the optimal path that meets the corresponding requirements by considering different routing factors, and further provide stable data transmission and QoS for the network by transmitting data to the destination node hop by hop. However, the current FSO network routing is still in the development stage, and the literature (Xie R, Yang W H, Kim Y C. Reconfigurable routing protocol for free space optical sensor networks [J]. Sensors, 2012, 12(4): 4824-4845) proposes a reconfigurable RRP routing protocol that adjusts the network virtual topology in real time by replying to reconfiguration packets to prevent dead nodes from harming data transmission of other nodes, thereby improving network connectivity and packet delivery rate and extending network lifetime. However, this routing protocol does not take into account the impact of adverse atmospheric conditions on FSO networks. The literature (Kiran K V, Kumar V, Turuk A K, et al. Quality Aware Design and Analysis of FSO Link in Wireless Optical System [C] / / 2017 14th IEEE India Council International Conference (INDICON). IEEE, 2017: 1-6) proposes a FSO network routing algorithm based on Link Margin (LM), which is used as one of the quality parameters for FSO link design and analysis. This algorithm calculates and sorts the LM of all source-destination node pairs in the network, thereby screening out reliable paths with LM higher than the threshold, and finally allocates network load according to LM, effectively improving FSO network routing reliability and reducing blocking probability, but this routing algorithm has problems such as single routing factor, high algorithm complexity and poor real-time performance.At the same time, the above routing algorithms are unable to adjust their routing strategies in real time according to changes in the atmospheric environment and node death of the FSO network. The deep reinforcement learning algorithm can process large-scale state and action spaces through the function approximation ability of neural networks and has good generalization. There are also many related studies on wireless network routing (Ding R, Xu Y, Gao F, et al. Deepinforcement learning for router selection in network with heavy traffic[J]. I EEE Access, 2019, 7: 37109-37120). However, traditional deep reinforcement learning algorithms will repeatedly sample invalid actions during training, resulting in a large amount of computational consumption and cannot adapt well to the dynamic changes of the atmospheric environment. Summary of the Invention

[0004] The purpose of the present invention is to provide an FSO adaptive routing method based on deep reinforcement learning to solve the problem of data service interruption caused by changes in the harsh atmospheric environment of the FSO network and node death.

[0005] The technical solution adopted by the present invention to solve the above technical problems is:

[0006] A FSO adaptive routing method based on deep reinforcement learning includes: building an undirected weighted topological graph based on the connection relationship of FSO network nodes and the current atmospheric environment of the FSO network, and constructing the FSO network environment as the deep reinforcement learning environment;

[0007] The deep reinforcement learning agent obtains data samples by interacting with the environment and continuously saves the data samples into the experience replay pool;

[0008] When the experience replay pool stores data that reaches the maximum storage number, data samples are randomly extracted from the experience replay pool to train the evaluation network in the agent. Through repeated training, the cumulative reward is maximized to achieve FSO adaptive routing.

[0009] The “deep reinforcement learning agent obtains data samples by interacting with the environment and continuously saves the data samples in the experience replay pool” includes:

[0010] Initialize the experience replay pool M and set the maximum storage number of the experience replay pool M;

[0011] Randomly initialize the source node, destination node position and initialize the node record table;

[0012] The FSO network environment constructs a mask vector according to the current node position;

[0013] Expand the LM matrix into a one-dimensional vector by row and concatenate it with the current node vector, the destination node vector, and the mask vector to construct the current input state;

[0014] Input the input state into the evaluation network, and evaluate the q vector according to the output of the input state;

[0015] The agent selects an action via a greedy strategy;

[0016] After the action is executed in the FSO network environment, the corresponding reward value is given and the next state is entered; and the environment determines whether the state at this time reaches the destination node and assigns a value to Done;

[0017] The input state, the action, the reward value, the next state and the Done are saved as a data sample in the replay experience pool M.

[0018] The “constructing a mask vector” includes:

[0019] Determine the current node position, record it as a traversed node, and update the node record table;

[0020] Constructing a first one-dimensional 0-1 vector according to the residual energy of all nodes in the FSO network and utilizing the death status of each node;

[0021] Constructing LM matrix based on LM of each link in FSO network environment;

[0022] Obtain the neighboring nodes that can communicate normally through the current node position and LM matrix, and construct the second one-dimensional 0-1 vector;

[0023] An “AND” operation is performed on the node record table, the first one-dimensional 0-1 vector, and the second one-dimensional 0-1 vector to obtain a mask vector at the current position.

[0024] The agent selects an action through a greedy strategy, including:

[0025] Normalize the eva l u_q vector output by the evaluation network, and then multiply the eva l u_q vector by the mask vector element by element to obtain the effective action space in the current state;

[0026] Initialize the probability and choose a compromise between the exploration mechanism and the greedy mechanism based on the probability, including:

[0027] Under the premise of updating the probability at a fixed decay rate, the agent adopts an exploration mechanism to randomly select an action At from the valid action space to execute; or,

[0028] The agent adopts a greedy mechanism to select the action At with the largest evaluation q value from the valid action space to execute.

[0029] The FSO adaptive routing method based on deep reinforcement learning also includes: a process of determining whether the amount of data in the experience replay pool M exceeds a set experience threshold:

[0030] If the data volume is not saturated, it is determined whether the current routing process is completed. If the process is not completed, the current state is updated to the next step state;

[0031] If the data volume is saturated, the new sample will replace the data sample with the earliest storage time.

[0032] Give the action a reward value corresponding to the action through a custom reward function;

[0033] Wherein, the custom reward function is:

[0034]

[0035] Among them, the movement penalty rmp=1, is the normalized link margin.

[0036] The “building an undirected weighted topology graph based on the connection relationship of the FSO network nodes and the current atmospheric environment of the FSO network to construct the FSO network environment” includes:

[0037] Build a network topology, and set the network nodes to die randomly with a probability of 1 / 6;

[0038] Randomly initialize and assign values ​​to link transmission distance and environmental parameters;

[0039] Determine the corresponding weather model based on the weather environment where the link is located;

[0040] Obtaining weather attenuation according to different weather models and link transmission distances d(i, j);

[0041] Calculate the geometric loss α based on the power loss caused by signal propagation geo (i,j), the specific formula is as follows:

[0042]

[0043] α geo (i,j)=10log 10 α tgeo (i,j); where is the beam illumination area, is the receiver capture area, where θ is the beam divergence angle and r is the receiver aperture;

[0044] The turbulence loss is calculated based on the atmospheric turbulence phenomenon. The specific formula is as follows:

[0045] In the formula is the refractive index structural parameter For near-ground horizontal links, The value of is almost constant from weak turbulence to strong turbulence The value of arrive

[0046] Based on weather attenuation, turbulence attenuation, geometric loss, and system loss, the LM is used to characterize the operational performance of the FSO link in different atmospheric environments. The specific formula is as follows:

[0047] LM(i,j)=P t (i)+|R s (j)|-α tur (i,j)-α geo (i,j)-α atmo (i,j)-α sys (i,j); where P t (i) is the transmitter power, R s (j) is the receiver sensitivity, α sys (i,j) is the system loss.

[0048] The weather model includes:

[0049] If the current weather is rainy, the rain attenuation coefficient α is calculated using the rainfall attenuation model. rain (dB / km), the specific formula is as follows:

[0050]

[0051] Where R is the rainfall intensity (mm / hr), and k1 and k2 are the rainfall coefficients. The rainfall coefficients depend on factors such as the wavelength of the FSO system, ambient temperature, and raindrop size. From the Charbonneau model, we can obtain k1 = 1.076 and k2 = 0.67.

[0052] If the current weather is snowy, the snow attenuation coefficient α is calculated using the snowfall attenuation model. snow (dB / km), the specific formula is as follows:

[0053] α snow =aS b

[0054] Where S is the snowfall intensity (mm / hr), a and b are snowfall coefficients;

[0055] If the current weather is fog or sunny, the fog attenuation coefficient a is calculated by the common empirical model of Mie scattering fog (dB / km), the specific formula is as follows:

[0056]

[0057] Where q is the size distribution coefficient of scattering, the specific q value can be obtained from the Kim model, where V is the visibility (km), and the wavelength λ = 1550 nm.

[0058] The "randomly sampling data samples from the experience replay pool to train the evaluation network in the agent, and learning to maximize the cumulative reward by repeated training" includes:

[0059] Randomly sampling m data from the experience replay pool M as training samples;

[0060] Taking the current state of the training sample as the input state of the evaluation network;

[0061] The evaluation network outputs the eval u_q vector according to the input state, and obtains the corresponding evaluation q value according to the action At corresponding to each training sample;

[0062] The target network in the agent obtains the target q vector according to the next step state of the current sample, and obtains the corresponding maximum target q value;

[0063] The agent calculates the loss function loss according to the evaluation q value and the target q value, and continuously adjusts the evaluation network weight to minimize the loss function;

[0064] The agent trains the evaluation network by batch processing m data samples, continuously optimizes the evaluation network weight, and learns the function mapping relationship between the state and the action;

[0065] The target network weight is adjusted in a soft update manner, which reduces the correlation between the evaluation network weight and the target network weight to a certain extent and improves the stability.

[0066] The method for obtaining the target q value includes:

[0067]

[0068] In the formula, γ is the discount factor; R t is the reward; Max[target_q(S t+1 )] is the maximum target q value; target_Q(S t ,A t) is the input state S t The target q value under .

[0069] The beneficial effects of the present invention are:

[0070] The present invention addresses various unpredictable environmental factors such as rain, fog, and snow, builds an FSO network atmospheric environment based on considerations of atmospheric turbulence, weather attenuation, geometric attenuation, and system loss, and comprehensively considers the survival status of network nodes. Training the FSO adaptive routing algorithm in this environment can improve its generalization and adaptability. At the same time, the present invention introduces deep reinforcement learning into the FSO network, fully utilizing the FSO network atmospheric environment information to implement routing selection. During routing selection, the number of network hops, node energy, and link margin are comprehensively considered, thereby improving the reliability of network data transmission and reducing energy consumption during routing. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A flow chart of the method of the present invention;

[0072] Figure 2 Schematic diagram of the FSO network topology structure of the present invention;

[0073] Figure 3 Flowchart for building FSO network environment;

[0074] Figure 4 Flowchart for constructing mask vector;

[0075] Figure 5 Flowchart for selecting actions for a greedy policy. DETAILED DESCRIPTION

[0076] The present application will be further described below with reference to the accompanying drawings.

[0077] To address data service interruptions caused by changes in the harsh atmospheric environment and node death in FSO networks, this paper proposes an FSO adaptive routing method (DQN_mask) based on the Deep Reinforcement Learning (DRL) architecture. The algorithm can adjust its routing strategy in real time based on dynamic changes in the FSO network's atmospheric environment and effectively avoid dead nodes. Furthermore, the present invention comprehensively considers the number of network hops, node energy, and link margin during route selection, effectively reducing routing energy consumption and improving the data delivery rate of FSO networks.

[0078] To facilitate understanding of this solution, let's first briefly introduce DRL. DRL is a reinforcement learning method that integrates deep neural networks and consists of two parts: an agent and an environment. The agent includes an evaluation network, a target network, and an experience replay pool. The environment is constructed based on the actual task requirements. For example, in this solution, the FSO network environment is constructed. The experience replay pool is used to store data samples obtained during the interaction between the agent and the environment, while mixing past and current data to reduce data correlation. In addition, the experience replay pool makes data samples reusable, thereby improving the algorithm's training efficiency. The evaluation network is designed to train an FSO adaptive routing strategy using data samples. The target network is used to stabilize the training process of the evaluation network.

[0079] The evaluation network and the target network use a BP neural network with the same structure, which is divided into three layers: input layer, hidden layer and output layer. The input layer is used to input the current state S t The hidden layer is used to abstract different input states and learn the functional mapping relationship between states and actions by continuously updating the neural network weights. Finally, according to different input states, the output layer can output the predicted value of executing each action and form a one-dimensional vector.

[0080] Specific embodiment 1:

[0081] An embodiment provided by the present invention:

[0082] like Figure 1 , an FSO adaptive routing method based on deep reinforcement learning, comprising the following steps:

[0083] Step 1: Build a 14-node undirected weighted topology graph based on the connection relationship of the FSO network nodes, as shown in Figure 2 As shown in the figure, the undirected weighted topology graph includes all nodes in the FSO network and the bidirectional links between them. Each link and each node is assigned a corresponding weight factor. Node weights are composed of the node's residual energy, while link weights are calculated using a theoretical formula using the Link Margin (LM) model. These factors account for various power losses, including weather attenuation, turbulence attenuation, geometric loss, and system loss. Once the link weights for each link are determined, the atmospheric environment of the current FSO network can be determined. Ultimately, the FSO network environment is constructed based on the network's topology and atmospheric environment.

[0084] In step 1, the process of building the FSO network environment is as follows: Figure 3 As shown, the following steps are included:

[0085] Step 1.1 First build Figure 2 The FSO network topology structure is set to have a random death probability of 1 / 6.

[0086] Step 1.2 randomly initializes the link distance and environmental parameters within a reasonable range.

[0087] Step 1.3 determines the weather environment of the link and determines the corresponding weather model.

[0088] Step 1.4 If the current weather is rainy, calculate the rain attenuation coefficient α using the rainfall attenuation model rain (dB / km), the specific formula is as follows:

[0089]

[0090] Where R is the rainfall intensity (mm / hr), and k1 and k2 are the rainfall coefficients. The rainfall coefficients depend on factors such as the wavelength of the FSO system, ambient temperature, and raindrop size. Using the Charbonneau model, we obtain k1 = 1.076 and k2 = 0.67.

[0091] Step 1.5 If the current weather is snowy, calculate the snow attenuation coefficient α using the snowfall attenuation model snow (dB / km), the specific formula is as follows:

[0092] α snow =aS b

[0093] Where S is the snowfall intensity (mm / hr), a and b are the snowfall coefficients. For snow, snow attenuation can be divided into dry snow and wet snow attenuation. When in a dry snow environment, the snowfall coefficient a=5.42×10 -5 +5.49, b=1.38, when in wet snow environment, snowfall coefficient a=1.02×10 -4 +3.78, b=0.72.

[0094] Step 1.6 If the current weather is foggy or sunny, calculate the fog attenuation coefficient α using the commonly used empirical model of Mie scattering fog (dB / km), the specific formula is as follows:

[0095]

[0096] Where q is the size distribution coefficient of scattering, and the specific q value can be obtained from the Kim model, where V is the visibility (km), and the wavelength λ = 1550nm.

[0097] Step 1.7 Calculate weather attenuation based on different weather models and transmission distance d(i,j)

[0098] Step 1.8 Calculate the geometric loss α based on the power loss caused by signal propagation geo(i,j), which is specifically formulated as follows:

[0099]

[0100] α geo (i,j) = 10 log 10 α tgeo (i,j)

[0101] wherein is the light beam illumination area, is the receiver capture area, wherein θ is the light beam divergence angle and r is the receiver aperture.

[0102] Step 1.9 calculates the turbulence loss according to the atmospheric turbulence phenomenon, which is specifically formulated as follows:

[0103]

[0104] wherein is the refractive index structure parameter For the near-ground horizontal link, the value of is almost constant, from weak turbulence to strong turbulence the value of is to

[0105] Step 1.10 comprehensively considers weather attenuation, turbulence attenuation, geometric loss and system loss, and represents the operating performance of the FSO link in different atmospheric environments through LM, which is specifically formulated as follows:

[0106] LM(i,j) = P t (i) + |R s (j) | - α tur (i,j) - α geo (i,j) - α atmo (i,j) - α sys (i,j)

[0107] wherein P t (i) is the transmitter power, R s (j) is the receiver sensitivity, and α sys (i,j) is the system loss. When LM is less than 0, the link cannot maintain reliable data transmission. On the contrary, as LM increases, the communication quality of the link will be better and better.

[0108] Step 1.11 judges whether the LM of all links has been calculated. If not, it goes to the next link and returns to step 1.2. Otherwise, it means that the atmospheric environment construction of the FSO network is completed.

[0109] This embodiment can also autonomously adjust its routing strategy according to the dynamic changes in the atmospheric environment of the FSO network, effectively avoiding dead nodes and selecting a path with the best communication quality.

[0110] Step 2: The agent obtains the current input state S by interacting with the FSO network environment. t , change the state S t Input the evaluation network, the evaluation network will output the evaluation q vector (evalu_q). Then use the ε greedy strategy to select action A t , that is, select the next hop node. Execute action A in the FSO network environment t After that, the environment will enter the next state S t+1 , and judge whether it reaches the destination node, and assign a value to Done. At the same time, the environment will use a custom reward function according to action A t Give reward R t Finally, {S t , A t , R t , S t+1 , Done} is saved as a data sample in the experience replay pool M until the experience replay pool reaches the maximum storage number.

[0111] In step 2, the detailed process of storing data samples in the experience replay pool is as follows:

[0112] Step 2.1 initializes the experience replay pool M and sets the experience threshold to 1,000,000, which is the maximum storage number of the experience replay pool.

[0113] Step 2.2 randomly initializes the source and destination node positions, constructing the current node vector n1 = [0, 1, …, 0, 0] and the destination node vector n2 = [0, 0, …, 0, 1], where 1 represents the position of the current and destination nodes, respectively. Also, initialize the node record table to a one-dimensional vector of length 14, all containing 1s. The node record table is used to record the nodes traversed during routing, preventing routing from falling into an infinite loop.

[0114] Step 2.3 The FSO network environment constructs a mask vector based on the current node position. The mask vector is used to shield invalid nodes; Figure 4 As shown, the process of constructing the mask vector is as follows:

[0115] 2.3.1 Determine the current node position and record it as a traversed node. Update the node record table to n3 = [1, 0, ..., 1, 1], where 0 is the traversed node.

[0116] 2.3.2 Based on the residual energy of all nodes in the network, determine the death status of each node and construct a one-dimensional 0-1 vector n4 = [0, 1, ..., 1, 0], where 0 represents a dead node and 1 represents a live node.

[0117] 2.3.3 A 14×14 LM matrix is ​​constructed based on the LM of each link in the FSO network environment, where the LM between unconnected nodes is set to -∞. Neighboring nodes with which normal communication is possible are determined using the current node position and the LM matrix, and the corresponding one-dimensional 0-1 vector n5 = [0, 1, …, 1, 0] is constructed. 0 represents a link with LM ≤ 0, and 1 represents a link with LM > 0.

[0118] 2.3.4 Perform an AND operation on the n3, n4, and n5 vectors to construct the mask vector m = [0, 1, ..., 1, 0] at the current position, where 0 represents an invalid node and 1 represents a valid node.

[0119] Step 2.4 Expand the LM matrix into a one-dimensional vector n6 by row, and concatenate it with the current node vector n1, the destination node vector n2 and the mask vector m to construct the current input state S t =[n6, n1, n2, m].

[0120] This embodiment combines the deep reinforcement learning (DQN) algorithm and sets up a mask mechanism in the discrete action space. The mask limits the action space of the DRL agent to valid actions at the current position, avoiding repeated sampling of invalid actions, and further improving the training efficiency and convergence speed of the algorithm.

[0121] Step 2.5: Set state S t Input the evaluation network, the evaluation network will be based on S t Output evaluation q vector (evalu_q). evalu_q is a one-dimensional vector of length 14, which represents the evaluation q value of selecting each node as the next hop node in the FSO network environment. Finally, the agent selects an action A through the greedy strategy. t ;like Figure 5 As shown, the greedy strategy selects action A t The detailed process is as follows:

[0122] 2.5.1 The evalu_q vector output by the evaluation network is normalized by the Softmax function, and then the evalu_q vector is element-wise multiplied with the mask vector according to the mask mechanism to obtain the effective action space in the current state.

[0123] 2.5.2 Initialize the probability ε value to 1, and make a compromise between the exploration mechanism and the greedy mechanism based on the probability ε.

[0124] 2.5.3 The agent has a probability of ε and uses an exploration mechanism to randomly select action A from the valid action space. t Otherwise, the agent will adopt a greedy mechanism and select the action A with the largest evaluation q value from the valid action space. t implement.

[0125] 2.5.4 With a fixed decay rate α = 5 × 10 -5 The probability ε is updated, that is, ε = ε - α, until ε = 0.05.

[0126] Step 2.6 Customize the reward function reward, based on the selected action A t Give reward value R t ;

[0127]

[0128] Among them, the movement penalty rmp=1, is the normalized link margin. min is the minimum value of all link margins, LM max is the maximum value of all link margins. The higher the link margin of the link between nodes and the node is valid, the higher the R t The higher the value.

[0129] Step 2.7 FSO network environment performs action A t After that, give the corresponding reward R t , and enter the next state S t+1 At the same time, the environment will determine whether the state has reached the destination node and assign a value to Done; if it has reached the destination node, Done = 1, otherwise Done = 0.

[0130] Step 2.8: The five-tuple {S t , A t , R t , S t+1 , Done} is saved as a data sample into the playback experience pool M;

[0131] Step 2.9 determines whether the data in the experience replay pool is saturated, that is, whether the data sample size in the experience pool exceeds the set experience threshold; if not, determine whether the current routing process is completed. If the process is not completed, the current state S t Update to next state S t+1 , and return to step 3.3; otherwise, jump out of the current routing process and return to step 3.2; if the data is saturated, the new sample will replace the oldest stored data sample and enter step 3;

[0132] Step 3: When the experience replay pool reaches the maximum storage number, randomly extract data samples from the experience replay pool to train the evaluation network. Through repeated training, the FSO adaptive routing algorithm that maximizes the cumulative reward is learned. The structural block diagram of the algorithm is as follows: Figure 1 As shown in Figure 2, the specific training process of the FSO adaptive routing algorithm is as follows:

[0133] 3.1 Randomly extract m data from the experience replay pool M as training samples.

[0134] 3.2 The S in each sample t As the input value of the evaluation network, the evaluation network is based on S t Output the evalu_q vector, and then obtain the corresponding evaluation q value evalu_Q(S according to the action At of each sample t ,A t )

[0135] 3.3 The target network is based on the next state S t+1 Output target q vector target_q(S t+1 ), and obtain the corresponding maximum target q value, that is, Max[target_q(S t+1 )]. The state S is calculated by the following formula t The target q value under:

[0136]

[0137] Where γ = 0.9 is the discount factor used to adjust the next state S t+1 The degree of influence on the action selected in the current state. Target q value target_Q(S t ,A t ) combined with the reward R t and S t+1 The maximum target q value under the current action reward, target_Q(S t ,A t ) also considers the impact of future states on current actions.

[0138] 3.4 The agent evaluates the q value evalu_Q(S t ,A t ) and the target q value target_Q(S t ,A t ) calculates the loss function loss, and uses the gradient descent method to continuously debug and evaluate the network weight evalu_ω to minimize the loss function. The ultimate goal is to make evalu_Q(S t ,A t ) is close to target_Q(S t ,At ). The loss function is as follows:

[0139]

[0140] 3.5 The agent trains the evaluation network by batch processing m data samples, continuously optimizing the evaluation network weights evalu_ω and learning the functional mapping relationship between state and action. At the same time, it uses soft updates to adjust the target network weights target_ω, to a certain extent reducing the correlation between the evaluation network weights and the target network weights and improving the stability of the algorithm. The update coefficient τ = 0.005, where the soft update formula is:

[0141] target_ω=τ×evalu_ω+(1-τ)target_ω.

[0142] 3.6 Steps 3.1 to 3.5 above represent the completion of one round of training. Through continuous training, the evaluation network will eventually learn the FSO adaptive routing algorithm that maximizes the cumulative reward. The agent only needs to use a greedy mechanism to select the largest evaluation q value in the effective action space to find the optimal action under different states and implement FSO adaptive routing.

[0143] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. The FSO adaptive routing method based on deep reinforcement learning is characterized by: include: Based on the connection relationship of FSO network nodes, an undirected weighted topological graph is constructed, as well as the current atmospheric environment of the FSO network, to build the FSO network environment as the environment for deep reinforcement learning; The deep reinforcement learning agent obtains data samples by interacting with the environment and continuously saves the data samples into the experience replay pool; When the experience replay pool reaches the maximum storage number, data samples are randomly extracted from the experience replay pool to train the evaluation network in the agent, and the cumulative reward is maximized through repeated training to achieve FSO adaptive routing; The deep reinforcement learning agent obtains data samples by interacting with the environment and continuously saves the data samples into the experience replay pool, including: Initialize the experience replay pool M and set the maximum storage number of the experience replay pool M; Randomly initialize the source node, destination node position and initialize the node record table; The FSO network environment constructs a mask vector according to the current node position; Expand the LM matrix into a one-dimensional vector by row and concatenate it with the current node vector, the destination node vector, and the mask vector to construct the current input state; Input the input state into the evaluation network, and evaluate the q vector according to the output of the input state; The agent selects an action via a greedy strategy; After the action is executed in the FSO network environment, a corresponding reward value is given and the next state is entered; and the environment determines whether the state at this time reaches the destination node and assigns a value to Done; The input state, the action, the reward value, the next state and the Done are saved as a data sample in the replay experience pool M.

2. The FSO adaptive routing method based on deep reinforcement learning according to claim 1, characterized in that The constructing of the mask vector comprises: Determine the current node position, record it as a traversed node, and update the node record table; Constructing a first one-dimensional 0-1 vector according to the residual energy of all nodes in the FSO network and utilizing the death status of each node; Constructing LM matrix based on LM of each link in FSO network environment; Obtain the neighboring nodes that can communicate normally through the current node position and LM matrix, and construct the second one-dimensional 0-1 vector; An AND operation is performed on the node record table, the first one-dimensional 0-1 vector, and the second one-dimensional 0-1 vector to obtain a mask vector at the current position.

3. The FSO adaptive routing method based on deep reinforcement learning according to claim 1, characterized in that The agent selects an action via a greedy strategy, consisting of: Normalize the evalu_q vector output by the evaluation network, and then multiply the evalu_q vector by the mask vector element by element to obtain the effective action space in the current state; Initialize the probability and choose a compromise between the exploration mechanism and the greedy mechanism based on the probability, including: Under the premise of updating the probability at a fixed decay rate, the agent adopts an exploration mechanism to randomly select an action At from the valid action space to execute; or, The agent adopts a greedy mechanism to select the action At with the largest evaluation q value from the valid action space to execute.

4. The FSO adaptive routing method based on deep reinforcement learning according to claim 1, characterized in that Also includes: The process of judging whether the amount of data in the experience replay pool M exceeds the set experience threshold: If the data volume is not saturated, it is determined whether the current routing process is completed. If the process is not completed, the current state is updated to the next step state; If the data volume is saturated, the new sample will replace the data sample with the earliest storage time.

5. The FSO adaptive routing method based on deep reinforcement learning according to claim 1, characterized in that: Give the action a reward value corresponding to the action through a custom reward function; Wherein, the custom reward function is: Among them, the movement penalty rmp=1, is the normalized link margin; LM(i,j) represents the link margin value between nodes; lm max and lm min They represent the maximum link margin value and the minimum link margin value of the current FSO network respectively.

6. The FSO adaptive routing method based on deep reinforcement learning according to claim 1, characterized in that: The method of constructing an undirected weighted topological graph based on the connection relationship of the FSO network nodes and the atmospheric environment of the current FSO network to construct the FSO network environment includes: Build a network topology, and set the network nodes to die randomly with a probability of 1 / 6; Randomly initialize and assign values ​​to link transmission distance and environmental parameters; Determine the corresponding weather model based on the weather environment where the link is located; Obtaining weather attenuation according to different weather models and link transmission distances d(i, j); Among them, α rain , α snow and α fog They represent the rain attenuation coefficient, snow attenuation coefficient, and fog attenuation coefficient respectively; d(i,j) represents the transmission distance between nodes; Calculate the geometric loss α based on the power loss caused by signal propagation geo (i,j), the specific formula is as follows: α geo (i,j)=10log 10 α tgeo (i,j); where is the beam illumination area, is the receiver capture area, where θ is the beam divergence angle and r is the receiver aperture; The turbulence loss is calculated based on the atmospheric turbulence phenomenon. The specific formula is as follows: In the formula is the refractive index structure parameter; for the near-ground horizontal link, The value of is almost constant from weak turbulence to strong turbulence The value of arrive Based on weather attenuation, turbulence attenuation, geometric loss, and system loss, the LM is used to characterize the operational performance of the FSO link in different atmospheric environments. The specific formula is as follows: LM(i,j)=P t (i)+|R s (j)|-α tur (i,j)-α geo (i,j)-α atmo (i,j)-α sys (i,j); where P t (i) is the transmitter power, R s (j) is the receiver sensitivity, α sys (i,j) is the system loss.

7. The FSO adaptive routing method based on deep reinforcement learning according to claim 6, characterized in that: The weather model includes: If the current weather is rainy, the rain attenuation coefficient α is calculated using the rainfall attenuation model. rain (dB / km), the specific formula is as follows: Where R is the rainfall intensity (mm / hr), k1 and k2 are the rainfall coefficients. The rainfall coefficients depend on factors such as the wavelength of the FSO system, ambient temperature, and raindrop size. From the Charbonneau model, we can obtain k1 = 1.076 and k2 = 0.

67. If the current weather is snowy, the snow attenuation coefficient α is calculated using the snowfall attenuation model. snow (dB / km), the specific formula is as follows: a snow =aS b Where S is the snowfall intensity (mm / hr), a and b are snowfall coefficients; If the current weather is foggy or sunny, the fog attenuation coefficient α is calculated using the commonly used empirical model of Mie scattering. fog (dB / km), the specific formula is as follows: Where q is the size distribution coefficient of scattering, and the specific q value can be obtained from the Kim model, where V is the visibility (km), and the wavelength λ = 1550nm.

8. The FSO adaptive routing method based on deep reinforcement learning according to claim 1, characterized in that: The "randomly extracting data samples from the experience replay pool to train the evaluation network in the agent, and learning to maximize the cumulative reward through repeated training" includes: Randomly extract m data from the experience replay pool M as training samples; Using the current state of the training sample as the input state of the evaluation network; The evaluation network outputs the evalu_q vector according to the input state, and then obtains the corresponding evaluation q value according to the action At corresponding to each training sample; The target network in the agent obtains the target q vector according to the next state of the current sample and obtains the corresponding maximum target q value; The agent calculates the loss function loss based on the evaluation q value and the target q value, and continuously adjusts the evaluation network weights to minimize the loss function; The agent trains the evaluation network by batch processing m data samples, continuously optimizing the evaluation network weights and the functional mapping relationship between learning states and actions; The target network weight is adjusted by soft update, which reduces the correlation between the evaluation network weight and the target network weight to a certain extent and improves stability.

9. The FSO adaptive routing method based on deep reinforcement learning according to claim 8, characterized in that: The method for obtaining the target q value includes: Where γ is the discount factor; R t is the reward; Max[target_q(S t+1 )] is the maximum target q value; target_Q(S t ,A t ) is the target q value under the input state St; if the destination node has been reached, Done = 1, if not, Done = 0.

Images