A cross-layer decision method for joint channel and routing selection based on Q-learning

Abstract

The present application provides a cross-layer decision method for joint channel and routing selection based on Q learning, the method comprising: obtaining coordinates; each routing node periodically broadcasts a message used to discover, establish, and keep alive neighbor relationships; when the neighbor node receives a Hello packet, it feeds back a confirmation message to establish a good neighbor relationship; when the source node has data to transmit, the source node selects the next hop node and channel transmission data packet according to the selection strategy, and the data packet contains the address of the selected next hop node; the neighbor point determines whether the data packet should be forwarded by itself; the next hop node replies to the source node through the control link, and the reply information contains an estimated signal-to-interference ratio and its latest Q table; the source node uses the feedback signal-to-interference ratio and the position obtained in the early stage to calculate the reward value; the source node updates the Q table and node selection strategy according to the reward value. This method improves the network transmission rate and anti-interference performance, and reduces the end-to-end delay and hop count.

Description

Technical Field

[0001] The present application relates to the technical field of wireless communication technology, and in particular to a cross-layer decision method for joint channel and routing selection based on Q learning. Background Art

[0002] Due to the wireless transmission characteristics of ad hoc networks, the network bandwidth it can provide is much lower than that of wired channels, and it is easily interfered by jammers. However, the current anti-interference methods of ad hoc networks are mainly concentrated in a single physical layer or network layer, resulting in limited anti-interference effects. Most physical layer methods adjust channels or transmission power based on learned interference patterns. However, in ad hoc networks, when routing nodes or transmission links are interfered with by external factors, routing failures will occur.

[0003] In this case, simply adjusting the signal or power is not feasible, and the path needs to be replanned. Network layer anti-interference research mainly focuses on multipath transmission, network coding and rerouting, but these methods are difficult to deal with dynamic interference and usually require a lot of additional overhead. The current small amount of cross-layer anti-interference research mainly focuses on using known topology information to find a subset of routes from the source node to the destination node, and then dynamically adjusts the strategy according to the current environment. However, this method requires a lot of prior information and requires high maintenance overhead.

[0004] Therefore, determining a strategy that requires only less information exchange to achieve effective anti-interference and good network performance has become the current research focus. Summary of the invention

[0005] The present application provides a cross-layer decision method for joint channel and routing selection based on Q learning, which can be used to solve the technical problem that cross-layer anti-interference research requires more prior information and has high maintenance costs.

[0006] The present application provides a Q-learning-based joint channel and routing cross-layer decision method, the method comprising:

[0007] Step 1, obtaining coordinates: In the early stage, the routing nodes in the network obtain their own real-time coordinate information through the equipped positioning system;

[0008] Step 2: Each routing node periodically broadcasts a message used to discover, establish, and maintain neighbor relationships, namely, a Hello packet. When the neighbor node receives the Hello packet, it feeds back a confirmation message containing its own address and coordinate information, namely, ACK. The routing node that receives the ACK records the neighbor information in the neighbor table and establishes a good neighbor relationship.

[0009] Step 3: When the source node has data to transmit, the source node selects the next hop node and channel according to the selection strategy; and adds the address of the selected next hop node and the address of the destination node to the data packet and transmits it outward.

[0010] Step 4: When a neighbor node receives a data packet, it compares the address of the next-hop node in the data packet with its own address to see if they are the same; if they are different, it considers that the data packet is not for it to forward, and discards it without processing or forwarding it; when the address of the next-hop node in the message is the same as its own address, it will continue to process the data packet and analyze the received signal to estimate the signal-to-interference ratio.

[0011] In multi-hop routing communication systems, the channel model is affected by large-scale fading. Jump to the node The channel gain of the hop node is denoted as , expressed as:

[0012] ;

[0013] in Indicates Jump to the node The distance of the hop node, represents the path fading factor; Channel gain between hop node and jammer Expressed as ;

[0014] Assume that the transmission power of routing nodes is the same as , the transmission power of the jammer is , then Jump Node The received signal-to-interference ratio for:

[0015] ;

[0016] in Indicates the communication channel The channel noise, The indicator function is expressed as follows:

[0017] ;

[0018] in Indicates Jump node and The communication channel between hop nodes, Indicates the channel where the interference is located; when When , it means that the channel accessed by the routing node is the same as the interference channel, and the data link is interfered.

[0019] Step 5: The next hop node replies to the source node through the control link. The reply information includes the estimated signal-to-interference ratio and its latest Q table.

[0020] Step 6: The source node calculates the reward value based on the feedback signal-to-interference ratio and the position obtained previously.

[0021] According to the received signal-to-interference ratio , No. Jump node and User communication rate between hop nodes for:

[0022] ;

[0023] in represents the channel bandwidth of the communication channel, Indicates the signal-to-interference ratio threshold for successful demodulation by the receiver; When it is lower than the demodulation threshold, The hop node cannot demodulate the signal correctly and thinks it is interfered, so the communication rate is 0;

[0024] Then calculate the reward value based on the transmission rate and location information:

[0025] ;

[0026] in, , These are two negative numbers with relatively large absolute values. Their function is to avoid interference as much as possible and choose the next hop action that is closer to the destination node. Respectively represent Jump node, The distance between the hop node and the source node to the destination node; are two weight coefficients that can adjust the transmission rate and the influence of the next hop node location to meet ;

[0027] No. Jump Node The next hop node and channel selection are only related to the Jump Node The next hop node and channel selection are related to each other, and have nothing to do with the node and channel selection of the previous link. The next-hop node and channel selection problem is regarded as a discrete-time Markov decision process MDP;

[0028] The MDP model is defined by a four-tuple , represents the state space, represents the action space, represents the state transition probability, Represents the reward value; the core elements in the MDP model are defined as follows:

[0029] Defining the state space :

[0030] ;

[0031] in Indicates The state of the jump node, , Indicates the current node. Indicates the channel where the current interference is located. Indicates the number of available channels. Representative The number of nodes that can be selected by the jump;

[0032] Defining the action space :

[0033] ;

[0034] in Indicates The action of jumping nodes, , Indicates The node selected by the jump Jump node, Indicates the connection Jump and The communication channel of the hop node, Representative The number of nodes that can be selected by the jump;

[0035] Define the state transition probability matrix : , , indicating that from the state Select Action Arrival Status The probability of

[0036] Defining Reward Values : Indicates in status Next select action The instant reward value obtained; considering that obtaining the hop count information requires knowing the topology information of the entire network in advance, it will cause a large overhead. The hop count is linked to the geographical location, and the routing node closer to the destination node is selected as the next hop; that is, the closer to the destination node, the fewer times it needs to be forwarded, and the farther away, the more times it needs to be forwarded; therefore, the reward value for each hop is defined as :

[0037] ;

[0038] in, , These are two negative numbers with relatively large absolute values. The purpose is to avoid interference as much as possible and choose an action that makes the next hop closer and closer to the destination node. is the weight coefficient, which can adjust the influence of transmission rate and next-hop node location to meet . Respectively represent Jump node, The distance between the jump node and the source node to the destination node; when When the current node selects a next hop farther from the destination node, the node selection is wrong and a negative reward is given. ;when , it means that the current node has selected a next hop closer to the destination node, and the node selection is correct; and in order to achieve the largest possible transmission rate, the above reward is given.

[0039] Step 7: The source node updates the Q table and node selection strategy according to the reward value.

[0040] For any Jump node, the update formula of Q table is:

[0041] ;

[0042] in is the learning rate, is the discount factor, For the current state Select Action The immediate return value, The state selects and executes actions at the routing node Later becomes state; Indicates the current state Next select action The Q value, represents the updated Q value;

[0043] Define routing nodes and channel selection strategies , The update formula is:

[0044] ;

[0045] ;

[0046] ;

[0047] in Indicates in status Next Select The maximum Q value corresponding to the node, Indicates that in the next state The maximum Q value that can be obtained under are the relevant parameters of the Boltzmann model:

[0048] ;

[0049] in, Indicates the beginning of the exploration phase. Indicates the end of the exploration phase. is the first parameter of the Boltzmann model, It is the second parameter of the Boltzmann model. It comprehensively considers the potential value of two consecutive hops and provides a more accurate and forward-looking routing selection strategy.

[0050] Step 8: The next hop node repeats the process of the source node from step 3 to step 7 until the data packet is sent to the destination node.

[0051] When the routing node transmits a data packet to the next node, the timing device is started. If the confirmation message from the next hop node is not received within the specified time, it is considered to be interfered, and a negative reward is obtained. The decision is made again and the data packet is sent.

[0052] Assume that a transmission needs to go through Jump, definition The communication rate with the minimum communication rate in the hop communication link is the communication rate of the data transmission:

[0053] ;

[0054] in, Indicates the transmission rate of data packets sent from the first hop node to the second hop node. Indicates Jump node to Transmission rate of the jumping node sending data packets

[0055] Considering that in the routing protocol, the hop count is an important measure of network performance, which directly affects network latency, congestion, and the integrity of data packets. The goal of each transmission of the routing node is to maximize the communication rate under the condition of minimizing the hop count of the routing path by optimizing the next-hop node and channel selection strategy of a single routing node. The utility function of a single transmission of the routing node is defined as:

[0056] ;

[0057] The specific algorithm is as follows:

[0058] Step 81: Initialize, set the simulation time T > 0, and the initial communication time t = 0;

[0059] Step 82: Set the learning parameters, discount factor, and relevant parameters of the Boltzmann model;

[0060] Step 83: Initialize the Q-table of each routing node = 0;

[0061] Step 84: Initialize the current routing node as the source node, sense the current interference channel as , and the initial state ;

[0062] Start the loop. When t < T, if the current node is not equal to the destination node ;

[0063] Step 85: Select the next-hop node and the communication channel according to the Boltzmann selection strategy of the current node, that is, the current node transmits data to the next-hop node through the channel ;

[0064] Step 86: After receiving the data packet, the next-hop node performs processing and analysis, estimates , and replies to the current node;

[0065] Step 87: The current node uses the reply information and combines it with the coordinate information to calculate the reward value, and then updates the selection strategy probability;

[0066] Step 88: The next-hop node senses the interference channel as , and updates the state ;

[0067] When the requirements are met, the transmission ends and the cycle ends.

[0068] The present invention solves the problem of establishing routing in an interference environment without central control and without relying on the topology information of the entire network, relying only on link status and geographic location information. The routing node dynamically senses the spectrum environment to identify spectrum holes, and combines the distance from its neighbors to the destination to find a suitable next hop and communication channel. Finally, the routing node combines these two aspects, makes a joint action, and uses Q learning to optimize the selection of the next hop node and channel. This method improves the network transmission rate and anti-interference performance, and reduces the end-to-end delay and number of hops. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A diagram showing the implementation steps of the routing optimization algorithm based on Q learning provided by the present invention;

[0070] Figure 2 A topological diagram provided by the present invention;

[0071] Figure 3 A channel selection probability graph of routing node 2 provided in an embodiment of the present application;

[0072] Figure 4 A next hop node selection probability graph of routing node 5 provided in an embodiment of the present application;

[0073] Figure 5 A channel selection probability graph of the routing node 5 provided in an embodiment of the present application;

[0074] Figure 6 A next hop node selection probability graph of routing node 7 provided in an embodiment of the present application;

[0075] Figure 7 A channel selection probability graph of routing node 7 provided in an embodiment of the present application;

[0076] Figure 8 A time-frequency diagram of a routing node and a jammer before convergence provided by the present invention;

[0077] Fig. 9 A time-frequency diagram of a routing node and a jammer before convergence provided by the present invention;

[0078] Fig.10 A comparison chart of utility values ​​provided by the present invention;

[0079] Fig.11 The delay variation curve diagram provided by the present invention;

[0080] Fig.12 The present invention provides a curve showing the effect of interference duration on utility value. DETAILED DESCRIPTION

[0081] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0082] The following first introduces the embodiments of the present application in conjunction with the accompanying drawings.

[0083] The present invention discloses an efficient and reliable anti-interference routing method, which models the distributed joint channel and node decision problem in a multi-channel scenario as a Markov decision process, and proposes a cross-layer routing anti-interference design for establishing routing based on geographic location and channel quality. The routing nodes in the network evaluate the geographic location and link quality of their adjacent nodes to select the appropriate next-hop node and communication channel, thereby reducing the cost of routing hops and maximizing the transmission rate. Considering that the update of the Q table only requires the exchange of reward values, the Q table is used instead of the routing table, reducing the amount of information interacted between neighbors. This method has low routing overhead and good anti-interference performance.

[0084] Figure 1 This is the implementation step of the routing optimization algorithm based on Q learning proposed by the present invention. First, each routing node obtains its own real-time position coordinate information through the equipped positioning. Then each routing node periodically broadcasts a Hello packet used to discover, establish, and keep the neighbor relationship alive. When the neighbor node receives the Hello packet, it sends a confirmation message to the routing node, which contains its own address and coordinates. After receiving the confirmation message, the routing node records the neighbor information in the neighbor table and establishes a good neighbor relationship. Secondly, when there is data to be transmitted, the source node selects the next hop node and channel according to the selection strategy. And the address of the selected next hop node and the address of the destination node are added to the data packet and broadcasted outward. When the neighbor node receives the data packet, it compares the address of the next hop node in the data packet with its own address to see if it is the same. If they are different, it is considered that the data packet is not a data packet to be forwarded by itself, and it is discarded without processing and forwarding. When the address of the next hop node in the message is the same as its own address, it will continue to process the data packet and analyze the received signal to estimate the signal-to-interference ratio. And reply to the source node through the control link, at this time, the reply information contains the estimated signal-to-interference ratio and its latest Q table. The signal-to-interference ratio is used to calculate the reward value, and the Q table is used to update the node selection strategy. Finally, the source node calculates the reward value and updates the Q table and selection strategy based on the reply information of the neighboring nodes.

[0085] When the source node transmits a data packet to the next node, a timing device is started. If no confirmation message is received from the next-hop node within the specified time, the selected communication channel is considered to be the same as the channel where the interference occurs, that is, it is interfered with. A negative reward is obtained and the Q table and selection strategy are updated.

[0086] Figure 2 This is a simulation topology diagram of the present invention. There are 15 routing nodes and 1 jammer in the system. The jammer's jamming mode is sweep frequency jamming. The user performs Q learning according to the transmission time slot, selects and adjusts the next hop node and communication channel.

[0087] Figure 3-7 Gives interference In the state, the action selection probability curve of each routing node when the data packet is sent from source node 2. The simulation results show that in the initial stage of reinforcement learning, the Q value of each action of the user in this state is 0, and the probability of selecting each action is equal. After a period of learning, the routing node's cognition of the interference environment and topology is continuously strengthened, and the interference change law and routing information are gradually mastered, so the routing node gradually tends to choose the optimal decision strategy. It can be found from the figure that in the late learning algorithm convergence stage, the routing node transmission path is stipulated as 2-5-7-15. In the state of interference 3, the routing node will choose the action of communication channel 1 with a probability of 1.

[0088] Figure 8-9 The time-frequency diagram of routing nodes and interference is given. The light-colored squares represent routing node channels, and the dark-colored squares represent interference channels. The overlapping of light and dark colors makes the darker squares represent routing nodes that are interfered. In the early stage of Q learning, Figure 8 As shown in , because the channel tends to be randomly selected, the probability of collision between routing nodes and interference is relatively large. In the later stage of Q learning, as Fig. 9 As shown in the figure, the routing nodes almost all avoid the interference of the jammer and achieve the purpose of anti-interference.

[0089] To demonstrate the effectiveness of the proposed algorithm, the proposed method is compared with two GPS-based methods: one is to select the routing node closest to the destination node and randomly select an uninterrupted channel based on perception; the other is to use reinforcement learning to optimize channel selection by identifying interference patterns, while fixedly selecting the node closest to the destination as the next hop node. The system performance of these three methods is analyzed. Fig.10 A comparison chart of the utility values ​​of the three methods is given. At the beginning of learning, the GPS-based perception channel selection algorithm selects nodes according to distance and does not need to be learned, that is, according to the fixed path 2-5-8-13-14-15, and perception can allow channel selection to avoid interference to a certain extent. Therefore, at the beginning, the GPS-based perception channel selection algorithm has the largest utility value. However, it can be seen that as the number of learning times increases, the method used will gradually become the optimal one.

[0090] Fig.11The delay variation curve is given, and it can be seen that the delay of the method used has also been significantly improved with the increase in the number of learning times. The delay is the time difference between the time when the source node sends the data and the time when the destination node receives the data. The components of each hop delay are: perception delay, transmission delay, data processing delay, and confirmation message return delay. The expression of transmission delay is:

[0091] ;

[0092] in Indicates the length of the data packet to be transmitted. Indicates the connection Jump node and The transmission rate of the channel of the hop node. Therefore, the main elements of the delay are: the number of routing hops, the transmission rate, and the ability to avoid interference. Specifically, the larger the number of hops, the more times the data needs to be processed, and the greater the cumulative data processing delay. The smaller the transmission rate, the greater the transmission delay. The weaker the ability to avoid interference, the more retransmissions, and the greater the cumulative delay.

[0093] This paper explores the impact of different interference durations on the utility function, such as Fig.12 The three histograms under one interference duration are based on GPS-perception, GPS-learning, and the proposed algorithm. As the interference duration decreases, the perception result becomes more unsatisfactory, and the superiority of the method of the present invention becomes more obvious.

[0094] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A Q-learning-based joint channel and routing cross-layer decision method, characterized in that: The method includes: Step 1, obtaining coordinates: In the early stage, the routing nodes in the network obtain their own real-time coordinate information through the equipped positioning system; Step 2, each routing node periodically broadcasts a message for discovering, establishing, and maintaining neighbor relationships, that is, a Hello packet; when a neighbor node receives the Hello packet, it feeds back an acknowledgment packet containing its own address and coordinate information, that is, an ACK; the routing node that receives the ACK records the neighbor information in the neighbor table and establishes good neighbor relationships; Step 3, when the source node has data to transmit, the source node selects the next-hop node and channel according to the selection strategy; and adds the address of the selected next-hop node and the address of the destination node to the data packet and transmits it outward; Step 4, when a neighbor node receives a data packet, it compares whether the address of the next-hop node in the data packet is the same as its own address; if not, it considers that the data packet is not for itself to forward and discards it without processing and forwarding; when the address of the next-hop node in the packet is the same as its own address, it continues to process the data packet and analyzes the received signal to estimate the signal-to-interference-plus-noise ratio; Step 5, the next-hop node replies to the source node through the control link, and the reply information includes the estimated signal-to-interference-plus-noise ratio and its latest Q table; Step 6, the source node calculates the reward value according to the feedback signal-to-interference-plus-noise ratio and the previously obtained position; Step 7, the source node updates the Q table and the node selection strategy according to the reward value; Step 8, the next-hop node repeats the process of the source node in Steps 3 to 7 until the data packet is sent to the destination node; The signal-to-interference-plus-noise ratio is determined by the following method: In a multi-hop routing communication system, the channel model is affected by large-scale fading. The channel gain from the l-th hop node to the l+1-th hop node is denoted as h l,l+1 , expressed as: in represents the distance from the lth hop node to the l+1th hop node, β represents the path fading factor; the channel gain h between the l+1th hop node and the jammer j,l+1 Expressed as Assume that the transmission power of routing nodes is the same as P t , the transmission power of the jammer is P j , then the l+1th hop node N l+1 The received signal-to-interference ratio SJNR l+1 for: where σ ft represents the communication channel f t The channel noise, ψ(f t -f j ) indicates that the function is expressed as follows: where f t represents the communication channel between the lth hop node and the l+1th hop node, f j Indicates the channel where the interference is located; when f t =f j When , it means that the channel accessed by the routing node is the same as the interference channel, and the data link is interfered; Step 6, the source node calculates the reward value according to the feedback signal-to-interference-plus-noise ratio and the previously obtained position, including: According to the received signal-to-interference ratio SJNR l+1 , the user communication rate C between the lth hop node and the l+1th hop node l,l+1 for: Where w represents the channel bandwidth of the communication channel, SJNR th Indicates the signal-to-interference ratio threshold for successful demodulation by the receiver; when receiving SJNR l+1 When the signal is lower than the demodulation threshold, the l+1th hop node cannot demodulate the signal correctly and considers it to be interfered, and the communication rate is 0; Calculating the reward value according to the transmission rate and position information again: Among them, r c 、r e are two negative numbers, which are used to avoid interference and choose the next hop closer to the destination node; d l,D ,d l+1,D ,d S,D Respectively represent the distances from the l-th hop node, the l+1-th hop node, and the source node to the destination node; w1 and w2 are two weight coefficients that can adjust the transmission rate and the influence of the next hop node position to satisfy w1+w2=1; The l-th hop node N l The next hop node and channel selection are only related to the l-1th hop node N l-1 The next hop node and channel selection are related to each other, and have nothing to do with the node and channel selection of the previous link. Therefore, N l The next-hop node and channel selection problem is regarded as a discrete-time Markov decision process MDP; The MDP model is defined by a four-tuple represents the state space, represents the action space, represents the state transition probability, Represents the reward value; the core elements in the MDP model are defined as follows: Defining the state space Where S l represents the state of the l-th hop node, N l Indicates the current routing node. Indicates the channel where the current interference is located, L indicates the number of available channels, Q l Represents the number of routing nodes that can be selected for the lth hop; Defining the action space where a l represents the action of the l-th hop node, a l =(N l+1 ,f t l+1 ), N l+1 represents the l+1th hop node selected by the lth hop node, f t l+1 represents the communication channel connecting the lth hop and the l+1th hop node, Q l+1 Represents the number of routing nodes that can be selected for the l+1th hop; Define the state transition probability matrix Indicates that from state S l Select action a l Arrival state S l+1 probability; Defining Reward Values R(S l ,a l ) indicates that in state S l Next select action a l The instant reward value is obtained; the number of hops is linked to the geographical location, and the routing node closer to the destination node is selected as the next hop; that is, the closer to the destination node, the fewer times the forwarding is required, and the farther the distance, the more times the forwarding is required; therefore, the reward value of each hop is defined as r(S l ,a l ): Among them, r c 、r e are two negative numbers, the purpose is to avoid interference as much as possible and choose the next hop action closer to the destination node; w1 and w2 are two weight coefficients, which can adjust the transmission rate and the influence of the next hop node position to satisfy w1+w2=1; d l,D ,d l+1,D ,d S,D Respectively represent the distances from the l-th hop node, the l+1-th hop node, and the source node to the destination node; when d l,D -d l+1,D When <0, it means that the current node has selected a next hop farther from the destination node. In this case, the node selection is wrong and a negative reward r is given. c When d l,D -d l+1,D When >0, it means that the current node selects a next hop closer to the destination node, and the node selection is correct; Step 7, the source node updates the Q table and the node selection strategy according to the reward value, including: For any l-th hop node, the update formula of the Q table is: Where α∈(0,1) is the learning rate, γ is the discount factor, and r l is the current state S l Select action a l The immediate return value, S l The state is in the routing node and selects and executes action a l Then it becomes S l+1 Status; Q l,t (S l ,a l ) indicates that in the current state S l Next select action a l Q value, Q l,t+1 (S l ,a l ) represents the updated Q value; Define routing nodes and channel selection strategy P l N , The update formula is: in Indicates that in state S l Next select N l+1 The maximum Q value corresponding to the node, max Q l+1 (S l+1 ,:,:) means in the next state S l+1 The maximum Q value that can be obtained under this condition; β is the relevant parameter of the Boltzmann model: Among them, β0 indicates the beginning of the exploration phase, β final Indicates the end of the exploration phase, τ is the first parameter of the Boltzmann model, and v is the second parameter of the Boltzmann model.

2. The method according to claim 1, characterized in that Step 8, the next-hop node repeats the process of the source node in Steps 3 to 7 until the data packet is sent to the destination node, including: When the routing node transmits the data packet to the next node, it starts a timing device. If the acknowledgment packet from the next-hop node has not been received by the specified time, it is considered to be interfered, obtains a negative reward, makes a new decision, and sends the data packet again; Assume that a transmission needs to pass through L hops, and define the communication rate with the minimum communication rate in the L-hop communication link as the communication rate of this data transmission: C=min(C 1,2 ,....,C L-1,L ) (14); Among them, C 1,2 Indicates the transmission rate of the data packet sent from the first hop node to the second hop node, C L-1,L It indicates the transmission rate of data packets sent from the l-1th hop node to the lth hop node; The goal of each transmission of the routing node is to maximize the communication rate under the condition of minimizing the number of hops of the routing path by optimizing the next-hop node and channel selection strategies of a single routing node; define the utility function of a single transmission of the routing node as: Step 8 includes the following steps: Step 81, perform initialization, set the simulation time T>0, and the initial communication time t = 0; Step 82, set the learning parameters, the discount factor, and the relevant parameters of the Boltzmann model; Step 83, initialize the Q table Q of each routing node n (S,a)=0; Step 84, initialize the current node N0 as the source node, and sense the current interference channel as Initial state Start the loop. When t<T, the current node N0 is not equal to the destination node D; Step 85: Select the next hop node N1 and the communication channel f according to the Boltzmann selection strategy of the current node. t 1 , that is, the current node N0 passes through channel f t 1 Transmit data to the next hop node N1; Step 86, after the next-hop node N1 receives the data packet, it processes and analyzes it, estimates SJNR0, and replies to the current node; Step 87, the current node N0 calculates the reward value using the reply information and the coordinate information, and then updates the probability of selecting the strategy; Step 88, the next hop node N1 senses the interference channel as Update Status When the requirements are met, the transmission ends and the cycle ends.

Images