Joint channel and route selection cross-layer decision-making method based on Q learning

Abstract

The invention provides a joint channel and route selection cross-layer decision-making method based on Q learning. The method comprises the following steps: acquiring coordinates; each routing node periodically broadcasts messages used for discovering, establishing and keeping alive neighbor relations; after receiving the Hello packet, the neighbor node feeds back an acknowledgement message and establishes a neighbor relationship; when data needs to be transmitted in the source node, the source node selects a next-hop node and a channel transmission data packet according to a selection strategy, and the data packet comprises the address of the selected next-hop node; the neighbor point judges whether the data packet should be forwarded by the neighbor point; the next-hop node replies to the source node through the control link, wherein reply information comprises the estimated signal-to-interference ratio and the latest Q table of the next-hop node; the source node calculates a reward value according to the feedback signal-to-interference ratio and the position obtained in the earlier stage; and the source node updates the Q table and the node selection strategy according to the reward value. According to the method, the network transmission rate and the anti-interference performance are improved, and the end-to-end time delay and hop count are reduced.

Description

Technical Field

[0001] The present application relates to the technical field of wireless communication technology, and in particular to a cross-layer decision method for joint channel and routing selection based on Q learning. Background Art

[0002] Due to the wireless transmission characteristics of ad hoc networks, the network bandwidth it can provide is much lower than that of wired channels, and it is easily interfered by jammers. However, the current anti-interference methods of ad hoc networks are mainly concentrated in a single physical layer or network layer, resulting in limited anti-interference effects. Most physical layer methods adjust channels or transmission power based on learned interference patterns. However, in ad hoc networks, when routing nodes or transmission links are interfered with by external factors, routing failures will occur.

[0003] In this case, simply adjusting the signal or power is not feasible, and the path needs to be replanned. Network layer anti-interference research mainly focuses on multipath transmission, network coding and rerouting, but these methods are difficult to deal with dynamic interference and usually require a lot of additional overhead. The current small amount of cross-layer anti-interference research mainly focuses on using known topology information to find a subset of routes from the source node to the destination node, and then dynamically adjusts the strategy according to the current environment. However, this method requires a lot of prior information and requires high maintenance overhead.

[0004] Therefore, determining a strategy that requires only less information exchange to achieve effective anti-interference and good network performance has become the current research focus. Summary of the invention

[0005] The present application provides a cross-layer decision method for joint channel and routing selection based on Q learning, which can be used to solve the technical problem that cross-layer anti-interference research requires more prior information and has high maintenance costs.

[0006] The present application provides a Q-learning-based joint channel and routing cross-layer decision method, the method comprising:

[0007] Step 1, obtaining coordinates: In the early stage, the routing nodes in the network obtain their own real-time coordinate information through the equipped positioning system;

[0008] Step 2: Each routing node periodically broadcasts a message used to discover, establish, and maintain neighbor relationships, namely, a Hello packet. When the neighbor node receives the Hello packet, it feeds back a confirmation message containing its own address and coordinate information, namely, ACK. The routing node that receives the ACK records the neighbor information in the neighbor table and establishes a good neighbor relationship.

[0009] Step 3: When the source node has data to transmit, the source node selects the next hop node and channel according to the selection strategy; and adds the address of the selected next hop node and the address of the destination node to the data packet and transmits it outward.

[0010] Step 4: When a neighbor node receives a data packet, it compares the address of the next-hop node in the data packet with its own address to see if they are the same; if they are different, it considers that the data packet is not for it to forward, and discards it without processing or forwarding it; when the address of the next-hop node in the message is the same as its own address, it will continue to process the data packet and analyze the received signal to estimate the signal-to-interference ratio.

[0011] In multi-hop routing communication systems, the channel model is affected by large-scale fading. Jump to the node The channel gain of the hop node is denoted as , expressed as:

[0012] ;

[0013] in Indicates Jump to the node The distance of the hop node, represents the path fading factor; Channel gain between hop node and jammer Expressed as ;

[0014] Assume that the transmission power of routing nodes is the same as , the transmission power of the jammer is , then Jump Node The received signal-to-interference ratio for:

[0015] ;

[0016] in Indicates the communication channel The channel noise, The indicator function is expressed as follows:

[0017] ;

[0018] in Indicates Jump node and The communication channel between hop nodes, Indicates the channel where the interference is located; when When , it means that the channel accessed by the routing node is the same as the interference channel, and the data link is interfered.

[0019] Step 5: The next hop node replies to the source node through the control link. The reply information includes the estimated signal-to-interference ratio and its latest Q table.

[0020] Step 6: The source node calculates the reward value based on the feedback signal-to-interference ratio and the position obtained previously.

[0021] According to the received signal-to-interference ratio , No. Jump node and User communication rate between hop nodes for:

[0022] ;

[0023] in represents the channel bandwidth of the communication channel, Indicates the signal-to-interference ratio threshold for successful demodulation by the receiver; When it is lower than the demodulation threshold, The hop node cannot demodulate the signal correctly and thinks it is interfered, so the communication rate is 0;

[0024] Then calculate the reward value based on the transmission rate and location information:

[0025] ;

[0026] in, , These are two negative numbers with relatively large absolute values. Their function is to avoid interference as much as possible and choose the next hop action that is closer to the destination node. Respectively represent Jump node, The distance between the hop node and the source node to the destination node; are two weight coefficients that can adjust the transmission rate and the influence of the next hop node location to meet ;

[0027] No. Jump Node The next hop node and channel selection are only related to the Jump Node The next hop node and channel selection are related to each other, and have nothing to do with the node and channel selection of the previous link. The next-hop node and channel selection problem is regarded as a discrete-time Markov decision process MDP;

[0028] The MDP model is defined by a four-tuple , represents the state space, represents the action space, represents the state transition probability, Represents the reward value; the core elements in the MDP model are defined as follows:

[0029] Defining the state space :

[0030] ;

[0031] in Indicates The state of the jump node, , Indicates the current node. Indicates the channel where the current interference is located. Indicates the number of available channels. Representative The number of nodes that can be selected by the jump;

[0032] Defining the action space :

[0033] ;

[0034] in Indicates The action of jumping nodes, , Indicates The node selected by the jump Jump node, Indicates the connection Jump and The communication channel of the hop node, Representative The number of nodes that can be selected by the jump;

[0035] Define the state transition probability matrix : , , indicating that from the state Select Action Arrival Status probability;

[0036] Defining Reward Values : Indicates in status Next select action The instant reward value obtained; considering that obtaining the hop count information requires knowing the topology information of the entire network in advance, it will cause a large overhead. The hop count is linked to the geographical location, and the routing node closer to the destination node is selected as the next hop; that is, the closer to the destination node, the fewer times it needs to be forwarded, and the farther away, the more times it needs to be forwarded; therefore, the reward value for each hop is defined as :

[0037] ;

[0038] in, , These are two negative numbers with relatively large absolute values. The purpose is to avoid interference as much as possible and choose an action that makes the next hop closer and closer to the destination node. is the weight coefficient, which can adjust the influence of transmission rate and next-hop node location to meet . Respectively represent Jump node, The distance between the jump node and the source node to the destination node; when When the current node selects a next hop farther from the destination node, the node selection is wrong and a negative reward is given. ;when , it means that the current node has selected a next hop closer to the destination node, and the node selection is correct; and in order to achieve the largest possible transmission rate, the above reward is given.

[0039] Step 7: The source node updates the Q table and node selection strategy according to the reward value.

[0040] For any Jump node, the update formula of Q table is:

[0041] ;

[0042] in is the learning rate, is the discount factor, For the current state Select Action The immediate return value, The state selects and executes actions at the routing node Later becomes state; Indicates the current state Next select action The Q value, represents the updated Q value;

[0043] Define routing nodes and channel selection strategies , The update formula is:

[0044] ;

[0045] ;

[0046] ;

[0047] in Indicates in status Next Select The maximum Q value corresponding to the node, Indicates that in the next state The maximum Q value that can be obtained under Are the relevant parameters of the Boltzmann model:

[0048] ;

[0049] in, Indicates the beginning of the exploration phase. Indicates the end of the exploration phase. is the first parameter of the Boltzmann model, It is the second parameter of the Boltzmann model. It comprehensively considers the potential value of two consecutive hops and provides a more accurate and forward-looking routing selection strategy.

[0050] Step 8: The next hop node repeats the process of the source node from step 3 to step 7 until the data packet is sent to the destination node.

[0051] When the routing node transmits a data packet to the next node, the timing device is started. If the confirmation message from the next hop node is not received within the specified time, it is considered to be interfered, and a negative reward is obtained. The decision is made again and the data packet is sent.

[0052] Assume that a transmission needs to go through Jump, definition The communication rate with the minimum communication rate in the hop communication link is the communication rate of the data transmission:

[0053] ;

[0054] in, Indicates the transmission rate of data packets sent from the first hop node to the second hop node. Indicates Jump node to Transmission rate of the jumping node for sending data packets

[0055] Considering that in the routing protocol, the hop count is an important measure of network performance, which directly affects network latency, congestion, and the integrity of data packets, etc. The goal of each transmission of the routing node is to maximize the communication rate under the condition of minimizing the hop count of the routing path by optimizing the next-hop node and channel selection strategy of a single routing node; the utility function of a single transmission of the routing node is defined as:

[0056] ;

[0057] The specific algorithm is as follows:

[0058] Step 81: Initialize, set the simulation time T > 0, and the initial communication time t = 0;

[0059] Step 82: Set the learning parameters, discount factor, and relevant parameters of the Boltzmann model;

[0060] Step 83: Initialize the Q-table of each routing node = 0;

[0061] Step 84: Initialize the current routing node as the source node, sense the current interference channel as , and the initial state ;

[0062] Start the loop. When t < T, the current node is not equal to the destination node ;

[0063] Step 85: Select the next-hop node and the communication channel according to the Boltzmann selection strategy of the current node, that is, the current node sends data to the next-hop node through the channel ;

[0064] Step 86: After receiving the data packet, the next-hop node performs processing and analysis, estimates , and replies to the current node;

[0065] Step 87: The current node uses the reply information and combines it with the coordinate information to calculate the reward value, and then updates the selection strategy probability;

[0066] Step 88: The next-hop node senses the interference channel as , and updates the state ;

[0067] When the requirements are met, the transmission ends and the cycle ends.

[0068] The present invention solves the problem of establishing routing in an interference environment without central control and without relying on the topology information of the entire network, relying only on link status and geographic location information. The routing node dynamically senses the spectrum environment to identify spectrum holes, and combines the distance from its neighbors to the destination to find a suitable next hop and communication channel. Finally, the routing node combines these two aspects, makes a joint action, and uses Q learning to optimize the selection of the next hop node and channel. This method improves the network transmission rate and anti-interference performance, and reduces the end-to-end delay and number of hops. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A diagram showing the implementation steps of the routing optimization algorithm based on Q learning provided by the present invention;

[0070] Figure 2 A topological diagram provided by the present invention;

[0071] Figure 3 A channel selection probability graph of routing node 2 provided in an embodiment of the present application;

[0072] Figure 4 A next hop node selection probability graph of routing node 5 provided in an embodiment of the present application;

[0073] Figure 5 A channel selection probability graph of the routing node 5 provided in an embodiment of the present application;

[0074] Figure 6 A next hop node selection probability graph of routing node 7 provided in an embodiment of the present application;

[0075] Figure 7 A channel selection probability graph of routing node 7 provided in an embodiment of the present application;

[0076] Figure 8 A time-frequency diagram of a routing node and a jammer before convergence provided by the present invention;

[0077] Fig. 9 A time-frequency diagram of a routing node and a jammer before convergence provided by the present invention;

[0078] Fig.10 A comparison chart of utility values ​​provided by the present invention;

[0079] Fig.11 The delay variation curve diagram provided by the present invention;

[0080] Fig.12 The present invention provides a curve showing the effect of interference duration on utility value. DETAILED DESCRIPTION

[0081] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0082] The following first introduces the embodiments of the present application in conjunction with the accompanying drawings.

[0083] The present invention discloses an efficient and reliable anti-interference routing method, which models the distributed joint channel and node decision problem in a multi-channel scenario as a Markov decision process, and proposes a cross-layer routing anti-interference design for establishing routing based on geographic location and channel quality. The routing nodes in the network evaluate the geographic location and link quality of their adjacent nodes to select the appropriate next-hop node and communication channel, thereby reducing the cost of routing hops and maximizing the transmission rate. Considering that the update of the Q table only requires the exchange of reward values, the Q table is used instead of the routing table, reducing the amount of information interacted between neighbors. This method has low routing overhead and good anti-interference performance.

[0084] Figure 1 This is the implementation step of the routing optimization algorithm based on Q learning proposed by the present invention. First, each routing node obtains its own real-time position coordinate information through the equipped positioning. Then each routing node periodically broadcasts a Hello packet used to discover, establish, and keep the neighbor relationship alive. When the neighbor node receives the Hello packet, it sends a confirmation message to the routing node, which contains its own address and coordinates. After receiving the confirmation message, the routing node records the neighbor information in the neighbor table and establishes a good neighbor relationship. Secondly, when there is data to be transmitted, the source node selects the next hop node and channel according to the selection strategy. And the address of the selected next hop node and the address of the destination node are added to the data packet and broadcasted outward. When the neighbor node receives the data packet, it compares the address of the next hop node in the data packet with its own address to see if it is the same. If they are different, it is considered that the data packet is not a data packet to be forwarded by itself, and it is discarded without processing and forwarding. When the address of the next hop node in the message is the same as its own address, it will continue to process the data packet and analyze the received signal to estimate the signal-to-interference ratio. And reply to the source node through the control link, at this time, the reply information contains the estimated signal-to-interference ratio and its latest Q table. The signal-to-interference ratio is used to calculate the reward value, and the Q table is used to update the node selection strategy. Finally, the source node calculates the reward value and updates the Q table and selection strategy based on the reply information of the neighboring nodes.

[0085] When the source node transmits a data packet to the next node, a timing device will be started. If no confirmation message is received from the next-hop node within the specified time, it is considered that the selected communication channel is the same as the channel where the interference occurs, that is, it is interfered with, and a negative reward is obtained and the Q table and selection strategy are updated.

[0086] Figure 2 This is a simulation topology diagram of the present invention. There are 15 routing nodes and 1 jammer in the system. The jammer's jamming mode is sweep frequency jamming. The user performs Q learning according to the transmission time slot, selects and adjusts the next hop node and communication channel.

[0087] Figure 3-7 Gives interference In the state, the action selection probability curve of each routing node when the data packet is sent from source node 2. The simulation results show that in the initial stage of reinforcement learning, the Q value of each action of the user in this state is 0, and the probability of selecting each action is equal. After a period of learning, the routing node's cognition of the interference environment and topology is continuously strengthened, and the interference change law and routing information are gradually mastered, so the routing node gradually tends to choose the optimal decision strategy. It can be found from the figure that in the late learning algorithm convergence stage, the routing node transmission path is stipulated as 2-5-7-15. In the state of interference 3, the routing node will choose the action of communication channel 1 with a probability of 1.

[0088] Figure 8-9 The time-frequency diagram of routing nodes and interference is given. The light-colored squares represent routing node channels, and the dark-colored squares represent interference channels. The overlapping of light and dark colors makes the darker squares represent routing nodes that are interfered. In the early stage of Q learning, Figure 8 As shown in , because the channel tends to be randomly selected, the probability of collision between routing nodes and interference is relatively large. In the later stage of Q learning, as Fig. 9 As shown in the figure, the routing nodes almost all avoid the interference of the jammer and achieve the purpose of anti-interference.

[0089] To demonstrate the effectiveness of the proposed algorithm, the proposed method is compared with two GPS-based methods: one is to select the routing node closest to the destination node and randomly select an uninterrupted channel based on perception; the other is to use reinforcement learning to optimize channel selection by identifying interference patterns, while fixedly selecting the node closest to the destination as the next hop node. The system performance of these three methods is analyzed. Fig.10 A comparison chart of the utility values ​​of the three methods is given. At the beginning of learning, the GPS-based perception channel selection algorithm selects nodes according to distance and does not need to be learned, that is, according to the fixed path 2-5-8-13-14-15, and perception can allow channel selection to avoid interference to a certain extent. Therefore, at the beginning, the GPS-based perception channel selection algorithm has the largest utility value. However, it can be seen that as the number of learning times increases, the method used will gradually become the optimal one.

[0090] Fig.11The delay variation curve is given, and it can be seen that the delay of the method used has also been significantly improved with the increase in the number of learning times. The delay is the time difference between the time when the source node sends the data and the time when the destination node receives the data. The components of each hop delay are: perception delay, transmission delay, data processing delay, and confirmation message return delay. The expression of transmission delay is:

[0091] ;

[0092] in Indicates the length of the data packet to be transmitted. Indicates the connection Jump node and The transmission rate of the channel of the hop node. Therefore, the main elements of the delay are: the number of routing hops, the transmission rate, and the ability to avoid interference. Specifically, the larger the number of hops, the more times the data needs to be processed, and the greater the cumulative data processing delay. The smaller the transmission rate, the greater the transmission delay. The weaker the ability to avoid interference, the more retransmissions, and the greater the cumulative delay.

[0093] This paper explores the impact of different interference durations on the utility function, such as Fig.12 The three histograms under one interference duration are based on GPS-perception, GPS-learning, and the proposed algorithm. As the interference duration decreases, the perception result becomes more unsatisfactory, and the superiority of the method of the present invention becomes more obvious.

[0094] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A Q-learning-based joint channel and routing cross-layer decision method, characterized in that: The method comprises: Step 1, obtaining coordinates: In the early stage, the routing nodes in the network obtain their own real-time coordinate information through the equipped positioning system; Step 2: Each routing node periodically broadcasts a message used to discover, establish, and maintain neighbor relationships, namely, a Hello packet. When the neighbor node receives the Hello packet, it feeds back a confirmation message containing its own address and coordinate information, namely, ACK. The routing node that receives the ACK records the neighbor information in the neighbor table and establishes a good neighbor relationship. Step 3: When the source node has data to transmit, the source node selects the next hop node and channel according to the selection strategy; and adds the address of the selected next hop node and the address of the destination node to the data packet and transmits it outward; Step 4: When a neighbor node receives a data packet, it compares the address of the next-hop node in the data packet with its own address to see if they are the same; if they are different, it considers that the data packet is not for it to forward, and discards it without processing or forwarding it; when the address of the next-hop node in the message is the same as its own address, it continues to process the data packet and analyzes the received signal to estimate the signal-to-interference ratio; Step 5: The next hop node replies to the source node through the control link. The reply information includes the estimated signal-to-interference ratio and its latest Q table. Step 6: The source node calculates the reward value based on the feedback signal-to-interference ratio and the position obtained in the previous step; Step 7: The source node updates the Q table and node selection strategy according to the reward value; Step 8: The next hop node repeats the process of the source node from step 3 to step 7 until the data packet is sent to the destination node.

2. The method according to claim 1, characterized in that The signal-to-interference ratio is determined by the following method: In multi-hop routing communication systems, the channel model is affected by large-scale fading. Jump to the node The channel gain of the hop node is denoted as , expressed as: ； in Indicates Jump to the node The distance of the hop node, represents the path fading factor; Channel gain between hop node and jammer Expressed as ; Assume that the transmission power of routing nodes is the same as , the transmission power of the jammer is , then Jump Node The received signal-to-interference ratio for: ； in Indicates the communication channel The channel noise, The indicator function is expressed as follows: ； in Indicates Jump node and The communication channel between hop nodes, Indicates the channel where the interference is located; when When , it means that the channel accessed by the routing node is the same as the interference channel, and the data link is interfered.

3. The method according to claim 2, characterized in that Step 6: The source node calculates the reward value based on the feedback signal-to-interference ratio and the position obtained earlier, including: According to the received signal-to-interference ratio , No. Jump node and User communication rate between hop nodes for: ； in represents the channel bandwidth of the communication channel, Indicates the signal-to-interference ratio threshold for successful demodulation by the receiver; When it is lower than the demodulation threshold, The hop node cannot demodulate the signal correctly and thinks it is interfered, so the communication rate is 0; Then calculate the reward value based on the transmission rate and location information: ； in, , They are two negative numbers, which are used to avoid interference and choose the next hop closer to the destination node; Respectively represent Jump node, The distance between the hop node and the source node to the destination node; are two weight coefficients that can adjust the transmission rate and the influence of the next hop node location to meet ; No. Jump Node The next hop node and channel selection are only related to the Jump Node The next hop node and channel selection are related to each other, and have nothing to do with the node and channel selection of the previous link. The next-hop node and channel selection problem is regarded as a discrete-time Markov decision process MDP; The MDP model is defined by a four-tuple , represents the state space, represents the action space, represents the state transition probability, Represents the reward value; the core elements in the MDP model are defined as follows: Defining the state space : ； in Indicates The state of the jump node, , Indicates the current routing node. Indicates the channel where the current interference is located. Indicates the number of available channels. Representative The number of routing nodes that can be selected by the hop; Defining the action space : ； in Indicates The action of jumping nodes, , Indicates The node selected by the jump Jump node, Indicates the connection Jump and The communication channel of the hop node, Representative The number of routing nodes that can be selected by the hop; Define the state transition probability matrix : , , indicating that from the state Select Action Arrival Status probability; Defining Reward Values : Indicates in status Next select action The instant reward value is obtained; the number of hops is linked to the geographical location, and the routing node closer to the destination node is selected as the next hop; that is, the closer to the destination node, the fewer times the forwarding is required, and the farther the distance, the more times the forwarding is required; therefore, the reward value for each hop is defined as : ； in, , are two negative numbers, the purpose is to avoid interference as much as possible and choose the next hop action that is closer to the destination node; are two weight coefficients that can adjust the transmission rate and the influence of the next hop node location to meet ; Respectively represent Jump node, The distance between the jump node and the source node to the destination node; when When the current node selects a next hop farther from the destination node, the node selection is wrong and a negative reward is given. ;when , it means that the current node has selected a next hop that is closer to the destination node, and the node selection is correct.

4. The method according to claim 3, characterized in that Step 7: The source node updates the Q table and node selection strategy based on the reward value, including: For any Jump node, the update formula of Q table is: ； in is the learning rate, is the discount factor, For the current state Select Action The immediate return value, The state selects and executes actions at the routing node Later becomes state; Indicates the current state Next select action The Q value, represents the updated Q value; Define routing nodes and channel selection strategies , The update formula is: ； ； ； in Indicates in status Next Select The maximum Q value corresponding to the node, Indicates that in the next state The maximum Q value that can be obtained under are the relevant parameters of the Boltzmann model: ； in, Indicates the beginning of the exploration phase. Indicates the end of the exploration phase. is the first parameter of the Boltzmann model, is the second parameter of the Boltzmann model.

5. The method according to claim 4, characterized in that Step 8: The next hop node repeats the process of the source node in steps 3 to 7 until the data packet is sent to the destination node, including: When the routing node transmits a data packet to the next node, the timing device is started. If the confirmation message from the next hop node is not received within the specified time, it is considered to be interfered, and a negative reward is obtained. The decision is made again and the data packet is sent. Assume that a transmission needs to go through Jump, definition The communication rate with the minimum communication rate in the hop communication link is the communication rate of the data transmission: ； in, Indicates the transmission rate of data packets sent from the first hop node to the second hop node. Indicates Jump node to The transmission rate of data packets sent by the hop node; The goal of each transmission of the routing node is to achieve the maximum communication rate with the minimum number of routing path hops by optimizing the next hop node and channel selection strategy of a single routing node; the utility function of a transmission of the routing node is defined as: ； Step 8 includes the following steps: Step 81, perform initialization, set simulation time T>0, initial communication time t=0; Step 82, setting learning parameters, discount factors and related parameters of the Boltzmann model; Step 83, initialize the Q table of each routing node =0; Step 84, initialize the current node is the source node, and the current interference channel is perceived as , initial state ; Start the loop. When t < T, the current node is not equal to the destination node ; Step 85: Select the next hop node according to the Boltzmann selection strategy of the current node and communication channel , that is, the current node Through the channel Give the next hop node transfer data; Step 86, next hop node After receiving the data packet, it is processed and analyzed to estimate the , and reply to the current node; Step 87, current node Calculate the reward value using the reply information and coordinate information, and then update the probability of selecting the strategy; Step 88, next hop node The perceived interference channel is , update status ; When the requirements are met, the transmission ends and the cycle ends.

Images