A mobile self-organizing network routing method based on deep reinforcement learning

Abstract

This invention provides a routing method for mobile ad hoc networks based on deep reinforcement learning, comprising: (1) exploring routing links through routing control packets using a dual deep Q-network model to obtain empirical data; (2) using the obtained empirical data as training data to train the dual deep Q-network model to obtain a trained dual deep Q-network model; (3) inputting the node of the data packet to be sent into the trained dual deep Q-network model to calculate the routing link, and transmitting the data packet according to the obtained routing link. On the one hand, neural networks can use more information as input, thereby expanding the state action space and facilitating the formulation of better strategies; on the other hand, neural networks can automatically extract hidden features from high-dimensional input data, effectively reducing the difficulty of manual feature extraction while obtaining the optimal end-to-end decision.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of wireless communication, and particularly relates to a mobile self-organizing network routing method based on deep reinforcement learning. BACKGROUND

[0002] A wireless self-organizing network, also known as a mobile ad hoc network, is composed of a group of mutually cooperative mobile nodes with wireless communication transceiver devices, is a multi-hop temporary non-central network, and each node in the network is equal in status. Each node can be a terminal or a router, has both communication and routing functions, and does not need to rely on inherent network infrastructure. Two terminal nodes that cannot directly communicate need to relay and forward through other nodes. Military use wireless transmission requires high confidentiality, anti-interception, and anti-interference. Certain specific application scenarios also require ultra-long transmission distance.

[0003] Compared with traditional centralized mobile communication systems, mobile ad hoc networks have the advantages of multi-hop, non-central and self-organizing, distributed control, dynamic topology, and independent networking. They can quickly build or dismantle wireless communication networks without using existing network infrastructure or centralized management equipment. Therefore, mobility and invulnerability are the most valuable characteristics of mobile ad hoc networks. However, mobile ad hoc networks also have the disadvantages of small wireless transmission bandwidth, weak wireless link security, poor scalability, limited node power energy, and strong mutual interference. Routing protocols play a crucial role in ad hoc networks and are the main research focus and difficulty. Due to the random movement of nodes, time-varying links, and limited bandwidth in ad hoc networks, routing interruptions or network congestion can easily occur. The related characteristics of mobile self-organizing networks make the distance vector routing protocol and link state routing protocol in traditional wireless networks no longer applicable, and special ad hoc network routing protocols must be researched. SUMMARY

[0004] The purpose of the present application is to provide a mobile self-organizing network routing method based on deep reinforcement learning to solve the above problems.

[0005] The present application is implemented by the following technical solutions:

[0006] A mobile self-organizing network routing method based on deep reinforcement learning, comprising:

[0007] (1) Combining a double deep Q network model, routing link exploration is performed through a routing control packet to obtain experience data;

[0008] (2) The obtained experience data is used as training data to train the double deep Q network model, and a trained double deep Q network model is obtained;

[0009] (3) inputting the node of the data packet to be sent into the trained double deep Q network model, calculating a routing link, and transmitting the data packet according to the routing link.

[0010] Preferably, in step (1), the double deep Q network model is a pre-trained double deep Q network model, which is obtained by pre-training the double deep Q network model using data of the AODV protocol as demonstration data.

[0011] Preferably, in step (1), the routing control packet includes an FR packet, an ACK packet and an ET packet; and step (1) includes:

[0012] S1, the source node generates an FR packet and sends the FR packet to a destination receiving node of the FR packet as a sending node of the FR packet; wherein the FR packet contains an FR packet sequence number, a sending node number of the FR packet, a destination receiving node number of the FR packet, a source node number and a destination node number;

[0013] S2, after the destination receiving node of the FR packet receives the FR packet, the destination receiving node of the FR packet updates a local routing information table according to the FR packet, generates an ACK packet and sends the ACK packet to the sending node of the FR packet; then the destination receiving node of the FR packet judges whether the destination receiving node of the FR packet is the destination node of the FR packet, if not, the sending node of the FR packet is updated to the destination receiving node of the FR packet and a next destination receiving node of the FR packet is calculated, and the updated FR packet is forwarded to the next destination receiving node of the FR packet; S2 is repeated until the destination node of the FR packet receives the FR packet or the sending node of the FR packet cannot send the FR packet to the destination receiving node of the FR packet.

[0014] S3, for each sending node of the FR packet, if the sending node of the FR packet receives the ACK packet, an experience data is added to a local experience data table, and it is judged whether the sending node of the ACK packet is the destination node, if yes, an ET packet containing the experience data is generated; if the sending node of the FR packet does not receive the ACK packet within a set time, an experience data is added to the local experience data table, and an ET packet containing the experience data is generated; the sending node of the FR packet sends the ET packet to the last destination receiving node of the FR packet according to the routing information table, the last destination receiving node of the FR packet receiving the ET packet adds the corresponding experience data in the local experience data table to the ET packet, the above process is repeated, and the ET packet is sent to the source node in the form of hop-by-hop forwarding.

[0015] Preferably, the destination receiving node of the FR packet is calculated by the sending node of the FR packet through the double deep Q network model.

[0016] Further, in S2, after receiving the FR packet, the destination receiving node of the FR packet updates the local routing information table according to the FR packet, specifically:

[0017] The destination receiving node of the FR packet searches the local routing information table according to the sending node number and the destination receiving node number in the FR packet to determine whether there is a routing entry from the sending node to the destination receiving node in the FR packet header, and if not, a new routing entry is added; if so, the routing entry is updated according to the FR packet.

[0018] Further, in the added or updated routing entry, the routing entry number is the sequence number of the received FR packet, the routing previous hop node is the sending node of the FR packet, the routing source node is the sending node of the FR packet, and the routing destination node is the destination receiving node of the FR packet.

[0019] Further, in S2, after receiving the FR packet, the destination receiving node of the FR packet generates an ACK packet and sends it to the sending node of the FR packet, specifically:

[0020] After receiving the FR packet, the destination receiving node of the FR packet checks the length of the local receiving queue in the local information table, checks the node remaining energy, and calculates the channel signal-to-noise ratio level according to the actual received effective signal strength and noise in the FR packet; packs the FR packet sequence number in the received FR packet, the length of the local receiving queue, the node remaining energy and the channel signal-to-noise ratio level into the ACK packet data part, and then encapsulates the local data header to obtain the ACK packet, and sends the ACK packet to the sending node of the FR packet.

[0021] Further, in S3, the experience entry of the experience data is e(s, a, r, s'), wherein:

[0022] s is the current state set, indicating the node number of the current network that has a routing control packet;

[0023] a is the action state set, indicating the possible actions of the routing control packet in the current network;

[0024] r is the reward function, indicating the reward obtained by forwarding the data packet from the current state to the next hop;

[0025] s' is the next hop state set, indicating the next hop node number that the routing control packet may move to after taking the action.

[0026] Further, the calculation method of the reward function in the experience entry is: based on the deep reinforcement learning routing algorithm, the neighbor channel signal-to-noise ratio level, the local receiving queue length and the node remaining energy in the ACK packet are used to calculate the cost function by weighting, and the reward function is calculated from the cost function.

[0027] Preferably, in step (1), the intelligent antenna is used for transmitting the routing control packet when the routing control packet explores the routing link.

[0028] Preferably, the method is implemented in the MAC layer, and the MAC layer uses the TDMA channel access mode.

[0029] Compared with the prior art, the application has the following beneficial effects:

[0030] The application uses a double deep Q network model to perform deep reinforcement learning on the experience data obtained by the routing control packet exploration, obtains the trained double deep Q network model, and then uses the trained double deep Q network model to perform routing calculation for data packet sending. In deep reinforcement learning, a neural network is used as a powerful approximator of a Q table, which brings the following two advantages to network application. On the one hand, the neural network can use more information as input, thereby expanding the state action space and being conducive to making a better strategy. On the other hand, the neural network can automatically extract hidden features from high-dimensional input data, thereby obtaining an end-to-end optimal decision while effectively reducing the difficulty of manually extracting features. The use of the double deep Q network model can reduce the correlation between the extracted experience data samples and solve the tilt problem of the value estimation generated by the deep Q network model.

[0031] Further, the double deep Q network model is pre-trained using data of the AODV protocol as demonstration data, which can optimize the initial performance of the algorithm, greatly reduce the trial-and-error cost of self-learning in the initial stage, and alleviate the cold start problem.

[0032] Further, the functions of the routing control packet of the application can be summarized as follows: the FR packet is generated by the source node, is forwarded hop by hop to the destination node or ends, the next node receiving the FR packet generates an ACK packet and returns it to the sending node of the FR packet, when the FR packet reaches the destination node or fails to explore, the FR packet sending node at this time generates an ET packet. The routing control packets in the network cooperatively complete the exploration of the route, and in each round of routing exploration process, the source node generates the FR packet which is forwarded hop by hop by the nodes in the network. When the FR packet reaches the destination node or fails to be transmitted to the next hop node, the exploration round ends. At this time, the ET packet carries the experience data of the explored path and is forwarded hop by hop to the source node, and the source node obtains the experience data of all experienced nodes on the exploration path.

[0033] Further, the application selects a path based on the neighbor channel signal-to-noise ratio level, the local receiving queue length, and the node residual energy comprehensive parameter, effectively improves the network congestion caused by the original link state routing algorithm considering only the path with the minimum hop count and the high frame error rate caused by the poor link quality, and simultaneously realizes real-time data transmission and QOS guarantee.

[0034] Furthermore, this invention is designed based on smart antenna underlying technology, using smart antennas to complete the transmission of routing control packets related to routing functions, which greatly improves the local range of distributed allocation and can meet the transmission needs in ultra-long-distance scenarios.

[0035] Furthermore, this invention employs a cross-layer design, fully utilizing information provided by other layers of the ad hoc network protocol layer. Based on the original protocol's hop count as the path metric, new constraint parameters obtained from the physical and data link layers are added, effectively improving the overall network performance. Moreover, routing control packets are transmitted only within the routing transmission subframe, reducing routing control overhead and improving system efficiency. This invention can flexibly adjust the TDMA frame structure to suit different application scenarios, considering the maximum number of nodes and hop count the network can support, exhibiting good scalability and compatibility. Attached Figure Description

[0036] Figure 1 This diagram illustrates the overall design scheme for the communication frame structure of a wireless ad hoc network.

[0037] Figure 2 This is a diagram illustrating the structure of the routing control packet.

[0038] Figure 3 Flowchart for collaborative route exploration for routing control packets. Detailed Implementation

[0039] To further understand the present invention, the present invention will be described below with reference to embodiments. These descriptions are only for further explaining the features and advantages of the present invention and are not intended to limit the claims of the present invention.

[0040] The present invention is a TDMA mobile ad hoc network routing method based on deep reinforcement learning, which mainly includes the following aspects: (1) The routing function is implemented across layers, and the flooding of all routing-related information and routing calculation are implemented in the MAC layer. The MAC layer adopts a TDMA channel access method with dynamic reservation mechanism: the time resources are divided into time units of superframes, and the superframes are divided into N frames; (2) beam switching is performed based on smart antennas to realize omnidirectional and directional transmission. All routing control packets are broadcast omnidirectionally and data packets are sent directionally; (3) the data in the routing information table is updated first through the broadcast of FR packets, and at the same time, it serves as a trigger signal for sending ACK packets and ET packets; (4) the local load and link status are sent to neighboring nodes through ACK packets, providing the necessary information for neighboring nodes to update their local information tables and neighboring information tables; (5) the experience data table is updated through ET packets, and at the same time, the experience data is transmitted to the source node to realize the function of routing calculation; (6) the FR packets generated by the source node calculate the next hop node according to the network parameters, and forward it hop by hop until the destination node is reached. At this time, the ET packets carry the explored experience data and forward it hop by hop to the source node; (7) the cost function and reward function are calculated based on the collected experience data to support the deep reinforcement learning algorithm to perform routing selection for each hop. This invention utilizes smart antennas to achieve full-network flooding of routing control packets and uses deep reinforcement learning to reflect the collected link information in routing path selection. It has advantages such as QoS guarantee, meeting different requirements, low routing overhead, high packet delivery rate, low latency, and high network performance.

[0041] The above aspects will be described in detail below:

[0042] (1) The routing scheme is designed across layers. The flooding of routing control packets and the execution of the routing algorithm are implemented at the MAC layer. The MAC layer adopts a TDMA channel access mechanism with a reservation mechanism. The overall routing algorithm adopts a passive link-state routing protocol. The source node explores the routing links through the three designed routing control packets and selects the optimal link by combining a deep reinforcement learning algorithm. The cross-layer design is to make full use of the information provided by other layers of the ad hoc network protocol layer, which can improve the performance of the entire network.

[0043] Wireless Ad Hoc Network Communication Frame Structure Design: The overall design scheme for the wireless ad hoc network communication frame structure is as follows: Figure 1 As shown, the entire network is divided into temporal resources using superframes as the unit of time. A superframe is divided into N frames, and each frame is divided into M time slots. Each superframe can be divided into two parts according to its function.

[0044] First, the K-frame is used to complete the fast network entry of nodes. Second, the subsequent NK-frame is a TDMA (Time Division Multiple Access) frame, in which each node accesses the shared channel using TDMA. Each TDMA frame can be divided into service segments, signaling segments, and beacon segments according to their functions, mainly completing time slot allocation, routing control packet flooding, and transmission of upper-layer service data and broadcast packets. In this embodiment of the invention, within a superframe, except for frames 00 and 10 which are used as access frames for node network entry, all other TDMA frames can be used for routing. Each TDMA frame is fixedly authorized to a certain node in the network to complete the task of broadcasting routing-related control information within its one-hop communication range.

[0045] The routing protocol designed in this invention mainly involves the service segment. The first 32 time slots of the service segment are service data transmission subframes used to transmit upper-layer service data. The last 6 time slots of the service segment are used for the transmission of routing control packets FR / ACK / ET. The FR packet is mainly responsible for updating the data in the routing information table, and also serves as a trigger signal for sending ACK and ET packets. The ACK packet is responsible for sending the local node load, remaining node energy, and link status to neighboring nodes, providing neighboring nodes with the necessary information to update their local and neighboring information tables.

[0046] (2) Constructing a routing decision model. Treating the entire network as a system, taking one routing decision process as an example: the source node periodically sends routing control packets to neighboring nodes, which then forward them to the next-hop node, experiencing multiple hops before finally reaching the destination node. Based on this, a four-tuple routing decision model is established, combined with reinforcement learning for route selection. Based on the analysis of deep reinforcement learning and mobile ad hoc network models, the agent in this process can be defined as the routing control packets generated by the source node, and the environment in which the agent exists is the entire network. The Markov decision model is rewritten as a four-tuple routing decision model (S, A, r, S′). Where:

[0047] S: The current state set represents the node numbers in the current network that possess routing control packets. For example, if node i has a routing control packet to be sent, then the current state set is S = {s}. i |i∈N}, where s i It is a one-dimensional array of length N, where the i-th element is equal to 1 and all other elements are equal to 0.

[0048] A: The action state set represents the actions that a routing control packet might take in the current network. For example, if node i plans to forward a routing control packet to a neighbor node j, then the action state set is A = {a}. i |l i,j ∈E}, where a i =j.

[0049] r: The reward function represents the reward received by a routing control packet when it is forwarded from the current state to the next hop. For example, the reward for a routing control packet forwarded from node i to node j is r(s). i ,a j ).

[0050] S': The next-hop state set represents the next-hop node number that the routing control packet might move to after taking an action. For example, after node i forwards the routing control packet to the next hop, the next-hop state set is S' = {s}. i Similarly, s i ' is a one-dimensional array of length N, where the i-th element is equal to 1 and all other elements are equal to 0.

[0051] (3) Design of Routing Control Packets: This routing protocol mainly uses three types of routing control packets, including Route Discovery (FR) packets, Link Feedback (ACK) packets, and Experience Transmission (ET) packets, such as... Figure 2 These three types of routing control packets are designed to explore the network topology and solve for the optimal routing strategy by synchronizing and updating local information tables. Combining smart antenna beamforming technology, beam switching is performed based on smart antennas, which are used to transmit and receive routing control packets. All routing control packets are broadcast omnidirectionally, while data packets are sent directionally. Within a TDMA subframe, all nodes broadcast (omnidirectional transmission) sequentially in designated time slots according to a certain order, while other nodes receive them directionally.

[0052] -FR Packet: Responsible for updating data in the routing information table, and also serves as a trigger signal for sending ACK and ET packets. It is transmitted using an omnidirectional transmission and directional reception mode. The header of the FR packet includes the packet type, FR packet sequence number, sending node number, and destination receiving node number. The data part of the FR packet carries the source node number, destination node number, maximum forwarding count, and routing neural network parameter information. Nodes listen in the time slots that are not their own, parse the received FR packets, and determine whether the destination receiving node in the FR packet header is the node they are targeting. If the destination receiving node is not the node they are targeting, the packet is flooded again. If the destination receiving node is the node they are targeting, the packet is forwarded, and the relevant information (routing entry number, routing source node, routing destination node, previous hop node, and routing neural network parameters) is stored in the routing information table.

[0053] -ACK packet: Responsible for sending the local node load, remaining node energy, and link status to neighboring nodes, providing necessary information for neighboring nodes to update their local and neighbor information tables. It uses an omnidirectional transmission and directional reception mode. The ACK packet header includes the packet type, ACK packet sequence number, sending node number, and destination receiving node number. The ACK packet data includes the local node number, local receive queue length, neighbor node number, remaining node energy, and neighbor channel signal-to-noise ratio (SNR). Nodes listen in time slots other than their own, parse received ACK packets, and determine if the destination receiving node in the ACK packet header is the node they are targeting. If the destination receiving node is not the node they are targeting, the packet is flooded out again; if the destination receiving node is the node they are targeting, the relevant reception information (local node number, local receive queue length, neighbor node number, remaining node energy, and neighbor channel SNR) is stored in the local and neighbor information tables. The local information table includes the local node number and local receive queue length. The local node number refers to the node itself, which is the destination receiving node of the ACK packet. The neighbor information table includes the neighbor node number, the node's remaining energy, and the neighbor channel signal-to-noise ratio level. The neighbor node number refers to the node that sent the ACK packet.

[0054] - ET Packet: Responsible for updating the experience data table and transmitting experience data to the source node for route calculation. It uses an omnidirectional transmission and directional reception mode for transmission. The ET packet header includes the packet type, ET packet sequence number, sending node number, and destination receiving node number. The ET packet data includes the ET packet sequence number, route source node number, and experience entry. It determines whether the node is the destination receiving node in the ET packet header. If not, the packet is flooded; if it is, the route source node number field in the ET packet is parsed and forwarded hop-by-hop, ultimately successfully feeding the experience data back to the source node.

[0055] (4) Node local cache information: Under the routing method of the present invention, the node needs to store a variety of information tables locally.

[0056] - Routing Information Table: Nodes store routing information tables based on the received FR packets; these tables contain routing entry numbers, source nodes, destination nodes, previous hop nodes, and routing neural network parameter information.

[0057] - Local Information Table: Nodes store their local information table based on the received ACK packets; it includes information such as the local node number and the length of the local receive queue.

[0058] - Neighbor Information Table: Nodes store their neighbor information table based on the received ACK packets; it includes information such as neighbor node number, node remaining energy, and neighbor channel signal-to-noise ratio level (from the physical layer, which can determine the link channel status between the two nodes).

[0059] - Experience Data Table: Nodes store their experience data tables based on the received ET packets; these experience data tables record information such as experience data sequence numbers and experience entries.

[0060] (5) Generation and processing of FR packets: FR packets continuously generated by the source node are forwarded by the destination receiving node in the network. The sending node number and destination receiving node number in the FR packet header are only updated when the receiving node is the destination receiving node. When a node in the network listens for an FR packet sent to it, the node determines whether it has processed the FR packet based on the FR packet's sequence number. If it has not processed it, it further extracts the information contained in the listened FR packet and performs corresponding processing based on the information. The specific processing is divided into three flows: forwarding flow, acknowledgment flow, and information update flow.

[0061] (a) Forwarding Process: Upon receiving an FR packet, the node first determines whether forwarding is necessary based on the maximum forwarding count information. If forwarding is required, it checks if the destination receiving node in the FR packet header matches the receiving node. If the destination receiving node is not the receiving node, the packet is flooded out. If the destination receiving node is the receiving node, it checks if the destination node is the receiving node. If the destination node is not the receiving node, it calculates the next-hop destination receiving node number using a dual-depth Q-network model based on the routing neural network parameters in the FR packet. Next, the sequence number, source node number, destination node number, routing neural network parameters, and the modified maximum forwarding count from the FR packet are packaged into the data portion of the FR packet. A new FR packet header, locally modified with the sending node number and the next-hop destination receiving node number, is then encapsulated. Finally, the reconstructed FR packet is flooded out through the transmitter.

[0062] (b) Information Update Process: If a node receiving an FR packet determines that it is the destination receiving node in the FR packet header, it checks the routing information table for a route entry from the sending node in the FR packet header to the destination receiving node, based on the sending node number and destination receiving node number in the FR packet header. If it does not exist, a new route entry needs to be added. New Route Entry Number: The sequence number of the received FR packet. New Route Previous Hop Node: The sending node that sent the FR packet (the sending node in the header). New Route Neural Network Parameters: The routing neural network parameters carried in the FR packet. New Route Source Node: The sending node that sent the FR packet (the sending node in the header). New Route Destination Node: The destination receiving node of the FR packet (the destination receiving node in the header).

[0063] If the routing information table already contains a routing entry from the sending node in the header of the FR packet to the destination receiving node, then it is only necessary to update the routing entry number, the previous hop node, and the routing neural network parameters based on the latest received FR packet.

[0064] (6) Generation and processing of ACK packets: In each round of route exploration, the generation of ACK packets is triggered when a node receives an FR packet.

[0065] Response Procedure: If the node receiving the FR packet determines that it is the destination receiving node in the FR packet header, it can proceed with the response procedure. First, it checks the length of the local receive queue in its local information table, checks the node's remaining energy, and calculates the channel signal-to-noise ratio (SNR) based on the actual received effective signal strength and noise in the FR packet information. Next, it packages the sequence number from the received FR packet, along with the aforementioned local receive queue length, remaining energy, and channel SNR information, into an ACK packet. This ACK packet is then encapsulated with a local data header, and the destination receiving node is set to the sending node in the FR packet header. Finally, the ACK packet is sent through the transmitter to the sending node of the FR packet, which is the node one hop upstream of the route entry corresponding to the FR packet sequence number in the receiving node's routing information table.

[0066] Meanwhile, since the destination receiving node number of the FR packet is calculated by the local node based on the parameters of the local routing neural network, whether a channel exists in the network from the sending node to the destination receiving node calculated by the neural network needs to be determined by the node after sending the FR packet by listening for and receiving the corresponding ACK packet. The specific process is as follows:

[0067] When a sending node sends an FR packet to a destination receiving node, the sending node starts a timer to check if it can receive an ACK packet before the timer expires. If it receives the packet, an experience data entry is added to the experience data table. The sequence number of this experience data entry is the sequence number of the received ACK packet, and the experience entry is e(s,a,r,s′), where s = s1, a = a2, r = r(s1,a2), and s′ = s2. If it does not receive the packet, for example, if node 3 does not receive the ACK packet sent by node 4, an experience data entry is also added. The sequence number of this experience data entry is the sequence number of the sent FR packet, and the experience entry is e(s,a,r,s′), where s = s3, a = a4, r = r(s3,a4), and s′ = s3, indicating that no such link exists.

[0068] A node listens in a time slot that is not its own, parses the received ACK packet, and determines whether the destination receiving node in the ACK packet header is the node itself. If the destination receiving node is not the node itself, the packet is flooded out. If the destination receiving node is the node itself, the relevant receiving information (local node number, local receiving queue length, neighbor node number, neighbor channel signal-to-noise ratio level) is stored in the local information table and the neighbor information table.

[0069] (7) Generation and processing of ET packets: A node generates an ET packet when it needs to end the current round of route exploration. There are two situations in which the current round of route exploration is terminated:

[0070] The first scenario is that the FR packet has been successfully forwarded to the destination node, meaning that the routing control packet has figured out a route from the source node to the destination node.

[0071] The second scenario is that the destination receiving node of the FR packet is not within the effective transmission range of the sending node of the current FR packet, causing the FR packet to fail to be forwarded to the next hop. In other words, the route explored by the routing control packet is invalid.

[0072] In both scenarios above, after the sending node of the current FR packet sends the FR packet, if the node receives an ACK packet from the destination node, or if the node does not receive an ACK packet before the timer expires, then the sending node of the current FR packet will generate an ET packet. The ET packet sequence number is the FR packet sequence number, the experience entry for the ET packet is the experience entry corresponding to the FR packet sequence number in the local experience data table, and the destination receiving node of the ET packet is the upstream node of the route corresponding to the FR packet sequence number in the local routing information table.

[0073] Finally, the node receiving the ET packet determines whether it is the destination receiving node in the ET packet header. If not, the packet is flooded out. If it is, the source node field in the ET packet is parsed and forwarded hop-by-hop, ultimately successfully feeding back the experience data to the source node. Specifically, the destination receiving node first looks up the experience entry in its local experience data table based on the ET packet's sequence number and appends it to the ET packet. Then, it forwards the repackaged ET packet to the previous hop node on the route corresponding to the ET packet's sequence number in its local routing information table. Finally, through hop-by-hop forwarding by each node, all experience data along the exploration path in this round is fed back to the source node.

[0074] (8) Deep Reinforcement Learning Network Framework Design: The main feature of the DQN (Deep Q-Network) model is the use of two neural networks to reduce the correlation between extracted empirical data samples. One neural network, called the Q estimation network, is used to represent the estimated value function Q(s,a;θ), as well as to select actions and update the current network model parameters θ; the other neural network, called the Q target network, represents the target value function Q(s,a;θ), and the parameters θ of the Q target network are... -Instead of updating in real-time with training iterations, the parameters in the Q-estimation network are assigned to the Q-target network at regular intervals. The goal of DQN is to minimize the difference between the optimal objective function and the estimated objective function in empirical replay, and to adjust the convergence behavior of the neural network parameters through backpropagation of gradient descent. However, DQN models are prone to the reuse of overestimated objective functions, leading to a skewed value estimation problem. To overcome this issue, the Dual Deep Q-Network (DDQN) model is adopted. The only difference between DDQN and DQN is their objective function. The implementation of DDQN revolves around an iterative strategy for the objective function, reducing the correlation between extracted empirical data samples and addressing the skewed value estimation problem. While the sampling of empirical data in DQN is random, some empirical data is more valuable for learning. Therefore, a priority sampling method is introduced to achieve more efficient convergence.

[0075] (9) Path Selection Method Design: This routing algorithm is a passive routing algorithm based on deep reinforcement learning, and it also has a QoS guarantee mechanism. The core idea of ​​the improved routing algorithm is to add an evaluation model to the routing algorithm to assess changes in physical layer link quality, the remaining energy status of nodes, and the load of data to be transmitted by nodes. The signal-to-noise ratio (SNR) level of the neighbor channel obtained through the physical layer is used to measure link quality, and the local receive queue length from the MAC layer is used as the measure of node load. The routing link cost measure is a single metric parameter composed of a weighted combination of multiple parameters. The data transmission cost function C is defined, which is composed of a weighted combination of multiple parameters that can measure path quality. The parameters used to calculate the cost function include the neighbor channel SNR level, the local receive queue length, and the remaining energy of the node. The reward function can be calculated from the cost function. The reward function encourages nodes to choose an effective action and penalizes ineffective actions. Effective actions include choosing a neighbor node as the next hop node or successfully transmitting the data packet to the destination node. Ineffective actions include choosing a non-neighbor node or a node with high congestion as the next hop node, or choosing a channel with a low SNR level to transmit data packets. Ultimately, a routing path meeting different QoS requirements can be calculated. Using a multi-metric weighted cost and reward function to select the optimal path offers greater flexibility while ensuring QoS quality, avoiding network congestion and poor link quality issues caused by a single metric. Considering the parameters used to measure the cost and reward functions, such as local receive queue length, neighbor channel signal-to-noise ratio, and node remaining energy, these parameters need to be obtained through flooding of the configured routing control packets, thus allowing the selection of an optimal route that meets different QoS requirements.

[0076] (10) Solving the cold start problem: Because deep reinforcement learning typically requires a large amount of data to achieve acceptable performance, and because reinforcement learning requires continuous experimentation during the learning process, the performance of deep reinforcement learning-based algorithms is not guaranteed. To alleviate the cold start problem, having the agent pre-learn an initial policy to optimize the initial performance of the algorithm can greatly reduce the trial-and-error cost of the agent's self-learning in the initial stage. Consider using data from the AODV protocol as demonstration data to pre-train the agent and optimize the initial performance of the algorithm.

[0077] Based on the above design aspects, the present invention provides a TDMA mobile ad hoc network routing method based on deep reinforcement learning, the specific process of which mainly includes the following steps:

[0078] (1) Using data from the AODV protocol as demonstration data, the deep reinforcement learning network, namely the dual-deep Q network model, is pre-trained to obtain the preliminary trained routing neural network parameters and the dual-deep Q network model.

[0079] (2) Combining pre-trained routing neural network parameters and a dual-deep Q-network model, routing links are explored through routing control packets (e.g., Figure 3 ), to obtain empirical data; the routing control packet includes FR packet, ACK packet and ET packet; the specific process is as follows.

[0080] 1) Generation and processing of FR packets: The source node generates FR packets and floods them out, and starts a timer; the FR packet header includes packet type, packet sequence number, sending node number, and destination receiving node number. The FR packet data includes FR packet sequence number, source node number, destination node number, maximum forwarding count, and routing neural network parameters. The destination receiving node number is calculated by the sending node using a dual-depth Q network model based on the routing neural network parameters.

[0081] When a node in the network detects a Forwarding (FR) packet sent to it, the node determines whether it has processed the FR packet based on its sequence number. If it has not processed the packet, it further extracts the information contained in the detected FR packet and takes appropriate action based on that information. The specific processing is divided into three steps: forwarding, acknowledgment, and information update.

[0082] (a) Forwarding Process: Upon receiving an FR packet, the node first determines whether forwarding is necessary based on the maximum forwarding count information. If forwarding is required, it checks if the destination receiving node in the FR packet header matches the receiving node. If the destination receiving node is not the receiving node, the FR packet is flooded out. If the destination receiving node is the receiving node, it checks if the destination node is the receiving node. If the destination node is not the receiving node, the receiving node is treated as the new sending node. Based on the routing neural network parameters in the FR packet, a pre-trained dual-deep Q-network model is used to calculate the next-hop destination receiving node number for forwarding. Next, the sequence number, source node number, destination node number, routing neural network parameters, and the modified maximum forwarding count from the FR packet are packaged into the data portion of the FR packet. A new FR packet header, with locally modified sending node number and next-hop destination receiving node number, is then encapsulated. Finally, the reconstructed FR packet is flooded out through the transmitter, and a timer is started.

[0083] (b) Information Update Process: If a node receiving an FR packet determines that it is the destination receiving node in the FR packet header, it checks the routing information table for a route entry from the sending node in the FR packet header to the destination receiving node, based on the sending node number and destination receiving node number in the FR packet header. If it does not exist, a new route entry needs to be added. New Route Entry Number: The sequence number of the received FR packet. New Route Previous Hop Node: The sending node in the FR packet header. New Route Neural Network Parameters: The routing neural network parameters carried in the FR packet. New Route Source Node: The sending node that sent the FR packet (the sending node in the header). New Route Destination Node: The destination receiving node of the FR packet (the destination receiving node in the header).

[0084] If the routing information table already contains a routing entry from the sending node in the header of the FR packet to the destination receiving node, then it is only necessary to update the routing entry number, the previous hop node, and the routing neural network parameters based on the latest received FR packet.

[0085] 2) Generation and processing of ACK packets:

[0086] Response Procedure: If the node receiving the FR packet determines that it is the destination receiving node in the FR packet header, it initiates the response procedure. First, it checks the length of the local receive queue in its local information table, checks the node's remaining energy, and calculates the channel signal-to-noise ratio (SNR) based on the actual received effective signal strength and noise in the FR packet information. Next, it packages the sequence number from the received FR packet, along with the aforementioned local receive queue length, node remaining energy, and channel SNR information, into the ACK packet data portion. This data is then encapsulated with a local data header (including the sequence number, sending node, and destination receiving node), with the destination receiving node set to the sending node in the FR packet header. Finally, the ACK packet is sent through the transmitter to the sending node of the FR packet, which is the node one hop upstream of the route entry corresponding to the FR packet sequence number in the receiving node's routing information table.

[0087] If the sending node of the FR packet receives the ACK packet, it adds an experience data entry to the experience data table. The sequence number of this experience data entry is the sequence number of the received ACK packet, and the experience entry for this experience data entry is e(s,a,r,s′), where s = s1, a = a2, r = r(s1,a2), and s′ = s2. If no ACK packet is received when the timer expires, for example, node 3 does not receive the ACK packet sent by node 4, an experience data entry is also added. The sequence number of this experience data entry is the sequence number of the sent FR packet, and the experience entry is e(s,a,r,s′), where s = s3, a = a4, r = r(s3,a4), and s′ = s3, indicating that no such link exists.

[0088] A node listens in a time slot that is not its own, parses the received ACK packet, and determines whether the destination receiving node in the ACK packet header is the node itself. If the destination receiving node is not the node itself, the packet is flooded out. If the destination receiving node is the node itself, the relevant receiving information (local node number, local receiving queue length, neighbor node number, neighbor channel signal-to-noise ratio level) is stored in the local information table and the neighbor information table.

[0089] The reward function in the experience item is calculated as follows: based on the deep reinforcement learning-based routing algorithm, the cost function is obtained by weighting the neighbor channel signal-to-noise ratio level, local receive queue length, and node remaining energy in the ACK packet, and the reward function is calculated from the cost function.

[0090] The cost function is defined by the following formula:

[0091]

[0092] Where σ represents the relative importance between the cost of reducing latency and the cost of ensuring throughput. The c() function is a cost normalization function proposed to make the cost function conform to real-world scenarios.

[0093]

[0094] Let c(0) represent the latency cost function at node j at time t due to queuing for computing resources. Even if there is no queue at node j, the basic time consumption for processing data packets still exists, i.e., c(0) = 0.5. This indicates that during the process of data packets being transmitted from node i to node j, there is interaction with channel l. i,j The cost function for guaranteed throughput is related to the signal-to-noise ratio level.

[0095] A reward function incentivizes nodes to choose an effective action and penalizes ineffective actions. Effective actions include choosing a neighboring node as the next-hop node or successfully transmitting the data packet to the destination node. Ineffective actions include choosing a non-neighboring node or a node with high congestion as the next-hop node, or choosing a channel with a low signal-to-noise ratio to transmit the data packet.

[0096]

[0097] 3) Generation and processing of ET packets:

[0098] If the sending node of the FR packet receives the ACK packet and determines that the sending node of the ACK packet is the destination node in the FR packet, or if the sending node of the FR packet does not receive the ACK packet when the timer expires, then the sending node of the FR packet will generate an ET packet and flood it out. The sequence number of the ET packet is the sequence number of the FR packet, and the experience entry of the ET packet is the experience entry corresponding to the sequence number of the FR packet in the local (sending node of the FR packet) experience data table. The destination receiving node in the ET packet header is the previous hop node of the route corresponding to the sequence number of the FR packet in the local (sending node of the FR packet) routing information table.

[0099] Upon receiving an ET packet, the node determines whether it is the destination receiving node listed in the ET packet header. If not, the packet is flooded out. If it is, the source node field in the ET packet is parsed and forwarded hop-by-hop, ultimately successfully feeding back the experience data to the source node. Specifically, the destination receiving node first looks up the experience entry in its local experience data table based on the ET packet's sequence number and appends it to the ET packet. Then, it forwards the repackaged ET packet to the node one hop above the route corresponding to that ET packet's sequence number in its local routing information table. This process is repeated, and finally, through hop-by-hop forwarding by each node, all experience data along the exploration path in this round is fed back to the source node.

[0100] (3) Use the empirical data obtained from multiple source nodes as training data to train the pre-trained dual-deep Q network model, and obtain the trained routing neural network parameters and dual-deep Q network model.

[0101] (4) Usage process

[0102] For a given node, a trained dual-deep Q-network model is used to perform routing calculations and transmit data packets.

Claims

1. A routing method for mobile ad hoc networks based on deep reinforcement learning, characterized in that, include: (1) Combining the dual-depth Q network model, we explore routing links through routing control packets to obtain empirical data; The routing control packet includes: FR packet, ACK packet, and ET packet; (2) Use the obtained empirical data as training data to train the dual-depth Q-network model and obtain the trained dual-depth Q-network model. (3) Input the node of the data packet to be sent into the trained dual-depth Q network model, calculate the routing link, and transmit the data packet according to the obtained routing link; Step (1) includes: S1, the source node generates an FR packet and sends the FR packet to the destination receiving node as the sending node of the FR packet; wherein, the FR packet contains the FR packet sequence number, the sending node number of the FR packet, the destination receiving node number of the FR packet, the source node number, and the destination node number; S2: After receiving the FR packet, the destination receiving node updates its local routing information table based on the FR packet, generates an ACK packet, and sends it to the sending node of the FR packet. Then, it determines whether the destination receiving node of the FR packet is the destination node of the FR packet. If not, it updates the sending node of the FR packet to the destination receiving node of the FR packet, calculates the destination receiving node of the next FR packet, and forwards the updated FR packet to the destination receiving node of the next FR packet. S2 is repeated until the destination node of the FR packet receives the FR packet or the sending node of the FR packet is unable to send the FR packet to the destination receiving node of the FR packet. S3, for each FR packet sending node, if the FR packet sending node receives an ACK packet, it adds an experience data entry to its local experience data table and determines whether the ACK packet sending node is the destination node. If it is the destination node, it generates an ET packet containing the experience data. If the FR packet sending node does not receive an ACK packet within a set time, it adds an experience data entry to its local experience data table and generates an ET packet containing the experience data. The FR packet sending node sends the ET packet to the destination receiving node of the previous FR packet according to the routing information table. The destination receiving node of the previous FR packet that receives the ET packet adds the corresponding experience data from its local experience data table to the ET packet. The above process is repeated, and the ET packet is sent to the source node in a hop-by-hop forwarding manner. The destination receiving node of the FR packet is calculated by the sending node of the FR packet using a dual-depth Q network model.

2. The routing method for mobile ad hoc networks based on deep reinforcement learning according to claim 1, characterized in that, In step (1), the dual-depth Q-network model is a pre-trained dual-depth Q-network model, which is obtained by using AODV protocol data as demonstration data to pre-train the dual-depth Q-network model to obtain the pre-trained dual-depth Q-network model.

3. The routing method for mobile ad hoc networks based on deep reinforcement learning according to claim 1, characterized in that, In S2, after the destination receiving node of the FR packet receives the FR packet, it updates its local routing information table according to the FR packet, specifically as follows: The destination receiving node of the FR packet checks its local routing information table for a route entry from the sending node to the destination receiving node in the header of the FR packet, based on the sending node number and the destination receiving node number in the FR packet. If the entry does not exist, a new route entry is added; if the entry exists, the route entry is updated according to the FR packet.

4. The routing method for mobile ad hoc networks based on deep reinforcement learning according to claim 3, characterized in that, In the newly added or updated routing entries, the routing entry number is the sequence number of the received FR packet, the previous hop node of the route is the sending node of the FR packet, the source node of the route is the sending node of the FR packet, and the destination node of the route is the destination receiving node of the FR packet.

5. The routing method for mobile ad hoc networks based on deep reinforcement learning according to claim 1, characterized in that, In S2, after the destination receiving node of the FR packet receives the FR packet, it generates an ACK packet and sends it to the sending node of the FR packet, specifically as follows: After receiving the FR packet, the destination receiving node checks the length of the local receiving queue in its local information table, checks the node's remaining energy, and calculates the channel signal-to-noise ratio (SNR) level based on the actual received effective signal strength and noise in the FR packet. It then packages the FR packet sequence number, the length of the local receiving queue, the node's remaining energy, and the channel SNR level into the ACK packet data part, encapsulates it with a local data header, and sends the ACK packet to the sending node of the FR packet.

6. The routing method for mobile ad hoc networks based on deep reinforcement learning according to claim 5, characterized in that, In S3, the empirical entries of the empirical data are: ,in: This represents the current state set, indicating the node numbers in the current network that contain routing control packets; This is a set of action states, representing the actions that routing control packets in the current network may take. Here, the reward function represents the reward a data packet receives for being forwarded from its current state to the next hop. This is the next-hop state set, representing the next-hop node number that the routing control packet may move to after taking action.

7. The routing method for mobile ad hoc networks based on deep reinforcement learning according to claim 6, characterized in that, The reward function in the experience entry is calculated as follows: based on the deep reinforcement learning-based routing algorithm, the cost function is obtained by weighting the neighbor channel signal-to-noise ratio level, local receive queue length, and node remaining energy in the ACK packet, and the reward function is calculated from the cost function.

8. The routing method for mobile ad hoc networks based on deep reinforcement learning according to claim 1, characterized in that, In step (1), when the routing control packet explores the routing link, a smart antenna is used to transmit the routing control packet.

9. The routing method for mobile ad hoc networks based on deep reinforcement learning according to claim 1, characterized in that, The method is implemented at the MAC layer, which uses TDMA channel access.

Images