Mobile ad hoc network routing optimization method and device based on deep reinforcement learning

Abstract

The application provides a mobile ad hoc network routing optimization method and device based on deep reinforcement learning, and belongs to the technical field of wireless communication networks.The method comprises the following steps: acquiring feature information of a current node; processing the feature information, converting a routing decision problem into a Markov decision process, so as to determine a state space, an action space, a transition probability and a reward function; the state space comprises state information of the current node; inputting the state information of the current node into a preset deep reinforcement learning model, and outputting a Q value corresponding to the current node; the deep reinforcement learning model is established based on a deep reinforcement learning network algorithm Dueling DQN; and determining a next hop node of the current node according to the Q value of the current node, so as to make a routing decision.The routing optimization method provided by the application applies a deep reinforcement learning algorithm to realize more accurate routing decisions, accelerates the convergence speed of reinforcement learning, and improves the efficiency of routing decisions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of wireless communication networks, and particularly relates to a mobile ad hoc network routing optimization method and device based on deep reinforcement learning. BACKGROUND

[0002] Mobile ad hoc networks have many new challenges in routing protocol design due to the characteristics of fast node movement, complex and changeable communication link environment, and high dynamic network topology. Traditional ad hoc network routing protocols cannot quickly respond when the network state changes due to the lack of design for high dynamic characteristics, resulting in a large number of invalid and redundant routing table entries in the network. With the increase of network dynamics, the routing performance rapidly decreases, and frequent routing updates also bring new burdens to the network. In addition, traditional ad hoc network routing protocols do not consider the energy, load and other state factors of nodes, and often select the shortest path for information transmission. When there are a large number of concurrent access demands, some busy nodes will exit the network due to energy depletion or excessive load, causing routing interruption and data transmission delay, which greatly affects the performance of the protocol.

[0003] In recent years, machine learning methods such as deep learning and reinforcement learning have been successfully applied to game strategy calculation, computer vision, natural language processing and other fields, and intelligent routing algorithms based on machine learning methods have become a research hotspot and shown great potential.

[0004] However, existing intelligent routing optimization algorithms based solely on deep learning models cannot guarantee their safety and robustness in complex and changeable network environments, and require high deployment costs; existing intelligent routing optimization algorithms based solely on reinforcement learning models have the problem of unguaranteed convergence in network scenarios with a large number of nodes. SUMMARY

[0005] The present application provides a mobile ad hoc network routing optimization method and device based on deep reinforcement learning, to solve the problems of poor safety and robustness of existing routing optimization algorithms and unguaranteed convergence.

[0006] In a first aspect, the present application provides a mobile ad hoc network routing optimization method based on deep reinforcement learning, comprising: obtaining feature information of a current node; processing the feature information, converting a routing decision problem into a Markov decision process to determine a state space, an action space, a transition probability and a reward function; the state space includes state information of the current node; inputting the state information of the current node into a preset deep reinforcement learning model to output a Q value corresponding to the current node; the deep reinforcement learning model is established based on a deep reinforcement learning network algorithm Dueling DQN; determining a next hop node of the current node according to the Q value of the current node to make a routing decision.

[0007] In a second aspect, a mobile ad hoc network routing optimization device based on deep reinforcement learning is provided, comprising:

[0008] A first processing module is configured to obtain feature information of a current node;

[0009] A second processing module is configured to process the feature information, convert a routing decision problem into a Markov decision process to determine a state space, an action space, a transition probability and a reward function; the state space includes state information of the current node;

[0010] A third processing module is configured to input the state information of the current node into a preset deep reinforcement learning model to output a Q value corresponding to the current node; the deep reinforcement learning model is established based on a deep reinforcement learning network algorithm Dueling DQN;

[0011] A fourth processing module is configured to determine a next hop node of the current node according to the Q value of the current node to make a routing decision.

[0012] In a third aspect, the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the mobile ad hoc network routing optimization method based on deep reinforcement learning according to any one of the above aspects when executing the program.

[0013] In a fourth aspect, the present application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the steps of the mobile ad hoc network routing optimization method based on deep reinforcement learning according to any one of the above aspects.

[0014] The mobile ad hoc network routing optimization method and device based on deep reinforcement learning provided by the present application have the following beneficial effects:

[0015] (1) Multi-information source fusion based on cross-layer design. The application can obtain hardware-based link quality estimation from the physical layer and software-based link quality estimation from the data link. By considering link quality, moving speed, residual energy, QoS demand, etc., the relevance and regularity hidden in multiple information sources are mined through a deep reinforcement learning model to provide more comprehensive and accurate basis for routing decision.

[0016] (2) Efficient exploration and utilization based on upper confidence. The application uses an improved upper confidence strategy to balance the contradiction between exploration and utilization. The number of times of selecting a node as a candidate forwarding node is considered in addition to the Q value. Therefore, the algorithm can explore an initially unexplored action and then tend to use this action to more efficiently explore some potential optimal routing paths.

[0017] (3) Penalty mechanism for end-to-end delay. The application considers the total waiting time of data packets in the buffer of the candidate forwarding node before they are transmitted on the communication channel as a factor of the penalty mechanism for end-to-end delay, thereby reducing the end-to-end delay and reducing the packet loss rate in blocking situations.

[0018] (4) Neural network architecture for convergence acceleration. To accelerate algorithm convergence to adapt to large-scale node number network environment, the application uses an optimized four-layer neural network instead of the traditional convolutional neural network to speed up the algorithm convergence speed. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0020] Figure 1 is a flowchart of the mobile ad hoc network routing optimization method based on deep reinforcement learning provided by the present application;

[0021] Figure 2 is a schematic diagram of a mobile ad hoc network composed of multiple nodes provided by the present application;

[0022] Figure 3 is a structure diagram of the neural network based on DDQN provided by the present application;

[0023] Figure 4 is a framework diagram of the routing decision process provided by the present application;

[0024] Figure 5 is a structure diagram of the mobile ad hoc network routing optimization device based on deep reinforcement learning provided by the present application;

[0025] Figure 6 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0026] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0027] It should be noted that in the description of the embodiments of the present application, the terms “comprise”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitation, the element defined by the sentence “comprises a…” does not exclude the presence of another identical element in the process, method, article or device comprising the element.

[0028] The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by “first”, “second” and the like are generally of a kind and do not limit the number of objects, for example, the first object can be one or more. In addition, “and / or” means at least one of the connected objects, and the character “ / ” generally means that the front and rear associated objects are in an “or” relationship.

[0029] The embodiments of the present application will be described below in conjunction with Figures 1-6 The method and device for mobile ad hoc network routing optimization based on deep reinforcement learning provided by the embodiments of the present application are described.

[0030] Figure 1 is a flowchart of the method for mobile ad hoc network routing optimization based on deep reinforcement learning provided by the present application, as Figure 1 shown, including but not limited to the following steps:

[0031] Step 101: acquiring feature information of a current node;

[0032] Step 102: processing the feature information, converting the routing decision problem into a Markov decision process to determine a state space, an action space, a transition probability and a reward function; the state space includes state information of a current node;

[0033] Step 103: inputting the state information of the current node into a preset deep reinforcement learning model to output a Q value corresponding to the current node; the deep reinforcement learning model is established based on a deep reinforcement learning network algorithm Dueling DQN;

[0034] Step 104: determining a next hop node of the current node according to the Q value of the current node to make a routing decision.

[0035] The implementation process of the above embodiment will be described in detail in combination with a specific flow.

[0036] (1) Network environment construction and mathematical modeling

[0037] Figure 2 is a schematic diagram of a mobile ad hoc network composed of multiple nodes provided by the application, as shown in Figure 2 The embodiment of the application designs a mobile ad hoc network composed of multiple unmanned aerial vehicles (UAVs) and unmanned surface vehicles (USVs).

[0038] It is assumed that the transmission power of each node is the same (i.e., the communication radius R is the same), and each node can communicate with adjacent nodes (i.e., candidate forwarding nodes) within the communication radius. Due to the limitation of node communication range, some nodes cannot directly communicate with other nodes and need to relay nodes to assist in forwarding data packets. If node i can directly communicate with adjacent node j, it is assumed that link L i,j (i≠j) exists. It is assumed that each node has a full-duplex transceiver, a unique identity (ID), and a positioning system for providing position, speed and direction information. In addition, each node can also obtain neighbor information by periodically receiving hello messages.

[0039] Considering that the communication link quality, node energy consumption and total message waiting time have a greater impact on routing decision in the above network environment, the application first models the link quality, energy consumption and total message waiting time.

[0040] 1) Link quality modeling

[0041] The application uses signal-to-noise ratio (SNR) and expected transmission count (ETX) to evaluate the link quality.

[0042] The key factors affecting the signal-to-noise ratio are path loss and Doppler shift. Assuming that the nodes use line-of-sight (LOS) communication means, the path loss of the transmitting and receiving nodes is caused by the radiation and dispersion of the transmitting power and the transmission characteristics of the channel. According to the Friis transmission equation, the distance between the receiver and the transmitter is proportional to the path loss. L i,j The path loss of the node i and its adjacent node j is expressed as:

[0043]

[0044] where z = 3 is the path loss exponent; d i,j is the Euclidean distance between the node i and its adjacent node j, the variable f is the carrier frequency, and c is the electromagnetic wave speed, which is approximately equal to the speed of light. The above formula calculates the dB value of the path loss, which is converted into the form absolute value:

[0045]

[0046] The size of the Doppler shift is proportional to the size of the relative velocity. When the node i communicates with the adjacent node j, the movement of the receiving node j causes a distance difference Δl of the received signal, and then the phase difference change value of the received signal can be expressed as:

[0047]

[0048] where θ is the angle between the incident direction of the radio wave and the movement direction of the node j; v j is the movement speed; and λ = c / f is the wavelength of the transmitted signal. Thus, the frequency change amount, i.e., the Doppler shift Δf i,j :

[0049]

[0050] Therefore, the signal-to-noise ratio loss caused by the Doppler shift is calculated as follows:

[0051]

[0052] where R d is the symbol rate. It is converted into the form absolute value:

[0053]

[0054] The signal-to-noise ratio between the node i and its adjacent node j is expressed as:

[0055]

[0056] where P i is the hello packet transmission power of the node i, h i.j is the link L i.jThe fading coefficient of the Nakagami-m channel is followed, including the effect of shadow fading. The expression N0 is the power of the additive white Gaussian noise (AWGN), PL i,j is the form absolute value of the path loss of the link L i.j , DF i,j is the form absolute value of the Doppler shift loss value.

[0057] In addition to the influence of the characteristics of the wireless communication channel, burst link anomalies or malicious link attacks by other devices can also cause packet loss. The expected transmission count (ETX) is used as another link quality assessment indicator in the present application. ETX measurement can be the number of expected data packet transmissions successfully received by the receiving node. The probability of successfully receiving hello messages at both ends of the bidirectional link is counted. The ETX of the link between node i and node j is represented as:

[0058]

[0059] wherein, is the probability of successful delivery of hello messages from node i to node j received by node j in the last w seconds, is the probability of successful delivery of hello messages from node j to node i received by node i in the last w seconds.

[0060] In summary, the link quality between node i and adjacent node j can be defined by considering both the signal-to-noise ratio and the expected transmission count as follows:

[0061]

[0062] wherein and are the normalized values of the link quality and the expected transmission count of the link L i,j , respectively. μ and v are the weights.

[0063] 2) Energy consumption modeling

[0064] The present application uses the ratio of the remaining energy of the node to the initial energy (i.e., the remaining energy level of the node) as an energy indicator to measure the energy consumption of the node. The remaining energy level E i of the i-th node can be represented as follows:

[0065]

[0066] wherein, E_res i is the remaining energy of node i, E_init i is the initial energy of node i. The larger the E i of the node, the smaller the energy consumption, and vice versa, the larger the energy consumption.

[0067] 3) Packet total waiting time modeling

[0068] When a node sends a packet to its neighbor, the unprocessed packet's waiting time in the neighbor node increases the end-to-end delay, and even increases the packet loss in heavy traffic. Therefore, the current forwarding node needs to consider the packet's waiting time in the neighbor node. Assuming that each node has the same buffer size, the scheduling mechanism of the data packets in the buffer is based on first-in-first-out (FIFO). When the current node i forwards a packet to node j, at the current time t, the total waiting time of the packet in node j can be calculated as:

[0069]

[0070] where P size is the size of the packet, is the processing delay of the kth packet in the buffer.

[0071] is the queue delay of the kth packet in the buffer, R i，j = W·log2(1+SNR i，j is the data packet transmission rate during the communication link L i，j , W is the bandwidth allocation of the communication link between node i and node j, and m is the total number of data packets in the buffer of node j. If there is only one data packet in the buffer of the candidate forwarding node, i.e., k = 1, the packet can be sent immediately. Otherwise, the packet will be queued and waiting for scheduling transmission.

[0072] (2) Markov decision process (MDP)

[0073] To solve the routing problem using reinforcement algorithm, the packet forwarding process is first modeled as a Markov decision process (MDP). MDP consists of four tuples {S, A, R, P}, where S is the state space, A is the action space, R is the reward function, and P is the state transition probability. The specific definitions of these tuples are as follows:

[0074] 1) State space: S = {s1, s2, s3,... s N}.

[0075]

[0076] where d i，j is the distance between the current forwarding node i and the candidate forwarding node j, d j，d is the distance between the candidate forwarding node j and the destination node. Nbr(i) represents the set of neighboring nodes within the communication range of node i.

[0077] The expression NFx = (d1, d2,..., d8) using the adjacent topology information is used to predict the location of the two-hop neighboring nodes of the current forwarding node in different areas. The predicted location of the two-hop neighboring nodes is represented as: where d j represents the farthest distance between the current forwarding node and the neighboring node of the jth area, l i represents the location of the node. represents the minimum distance between the predicted two-hop neighboring node and the destination node of the n areas. d sum is smaller, the greater the probability of route void alleviation, l d and l j are the locations of the destination node d and the candidate forwarding node j, respectively, LQ i，j is the link quality of the destination link l i , is the total packet waiting time of the candidate forwarding node j at the current time.

[0078] The feature DIR i，d is used to learn to avoid selecting candidate forwarding nodes that are far away from the destination node, and its specific expression is as follows:

[0079]

[0080] where, represents the velocity vector of the candidate forwarding node j, l l is the location of the last-hop forwarding node, and is the vector from node i to the destination node d. If 0 < DIR i，d < 1, the candidate forwarding node j is close to node d; -1 < DIR i，d < 0, j is far away from the destination node. All these features are designed to enable the agent to quickly learn the optimal strategy.

[0081] In addition, different features have different units and scales. In order to facilitate data processing and accelerate the convergence of the Q network, these dimensional features are normalized to dimensionless features using the min-max normalization method.

[0082] 2) Action space: A = {a1, a2, a3,... an n}, which is the set of neighbor nodes within the communication range of the current forwarding node i, and n is the total number of neighbor agents. When the current forwarding node needs to forward a packet, an action is selected.

[0083] 3) Transition probability: P: S i ×a i ×S j → [0, 1], which defines the probability of transitioning to the next state after performing an action in the current state.

[0084] 4) Reward function: S i ×A i →R is the key component. A well-designed reward function can ensure faster convergence of the algorithm and improve network performance.

[0085]

[0086] In the above formula, -1 represents that the reward function decreases by 1 for each additional hop, E j is the remaining energy level of the next hop node j, and σ is its weight. δ is the weight of the link quality between nodes i and j, is the total waiting delay of the data packet in the candidate forwarding node j. and are the total waiting time of the data packet in node j and the normalized value of the link quality of link L i，j If the candidate forwarding node j is the destination node, 100 rewards are awarded immediately when node j is selected as the next hop forwarding node.

[0087] (3) Routing decision learning based on DDQN algorithm

[0088] After modeling the data packet forwarding process as an MDP, deep reinforcement learning (Dueling Deep QNetwork, DDQN) is used to learn the routing decision, with the goal of ensuring that the node learns the optimal policy. The present application learns the optimal policy by indirectly learning the state function V * = maxV π (s) and the action function Q * = maxQ π (s, a).

[0089] The state function is the expected sum of all rewards from the current state s when the kth data packet is forwarded to the final state d using the policy π, which is expressed as:

[0090]

[0091] where γ ∈ [0, 1] is the discount factor, used to determine the trade-off between current and future rewards. According to the Bellman equation, the action function can be expressed as:

[0092]

[0093] Since the Q value outputted in the DQN network can be inaccurate, a DDQN is selected to learn the routing decision by optimizing the structure of the neural network. In order to accelerate the learning convergence, a four-layer neural network is used instead of a traditional convolutional neural network (CNN) as the network structure to simplify the algorithm. The simple four-layer neural network has an input layer, two hidden layers and an output layer, as shown in Figure 3 . Figure 3 Figure 1 is a structural diagram of a neural network based on DDQN provided by the present application.

[0094] For the DDQN, the Q value is divided into two parts. The first part is only related to the state s and is independent of the specific action a, and is referred to as a state function V(s; w, a). The second part is related to both the state s and the action a, and is referred to as an advantage function A(s, a; w, b). Each value of the vector corresponds to an action, and the Q value can be expressed as:

[0095] Q(s, a; w, a, b) = V(s; w, a) + A(s, a; w, b)

[0096] where w is the network parameter vector of the common part, a is the network parameter vector of the state function, and b is the network parameter vector of the advantage function.

[0097] However, there is a problem that cannot be identified, that is, given a Q value, it is impossible to obtain a unique state function and advantage function.

[0098] In order to overcome this problem, the advantage function must be processed, and the Q value can be rewritten as:

[0099]

[0100] In the designed DDQN algorithm structure, there are two Q networks: one is the current Q network Q(s, a; w, a, b), and the other is the target Q network Q'(s, a; w', a', b'), where w, a, b are the parameter vectors of the current Q network, w', a', b' are the parameter vectors of the target Q network, and a' represents an action in the set of selectable actions in the current state.

[0101] The objective function of the Q network is to minimize the loss function L(w t , a t , b t ).

[0102]

[0103] where The discount factor g is a predetermined hyperparameter; represents the mean function.

[0104] Then the gradient of the loss function is optimized using adaptive moment estimation (Adam). The gradient of the loss function is expressed as:

[0105]

[0106] In order to save memory space and stabilize the learning process, the application uses soft update instead of hard copy to update the parameters of the target Q-network.

[0107] The calculation method of soft update is as follows:

[0108]

[0109] Where τ∈[0,1] represents the weight of the target Q-network and the weight of the current Q-network. Although soft update takes more time, it increases the stability of learning.

[0110] In the training stage, in order to balance exploration and utilization, an improved upper-confidence-bound (IUCB) strategy is proposed to select actions, expressed as:

[0111]

[0112] Where represents the number of times of accessing state s i in a fixed time, represents the number of times of selecting a node as the candidate forwarding node Q i value of the y-th largest in a fixed time, Q i (s i , a i ) represents the Q value corresponding to the current node i.

[0113] The meanings of parameters and are different from those of the original UCB strategy. The improved IUCB not only considers Q value, but also considers the number of times of selecting the candidate forwarding node Q value of the y-th largest, so that the application can fully explore the actions that have not been explored at the beginning, and then it tends to use this action. After continuous iteration, the algorithm realizes convergence. The value of y can be set according to needs.

[0114] In the test stage, the softmax strategy is used to select the next hop forwarding node, that is, the next hop node is selected according to the probability instead of the Q value. The advantage of softmax is to avoid local optimization and divide the traffic. Its expression is:

[0115]

[0116] Where, A iis the action set for node i, and u is the temperature parameter. When u is large, the probability distribution of different candidate forwarding nodes is less different, resulting in a larger loss. When u is small, the probability distribution of different candidate forwarding nodes is more different, resulting in a smaller loss.

[0117] In order to more clearly describe the overall technical scheme of the present application, the scheme of the present application will be described in the whole from the overall technical scheme of the present application.

[0118] Figure 4 is the framework diagram of the routing decision process provided by the present application, as shown in Figure 4 The core module of the present application is a network layer geographic routing decision system based on deep reinforcement learning (Dueling Deep Q Network, DDQN). This module is responsible for learning the Q value of each node according to the input state information. The signal-to-noise ratio (SNR) and expected transmission count (ETX) indicators obtained by the data link layer and the physical layer are used to estimate the link quality. At the same time, each node needs to extract additional state information from the established neighbor table and perform a free routing loop operation.

[0119] According to the designed routing protocol architecture, the routing process is divided into two stages: the first stage is the establishment and maintenance of the neighbor table, and the second stage is the routing decision.

[0120] First stage: establishment and maintenance of the neighbor table

[0121] Each node broadcasts a hello packet at a certain time interval to exchange useful state information, including the geographical position, speed, remaining energy, probability value (d r ), total packet waiting time and node ID of the node, etc. When a node receives the hello packet sent by the neighbor, a table entry is established in the neighbor table, and the relevant information of the hello packet is recorded. After receiving the hello packet, the receiving node returns an ACK packet to the current forwarding node to confirm that the link has been established. In addition, the ACK packet contains information such as node ID and position, ACK confirmation, signal-to-noise ratio and probability value.

[0122] In the data packet forwarding process, additional information needs to be recorded in the neighbor table to assist in routing decision. The data packet includes information such as node ID and position of the source node and destination node, nodes (HVN) that the data packet has visited (used to avoid routing loops), a unique sequence number to determine the freshness of the data packet, ID of the last node and data packet content.

[0123] Second stage:

[0124] After the establishment of the routing decision neighbor table, when the source node needs to send a data packet to the destination node, the routing decision process based on the DDQN algorithm is triggered. With the help of relay node forwarding, the data packet finally reaches the destination node.

[0125] If the current forwarding node i needs to forward a data packet, the routing decision process is as follows.

[0126] First, the current forwarding node i must determine the set of its optional actions A opti =A i -HVN. If the current node i discards the packet. If the next step is performed.

[0127] Node i extracts feature information from the neighbor table and the received data packet: and so on.

[0128] where l i is the position of the current forwarding node i, l l is the position of the last hop forwarding node l, l d is the position of the destination node d, l x is the position of the neighbor node x, LQ i,x is the link quality of the link l i , and is the total waiting time of the data packet in the buffer of the neighbor node x, E j is the remaining energy, and NFx is the neighbor topology information.

[0129] According to these features, the state information of node i is:

[0130]

[0131] The normalized state s i is input into the DDQN network.

[0132] Finally, the current node i selects the next hop node according to the Q value output by the Q network.

[0133] Specifically, if the Q-network is in the training phase, the selection strategy of the next hop node depends on the expression: The optimal decision is obtained through iterative learning until the Q network converges.

[0134] In the test phase, each node selects the next hop node according to the softmax function .

[0135] Figure 5 is a structural diagram of the mobile ad hoc network routing optimization device based on deep reinforcement learning provided by the present application, as Figure 5As shown, the device includes:

[0136] The first processing module 501 is used to obtain the feature information of the current node;

[0137] The second processing module 502 is used to process the feature information and transform the routing decision problem into a Markov decision process to determine the state space, action space, transition probability, and reward function; the state space includes the state information of the current node.

[0138] The third processing module 503 is used to input the state information of the current node into a preset deep reinforcement learning model and output the Q value corresponding to the current node; the deep reinforcement learning model is established based on the deep reinforcement learning network algorithm Dueling DQN;

[0139] The fourth processing module 504 is used to determine the next hop node of the current node based on the Q value of the current node in order to make routing decisions.

[0140] It should be noted that the mobile ad hoc network routing optimization device based on deep reinforcement learning provided in this embodiment of the invention can execute the mobile ad hoc network routing optimization method based on deep reinforcement learning described in any of the above embodiments during specific operation, and will not be elaborated on in this embodiment.

[0141] In summary, the mobile ad hoc network routing optimization method and apparatus based on deep reinforcement learning provided by this invention have the following beneficial effects:

[0142] This invention provides hardware-based link quality estimation from the physical layer and software-based link quality estimation from the data link. By comprehensively considering link quality, mobility speed, remaining energy, and QoS requirements, it uses a deep reinforcement learning model to uncover the correlations and patterns hidden in multiple information sources, providing a more comprehensive and accurate basis for routing decisions.

[0143] This invention utilizes an improved high-confidence strategy. To balance the conflict between exploration and exploitation, it considers not only the Q-value but also the number of times a node is selected as a candidate forwarding node. Therefore, the algorithm can explore actions that were not initially explored and then tend to exploit these actions, thus more efficiently exploring potential optimal routing paths.

[0144] This invention addresses end-to-end latency by considering the total waiting time of data packets in the candidate forwarding node buffer before propagation on the communication channel as a penalty mechanism, thereby reducing end-to-end latency and decreasing packet loss rate under blocking conditions.

[0145] In order to accelerate algorithm convergence and adapt to network environments with a large number of nodes, this invention uses an optimized four-layer neural network instead of a traditional convolutional neural network, thereby speeding up the algorithm's convergence speed.

[0146] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640. The processor 610, communication interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a mobile ad hoc network routing optimization method based on deep reinforcement learning. This method includes: acquiring feature information of the current node; processing the feature information to transform the routing decision problem into a Markov decision process to determine the state space, action space, transition probabilities, and reward function; the state space includes the state information of the current node; inputting the state information of the current node into a preset deep reinforcement learning model and outputting the Q-value corresponding to the current node; the deep reinforcement learning model is established based on the Dueling DQN deep reinforcement learning network algorithm; and determining the next-hop node of the current node based on the Q-value of the current node for routing decision-making.

[0147] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the mobile ad hoc network routing optimization method based on deep reinforcement learning provided in the above embodiments. The method includes: acquiring feature information of the current node; processing the feature information to transform the routing decision problem into a Markov decision process to determine the state space, action space, transition probability, and reward function; the state space includes the state information of the current node; inputting the state information of the current node into a preset deep reinforcement learning model and outputting the Q value corresponding to the current node; the deep reinforcement learning model is established based on the deep reinforcement learning network algorithm Dueling DQN; and determining the next hop node of the current node based on the Q value of the current node to perform routing decision.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0149] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for mobile ad hoc network routing optimization based on deep reinforcement learning, the mobile ad hoc network comprising a plurality of nodes, characterized in that, The method comprises the following steps: obtaining feature information of a current node; processing the feature information to convert a routing decision problem into a Markov decision process to determine a state space, an action space, a transition probability, and a reward function; the state space includes state information of the current node; inputting the state information of the current node into a preset deep reinforcement learning model to output a Q value corresponding to the current node; the deep reinforcement learning model is established based on a deep reinforcement learning network algorithm Dueling DQN; determining a next-hop node of the current node according to the Q value of the current node to make a routing decision; Wherein, the state space S = {s1, s2, s3, … s N}; the state information of the current node is represented as: where d i,j is the distance between the current node i and the candidate forwarding node j, d j,d is the distance between the candidate forwarding node j and the destination node d; denotes the predicted minimum distance between the two-hop neighboring node and the destination nodes of e regions, LQ i,j is the link quality between node i and node j; Utilizing a characteristic DIR i,d to learn to avoid selecting a candidate forwarding node that is far away from the destination node, wherein denotes a velocity vector of the candidate forwarding node j, l l is a position of the last-hop forwarding node, is a vector from node l to the destination node d; if 0 < DIR i,d < 1, the candidate forwarding node j is close to the destination node d; -1 < DIR i,d < 0, j is far away from the destination node; is the total waiting time of the data packet at the candidate forwarding node j at the current time; Nbr(i) represents the set of neighboring nodes within the communication range of node i; The smaller dsum is, the greater the probability of route void alleviation is, l i 、l d and l j are the positions of node i, destination node d and candidate forwarding node j respectively. the action space is a set of neighbor nodes within a communication range of the current node i, and n is the total number of neighbor agents; the transition probability represents a probability of transitioning from a current state to a next state after an action is performed; the reward function is used to improve the decision efficiency of the routing decision; wherein, in a training phase of a current Q network, the next-hop node of the current node is determined according to the Q value of the current node, comprising: an improved upper confidence bound strategy is used to select the next-hop node of the current node; an expression of the upper confidence bound strategy is: wherein, denotes the number of times the state s i is visited within a fixed time, denotes the number of times the yth largest candidate forwarding node for the state s i is selected within a fixed time, Q i (s i , a i ) denotes the Q-value of the current node i, A i is the action set of the current node i. 2.The method of claim 1, wherein, the feature information of the current node is obtained, comprising: under the condition that the set of available actions of the current node is not empty, feature information is extracted from a neighbor table of the current node and a received data packet. 3.The method of claim 1, wherein, the current Q network of the deep reinforcement learning model is a four-layer neural network, the four-layer neural network has an input layer, two hidden layers, and an output layer, and a Q value function of the current Q network is determined according to a state function and an advantage function; the step of updating the current Q network comprises: based on the Q value function, a minimization loss function for updating the current Q network is constructed; based on the minimization loss function, a gradient algorithm is used to update parameters of the current Q network to generate a target Q network. 4.The method of claim 3, wherein, the updating mode of network parameters of the target Q network is soft updating. 5.The method of claim 3, wherein, in a test phase after the current Q network converges, the next-hop node of the current node is determined according to the Q value of the current node, comprising: a softmax strategy is used to select the next-hop node of the current node; an expression of the softmax strategy is: where A i is the action set of the current node i, and u is a temperature parameter.

6. A deep reinforcement learning based mobile ad hoc network routing optimization apparatus, characterized by, the method for implementing the mobile ad hoc network routing optimization method according to any one of claims 1 to 5 comprises: a first processing module for obtaining feature information of a current node; a second processing module for processing the feature information to convert a routing decision problem into a Markov decision process to determine a state space, an action space, a transition probability, and a reward function; the state space includes state information of the current node; a third processing module for inputting the state information of the current node into a preset deep reinforcement learning model to output a Q value corresponding to the current node; the deep reinforcement learning model is established based on a deep reinforcement learning network algorithm Dueling DQN; a fourth processing module for determining a next-hop node of the current node according to the Q value of the current node to make a routing decision.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, the processor executes the computer program to implement the steps of the deep reinforcement learning-based mobile ad hoc network routing optimization method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method for mobile ad hoc network routing optimization based on deep reinforcement learning according to any one of claims 1 to 5.

Images