A DQN-OLSR routing method based on deep reinforcement learning DQN

Abstract

The application discloses a DQN-OLSR routing method based on deep reinforcement learning DQN, which comprises the following steps: adding node load capacity information in a HELLO message, adding coordinate information and speed information in the HELLO message, calculating weighted link survival time, and adding node residual energy in the HELLO message; in the process of routing calculation, a routing learns and trains a Q function through deep reinforcement learning DQN according to four parameters of a node neighbor number, node load capacity, weighted link survival time and residual energy; when calculating an MPR set, the weights of the four parameters are obtained according to the Q function, the comprehensive quality of the node is calculated, and then the node with the maximum comprehensive quality is selected as an MPR node; when the node sends a data packet, a routing table is calculated based on the principle of the shortest hop number, and message forwarding is performed through the optimal MPR node. The method improves the stability of the whole network, improves the network throughput and reduces the packet loss rate.

Description

Technical Field

[0001] This invention belongs to the field of mobile ad hoc network OLSR routing technology, and specifically relates to a DQN-OLSR routing method based on deep reinforcement learning DQN (Deep Q Network). Background Technology

[0002] Optimized Link State Routing (OLSR) is an optimized version of the classic link state routing algorithm, commonly used in mobile ad hoc networks. Its most significant feature compared to traditional link state protocols is multipoint relays (MPRs). Only MPR nodes can forward control information, which significantly reduces the number of control messages in the network, lowers the overhead of routing protocols, and improves network performance. Therefore, the quality of the MPR set directly affects the overall network quality.

[0003] Each node periodically sends HELLO messages to achieve neighbor discovery and information sharing. However, topology discovery and route calculation rely on TC messages. Only MPR nodes can forward TC messages, therefore the MPR set must cover all two-hop neighbor nodes. MPR selection has been proven to be an NP-complete problem, and traditional greedy algorithms struggle to find the optimal solution.

[0004] With technological advancements and reduced production costs, drones have found wider applications in fields such as rescue and reconnaissance. Drone networks are characterized by high-speed mobility and energy constraints, making the original OLSR protocol insufficient for the needs of self-organizing drone networks. Firstly, the original OLSR protocol simply uses the number of neighboring nodes as the MPR (Multi-Level Position) selection factor, which is ill-suited to the frequent topology changes caused by high-speed movement in drone networks, thus compromising communication stability. Secondly, as energy-constrained network nodes, drones will experience energy shortages and communication interruptions if they continuously act as MPR nodes forwarding TC (Transmission Control Point) messages. Summary of the Invention

[0005] The purpose of this invention is to address the problems mentioned in the background section by providing a DQN-OLSR routing method based on deep reinforcement learning (DQN). This method proposes three parameters: node load capacity, weighted link lifetime, and remaining energy. The weights of these three parameters are calculated using the DQN algorithm, thereby determining the optimal set of maximum link ratios (MPRs). This overcomes the problem of traditional methods being unable to adapt to rapid changes in network topology. Compared to traditional OLSR routing methods, this invention significantly improves routing stability, increases network throughput, and reduces packet loss rate.

[0006] In order to achieve the object of the present application, the present application discloses a DQN-OLSR routing method based on deep reinforcement learning DQN, which specifically comprises the following steps:

[0007] Step M1, in the OLSR protocol, the node quality measurement index of the number of node neighbors is combined, three node quality measurement indexes suitable for high-speed mobile and energy-limited networks are added, the three added node quality measurement indexes include node load capacity, weighted link survival time and node residual energy; the node quality measurement index is added to the HELLO message of the node;

[0008] Step M2, the node directly discovers neighbors and shares information through the HELLO message, and calculates to obtain a neighbor table;

[0009] Step M3, the node calculates the weight of the four node quality measurement indexes by combining the neighbor table and deep reinforcement learning DQN, calculates the comprehensive quality of the node according to the weight, and then calculates the optimal MPR set from large to small according to the comprehensive quality of the node;

[0010] Step M4, the nodes in the MPR set forward TC messages to realize the discovery of the network topology and obtain a topology table;

[0011] Step M5, based on the topology table, the best routing table is calculated according to the principle of the shortest hop number, and the routing calculation is completed.

[0012] Further, the node load capacity L is calculated by the current message queue length of the node and the maximum length of the message queue:

[0013]

[0014] Wherein, S load represents the current message queue length of the node, and l represents the maximum length of the message queue.

[0015] Further, the weighted link survival time is calculated by the link survival time between nodes and the comprehensive link survival time; it is assumed that nodes O and Q are one-hop neighbor nodes, the speeds of the nodes O and Q are V O and V Q , respectively, then the speed V QO of Q relative to O is:

[0016] V QO = V Q -V O

[0017] The link survival time t OQ between the nodes O and Q is:

[0018]

[0019] where |OQ| is the distance between node O and point Q, R is the communication radius of node O, vector is the motion vector of point Q, point D is the intersection of the motion vector of point Q and the aforementioned circle, |OD| is the distance between node O and point D, and angle β is the acute angle between vector and vector ;

[0020] By adding up the link life time t Qi of all one-hop neighbors of node Q and taking the average, the comprehensive link life time T

[0021]

[0022] where N1(Q) is the set of symmetric one-hop neighbors of node Q, Num N1(Q) is the number of N1(Q);

[0023] The final weighted link life time is T OQ :

[0024]

[0025] where α=0.7 and β=0.3.

[0026] Further, the residual energy E of a node represents the energy remaining for the node to use for communication; if the residual energy of a node is small, it is not suitable to make the node an MPR node, so as to avoid the occurrence of routing errors due to the disconnection of the node from the network because of insufficient energy;

[0027] It is specified that the initial energy of a communication node is E0, and the communication node has four states when performing wireless communication, namely, sleep state Sleep, idle state Idle, message sending state Tx, and message receiving state Rx. Generally, the power consumption of the communication node in the message sending state Tx and the message receiving state Rx is high, and the power consumption of the communication node in the sleep state Sleep is low. The power consumption values of the four states are multiplied by the working time of the corresponding states, and then added up to obtain the energy E cost consumed by the node for communication:

[0028] E cost = P tx × T tx + P rx × T rx + P idle × T iale + P sleep × T sleep

[0029] where P tx , Prx , P idle , and P sleep represent the working power of the four states respectively, T tx , T rx , T idle , and T sleep represent the working time length of the four states respectively;

[0030] With the initial energy and the consumed energy, the energy E of the computing node remaining is calculated:

[0031] E = E0 - E cost

[0032] E = E0 - (P tx × T tx + P rx × T rx + P idle × T idle + P sleep × T sleep ).

[0033] Further, the calculation of the weight in step M3 by using the DQN specifically includes:

[0034] The adjustment problem of the four metric weights of the number of node neighbors, the weighted link survival time, the node load capacity, and the node remaining energy is described as a Markov Decision Process (MDP) process: each node dynamically interacts with the environment in the process of selecting the MPR node, and the best action in the current state is obtained through the reinforcement learning method; in the reinforcement learning model, the agent is each node, the environment is the entire communication network, the state is the value size of the four metric indicators, and the action is the weight size of the four metrics;

[0035] In the reinforcement learning model, the utility function is defined as:

[0036] U = α * lb(L) + β * lb(T) - γ * lb(E)

[0037] wherein L is the load capacity of the node, T is the average value of the weighted link survival time of the computing node and all MPR nodes thereof, E is the average value of the remaining energy of all MPR nodes of the computing node; α, β, and γ are the reward size coefficients of the aforementioned three parameters respectively;

[0038] It can be known from the analysis of the utility function U that it can select the neighbor node with long communication time and large remaining energy as the MPR node while ensuring that the node has strong load capacity; therefore, the energy consumption can be optimized, and the network performance can be enhanced;

[0039] The utility function U at the front and back time is subtracted, and the return function R can be defined t :

[0040]

[0041] Wherein, δ is used to adjust the threshold value of the size of the return function; the difference between the U t of the environment gives U t The size of the return; otherwise, the return is zero; when the U t of the next time is greater than the previous time, the return is positive, that is, the reward; otherwise, the return is negative, that is, the punishment;

[0042] After describing the problem as an MDP process, the deep reinforcement learning DQN is used to set the weight size of the node quality measurement value.

[0043] Further, the optimal MPR set calculation method in step M3 includes:

[0044] Set the MPR node currently being calculated as A, define N1_A as the one-hop neighbor subset of node A, N2_A as the two-hop neighbor subset of node A, and MPR_A as the MPR set of node A, and the specific calculation steps are as follows:

[0045] Step M3-1, add the nodes with always willing to forward message forwarding willingness in set N1_A to MPR_A; delete the N1_A nodes added to MPR_A, and delete the nodes in set N2_A covered by MPR_A set;

[0046] Step M3-2, if there is a node in N1_A that is the only one-hop neighbor node of a node in N2_A, add the node in set N1_A to MPR_A set; delete the N1_A nodes added to MPR_A, and delete the nodes in set N2_A covered by MPR_A set;

[0047] Step M3-3, if N2_A is not empty, that is, there are still nodes in N2_A that are not covered by MPR_A, for each node Y in set N1_A, calculate and count the number of one-hop neighbors C Y , load condition L Y , link L AY The weighted link survival time T AY of the node Y and the remaining energy E Y of the node Y;

[0048] Step M3-4, input the current state information, that is, the size of the four measurement index values, into the deep reinforcement learning DQN algorithm, and the action obtained, that is, the measurement index weight size, is output as the different weights of the four measurement indexes in step M3;

[0049] Step M3-5, according to the four metrics of step M3 and the different weights of step M4, the node comprehensive quality Comp of node Y is calculated Y :

[0050] Comp Y = a * C Y + b * L Y + g * T AY + d * E Y #

[0051] Step M3-6, the node comprehensive quality Comp of the node is selected from N1_A Y The node with the maximum node comprehensive quality Comp is added to MPR_A, the N1_A node added to MPR_A is deleted, the node covered by the MPR_A set in the set N2_A is deleted, and the judgment is returned to step M3-3 until N2_A is empty, and then the MPR_A calculation is completed.

[0052] Compared with the prior art, the significant progress of the present application is that: 1) under the premise of meeting the normal routing function, the optimal MPR set suitable for frequent changes of network topology can be calculated, the occurrence of routing errors can be significantly reduced, the data packet loss can be reduced, and the communication performance of the wireless ad hoc network can be improved; 2) the DQN-OLSR routing method based on deep reinforcement learning DQN adds three metrics of node load capacity, weighted link survival time and current node residual energy to help the calculation of the MPR set, and overcomes the shortcomings of poor routing stability and low network performance of the traditional MPR set in the high-speed mobile and energy-limited network.

[0053] To more clearly illustrate the functional characteristics and structural parameters of the present application, the following further describes the present application in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0054] The accompanying drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0055] Figure 1 is the overall framework diagram of the method of the present application;

[0056] Figure 2 is a schematic diagram of a DQN neural network;

[0057] Figure 3 is a geometric diagram for calculating the link survival time. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of the present application.

[0059] The present application will be further described below with reference to the drawings of the embodiments of the present application. Figure 1 , the drawings of the embodiments of the present application. Figure 2 and the drawings of the embodiments of the present application. Figure 3 , the drawings of the embodiments of the present application. The overall flow of the present application is specifically as follows:

[0060] Step 1, three node quality measurement indexes suitable for high-speed moving and energy-limited networks are proposed, and the indexes are added to the HELLO message of the node.

[0061] The three indexes are: node load capacity, weighted link survival time and node residual energy. The node load capacity is calculated from the current message queue length and the maximum message queue length of the node. The weighted link survival time can be calculated by mathematical modeling through node position information and speed information. The node residual energy can be calculated by the initial energy and the consumed energy.

[0062] Step 2, the node performs neighbor discovery and information sharing through the HELLO message, and calculates the neighbor table.

[0063] Step 3, the values of the four node quality measurement indexes are obtained through the HELLO message, the weight size of the indexes is calculated with the help of deep reinforcement learning DQN, and the comprehensive quality of the node is calculated according to the weight.

[0064] In the reinforcement learning model, the agent is each communication node, the agent constantly communicates with other nodes, makes the best action in different states, and calculates the best MPR set. In the reinforcement learning model, the state space is the values of the four node quality measurement indexes, and the action space is the weight size of the four node quality measurement indexes.

[0065] Step 4, start calculating MPR. Set the node currently calculating MPR as A, define N1_A as the one-hop neighbor subset of node A, N2_A as the two-hop neighbor subset of node A, and MPR_A as the MPR set of node A, and the specific calculation steps are as follows: add the nodes with always willing to forward message forwarding willingness in set N1_A to MPR_A. Delete the N1_A nodes added to MPR_A, and delete the nodes covered by MPR_A set in set N2_A.

[0066] Step 5, if a node in N1_A is the only hop neighbor node of a node in N2_A, the node in N1_A is added to the MPR_A set. The N1_A node added to MPR_A is deleted, and the nodes in N2_A covered by the MPR_A set are deleted.

[0067] Step 6, if N2_A is not empty, i.e. there are still nodes in N2_A not covered by MPR_A, the node with the maximum comprehensive quality Comp Y is added to MPR_A, the N1_A node added to MPR_A is deleted, and the nodes in N2_A covered by the MPR_A set are deleted.

[0068] Step 7, return to step 6 for judgment until N2_A is empty, and then the MPR_A calculation is completed, and the MPR set is the best MPR set suitable for high-speed mobile and energy-limited networks.

[0069] Step 8, the MPR node forwards the TC message to realize the discovery of the network topology structure and obtain a topology table.

[0070] Step 9, based on the topology table, the best routing table is calculated according to the shortest hop principle and with the aid of the optimal MPR set, and thus the DQN-OLSR routing method based on deep reinforcement learning DQN is calculated.

[0071] The technical scheme is specifically implemented as follows:

[0072] In a first aspect, the application proposes three parameters of node load capacity, weighted link survival time and node residual energy as the measurement factors for MPR set selection. The three newly proposed measurement factors can better solve the fast-moving communication scenario, and are specifically as follows:

[0073] The node load capacity L is calculated from the current message queue length S load and the maximum message queue length l of the node:

[0074]

[0075] The weighted link survival time is calculated by weighting the link survival time and the comprehensive link survival time. Assuming that nodes O and Q are one-hop neighbor nodes, and the speeds of nodes O and Q are V O and V Q respectively, then the speed V QO of Q relative to O is:

[0076] V QO = V Q -V O

[0077] Link lifetime between nodes O and Q OQ is:

[0078]

[0079] where |OQ| is the distance between point O and point Q, point D is the intersection of the motion vector of point Q and the circle with radius R, |OD| is the distance between point O and point D, and angle β is the acute angle between the vector and the vector .

[0080] As shown in the accompanying Figure 3 , the figure is a geometric illustration of the calculation of the link lifetime of point O and point Q. R is the communication radius of node O, and a circle is drawn with the communication radius R of the node. The vector is the motion vector of point Q, and point D is the intersection of the motion vector of point Q and the aforementioned circle. The acute angle between the vector and the vector is denoted as angle β. Then the distance |QD| that point Q needs to move to leave the communication range of point O can be calculated by the knowledge of triangles. Finally, the distance |QD| divided by the relative speed of point Q with respect to point O can obtain the link lifetime.

[0081] After the link lifetime is calculated, the link lifetime of a node (for example, node Q) with all its neighbor nodes is accumulated and then averaged to obtain the comprehensive link lifetime of node Q

[0082]

[0083] where N1(Q) is the symmetric one-hop neighbor set of node Q, Num N1(Q) is the number of N1(Q).

[0084] Then T OD and are added with a certain weight to obtain the weighted link lifetime T OQ between node O and node Q:

[0085]

[0086] where α = 0.7 and β = 0.3.

[0087] The node residual energy E represents the energy remaining for the current node to communicate; if the node has less energy remaining, it is not appropriate to use it as an MPR node, avoiding the occurrence of routing errors due to insufficient energy to disconnect from the network. The initial energy of the communication node is defined as E0, and the communication node has four main states when performing wireless communication, which are: sleep state (Sleep), idle state (Idle), message sending state (Tx), and message receiving state (Rx). The power consumption values of the four states are multiplied by the working time of the corresponding state, and then accumulated to obtain the energy E consumed by the node communication cost .

[0088] E cost = P tx × T tx + P rx × T rx + P idle × T idle + P sleep × T sleep

[0089] With the initial energy and the consumed energy, the residual energy E of the node can be calculated:

[0090] E = E0-E cost

[0091] E = E0-(P tx × T tx + P rx × T rx + P idle × T idle + P sleep × T sleep ).

[0092] In a second aspect, the present application uses the above-mentioned three node metric indicators to realize the calculation of the optimal MPR set by combining deep reinforcement learning DQN, and the specific implementation is as follows:

[0093] Let the current MPR node being calculated be A, define N1_A as a one-hop neighbor subset of node A, N2_A as a two-hop neighbor subset of node A, and MPR_A as the MPR set of node A, and the specific calculation steps are as follows:

[0094] (1) Add the nodes with always willing to forward message forwarding willingness in the set N1_A to MPR_A. Delete the N1_A nodes added to MPR_A, and delete the nodes in the set N2_A covered by the MPR_A set;

[0095] (2) If there is a node in N1_A which is the only hop neighbor node of a node in N2_A, add the node in N1_A to the MPR_A set. Remove the N1_A node added to the MPR_A, and remove the nodes in N2_A covered by the MPR_A set;

[0096] (3) If N2_A is not empty, that is, there are still nodes in N2_A not covered by MPR_A, for each node Y in N1_A, calculate and count the number C of one-hop neighbors of Y Y , load L Y , link L AY , weighted link survival time T AY and the remaining energy E Y of the node Y;

[0097] (4) Input the current state information, that is, the size of the four measurement indicators, into the deep reinforcement learning DQN algorithm, and the action, that is, the size of the measurement indicator weight, is output as the different weights of the four measurement indicators in step (3);

[0098] (5) According to the four measurement indicators in step (3) and the different weights obtained in step (4), the node comprehensive quality Comp Y of node Y is calculated:

[0099] Comp Y =α*C Y +β*L Y +γ*T AY +δ*E Y #

[0100] (6) Select the node with the maximum node comprehensive quality Comp Y from N1_A to join MPR_A, remove the N1_A node added to MPR_A, and remove the nodes in N2_A covered by the MPR_A set. Return to (3) for judgment until N2_A is empty, and then the MPR_A calculation is completed.

[0101] At this point, the optimal MPR set calculation is completed. The calculation method not only considers the comprehensive link survival time, and preferentially selects the node with long link survival time as the MPR to ensure the stability of communication, but also considers the load capacity and remaining energy of the node, and avoids the occurrence of network congestion or network interruption phenomenon caused by high node load or low energy.

[0102] In a third aspect, the deep reinforcement learning DQN is used to calculate the weight of the measurement factor, which is also an important part of the present application.

[0103] Reinforcement learning is an important component of machine learning, which is used to describe and solve the problem that an agent learns a strategy to maximize the return or achieve a specific goal in the process of interacting with the environment. The agent corrects the action by obtaining the return through interaction with the environment, and the behavior steps can be expressed as a Markov decision process (MDP) process. The most important thing in a Markov decision process is the five-tuple <S, A, P, R, γ>, where S is the state set, A is the action set, P is the state transition probability matrix, and R is the reward function, and γ is a discount coefficient between 0 and 1.

[0104] Specifically, we describe the adjustment problem of the four metric weights as an MDP process: each node dynamically interacts with the environment in the process of selecting the MPR node, and obtains the optimal action in the current state through reinforcement learning. In this reinforcement learning model, the agent is each node, the environment is the entire communication network, the state is the value of the four metric indicators, and the action is the weight of the four metrics.

[0105] In this reinforcement learning model, the utility function is defined as:

[0106] U = α * lb(L) + β * lb(T) - γ * lb(E)

[0107] Where L is the load capacity of the node, T is the average of the weighted link survival time of the computing node and all its MPR nodes, and E is the average of the residual energy of all MPR nodes of the computing node. α, β and γ are the reward size coefficients of the three parameters mentioned above.

[0108] From the analysis of the utility function U, it can be seen that it can select neighbor nodes with long communication time and large residual energy as MPR nodes while ensuring that the node has strong load capacity. Therefore, energy consumption can be optimized and network performance can be enhanced.

[0109] The difference between the utility functions U at the front and rear moments can be defined as the return function R t

[0110]

[0111] Where δ is used to adjust the threshold value of the return function size. When the difference between the U t at the front and rear moments is greater than δ, the environment gives the U t at the rear moment a return value proportional to the size of the difference; otherwise, the return is zero. When the U t at the rear moment is greater than the U at the front moment, the return is positive, i.e. reward; otherwise, the return is negative, i.e. punishment.

[0112] ​DQN is an advanced version of Q-learning reinforcement learning, DQN adds a neural network to fit the Q function, solves the problem that Q-learning cannot solve. The core of Q-learning is the Q table, in this invention, our state is the size of four measurement indicators, and the action is the size of four measurement indicators weight. For Q table, the dimension is large, Q-learning cannot solve this problem, so we choose to use deep reinforcement learning DQN to solve it. We only need to store the structure and parameters of the neural network, without recording the huge Q table; In addition, similar state input will get similar action output, with stronger generalization ability. The specific DQN neural network diagram is shown in the accompanying Figure 2 .

[0113] But if it is simply combined with Q-learning and neural network will bring two problems: first, the samples of neural network are independent of each other, which is different from the state correlation change of reinforcement learning; Second, the introduction of nonlinear function, using neural network to approximate Q table, the training result may not converge. In order to solve these two problems, experience pool and fixed Q-target strategy are introduced. Experience pool uses off-policy strategy, which stores the experience obtained by exploration or action in experience pool, and then target-police randomly extracts experience from experience pool to update the network, cutting off the correlation of experience and improving the utilization rate of experience. Fixed Q-target is used to speed up the convergence of training. There are two networks with the same structure but different parameters in DQN, one is used to predict Q estimate value (MainNet), and the other is used to predict Q real value (target). The prediction of real value uses the parameters between the two, and the estimate value uses the latest parameters:

[0114] targetQ=r+γ*Qmax(s′,a′,θ)

[0115] According to the loss of targetQ and Q estimate, the loss function we use is mean square error loss:

[0116] LOSS(θ)=E[((TargetQ-Q(s,a,θ)) 2 ]#

[0117] Specifically: first, initialize MainNet and target, then update the parameters of MainNet according to the loss function, note that the target is fixed; After several iterations, the parameters of MainNet are assigned to the target network, so as to realize the fixed Q-target mechanism. The above process is iterated continuously until the training converges. In this process, targetQ is fixed for a period of time, which makes the update of the algorithm more stable.

[0118] In the fourth aspect, after the optimal MPR set is calculated by the deep reinforcement learning DQN, the routing table is calculated by the topology table information. The topology table is obtained by the MPR node forwarding the TC message, and the optimal topology table can be obtained with the optimal MPR node. The routing table is calculated according to the shortest hop principle on the basis of the topology table to obtain the optimal routing path.

[0119] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one entity or action from another, without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0120] Although the embodiments of the present application have been shown and described, it is to be understood that various changes, modifications, substitutions, and alterations can be made to the embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A DQN-OLSR routing method based on deep reinforcement learning DQN, characterized in that, Specifically comprising the following steps: Step M1, combining the node quality measurement index of the number of node neighbors in the OLSR protocol, adding three node quality measurement indexes suitable for high-speed mobile and energy-limited networks, the three added node quality measurement indexes include node load capacity, weighted link survival time, and node residual energy; the three added node quality measurement indexes are added to the HELLO message of the node; Step M2, the node directly discovers neighbors and shares information through the HELLO message, and calculates a neighbor table; Step M3, the node calculates the weights of the four node quality measurement indexes by combining the neighbor table and deep reinforcement learning DQN, calculates the comprehensive quality of the node according to the weights, and then calculates the optimal MPR set according to the comprehensive quality of the node from large to small; Step M4, the nodes in the MPR set forward the TC message to realize the discovery of the network topology and obtain a topology table; Step M5, based on the topology table, the best routing table is calculated according to the shortest hop principle, and the routing calculation is completed; The node load capacity L is calculated from the current message queue length and the maximum length of the message queue of the node: where S load denotes the current message queue length of the node, and l denotes the maximum length of the message queue. 2.The DQN-OLSR routing method based on deep reinforcement learning DQN of claim 1, wherein, The weighted link lifetime is calculated by the link lifetime and the comprehensive link lifetime between nodes; assuming that nodes O and Q are one-hop neighbor nodes, and the speeds of nodes O and Q are V O and V Q respectively, then the speed V QO of Q relative to O is: V QO =V Q -V O Link lifetime t between nodes O, Q OQ is: where |OQ| is the distance between node O and point Q, the circle is drawn with the communication radius R of node O, vector is the motion vector of point Q, point D is the intersection of the motion vector of point Q and the aforementioned circle, |OD| is the distance between node O and point D, and angle β is the acute angle between vector and vector ; By accumulating the link lifetimes t of all one-hop neighbors of node Q Qi The integrated link lifetime can be calculated by accumulating and averaging where N1(Q) is the set of symmetric one-hop neighbors of node Q, Num N1(Q) is the number of N1(Q); The final weighted link lifetime is T OQ : Wherein, α=0.7, β=0.

3. 3.The DQN-OLSR routing method based on deep reinforcement learning DQN of claim 1, wherein, The node residual energy E represents the residual energy of the current node for communication; The initial energy of the communication node is E0, the communication node includes four states when performing wireless communication, which are sleep state Sleep, idle state Idle, message sending state Tx, and message receiving state Rx; the power consumption values of the four states are multiplied by the working time of the corresponding state, and then accumulated to obtain the energy E consumed by the node communication cost : E cost = P tx x T tx + P rx x T rx + P idle x T idle + P sleep x T sleep where P tx , P rx , P idle and P sleep represent the working power of the four states respectively, T tx , T rx , T idle and T sleep represent the working time length of the four states respectively. According to the initial energy and the consumed energy, the residual energy E of the node is calculated: E = E0- E cost E = E0- (P tx x T tx + P rx x T rx + P idle x T idle + P sleep x T sleep ).

4. The DQN-OLSR routing method based on deep reinforcement learning DQN of claim 1, characterized in that, In step M3, the DQN is used to calculate the weights, which specifically includes: The adjustment of the weights of the four measurement indexes is described as a Markov decision MDP process: in the process of selecting the MPR node, each node dynamically interacts with the environment, and the best action in the current state is obtained through deep reinforcement learning method; in the reinforcement learning model, the agent is each node, the environment is the entire communication network, the state is the value of the four measurement indexes, and the action is the weight of the four measurement indexes; In the reinforcement learning model, the utility function is defined as: U=α*lb(L)+β*lb(T)-γ*lb(E) Wherein, L is the load capacity of the node, T is the average value of the weighted link survival time of the computing node and all MPR nodes of the node, E is the average value of the residual energy of all MPR nodes of the computing node; α, β and γ are the reward size coefficients of the three aforementioned parameters; The difference between the utility functions U at the preceding and succeeding times can define the return function R t : Wherein, δ is used to adjust the threshold of the return function size; the current and the next time U t When the difference is greater than δ, the environment gives U t The difference size reward; otherwise, the reward is zero; when the next time U t Is greater than the previous time, the reward is positive, that is, the reward; otherwise, the reward is negative, that is, the punishment; After describing the problem as an MDP process, the deep reinforcement learning DQN is used to set the weight size of the node quality measurement value. 5.The DQN-OLSR routing method based on deep reinforcement learning DQN of claim 4, wherein, The optimal MPR set calculation method in step M3 includes: Set the MPR node currently being calculated as A, define N1_A as a one-hop neighbor subset of node A, N2_A as a two-hop neighbor subset of node A, and MPR_A as the MPR set of node A; the specific calculation steps are as follows: Step M3-1, add the nodes with always willing to forward message forwarding willingness in set N1_A to MPR_A; delete the N1_A nodes added to MPR_A, and delete the nodes covered by the MPR_A set in set N2_A; Step M3-2, if a node in N1_A is the only hop neighbor node of a node in N2_A, add the node in N1_A to the MPR_A set; delete the N1_A node added to the MPR_A, and delete the node in the set N2_A covered by the MPR_A set; Step M3-3, if N2_A is not empty, i.e. there are still nodes in N2_A not covered by MPR_A, for each node Y in set N1_A, calculate and count the one-hop neighbor number C of Y Y , load situation L Y , link L AY weighted link survival time T AY and the residual energy E of node Y Y ; Step M3-4, input the current state information, i.e. the size of the four measurement index values, into the deep reinforcement learning DQN algorithm, and obtain the action, i.e. the size of the measurement index weight, as the different weights of the four measurement indexes in step M3. Step M3-5, according to the four measurement indexes of step M3 and the different weights of step M4, the node comprehensive quality Comp of node Y is calculated Y : Comp Y = a * C Y + b * L Y + g * T AY + d * E Y Step M3-6, select node comprehensive quality Comp from N1_A Y The maximum node joins MPR_A, delete N1_A node which joins MPR_A, delete node which is covered by MPR_A set in set N2_A; return to step M3-3 to judge until N2_A is empty, then MPR_A calculation ends.

Images