A network state-based unmanned cluster routing protocol switching method

Abstract

The present application relates to the field of unmanned cluster routing protocol switching, and provides a network state-based unmanned cluster routing protocol switching method, the main modules of which can be divided into a network state collection module, a key parameter analysis module and a switching threshold optimization module. The main process is as follows: based on the current network state information of the unmanned aerial vehicle node, whether the business transmission demand can be met is analyzed, the feedback and reinforcement of the continuous perception of the network state environment and the adjustment of the routing protocol switching threshold are based on reinforcement learning, the optimal routing protocol switching threshold suggestion is generated, the applicable routing protocol based on the current network state analysis is realized, and smooth switching is realized. The present application designs a double-threshold switching method to reduce the routing protocol oscillation caused by the fluctuation near a certain threshold. The method provided by the present application can complete flexible and adaptive switching between different routing protocols according to the dynamic changes of the network environment, support intelligent adaptation of the unmanned cluster routing protocol, and communication guarantee.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned cluster routing protocol switching, in particular to a network state-based unmanned cluster routing protocol switching method. BACKGROUND

[0002] With the accelerated application of frontier technology in the military field, unmanned cluster system has become an important part of modern war, and major military powers in the world have attached great importance to the application of intelligent unmanned system technology in the military field. In recent years, the combat performance and lethality of unmanned cluster combat platform have been significantly improved, and it can replace people to complete various combat tasks. The application of unmanned cluster combat platform in modern war has obvious advantages, such as strong anti-detection ability, long working duration, no concern for personnel casualties, strong operability, strong detection ability, etc. It has been gradually applied to perform various dangerous and complex tasks, and can provide new combat forces for future unmanned combat intelligence reconnaissance, surveillance and strike, communication support, etc.

[0003] Unmanned cluster ad hoc network has excellent characteristics such as no center, no dependence on fixed infrastructure and fast formation, but also has the following limitations: variable network environment, dynamic topology change, unreliable wireless link and poor security. Therefore, the routing protocol algorithm in traditional wired network cannot be directly used in unmanned cluster ad hoc network. Different types of routing protocols have their own advantages and disadvantages, and each routing protocol needs to be matched with a specific network scenario to maximize its performance. A single routing protocol is difficult to maintain good performance in a complex and variable network scenario. Therefore, it is of great significance to introduce a flexible and adaptive switching strategy in the routing protocol. Adaptive routing protocol switching is based on dynamic perception of network state, generates routing protocol switching threshold recommendation, matches the most suitable routing protocol for the network, so that each routing protocol can play to its strengths in its application scenario and ensure network performance. SUMMARY

[0004] The present application focuses on solving the problem that a single routing protocol in an unmanned cluster system with dynamic and variable topology and unstable communication cannot guarantee business transmission requirements, and proposes a network state-based unmanned cluster routing protocol switching method, which can be applied to flexible adaptation of various routing protocols and current network state.

[0005] The technical scheme adopted by the present application is as follows:

[0006] A network state-based unmanned cluster routing protocol switching method, comprising the following steps:

[0007] Step 1: Collecting the network state information of the UAV node at a set time interval, and integrating the collected local neighbor table and topology table information, accumulating the path interruption times and calculating the interruption time, generating a local information library containing network key situation characteristics, i.e. topology record table and path interruption record table;

[0008] Step 2: Extracting the key parameter features in the local information library, calculating the path average interruption time in the corresponding statistical period, defining the current communication state of the network by combining the path interruption times and the path average interruption time, and defining the reward function of the current UAV node selection action in reinforcement learning in combination with the network packet loss rate;

[0009] Step 3: The UAV node interacts with the network state environment through the reinforcement learning method of Q-learning, and selects the threshold value of the UAV node switching routing protocol as the action in reinforcement learning. Each state-action pair corresponds to a Q value. The Q value saves the cumulative reward value of taking action a in state s. Based on the reward function feedback, it is iterated multiple times until the learning converges, and finally the optimal threshold value is generated, including the upper limit switching threshold value and the lower limit switching threshold value;

[0010] Step 4: Based on the current network communication state of the UAV node, the applicability judgment and switching of the routing protocol are performed according to the upper limit switching threshold value and the lower limit switching threshold value generated in step 3.

[0011] Further, the topology record table in step 1 contains topology tuples (T_record, T_now, addr_dest, addr_last, T_time), which respectively represent the initial record timestamp, the update timestamp, the destination node main address, the next hop node main address and the valid time, and a path is uniquely determined by the sequence number and the destination node main address; The path interruption record table contains the destination node main address, the next hop node main address, the interruption time array and the interruption times. The destination node and the next hop node main address are consistent with the topology record table. The interruption time array records the duration of each path interruption, and the interruption times are accumulated after each path interruption;

[0012] For the topology set of OLSR protocol, each topology tuple (T_record, T_now, addr_dest, addr_last, T_time) is recorded. In each update, the topology tuples in the topology record table are analyzed one by one. If the topology tuple in the topology record table exists in the latest topology set, it indicates that the path state is normal, and the valid time T_time of the topology tuple in the topology record table is updated; if the topology tuple has been deleted from the latest topology set, it indicates that the path state is abnormal, and the path has been interrupted. Set the valid time T_time to a negative value, and accumulate the interruption times i tAnd the interruption time I_time is calculated as follows:

[0013] I_time = T_now - (T_time + T_record)

[0014] The effective time T_time greater than zero indicates that the path state is normal and uninterrupted, and the effective time T_time less than zero indicates that the path is interrupted, and the path information is recorded in the path interruption record table;

[0015] According to the topology record table and the path interruption record table, the characteristic parameters, i.e., the interruption frequency i t and the average interruption time AI_time, are obtained, and the average interruption time is calculated as follows:

[0016]

[0017] Further, in step 2, the current communication state of the network is The reward at time t is Where i t is the path interruption frequency, AI_time is the average interruption time, p t is the packet loss rate, ω1, ω2, ω m and ω n are weight parameters with values between 0 and 1.

[0018] Further, step 3 is specifically:

[0019] The switching threshold optimization of the routing protocol is performed using a Q-learning-based reinforcement learning method, and the specific process is as follows:

[0020] Step 301: initialization, using random numbers to initialize the Q value table, while initializing the learning rate α, the discount factor γ, the convergence judgment condition δ and the greed coefficient ε, and selecting the threshold value of the unmanned aerial vehicle node as the action in reinforcement learning, each state-action corresponds to a Q value, and the Q value saves the cumulative reward value of taking action a in state s;

[0021] Step 302: when each new task arrives, observe the network state s t at the current time t, the network state s t includes the interruption frequency and the packet loss rate in the last statistical period;

[0022] Step 303: judge whether the Q value table converges, if ΔQ(s, a) > δ, the convergence condition is not met, according to the environment state s t at the current time, select action a t according to the probability 1-ε; if ΔQ(s, a) < δ, the convergence condition is met, select the action a t; wherein, AQ(s, a) is the difference between the Q value before and after the update;

[0023] Step 304: performing action a t , i.e. adjusting the routing protocol according to the selected upper switching threshold and lower switching threshold, calculating the reward r t , and obtaining the next state s t+1 of the network.

[0024] Step 305: updating the Q value table, repeating steps 302 to 304.

[0025] wherein the Q value function is updated as follows:

[0026] Q(s t , a t )←Q(s t , a t )+α[r t +γmaxQ(s t+1 , a t )-Q(s t , a t )]

[0027] wherein Q(s t , a t ) represents the action value at the current time t in state s t , taking action a t , reaching state s t+1 at t+1 and obtaining reward r t+1 , at this time, finding the action a in the Q value table that can make the value in state s t+1 maximum, obtaining Q(s t+1 , a), updating Q(s t , a t ).

[0028] Further, step 4 is specifically:

[0029] calculating the current network communication state state, and judging the applicability of the routing protocol:

[0030] if state is higher than the upper switching threshold, indicating that the network communication state is good, meeting the service transmission quality requirement, using OLSR routing protocol;

[0031] if state is lower than the upper switching threshold, and higher than the lower switching threshold, indicating that the network communication state is poor, intermittent, maintaining OLSR routing protocol;

[0032] if state is lower than the lower switching threshold, indicating that the network communication state is extremely poor, unable to complete service transmission, switching to DTN routing protocol.

[0033] Compared with the prior art, the beneficial effects achieved by this invention are as follows:

[0034] This invention generates switching thresholds for routing protocols by analyzing network environment parameters, thereby improving the communication robustness and adaptability of unmanned swarm systems under different task scenarios. Furthermore, this invention designs a dual-threshold switching method to reduce routing protocol oscillations caused by fluctuations around a certain threshold. It is applicable to switching between multiple different routing protocols. Taking Optimized Link State Routing (OLSR) and Delay / Disruption Tolerant Network (DTN) as examples: OLSR is used when network quality is good / poor; when network quality is extremely poor and basic communication is difficult to maintain, the method switches to DTN. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the execution of a network-state-based unmanned cluster routing protocol switching method according to the present invention.

[0036] Figure 2 This is a schematic diagram of a threshold optimization method based on reinforcement learning according to the present invention.

[0037] Figure 3 This is a flowchart of a routing protocol switching method based on dual thresholds according to the present invention.

[0038] Figure 4 This is a flowchart of a threshold optimization algorithm based on Q-learning according to the present invention.

[0039] Figure 5 This is a schematic diagram of the network nodes in the simulation scenario of the present invention.

[0040] Figure 6 This is a schematic diagram of the routing protocol switching process in the simulation scenario of this invention.

[0041] Figure 7 This is a performance comparison chart of the routing strategy under the simulation scenario of this invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] like Figure 1 As shown, a method for switching unmanned cluster routing protocols based on network status is implemented using a network status collection module, a key parameter analysis module, and a switching threshold optimization module. Specifically, it includes the following processes:

[0044] (1) Network state collection module, through the statistics unmanned aerial vehicle node periodically perceive its own network state information, and the collected local neighbor table and topology table information integration, generate containing network key situation characteristics of local information library, namely topology record table and path interruption record table.

[0045] Topology record table entry (T_record, T_now, addr_dest, addr_last, T_time) is shown in Table 1, a path is uniquely determined by the sequence number and the destination node main address, there may be multiple paths with the same destination node and different next hop nodes, and the valid time greater than zero indicates that the path state is normal and not interrupted; the valid time less than zero indicates that the path has been interrupted, and the path information is recorded in the interruption record table.

[0046] Table 1 Topology record table entry

[0047]

[0048] Path interruption record table entry is shown in Table 2, including destination node main address, next hop node main address, interruption time array and interruption times, the destination node main address and the next hop node main address are consistent with the topology record table, the interruption time array records the duration of each path interruption, and the interruption times are accumulated after each path interruption.

[0049] Table 2 Path interruption record table entry

[0050]

[0051] For the topology set of OLSR protocol, each topology tuple (T_record, T_now, addr_dest, addr_last, T_time) is recorded. In each update, the topology tuples in the topology record table are analyzed one by one, if the topology tuple in the topology record table exists in the latest topology set, it indicates that the path state is normal, and the valid time T_time of the topology tuple in the topology record table is updated; if the topology tuple has been deleted from the latest topology set, it indicates that the path state is abnormal, it is considered that the path has been interrupted, the valid time T_time is set to negative value, and the topology tuple information is recorded in the path interruption record table, and the interruption times i are accumulated t And calculate the interruption time I_time. The interruption time is calculated as follows:

[0052] I_time = T_now - (T_time + T_record)

[0053] The valid time T_time greater than zero indicates that the path state is normal and not interrupted, and the valid time T_time less than zero indicates that the path has been interrupted, and the path information is recorded in the path interruption record table.

[0054] The characteristic parameters, i.e. the interruption times i t and the average interruption time AI_time, are obtained according to the topological record table and the path interruption record table, and the average interruption time is calculated as follows:

[0055]

[0056] (2) The key parameter analysis module extracts the key parameter characteristics in the local information library according to the perceived network state of the unmanned aerial vehicle node, calculates the path average interruption time in the corresponding statistical period, and defines the current network communication state state In reinforcement learning, the reward function of the current unmanned aerial vehicle node selection action is defined in combination with the network service transmission state characteristics, i.e. the packet loss rate The higher the reward value, the better the network state feedback obtained by the current action; wherein i t is the path interruption times, AI_time is the average interruption time, p t is the packet loss rate, ω1, ω2, ω m and ω n are weight parameters with values between 0 and 1.

[0057] (3) The switching threshold optimization module, the unmanned aerial vehicle node interacts with the network state environment through the reinforcement learning method of Q-learning, and the threshold selection of the unmanned aerial vehicle node switching routing protocol is taken as the action in reinforcement learning. Each state-action pair corresponds to a Q value, and the Q value saves the cumulative reward value of taking action a in state s. Based on the reward function feedback, it is iterated multiple times until the learning converges, and finally the optimal threshold is generated, including the upper limit switching threshold and the lower limit switching threshold, as shown in Figure 2 The switching threshold optimization problem can be described as a Markov decision process. After the agent (unmanned aerial vehicle node) performs a certain action, the environment is converted to a new state and gives a reward, and then the agent performs a new action according to the information state and the reward feedback from the environment, as shown in Figure 4 The specific algorithm is described as follows:

[0058] Step 301: initialization, using random numbers to initialize the Q value table, and initializing the learning rate α, the discount factor γ, the convergence judgment condition δ and the greed coefficient ε at the same time, and taking the threshold selection of the unmanned aerial vehicle node as the action in reinforcement learning. Each state-action corresponds to a Q value, and the Q value saves the cumulative reward value of taking action a in state s;

[0059] Step 302: when each new task arrives, observe the network state s t at the current time t, the network state st This includes the number of interruptions and packet loss rate in the previous statistical period;

[0060] Step 303: Determine if the Q-value table has converged. If ΔQ(s, a) > δ, convergence is not satisfied. Then, based on the current environmental state s... t Choose action a according to probability 1-ε t If ΔQ(s, a) < δ, the convergence condition is satisfied, and the action a with the largest Q value is selected. t Where ΔQ(s, a) is the difference between the Q value before and after the update;

[0061] Step 304: Perform action a t That is, adjusting the routing protocol according to the selected upper and lower handover thresholds, and calculating the reward r. t And obtain the next network state s t+1 ;

[0062] Step 305: Update the Q-value table, repeat steps 302 to 304;

[0063] The Q-value function is updated as follows:

[0064] Q(s t a t )←Q(s t a t )+α[r t +γmaxQ(s t+1 a t )-Q(s t a t )]

[0065] Where Q(s) t a t () indicates that at the current time t, the state is s. t The value of the action, taking action a t It then reaches state s at time t+1. t+1 And receive a reward r t+1 At this point, find the value in the Q-value table that enables the state s t+1 Take the most valuable action 'a' to obtain Q(s). t+1 ,a), for Q(s) t a t )renew.

[0066] (4) The switching threshold optimization module, based on the current network communication status of the UAV node, performs a routing protocol applicability judgment and switching according to the upper and lower switching thresholds generated in step (3). Figure 3 As shown, the algorithm flow is as follows:

[0067] The current network communication state state is calculated, and the applicability of the routing protocol is determined:

[0068] If the state is higher than the upper switching threshold, it indicates that the network communication state is good, and the service transmission quality requirement is met, and the OLSR routing protocol is used.

[0069] If the state is lower than the upper switching threshold and higher than the lower switching threshold, it indicates that the network communication state is poor, and the OLSR routing protocol is maintained.

[0070] If the state is lower than the lower switching threshold, it indicates that the network communication state is extremely poor, and the service transmission cannot be completed, and the DTN routing protocol is switched.

[0071] Embodiment:

[0072] In the simulation network environment, four unmanned aerial vehicle nodes are configured, as shown in Figure 5 , which are source node S, relay nodes R1 and R2, and destination node D, each intelligent computing unmanned aerial vehicle node uses a simulation computer instead, each node is configured with two wireless network cards, and two working channels are set respectively, and the network interfaces of all nodes are added to the local area network.

[0073] Table 3 Network node configuration table

[0074] Node name Function EID identification IP address identification S Packet sending / / node1 10.22.0.1 / 192.168.0.1 R1 Packet relay forwarding / / node2 10.22.0.2 / 192.168.0.2 R2 Packet relay forwarding / / node3 10.22.0.3 / 192.168.0.3 D Packet receiving / / node4 10.22.0.4 / 192.168.0.4

[0075] The parameter configuration of the OLSR protocol is shown in the following table:

[0076] Table 4 OLSR protocol parameter configuration item

[0077] Link type mesh Link quality algorithm etc_ff Link quality decay factor 0.05 Hello message broadcast interval 1s Hello message valid time 20s TC message broadcast interval 2s TC message valid time 20s

[0078] Based on Q-learning, the threshold value learning is performed through 10000 iterations, the upper threshold is 0.3, and the lower threshold is 0.1, that is, when the network quality is higher than 0.3, it is considered that the communication quality is good, and the OLSR should be used for data packet transmission; when the network quality is lower than 0.3 and higher than 0.1, it is considered that the communication quality is poor, and the OLSR routing protocol is still used for data packet transmission; when the network quality is lower than 0.1, it is considered that the communication quality is extremely poor, and the DTN routing protocol should be used for data packet transmission.

[0079] The initial state is that the node S and the nodes R1 and R2 are all in the indoor, the mobile node D is arranged outdoors to make the node D and the nodes R1 and R2 not in communication with each other, the distance is pulled apart, and the node D is not in communication with the three nodes in the indoor. The nodes in the network run the OLSR routing protocol and the threshold switching algorithm simultaneously, at the beginning of the simulation, data packets with the destination node D are sent at a data packet rate of 500kbps at the source node S, and the routing information of the source node is observed and recorded.

[0080] At this time, the data packets are transmitted through the DTN routing protocol.

[0081] Current routing mode DTN routing protocol

[0082] After 10s of the beginning of the experiment, the mobile destination node D is moved, the node D is made to be in communication with the relay nodes R1 and R2 after 5s, and not in direct communication with the source node S, the routing information is observed and recorded, at this time, the data packets are transmitted through the OLSR routing protocol.

[0083] Current routing mode OLSR routing protocol Current selected path S→R2→D

[0084] After maintaining the above state for 15s, the destination node D is moved again to make the node D out of the network, after that, the state is maintained for 10s, at this time, the routing protocol at the source node S is switched to the DTN mode again, the experiment is ended, and the source node stops sending data packets.

[0085] Current routing mode DTN routing protocol

[0086] The switching process of the routing mode in the simulation experiment is as shown in Figure 6 .

[0087] The data packet delivery rate of the routing protocol switching strategy of the application technology is compared with the data packet delivery rate of the routing protocol strategy of keeping OLSR as shown in Figure 7 . It can be verified that the routing protocol switching method based on the network state can effectively reduce the loss of data packets and improve the data packet delivery rate due to the timely switching of the routing protocol according to the communication state of the network nodes, and has obvious performance improvement compared with the data packet transmission strategy of keeping OLSR routing protocol all the time.

Claims

1. A network status based unmanned cluster routing protocol switching method, characterized in that, Comprising the following steps: Step 1: Collecting the network state information of the unmanned aerial vehicle node network at a set time interval, and integrating the collected local neighbor table and topology table information, accumulating the path interruption times and calculating the interruption time, generating a local information library containing network key situation characteristics, namely a topology record table and a path interruption record table; Step 2: Extracting the key parameter features in the local information library, calculating the average path interruption time in the corresponding statistical period, defining the current communication state of the network by combining the path interruption times and the average path interruption time, and defining the reward function of the current unmanned aerial vehicle node action in reinforcement learning in combination with the network packet loss rate; Step 3: the unmanned aerial vehicle node interacts with the network state environment through the reinforcement learning method of Q-learning, and the threshold selection of the unmanned aerial vehicle node switching routing protocol is taken as the action in the reinforcement learning, each state-action pair corresponds to a Q value, the Q value saves the cumulative reward value of taking action a t at the current time network communication state s t , and the iteration is fed back multiple times based on the reward function until the learning converges, and finally the optimal threshold is generated, including the upper limit switching threshold and the lower limit switching threshold; Step 4: Based on the network communication state of the current unmanned aerial vehicle node, the upper and lower switching thresholds generated in step 3 are used to judge and switch the applicability of the routing protocol.

2. The network state based unmanned cluster routing protocol switching method according to claim 1, characterized in that, The topology record table in step 1 contains topology tuples (T_record, T_now, addr_dest, addr_last, T_time), which represent the initial record timestamp, update timestamp, destination node main address, next hop node main address and valid time, respectively, and a path is uniquely determined by the sequence number and destination node main address; The path interruption record table contains the destination node main address, the next hop node main address, the interruption time array and the interruption times, the destination node and the next hop node main address are consistent with the topology record table, the interruption time array records the duration of each path interruption, and the interruption times are accumulated after each path interruption; For the topology set of OLSR protocol, record each topology tuple (T_record, T_now, addr_dest, addr_last, T_time). Analyze the topology tuple in the topology record table piece by piece at each update. If the topology tuple in the topology record table exists in the latest topology set, it indicates that the path state is normal, and the valid time T_time of the topology tuple in the topology record table is updated. If the topology tuple has been deleted from the latest topology set, it indicates that the path state is abnormal, and the path has been interrupted. Set the valid time T_time as a negative value, and accumulate the interruption times i t And calculate the interruption time I_time. The interruption time is calculated as follows: I_time = T_now - (T_time + T_record) The valid time T_time greater than zero indicates that the path state is normal and uninterrupted, and the valid time T_time less than zero indicates that the path has been interrupted, and the path information is recorded in the path interruption record table; The characteristic parameters, i.e. the number of interruptions i and the average interruption time AI_time, are obtained from the topology record table and the path interruption record table t and the average interruption time AI_time, which is calculated as follows:

3. The network state based unmanned cluster routing protocol switching method according to claim 1, wherein, In step 2, the network communication state at the current time t is The reward at time t is where i t is the number of path interruptions, AI_time is the average interruption time, p t is the packet loss rate, ω1, ω2, ω m and ω n are weight parameters with values between 0 and 1.

4. The network state based unmanned cluster routing protocol switching method according to claim 3, characterized in that, Step 3 is specifically: The Q-learning-based reinforcement learning method is used to optimize the switching threshold of the routing protocol, and the specific process is as follows: Step 301: initialization, initializing the Q value table using a random number, while initializing the learning rate a, the discount factor g, the convergence judgment condition d and the greed coefficient e, and selecting the threshold value of the unmanned aerial vehicle node as the action in reinforcement learning, each state-action corresponds to a Q value, and the Q value saves the cumulative reward value of taking action a t at the current time network communication state s t ; Step 302: observing the network communication state s at the current time t t The network communication state s t includes the number of interruptions and the packet loss rate in the last statistical period; Step 303: judging whether the Q value table converges, if AQ(s t ,a t ) < δ, the convergence condition is satisfied, and the action with the maximum Q value is selected; if AQ(s t ,a t ) > δ, it is not converged, and step 304 is executed; wherein AQ(s t ,a t ) is the difference between before and after the Q value is updated. Step 304: according to the network communication state s t , the action a with the maximum Q value is selected according to the probability 1-ε t , the action a is performed according to the upper limit switching threshold and the lower limit switching threshold t , the reward r is calculated t , the routing protocol is adjusted according to the selected upper limit switching threshold and the lower limit switching threshold, and the network state feedback is obtained, the Q value table is updated according to the new network communication state s t+1 , and the step 303 is returned; Wherein, the Q value function is updated as follows: Q(s t ,a t )←Q(s t ,a t )+α[r t +γmaxQ(s t+1 ,a t )-Q(s t ,a t )] where Q(s t ,a t ) represents the action value of the current time t in the network communication state s t , and after taking the action a t , the network communication state s t+1 is reached at t+1 and the reward r t+1 is obtained, at this time, the action a t+1 that can make the value maximum in the network communication state s t is found in the Q value table, Q(s t+1 ,a t ) is obtained, and Q(s t ,a t ) is updated.

5. The network state based unmanned cluster routing protocol switching method according to claim 1, wherein, Step 4 is specifically: calculating a network communication state s at the current time t and judging the applicability of the routing protocol: If s t If the network communication state is good, the upper limit switching threshold is higher, which means that the OLSR routing protocol is used to meet the service transmission quality requirement. if s t Below the upper switching threshold, while higher than the lower switching threshold, indicates that the network communication state is poor, time to time, maintain OLSR routing protocol; If s t Below the lower threshold, it means the network communication state is very poor, and the service transmission cannot be completed. The DTN routing protocol is switched.

Images