An intelligent backoff method for UAV multi-hop SPMA network based on deep reinforcement learning

Abstract

The present invention proposes an intelligent backoff method for a multi-hop SPMA network of an unmanned aerial vehicle based on deep reinforcement learning, comprising: taking channel state information as input of an intelligent backoff network in the multi-hop SPMA network, wherein the channel state information includes channel occupancy and its historical value, and the number of data packets to be forwarded in each priority queue; outputting a backoff duration according to the intelligent backoff network, and updating the channel state of the intelligent backoff network; evaluating the backoff effect according to the updated channel occupancy, evaluating the data packet forwarding success rate according to the updated number of data packets to be forwarded in each priority queue, and obtaining a reward value according to the backoff effect and the data packet forwarding success rate; calculating the maximum reward value corresponding to different actions through the intelligent backoff network, selecting the action with the largest reward value as the current input, and training the intelligent backoff network until convergence; obtaining target channel state information, inputting the target channel state information into the trained intelligent backoff network, and obtaining an optimal backoff strategy.

Description

Technical Field

[0001] The present invention belongs to the field of mobile ad hoc networks. Background Art

[0002] Compared to traditional media access control (MAC) protocols such as carrier sense multiple access (CSMA) and time division multiple access (TDMA), the statistical priority multiple access (SPMA) protocol, based on multi-priority services, can set different sending thresholds for services of different priorities, ensuring low latency and high reliability for high-priority services. Therefore, SPMA has attracted increasing attention in the field of tactical network communications.

[0003] In SPMA, data packets are classified by priority, and the threshold of each priority queue is usually set to a fixed value, so that high-priority traffic occupies a large amount of bandwidth resources to meet its QoS (quality of service) requirements, and correspondingly low-priority traffic will lose a certain amount of broadband resources.

[0004] The backoff conditions of SPMA based on channel occupancy statistics are different from those of traditional MAC protocols. Traditional backoff algorithms, such as binary exponential backoff (BER) and multiplicative increasing linear descent (MILD) algorithms, are designed based on actual collisions and are therefore not suitable for SPMA.

[0005] Furthermore, existing SPMA backoff algorithms are calculated based on full-pass networks and fail to consider the packet forwarding requirements of multi-hop networks. Compared to full-pass networks, multi-hop networks require multiple forwarding cycles before reaching their destination. The backoff time for forwarded packets should be minimized to ensure the delay between the source and destination. However, existing SPMA backoff algorithms do not consider the forwarding rate as a parameter in backoff calculation. Therefore, SPMA backoff algorithms based on full-pass networks are no longer suitable for multi-hop networks.

[0006] This paper studies the setting of backoff time in multi-hop transmission networks and uses the DQN algorithm to generate a more reasonable backoff duration based on the channel occupancy rate, its historical value, and the number of forwarded data packets in the queue. This ensures the success rate of data packets sent by each node after backoff, especially the success rate and delay of data packet forwarding, thereby improving network throughput. Summary of the Invention

[0007] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0008] To this end, the purpose of the present invention is to propose an intelligent backoff method for drone multi-hop SPMA network based on deep reinforcement learning, which is used to improve the success rate of multi-hop SPMA data packets, reduce data packet transmission delay and improve the total network throughput.

[0009] To achieve the above objectives, the first embodiment of the present invention proposes an intelligent backoff method for a drone multi-hop SPMA network based on deep reinforcement learning, comprising:

[0010] Channel state information is used as input to an intelligent backoff network in a multi-hop SPMA network, wherein the channel state information includes channel occupancy and its historical value, and the number of packets to be forwarded in each priority queue;

[0011] Outputting a backoff duration according to the intelligent backoff network and updating a channel state of the intelligent backoff network;

[0012] Evaluate the backoff effect according to the updated channel occupancy rate, evaluate the packet forwarding success rate according to the updated number of packets to be forwarded in each priority queue, and obtain a reward value based on the backoff effect and the packet forwarding success rate;

[0013] Calculating the maximum reward values ​​corresponding to different actions through the intelligent backoff network, selecting the action with the largest reward value as the current input, and training the intelligent backoff network until convergence;

[0014] Target channel state information is obtained, and the target channel state information is input into a trained intelligent backoff network to obtain an optimal backoff strategy.

[0015] In addition, the intelligent backoff method for a drone multi-hop SPMA network based on deep reinforcement learning according to the above embodiment of the present invention may also have the following additional technical features:

[0016] Furthermore, in one embodiment of the present invention, using the channel state information as input to the intelligent backoff network in the multi-hop SPMA network includes:

[0017] The channel occupancy rate of the sending node in the previous several time slots and the number of packets to be forwarded in each priority queue are used as the input of the DQN model of the intelligent backoff network through the intelligent backoff network. The input state is s t ={c t ,c t-1 ,...,c t-5 ,k0,k1,…,k M-1}, where t is the current time, tj is the j time slots before the current time; c t is the channel occupancy rate at time t; k j is the number of packets to be forwarded in the j-priority queue.

[0018] Furthermore, in one embodiment of the present invention, outputting a backoff duration according to the intelligent backoff network and updating a channel state of the intelligent backoff network includes:

[0019] The intelligent backoff network interacts with the environment to obtain the state s t ={c t ,c t-1 ,...,c t-5 ,k0,k1,…,k M-1}, then take action a t ∈{0,1,2,…,M-1} to generate the backoff time, where M represents the number of actions. The backoff time expression is as follows:

[0020]

[0021] Where T is the length of a time slot.

[0022] Furthermore, in one embodiment of the present invention, the reward function of the intelligent backoff network is defined as:

[0023]

[0024] Where C is the number of all packets in the priority queue at the start of backoff; ΔN is the change in the number of forwarded packets before and after backoff; B indicates whether the packet can be sent after backoff, 1 indicates it can be sent, and 0 indicates it cannot be sent; t is the time when backoff is completed; t s Indicates the time when the retreat starts; T th It represents the backoff time threshold for obtaining the upper limit reward; ρ is the decay coefficient.

[0025] Furthermore, in one embodiment of the present invention, the update formula of the reward value is defined as: Where Q(s t ,a t ) is in a given state s t Next, perform action a t The obtained Q value is the reward value; α is the learning rate; r t is the return at time t; γ is the decay factor; Indicates s t The maximum Q value of the next state that can be reached after executing an action in the state; the DQN model selects the action with the largest Q value in the current state as the output, and the formula is

[0026] To achieve the above objectives, the second embodiment of the present invention proposes a deep reinforcement learning drone multi-hop SPMA network intelligent backoff device, including the following modules:

[0027] An input module is configured to use channel state information as input to the intelligent backoff network in the multi-hop SPMA network, wherein the channel state information includes channel occupancy rate and its historical value, and the number of data packets to be forwarded in each priority queue;

[0028] An updating module, configured to output a backoff duration according to the intelligent backoff network and update a channel state of the intelligent backoff network;

[0029] An evaluation module is configured to evaluate a backoff effect based on the updated channel occupancy, evaluate a packet forwarding success rate based on the updated number of packets to be forwarded in each priority queue, and obtain a reward value based on the backoff effect and the packet forwarding success rate;

[0030] A training module, configured to calculate the maximum reward values ​​corresponding to different actions through the intelligent backoff network, select the action with the largest reward value as the current input, and train the intelligent backoff network until convergence;

[0031] The generation module is used to obtain target channel state information and input the target channel state information into a trained intelligent backoff network to obtain an optimal backoff strategy.

[0032] Furthermore, in one embodiment of the present invention, the input module is further configured to:

[0033] The channel occupancy rate of the sending node in the previous several time slots and the number of packets to be forwarded in each priority queue are used as the input of the DQN model of the intelligent backoff network through the intelligent backoff network. The input state is s t ={c t ,c t-1 ,...,c t-5 ,k0,k1,…,k M-1}, where t is the current time, tj is the j time slots before the current time; c t is the channel occupancy rate at time t; k j is the number of packets to be forwarded in the j-priority queue.

[0034] Furthermore, in one embodiment of the present invention, the update module is further configured to:

[0035] The intelligent backoff network interacts with the environment to obtain the state s t ={c t ,c t-1 ,...,c t-5 ,k0,k1,…,kM-1}, then take action a t ∈{0,1,2,…,M-1} to generate the backoff time, where M represents the number of actions. The backoff time expression is as follows:

[0036]

[0037] Where T is the length of a time slot.

[0038] To achieve the above-mentioned purpose, the third aspect embodiment of the present invention proposes a computer device, characterized in that it includes a memory, a processor and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements the above-mentioned drone multi-hop SPMA network intelligent backoff method based on deep reinforcement learning.

[0039] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, it implements the above-mentioned drone multi-hop SPMA network intelligent backoff method based on deep reinforcement learning.

[0040] An embodiment of the present invention proposes an intelligent backoff method for statistical priority multiple access (SPMA) of unmanned aerial vehicles in a multi-hop transmission network. When there are a large number of forwarded data packets in multi-hop transmission, SPMA enters the backoff phase when the channel occupancy rate exceeds the priority threshold. The DQN (Deep Q-Network) algorithm in deep reinforcement learning (DRL) is used to guide nodes to generate a backoff duration, thereby improving the success rate of sending multi-hop SPMA data packets, reducing data packet transmission delay and increasing the total network throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0042] Figure 1 A flowchart of an intelligent backoff method for a multi-hop SPMA network of a drone based on deep reinforcement learning is provided in an embodiment of the present invention.

[0043] Figure 2 This is the workflow diagram of SPMA using the DQN-based intelligent backoff method;

[0044] Figure 3 This is a diagram of the intelligent backoff network structure based on DQN;

[0045] Figure 4These are the two topologies used in the experimental simulation;

[0046] Figure 5 This is a simulation diagram of the throughput performance of the DQN-based backoff method and other methods;

[0047] Figure 6 This is a simulation diagram of the throughput performance of the DQN-based backoff method under different network structures;

[0048] Figure 7 This is a simulation diagram of the success rate of data packet transmission at each priority level based on the DQN backoff method and other methods;

[0049] Figure 8 This is a simulation diagram of the total backoff time based on the DQN backoff method and other methods.

[0050] Figure 9 Schematic diagram of an intelligent backoff device for a drone multi-hop SPMA network based on deep reinforcement learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0052] The following describes an intelligent backoff method for a drone multi-hop SPMA network based on deep reinforcement learning according to an embodiment of the present invention with reference to the accompanying drawings.

[0053] Figure 1 A flowchart of an intelligent backoff method for a multi-hop SPMA network of a drone based on deep reinforcement learning is provided in an embodiment of the present invention.

[0054] like Figure 1 As shown in FIG, the intelligent backoff method of the drone multi-hop SPMA network based on deep reinforcement learning includes the following steps:

[0055] S101: Using channel state information as input to an intelligent backoff network in a multi-hop SPMA network, wherein the channel state information includes channel occupancy and its historical value, and the number of packets to be forwarded in each priority queue;

[0056] S102: Outputting a backoff duration according to the intelligent backoff network and updating the channel status of the intelligent backoff network;

[0057] S103: Evaluate the backoff effect based on the updated channel occupancy, evaluate the packet forwarding success rate based on the updated number of packets to be forwarded in each priority queue, and obtain a reward value based on the backoff effect and the packet forwarding success rate;

[0058] S104: Calculate the maximum reward value corresponding to different actions through the intelligent backoff network, select the action with the largest reward value as the current input, and train the intelligent backoff network until convergence;

[0059] S105: Obtain target channel state information, input the target channel state information into the trained intelligent backoff network, and obtain an optimal backoff strategy.

[0060] Furthermore, in one embodiment of the present invention, using the channel state information as input to an intelligent backoff network in a multi-hop SPMA network includes:

[0061] The channel occupancy rate of the sending node in the previous several time slots and the number of packets to be forwarded in each priority queue are used as the input of the DQN model of the intelligent backoff network through the intelligent backoff network. The input state is s t ={c t ,c t-1 ,...,c t-5 ,k0,k1,…,k M-1}, where t is the current time, tj is the j time slots before the current time; c t is the channel occupancy rate at time t; k j is the number of packets to be forwarded in the j-priority queue.

[0062] Furthermore, in one embodiment of the present invention, outputting a backoff duration according to the intelligent backoff network and updating a channel state of the intelligent backoff network include:

[0063] Interact with the environment through an intelligent backoff network to obtain the state s t ={c t ,c t-1 ,...,c t-5 ,k0,k1,…,k M-1}, then take action a t ∈{0,1,2,…,M-1} to generate the backoff time, where M represents the number of actions. The backoff time expression is as follows:

[0064]

[0065] Where T is the length of a time slot.

[0066] Furthermore, in one embodiment of the present invention, the reward function of the intelligent backoff network is defined as:

[0067]

[0068] Where C is the number of all packets in the priority queue at the start of backoff; ΔN is the change in the number of forwarded packets before and after backoff; B indicates whether the packet can be sent after backoff, 1 indicates it can be sent, and 0 indicates it cannot be sent; t is the time when backoff is completed; t s Indicates the time when the retreat starts; T th It represents the backoff time threshold for obtaining the upper limit reward; ρ is the decay coefficient.

[0069] Furthermore, in one embodiment of the present invention, the update formula of the reward value is defined as: Where Q(s t ,a t ) is in a given state s t Next, perform action a t The obtained Q value is the reward value; α is the learning rate; r t is the return at time t; γ is the decay factor; Indicates s t The maximum Q value of the next state that can be reached after executing an action in the state; the DQN model selects the action with the largest Q value in the current state as the output, and the formula is

[0070] Figure 2 This is a diagram of the SPMA workflow using a DQN-based transmission and backoff method. Before sending a packet, the channel occupancy rate is compared with the priority threshold. If the channel occupancy rate exceeds the priority threshold, the packet is not sent and enters the intelligent backoff network, which generates a backoff time based on the current network status.

[0071] Figure 3 This is a diagram of the DQN-based intelligent backoff network structure. The intelligent backoff network is used to generate the backoff duration. The intelligent backoff network uses the historical channel occupancy rate and the number of packets to be forwarded in each priority queue to determine the backoff time. The specific steps include the following:

[0072] 1) In the DQN-based intelligent backoff model, the intelligent backoff network takes the channel occupancy rate of the sending node in the previous several time slots and the number of packets to be forwarded in each priority queue as the input of the model. The input state is s t ={c t ,c t-1 ,...,c t-5 ,k0,k1,…,k M-1}, where t is the current time, tj is the j time slots before the current time; c tis the channel occupancy rate at time t; k j is the number of packets to be forwarded in the j-priority queue.

[0073] 2) Intelligent backoff network interacts with the environment to obtain state s t ={c t ,c t-1 ,...,c t-5 ,k0,k1,…,k M-1}, then take action a t ∈{0,1,2,…,M-1} to generate the backoff time, where M represents the number of actions. In this invention, M=45. The backoff time expression is as follows:

[0074]

[0075] Where T is the length of time of a time slot.

[0076] 3) If the data packet can be sent after the backoff, a reward will be obtained. The value of the reward is related to the length of the backoff time. The shorter the backoff time, the fewer the number of forwarded data packets, and the more rewards will be obtained. Conversely, if the data packet is not sent after the backoff, or the number of data packets in the priority queue increases, a penalty will be obtained. Therefore, the reward function of the smart backoff network is defined as:

[0077]

[0078] Where C is the number of all packets in the priority queue at the start of backoff; ΔN is the change in the number of forwarded packets before and after backoff; B indicates whether the packet can be sent after backoff, 1 indicates it can be sent, and 0 indicates it cannot be sent; t is the time when backoff is completed; t s Indicates the time when the retreat starts; T th It represents the backoff time threshold for obtaining the upper limit reward; ρ is the decay coefficient.

[0079] 4) Given an input state, the DQN model calculates the Q values ​​corresponding to different actions and selects the action with the highest Q value as the current input behavior. The action space is selected using a greedy algorithm, with the parameter ε gradually increasing from 0 to 0.9. The Experience Replay size is 2000, and the mini-batch size is 256. Every 500 steps, the parameters of the target network are replaced by those of the Q-Network.

[0080] The update formula of Q value is: Where Q(s t ,a t ) is in a given state s t Next, perform action a t The obtained Q value; α is the learning rate; rt is the return at time t; γ is the decay factor; Indicates s t The maximum Q value of the next state that can be reached after executing an action in the state. The DQN model selects the action with the largest Q value in the current state as the output. The formula is

[0081] Figure 4 These are two network topologies selected for experimental simulation to verify the performance and advantages of the present invention in a multi-hop transmission environment. Nodes in the same dotted box can communicate directly, while nodes not in the same dotted box require forwarding from other nodes to achieve inter-node communication.

[0082] Figure 5 This is a simulation diagram of the SPMA multi-hop transmission network using DQN-based smart backoff (DQN-BMN), fixed threshold setting random backoff (FTS-RB) and fair priority threshold and conflict backoff (PCA). Figure 4 It can be seen that when the network load is low (0-1Mbps), the DQN-BMN method does not significantly improve the system throughput. As the packet arrival rate increases, the advantages of DQN-BMN gradually become apparent. At the same packet arrival rate, DQN-BMN can achieve higher system throughput. When the load is between 2Mbps and 10Mbps, the improvement in network throughput by the DQN-BMN method gradually increases. As the load increases further, the transmission and forwarding of high-priority packets occupy the vast majority of the network capacity, and the network's ability to send low-priority packets is gradually limited. At this point, the improvement in system throughput mainly comes from the appropriate setting of the backoff time by the intelligent backoff network.

[0083] Figure 6 This is a simulation diagram of the throughput of three methods under different network topologies. It can be seen from the figure that when the system load is low, the throughput of the DQN-BMN method in different topologies is roughly the same. This is because the channel occupancy rate is relatively low and data packets can be sent and forwarded freely. However, as the system load increases, when the load is between 2.5Mbps and 12.5Mbps, the throughput of network topology 2 is better than that of topology 1. This is because there are more forwarding data packets in network structure (2). The performance of network structure 2 is better than that of network structure 1 due to the throughput of forwarding data packets after the backoff time is reasonably set. However, there is no obvious performance improvement in this load range using PCA and FTS-RB methods, which further proves that the performance improvement of the DQN-BMN method comes from the setting of the backoff time.

[0084] Figure 7This figure shows a simulation of the transmission success rate of packets at various priority levels based on the DQN transmission and backoff model. The transmission success rate is the ratio of the number of packets successfully received by the destination node to the number of packets generated by the source node. As can be seen from the figure, the transmission success rate for each priority level decreases with increasing provided load. Under the same provided load, the DQN-BMN method consistently achieves higher transmission success rates than the FTS-RB and PCA methods for priorities 1, 2, 3, and 4. In the first case, the performance of priorities 1, 2, 3, and 4 improves by approximately 6%, 20%, 34%, and 50%, respectively, compared to FTS-RB. This comparison shows that the rational backoff time setting ensures the success rate of packet forwarding, thereby improving the transmission success rate of the corresponding packets.

[0085] Figure 8 This is a simulation of the total backoff time for the DQN-based transmission and backoff model. As can be seen from the figure, the total backoff time of the DQN-BMN method is consistently lower than that of the FTS-RB method. In most cases, the total backoff time of DQN-BMN is lower than that of PCA. In terms of average backoff time, DQN-BMN is lower than both FTS-RB and PCA.

[0086] An embodiment of the present invention proposes an intelligent backoff method for statistical priority multiple access (SPMA) of unmanned aerial vehicles in a multi-hop transmission network. When a large number of forwarded packets are transmitted in multi-hop transmission, SPMA enters a backoff phase when the channel occupancy exceeds the priority threshold. The DQN (Deep Q-Network) algorithm from Deep Reinforcement Learning (DRL) is used to guide nodes to generate backoff durations, thereby improving the success rate of multi-hop SPMA packet transmission, reducing packet transmission latency, and increasing the total network throughput.

[0087] In order to implement the above embodiments, the present invention also proposes a drone multi-hop SPMA network intelligent backoff device based on deep reinforcement learning.

[0088] Figure 9 A schematic structural diagram of a drone multi-hop SPMA network intelligent backoff device based on deep reinforcement learning provided in an embodiment of the present invention.

[0089] like Figure 9 As shown, the UAV multi-hop SPMA network intelligent backoff device based on deep reinforcement learning includes: an input module 100, an update module 200, an evaluation module 300, and a generation module 400, wherein,

[0090] An input module is configured to use channel state information as input to the intelligent backoff network in the multi-hop SPMA network, wherein the channel state information includes channel occupancy rate and its historical value, and the number of packets to be forwarded in each priority queue;

[0091] An update module, configured to output a backoff duration according to the intelligent backoff network and update a channel state of the intelligent backoff network;

[0092] An evaluation module is used to evaluate the backoff effect based on the updated channel occupancy rate, evaluate the packet forwarding success rate based on the updated number of packets to be forwarded in each priority queue, and obtain a reward value based on the backoff effect and the packet forwarding success rate;

[0093] The training module is used to calculate the maximum reward value corresponding to different actions through the intelligent backoff network, select the action with the largest reward value as the current input, and train the intelligent backoff network until convergence;

[0094] The generation module is used to obtain the target channel state information and input the target channel state information into the trained intelligent backoff network to obtain the optimal backoff strategy.

[0095] Furthermore, in one embodiment of the present invention, the input module is further configured to:

[0096] The channel occupancy rate of the sending node in the previous several time slots and the number of packets to be forwarded in each priority queue are used as the input of the DQN model of the intelligent backoff network through the intelligent backoff network. The input state is s t ={c t ,c t-1 ,...,c t-5 ,k0,k1,…,k M-1}, where t is the current time, tj is the j time slots before the current time; c t is the channel occupancy rate at time t; k j is the number of packets to be forwarded in the j-priority queue.

[0097] Furthermore, in one embodiment of the present invention, the update module is further configured to:

[0098] Interact with the environment through an intelligent backoff network to obtain the state s t ={c t ,c t-1 ,...,c t-5 ,k0,k1,…,k M-1}, then take action a t ∈{0,1,2,…,M-1} to generate the backoff time, where M represents the number of actions. The backoff time expression is as follows:

[0099]

[0100] Where T is the length of a time slot.

[0101] To achieve the above-mentioned purpose, the third aspect embodiment of the present invention proposes a computer device, characterized in that it includes a memory, a processor and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements the above-mentioned drone multi-hop SPMA network intelligent backoff method based on deep reinforcement learning.

[0102] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, it implements the above-mentioned drone multi-hop SPMA network intelligent backoff method based on deep reinforcement learning.

[0103] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0104] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0105] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limiting the present invention. A person skilled in the art may change, modify, replace, and modify the above embodiments within the scope of the present invention.

Claims

1. An intelligent backoff method for drone multi-hop SPMA network based on deep reinforcement learning, characterized by: The following steps are involved: Channel state information is used as input to an intelligent backoff network in a multi-hop SPMA network, wherein the channel state information includes channel occupancy and its historical value, and the number of packets to be forwarded in each priority queue; Outputting a backoff duration according to the intelligent backoff network and updating a channel state of the intelligent backoff network; Evaluate the backoff effect according to the updated channel occupancy rate, evaluate the packet forwarding success rate according to the updated number of packets to be forwarded in each priority queue, and obtain a reward value based on the backoff effect and the packet forwarding success rate; Calculating the maximum reward values ​​corresponding to different actions through the intelligent backoff network, selecting the action with the largest reward value as the current input, and training the intelligent backoff network until convergence; Obtaining target channel state information, and inputting the target channel state information into a trained intelligent backoff network to obtain an optimal backoff strategy; The reward function of the intelligent backoff network is defined as: Where C is the number of all packets in the priority queue at the start of backoff; ΔN is the change in the number of forwarded packets before and after backoff; B indicates whether the packet can be sent after backoff, 1 indicates it can be sent, and 0 indicates it cannot be sent; t is the time when backoff is completed; t s Indicates the time when the retreat starts; T th It represents the backoff time threshold for obtaining the upper limit reward; ρ is the decay coefficient.

2. The method according to claim 1, characterized in that The method of using the channel state information as an input of an intelligent backoff network in a multi-hop SPMA network includes: The channel occupancy rate of the sending node in the previous several time slots and the number of packets to be forwarded in each priority queue are used as the input of the DQN model of the intelligent backoff network through the intelligent backoff network. The input state is s t ={c t ,c t-1 ,...,c t-5 ,k0,k1,…,k M-1 }, where t is the current time, tj is the j time slots before the current time; c t is the channel occupancy rate at time t; k j is the number of packets to be forwarded in the j-priority queue.

3. The method according to claim 1, characterized in that Outputting a backoff duration according to the intelligent backoff network and updating a channel state of the intelligent backoff network include: The intelligent backoff network interacts with the environment to obtain the state s t ={c t ,c t-1 ,...,c t-5 ,k0,k1,…,k M-1 }, then take action a t ∈{0,1,2,…,M-1} to generate the backoff time, where M represents the number of actions. The backoff time expression is as follows: Where T is the length of a time slot.

4. The method according to claim 1, wherein The update formula of the return value is defined as: Where Q(s t ,a t ) is in a given state s t Next, perform action a t The obtained Q value is the reward value; α is the learning rate; r t is the return at time t; γ is the attenuation factor; Indicates s t The maximum Q value of the next state that can be reached after executing an action in the state; the DQN model selects the action with the largest Q value in the current state as the output, and the formula is 5. A drone multi-hop SPMA network intelligent backoff device based on deep reinforcement learning, characterized in that: Includes the following modules: An input module is configured to use channel state information as input to the intelligent backoff network in the multi-hop SPMA network, wherein the channel state information includes channel occupancy rate and its historical value, and the number of data packets to be forwarded in each priority queue; An updating module, configured to output a backoff duration according to the intelligent backoff network and update a channel state of the intelligent backoff network; An evaluation module is configured to evaluate a backoff effect based on the updated channel occupancy, evaluate a packet forwarding success rate based on the updated number of packets to be forwarded in each priority queue, and obtain a reward value based on the backoff effect and the packet forwarding success rate; A training module, configured to calculate the maximum reward values ​​corresponding to different actions through the intelligent backoff network, select the action with the largest reward value as the current input, and train the intelligent backoff network until convergence; A generation module is used to obtain target channel state information and input the target channel state information into a trained intelligent backoff network to obtain an optimal backoff strategy; The reward function of the intelligent backoff network is defined as: Where C is the number of all packets in the priority queue at the start of backoff; ΔN is the change in the number of forwarded packets before and after backoff; B indicates whether the packet can be sent after backoff, 1 indicates it can be sent, and 0 indicates it cannot be sent; t is the time when backoff is completed; t s Indicates the time when the retreat starts; T th It represents the backoff time threshold for obtaining the upper limit reward; ρ is the decay coefficient.

6. The device according to claim 5, characterized in that The input module is further used for: The channel occupancy rate of the sending node in the previous several time slots and the number of packets to be forwarded in each priority queue are used as the input of the DQN model of the intelligent backoff network through the intelligent backoff network. The input state is s t ={c t ,c t-1 ,...,c t-5 ,k0,k1,…,k M-1 }, where t is the current time, tj is the j time slots before the current time; c t is the channel occupancy rate at time t; k j is the number of packets to be forwarded in the j-priority queue.

7. The device according to claim 5, characterized in that The update module is further configured to: The intelligent backoff network interacts with the environment to obtain the state s t ={c t ,c t-1 ,...,c t-5 ,k0,k1,…,k M-1 }, then take action a t ∈{0,1,2,…,M-1} to generate the backoff time, where M represents the number of actions. The backoff time expression is as follows: Where T is the length of a time slot.

8. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for intelligent backoff of a multi-hop SPMA network of a drone based on deep reinforcement learning as described in any one of claims 1 to 4 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the intelligent backoff method for a drone multi-hop SPMA network based on deep reinforcement learning is implemented.

Images