A method and system for channel access of unmanned aerial vehicle ad hoc network in a broadcast scenario

Abstract

The application discloses a kind of broadcast scene under unmanned aerial vehicle ad hoc network channel access method and system, comprising: when unmanned aerial vehicle node has sending demand, node statistics last transmission feedback, according to Q-Learning algorithm output a backoff window value CW, after backoff, node accesses channel;Before sending, node selects the number of forwarding node according to the reception success rate threshold, the number of selected forwarding node is numbered and CW is attached in data packet and is broadcasted to send;Node receives broadcast packet and reads MAC header field, statistics neighbor CW value, maintains neighbor table and CW popularity table, and compares whether it is forwarding node itself;If current node is selected forwarding node, then node needs to forward the received data packet in the form of broadcast;Source node arrives when ACK time limit, the effective ACK number received this time is counted, the feedback of this transmission is calculated, and the weight table of Q-Learning is updated before next transmission;Node repeats the above process, and iteratively updates adaptive channel access strategy.

Description

Technical Field

[0001] This invention relates to the field of channel access technology, and in particular to a method and system for unmanned aerial vehicle (UAV) ad hoc network channel access in a broadcast scenario. Background Technology

[0002] Due to their system stability and wide coverage, ad hoc UAV networks are widely used in military and civilian fields such as environmental reconnaissance, long-range strikes, and emergency rescue, and have significant research value. To leverage the advantages of multi-UAV collaboration, UAV nodes in a network typically need to exchange key information such as position, speed, and mission parameters in real time to update their decision-making. Therefore, broadcast communication has become an important scenario for UAV networking. The Media Access Control (MAC) protocol governs the rules followed by nodes throughout the network when accessing wireless channels and determines how to maximize the use of limited channel bandwidth. The quality of the channel access mechanism directly determines the overall network performance, making MAC protocol research crucial for the development of FANET.

[0003] Current UAV ad hoc networks primarily employ contention-based protocols from traditional Ad Hoc networks for channel access control, with Carrier Sense Multiple Access (CSMA / CA) being a representative example. However, CSMA / CA struggles to apply ACK mechanisms in broadcast scenarios, preventing it from leveraging its dynamic backoff advantage and ultimately failing to meet performance requirements in the highly flexible topology of UAV ad hoc networks.

[0004] Based on CSMA / CA, this invention proposes a dynamic forwarding mechanism for broadcast scenarios. Each UAV node is treated as a decision-making agent, and the backoff algorithm is optimized using multi-agent reinforcement learning (MARL) technology. This enables the node to iteratively obtain a highly adaptive channel access strategy through interactive learning with the environment. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method and system for unmanned aerial vehicle (UAV) ad hoc network channel access in a broadcast scenario, which allows the backoff period (CW) to be adaptively adjusted with changes in the environment based on the CSMA / CA protocol, so as to improve network throughput, end-to-end latency performance and node fairness.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A channel access method for unmanned aerial vehicle (UAV) ad hoc networks is disclosed. This method is applied to broadcast communication scenarios within UAV ad hoc networks. Each node in the UAV ad hoc network is treated as a decision-maker. Based on the Q-Learning algorithm in reinforcement learning, the decision-makers interact and learn from the environment to iteratively obtain an adaptive access strategy. The access method specifically includes the following steps:

[0008] Step S1: When there are data packets to be sent in the MAC queue of a node, the node first performs carrier sensing on the channel. If the channel is busy, the access is postponed and the node waits for the next time slot to perform sensing. If the channel is idle, the node updates the weight of each decision according to the feedback obtained from the previous transmission through the Q-Learning algorithm, calculates and outputs the decision value at the current moment, and uses this decision value as the backoff window CW.

[0009] Step S2: The node selects a random integer value within [0, CW] based on a uniform distribution, uses it as the initial value of the backoff counter, and begins backoff. During the backoff period, the node listens to the channel in each time slot. If the channel is idle, the value of the backoff counter is decremented by one; otherwise, the node waits for the next time slot to continue listening. When the value of the backoff counter is reduced to zero, the node accesses the channel and prepares to transmit.

[0010] Step S3: Define a dynamic forwarding mechanism. Before sending a data packet, a node will randomly select M nodes from the neighbor table as forwarding nodes. The selected nodes need to forward the data packet in the form of broadcast after receiving it, which also serves as flooding and ACK.

[0011] Step S4: When a node sends a data packet, it will attach the CW (Content Written) to the MAC header field of the packet. After receiving the data packet, the neighboring node will count this value to estimate the network congestion level.

[0012] Step S5: When a node receives a data packet, it first determines whether the data packet is the original data packet or a forwarded data packet. If it is the original data packet and the node is selected to forward it, the node uses the CW attached to the data packet as a backoff window to back off and then completes the forwarding. If it is a forwarded data packet, the node needs to determine whether the packet is a valid ACK data packet and include it in the feedback statistics. Finally, the source node calculates the reward value for this transmission based on the backoff delay caused by the CW, the fairness comparison, and the number of ACKs.

[0013] Furthermore, in the broadcast communication scenario based on UAV ad hoc networks, there is a UAV swarm consisting of N nodes and one channel, where 50 ≤ N ≤ 100. After competing for access to the channel, the nodes send data packets via broadcast to exchange information needed for network formation. Based on the characteristic of periodic broadcasting, each node can receive several data packets sent by all surrounding nodes within a certain period of time, and the nodes maintain a neighbor table based on the source of the data packets. Considering the highly dynamic characteristics of the UAV ad hoc network topology, the nodes update the neighbor table based on the source of the broadcast packets received in the last second to ensure that the neighbors are within the communication range.

[0014] Furthermore, step S1 includes:

[0015] S101: Model the channel access problem for UAV nodes, and model the access process as a distributed partially observable Markov decision process (Dec-POMDP), defining it as a...<S,A,P,R> A tuple, where S is the state space, representing the decision-maker's observation of the environment, A represents the set of possible actions in state S, and P is P0. a (s, s') represents the probability of transitioning from the old state s∈S to the new state s'∈S by performing the action a∈A, where R is the probability of transitioning from the old state s∈S to the new state s'∈S. a (s,s') is the immediate reward (penalty) obtained by the agent after transitioning from state s to s' through action a; policy π is the mapping from state space to action, i.e., π:S→A; the goal of Dec-POMDP is to find the optimal policy to maximize the total reward function over a period of time.

[0016] S102: The modeling of Dec-POMDP is concretized, with each node acting as an individual decision-making body. The node's overall observation of the channel state and the historical decisions of its neighboring nodes is represented as state s. t ,definition Where s L,t =CW t-1 This represents the local state, specifically the backoff window used in the previous transmission; s G,t =CW′ t This indicates the overall state, representing the CW level of the average CW received from neighboring nodes' broadcast packets. It reflects the degree of channel congestion to some extent.

[0017] S103: The action definition of a node is... That is, the node's decision is based on the backoff window of the previous transmission, either halving, keeping it unchanged, or doubling the window value; the update of the backoff window CW is defined as follows:

[0018] S104: The reward r that a node receives after each transmission is completed. t The feedback obtained during transmission is related to the number of ACKs obtained within the time limit, the backoff delay, and the CW value of neighboring nodes, and is calculated in detail during the node's reception process;

[0019] S105: After detecting that the channel is idle, the node invokes the Q-Learning algorithm. The input to the algorithm is the node's current observation of the environment, s. t The output is the node's action 'a'. t This refers to the backoff window value CW for this transmission, which is used as the initial value for the backoff counter to begin backoff.

[0020] Furthermore, in step S1, the value function of Dec-POMDP is defined as V. πLet S→R represent the expected value obtained in each state s∈S under policy π. The value function represents the quality of the policy, and its expression is:

[0021]

[0022] Every state-action pair has a weight, or Q-value, and the optimal Q-value function is represented as Q. * (s,a)=r t (s t ,a t )+γE π [V π (s t+1 Thus, the optimal value function is expressed as: Therefore, the problem is narrowed down to solving Q. * The maximum value of (s,a) can be achieved through continuous iteration; specifically, the elements contained in one state transition are (s... t a t r t s t+1 The update rule for the Q-value function is defined as follows:

[0023]

[0024] In equation (2), the discount factor γ determines the ratio of current reward to future reward. If γ = 0, the decision-maker only focuses on maximizing immediate reward. Conversely, γ = 1 indicates that the decision-maker will strive for a longer-term reward value. α is the learning rate of the decision-maker, which quantifies the degree to which newly acquired information covers historical information. When α = 1, the decision-maker only considers recently acquired information, while α = 0 means that the decision-maker will not learn any new information.

[0025] Furthermore, step S3 includes:

[0026] S301: After a node sends a broadcast data packet, it needs to transmit feedback to indicate whether the transmission was successful or not. In a unicast network, feedback is achieved by an ACK data frame. However, in a broadcast network, there is no condition for feedback. Therefore, the dynamic forwarding mechanism defines that before sending, a node will randomly select M (2≤M≤N) nodes from its neighbor table as forwarding nodes. After receiving the broadcast packet, the designated forwarding node needs to forward the packet, which improves channel utilization and provides a feedback mechanism similar to ACK for the source node.

[0027] S302: Considering that too few forwarding nodes may lead to inaccurate feedback, and too many forwarding nodes may lead to network overload, "successful reception" is defined as follows: if the source node receives ACKs from at least 2 forwarding nodes within the valid time after a transmission, then the source node has successfully received the data.

[0028] S303: Success rate of reception p s Defined as the ratio of the number of times a node successfully receives data to the total number of data packets sent up to the current moment; the reception rate threshold p th It is a constant and can take a value of 0.5; when p s ≥p th When p, the node selects 3 forwarding nodes; s <p th When a node selects two forwarding nodes, it attaches the forwarding node's number to the data packet and broadcasts it.

[0029] Furthermore, in step S5, CW is defined. levels ={3,7,15,31,63,127,255} is the set of backoff window levels, and ∑CW is the neighbor CW queue; when a node receives a broadcast packet, it records the CW carried in the MAC header field and puts it into ∑CW. The number of times each CW level appears in the queue is counted, and this is used as the basis for classifying the CW. levels The frequency of occurrence is sorted in descending order; ∑CW is updated every 1 second to ensure that the queue only stores CW received within 1 second.

[0030] Furthermore, in step S5, the fairness reward function is defined as follows: in Includes CW levels The corresponding 7 reward values, CW t The higher the frequency of occurrence in the ∑CW queue, the greater the reward for this decision.

[0031] The delay reward function is defined as follows:

[0032]

[0033] In equation (3), N ACK The number of valid ACKs received by a node within a specified time limit after a single transmission; a smaller CW means better latency performance, thus the reward value is higher when the transmission is successful, but the corresponding penalty is also greater when the transmission fails.

[0034] Combining the fairness reward function and the time-delay reward function yields the local reward function:

[0035] R Local (CW t N ACK )=w×R delay (CW t N ACK )+(1-w)×R fairness (CW t (10)

[0036] In equation (4), w is the performance weight coefficient, which represents the node's emphasis on latency and fairness, and is usually 0.6≤w≤0.8;

[0037] Define R Global (CW t ') represents the overall reward function:

[0038]

[0039] The above sub-reward functions are integrated into the overall reward function in Dec-POMDP:

[0040] r t =N ACK ×[K×R Local (CW t N ACK )+(1-K)×R Global (CW t ′)] (12)

[0041] In equation (5), K is the local weight coefficient, which can be 0.7.

[0042] This invention also provides a UAV ad hoc network channel access system for broadcast scenarios, comprising:

[0043] The channel listening and parsing module is used to determine whether the channel is idle, whether the received signal power is sufficient to complete packet reception, and the type of packet. After completing the data packet reception, it parses the MAC frame and records the data.

[0044] The neighbor node perception module maintains the neighbor node table and the neighbor CW table based on the MAC frame data parsed by the channel listening module, and updates them every 1 second.

[0045] The parameter calculation module is used to calculate and update the parameters of the Q-table in the Q-Learning algorithm, and to sort the neighbor CW table according to the frequency of occurrence of each CW, and calculate the successful reception rate p. s ;

[0046] The channel access module, based on the Q-table parameters and the successful reception rate p... s Determine the backoff and forwarding strategies, and access the channel for broadcast transmission.

[0047] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of the UAV ad hoc network channel access method in a broadcast scenario.

[0048] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the UAV ad hoc network channel access method in a broadcast scenario.

[0049] The beneficial effects of this invention are:

[0050] (1) The channel access problem is modeled as a distributed partially observable Markov decision process (Dec-POMDP), which makes full use of the characteristics of node broadcast communication in FANET, allowing each node to exchange MAC protocol status information in real time.

[0051] (2) The CSMA / CA algorithm in the traditional MAC protocol is optimized by adopting multi-decision body Q-Learning, so that the backoff period CW can be adaptively adjusted according to the channel congestion level.

[0052] (3) A dynamic forwarding mechanism is proposed, which provides a feedback mechanism for both broadcast CSMA / CA and Q-Learning algorithms. This mechanism avoids the blocking of the MAC queue when the source node is waiting for ACK, and also plays a flooding role, which helps the node to make full use of the idle channel to transmit important information in the FANET.

[0053] Compared with existing technologies, this invention adopts multi-agent reinforcement learning technology on the basis of CSMA / CA protocol, which enables the backoff period CW to be adaptively adjusted with changes in the environment, effectively improving network throughput, end-to-end latency performance and node fairness, and has considerable application prospects. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating a method for accessing a UAV ad hoc network channel in a broadcast scenario, as provided in Example 1.

[0055] Figure 2 This is a schematic diagram of the Dec-POMDP model provided in Example 1;

[0056] Figure 3 This is a schematic diagram of the MAC operation based on the dynamic forwarding mechanism proposed in Example 1. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Example 1

[0059] refer to Figures 1-3 This embodiment provides a method for UAV ad hoc network channel access in a broadcast scenario. This method treats each node in the UAV ad hoc network as a decision-maker, and based on the Q-Learning algorithm in reinforcement learning, enables the decision-maker to interact and learn from the environment, iteratively obtaining an adaptive access strategy. The method includes the following steps:

[0060] Step 1: When there are data packets to be sent in the MAC queue of a node, the node first performs carrier sensing on the channel. If the channel is busy, the access is postponed and the node waits for the next time slot to listen. If the channel is idle, the node updates the weight of each decision based on the feedback obtained from the previous transmission using the Q-Learning algorithm, calculates and outputs the decision value at the current moment, and uses this decision value as the backoff window CW.

[0061] Step 2: The node selects a random integer value within [0, CW] based on a uniform distribution, uses it as the initial value of the backoff counter, and begins backoff. During backoff, the node listens to the channel in each time slot. If the channel is idle, the value of the backoff counter is decremented by one; otherwise, the node waits for the next time slot to continue listening. If a data packet is received from another node during the backoff process, the node retains the value of the backoff counter, temporarily stops backoff, and enters the receiving state. When the value of the backoff counter is reduced to zero, the node accesses the channel and prepares to send.

[0062] Step 3: In order to enable nodes to receive feedback after sending data packets, a dynamic forwarding mechanism is defined. Before sending a data packet, a node will randomly select M nodes from the neighbor table as forwarding nodes. The selected nodes need to forward the data packet in the form of broadcast after receiving it, which also serves as flooding and ACK.

[0063] Step 4: When a node sends a data packet, it will attach the CW (Content Written) to the MAC header field of the packet. After receiving this data packet, neighboring nodes will count this value to estimate the network congestion level.

[0064] Step 5: When a node receives a data packet, it first determines whether the data packet is the original data packet or a forwarded data packet. If it is the original data packet and the node is selected to forward it, the node uses the CW attached to the data packet as a backoff window to back off and then completes the forwarding. If it is a forwarded data packet, the node needs to determine whether the packet is a valid ACK data packet and include it in the feedback statistics. Finally, the source node calculates the reward value for this transmission based on the backoff delay caused by the CW, the fairness comparison, and the number of ACKs.

[0065] Specifically, in this embodiment, the broadcast communication scenario of the UAV ad hoc network includes a UAV swarm consisting of N nodes and one channel, where 50 ≤ N ≤ 100. After competing for access to the channel, the nodes send data packets in a broadcast manner to exchange the information required for network formation. Based on the characteristic of periodic broadcasting, each node can receive several data packets sent by all surrounding nodes within a certain period of time, and the nodes maintain a neighbor table according to the source of the data packets. Considering the highly dynamic characteristics of the UAV ad hoc network topology, the nodes update the neighbor table according to the source of the broadcast packets received in the last second to ensure that the neighbors are within the communication range.

[0066] Specifically, in this embodiment, the channel access problem of UAV nodes can be modeled as a distributed partially observable Markov decision process (Dec-POMDP), which is defined as a...<S,A,P,R> A tuple, where S is the state space, representing the decision-maker's observation of the environment, A represents the set of possible actions in state S, and P is P0. a (s, s') represents the probability of transitioning from the old state s∈S to the new state s'∈S by performing the action a∈A, where R is the probability of transitioning from the old state s∈S to the new state s'∈S. a (s,s') is the immediate reward (penalty) that the agent receives after transitioning from state s to s' through action a; policy π is the mapping from the state space to actions, i.e., π:S→A; the goal of Dec-POMDP is to find the optimal policy to maximize the total reward function over a period of time.

[0067] Specifically, the modeling of Dec-POMDP is as follows: Figure 2 As shown, each node acts as an independent decision-making entity, and the node's overall observation of the channel state and the historical decisions of its neighboring nodes is represented by the state s. t ,definition Where s L,t =CW t-1 This represents the local state, specifically the backoff window used in the previous transmission; s G,t =CW′ t This indicates the overall state, representing the CW level of the average CW received from neighboring nodes' broadcast packets. It reflects the degree of channel congestion to some extent.

[0068] The action of a node is defined as follows That is, the node's decision is based on the backoff window of the previous transmission, either halving, keeping it unchanged, or doubling the window value; the update of the backoff window CW is defined as follows:

[0069] The reward r that a node receives after each transmission is completed t The feedback obtained during transmission is related to the number of ACKs obtained within the time limit, the backoff delay, and the CW value of neighboring nodes, and is calculated in detail during the node's reception process;

[0070] After detecting that the channel is idle, the node invokes the Q-Learning algorithm. The input to the algorithm is the node's observation of the environment at the current moment. t The output is the node's action 'a'. t This refers to the backoff window value CW for this transmission, which is used as the initial value for the backoff counter to begin backoff.

[0071] Specifically, in this embodiment, if the backoff window CW is selected when the node sends at time t-1... t-1 If the mean of the neighbor CW counted by this node at time t is 63, and the mean of the neighbor CW counted at time t is between [64, 127], then s t-1 = (63, 127), at this point if the output of the Q-Learning algorithm is CW doubled, then CW t The value is 127. If the neighbor's CW rank does not change at time t, then s t = (63, 127).

[0072] Specifically, in this embodiment, the transition probability P a (s,s') is determined by the joint behavior of multiple drones, while the behavior of other drones is difficult to know, making it difficult to evaluate the policy π. In reinforcement learning, Q-Learning is an effective learning algorithm that can learn without knowing the prior P. a Solve this Dec-POMDP given (s, s'). Define the value function of the Dec-POMDP as V. π Let S→R represent the expected value obtained in each state s∈S under policy π. The value function represents the quality of the policy, and its expression is:

[0073]

[0074] Every state-action pair has a weight, or Q-value, and the optimal Q-value function is represented as Q. * (s,a)=r t (s t ,a t )+γE π [V π (s t+1 Thus, the optimal value function is expressed as: Therefore, the problem is narrowed down to solving Q. * The maximum value of (s,a) can be achieved through continuous iteration.

[0075] Specifically, the elements contained in a single state transition are (s t a t r t s t+1 The update rule for the Q-value function is defined as follows:

[0076]

[0077] In equation (2), the discount factor γ determines the ratio of current reward to future reward. If γ = 0, the decision-maker only focuses on maximizing immediate reward. Conversely, γ = 1 indicates that the decision-maker will strive for a longer-term reward value. α is the learning rate of the decision-maker, which quantifies the degree to which newly acquired information covers historical information. When α = 1, the decision-maker only considers recently acquired information, while α = 0 means that the decision-maker will not learn any new information.

[0078] Specifically, in this embodiment, the dynamic forwarding mechanism can simultaneously function as both a flooding and ACK mechanism. The specific method is as follows:

[0079] Step 301: After sending a broadcast data packet, a node needs to transmit feedback to indicate whether the transmission was successful or not. In a unicast network, feedback is achieved by an ACK data frame. However, in a broadcast network, there is no condition for feedback. Therefore, the dynamic forwarding mechanism defines that before sending, a node will randomly select M (2≤M≤N) nodes from its neighbor table as forwarding nodes. After receiving the broadcast packet, the designated forwarding node needs to forward the packet, which improves channel utilization and provides a feedback mechanism similar to ACK for the source node.

[0080] Step 302: Considering that too few forwarding nodes may lead to inaccurate feedback, and too many forwarding nodes may lead to network overload, "successful reception" is defined as follows: if the source node receives ACKs from at least 2 forwarding nodes within the valid time after a transmission, then the source node has successfully received the data.

[0081] Step 303: Success rate p s Defined as the ratio of the number of times a node successfully receives data to the total number of data packets sent up to the current moment; the reception rate threshold p th It is a constant, with a value of 0.5; when p s ≥p th When p, the node selects 3 forwarding nodes; s <p th When a node selects two forwarding nodes, it attaches the forwarding node's number to the data packet and broadcasts it.

[0082] Specifically, in this embodiment, when a node counts its neighbor CW, it defines CW. levels ={3,7,15,31,63,127,255} is the set of backoff window levels, and ΣCW is the neighbor CW queue. When a node receives a broadcast packet, it records the CW carried in the MAC header field and puts it into ΣCW. The number of times each CW level appears in the queue is counted, and this is used as the basis for classifying the CW. levelsThe frequencies of occurrence are sorted in descending order. ΣCW is updated every 1 second to ensure that the queue only stores CWs received within 1 second.

[0083] Specifically, in this embodiment, the node's reward function takes into account three aspects: latency, fairness, and reception success rate.

[0084] The fair reward function is defined as follows: in Includes CW levels The corresponding 7 reward values, CW t The higher the frequency of occurrence in the ∑CW queue, the greater the reward for this decision.

[0085] The time-delay reward function is defined as follows:

[0086]

[0087] In equation (3), N ACK The number of valid ACKs received by a node within a specified time limit after a single transmission, 0 ≤ N ACK ≤M. When reception is successful, a smaller CW means better latency performance, so the reward value is higher when transmission is successful. However, the corresponding penalty should also be greater when transmission fails, following the principle of "the greater the risk, the greater the reward".

[0088] Combining the fairness reward function and the time-delay reward function yields the local reward function:

[0089] R Local (CW t N ACK )=w×R delay (CW t N ACK )+(1-w)×R fairness (CW t (16)

[0090] In equation (4), w is the performance weight coefficient, which represents the node's emphasis on latency and fairness. The larger w is, the more the node tends to iterate and learn in the direction of low latency, and vice versa. Usually, 0.6≤w≤0.8.

[0091] Define R Global (CW t ') represents the overall reward function, which indicates the average latency performance of the neighboring nodes. Its expression is as follows:

[0092]

[0093] The above sub-reward functions are integrated into the overall reward function in Dec-POMDP:

[0094] r t =N ACK ×[K×R Local (CW t N ACK )+(1-K)×R Global (CW t ′)] (18)

[0095] In equation (5), K is the local weight coefficient. The larger K is, the more a node focuses on its own benefits, and vice versa. Usually, K = 0.7.

[0096] This invention also provides a UAV ad hoc network channel access system for broadcast scenarios, comprising:

[0097] The channel listening and parsing module is used to determine whether the channel is idle, whether the received signal power is sufficient to complete packet reception, and the type of packet. After completing the data packet reception, it parses the MAC frame and records the data.

[0098] The neighbor node perception module maintains the neighbor node table and the neighbor CW table based on the MAC frame data parsed by the channel listening module, and updates them every 1 second.

[0099] The parameter calculation module is used to calculate and update the parameters of the Q-table in the Q-Learning algorithm, and to sort the neighbor CW table according to the frequency of occurrence of each CW, and calculate the successful reception rate p. s ;

[0100] The channel access module, based on the Q-table parameters and the successful reception rate p... s Determine the backoff and forwarding strategies, and access the channel for broadcast transmission.

[0101] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of unmanned aerial vehicle (UAV) self-organizing network channel access in a broadcast scenario.

[0102] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the UAV ad hoc network channel access method in a broadcast scenario.

[0103] In summary, this invention employs multi-agent reinforcement learning technology based on the CSMA / CA protocol, enabling the backoff period (CW) to adaptively adjust with changes in the environment. This effectively improves network throughput, end-to-end latency performance, and node fairness, and has considerable application prospects.

[0104] Any aspects of this invention not described in detail are well-known to those skilled in the art.

[0105] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for unmanned aerial vehicle (UAV) ad hoc network channel access in a broadcast scenario, characterized in that, include: When a drone node has a transmission requirement, the node counts the feedback from the previous transmission and outputs a backoff window value CW according to the Q-Learning algorithm. After the backoff is completed, the node accesses the channel. Before sending a packet, the node selects the number of forwarding nodes according to the reception success rate threshold, and attaches the selected forwarding node number and CW to the data packet for broadcast. After receiving a broadcast packet, the node reads the MAC header field, calculates the neighbor CW value, maintains the neighbor table and CW popularity table, and compares whether it is a forwarding node. If the current node is the selected forwarding node, the node needs to forward the received data packets in the form of broadcast; when the ACK timeout arrives, the source node counts the number of valid ACKs received in this transmission, calculates the feedback of this transmission, and updates the weight table of Q-Learning before the next transmission; the node repeats the above process to iteratively update the adaptive channel access strategy. Specifically, the steps include the following: Step S1: When there are data packets to be sent in the MAC queue of a node, the node first performs carrier sensing on the channel. If the channel is busy, the access is postponed and the node waits for the next time slot to perform sensing. If the channel is idle, the node updates the weight of each decision according to the feedback obtained from the previous transmission through the Q-Learning algorithm, calculates and outputs the decision value at the current moment, and uses this decision value as the backoff window CW. Step S2: The node selects a random integer value within [0, CW] based on a uniform distribution, uses it as the initial value of the backoff counter, and begins backoff. During the backoff period, the node listens to the channel in each time slot. If the channel is idle, the value of the backoff counter is decremented by one; otherwise, the node waits for the next time slot to continue listening. When the value of the backoff counter is reduced to zero, the node accesses the channel and prepares to transmit. Step S3: Before sending a data packet, the node will randomly select M nodes from the neighbor table as forwarding nodes. The selected nodes need to forward the data packet in the form of broadcast after receiving it, which also serves as flooding and ACK. Step S4: When a node sends a data packet, it will attach the CW (Content Written) to the MAC header field of the packet. After receiving the data packet, the neighboring node will count this value to estimate the network congestion level. Step S5: When a node receives a data packet, it first determines whether the data packet is the original data packet or the forwarded data packet. If it is the original data packet and the node is selected to forward it, the node uses the CW attached to this data packet as the backoff window to back off, and then completes the forwarding. If the data packet is forwarded, the node needs to determine whether the packet is a valid ACK data packet and include it in the feedback statistics; finally, the source node calculates the reward value for this transmission based on the backoff delay caused by this CW selection, fairness comparison, and the number of ACKs. Step S1 includes: S101: Model the channel access problem for UAV nodes, and model the access process as a distributed partially observable Markov decision process (Dec-POMDP), defining it as a...<S, A, P, R> A tuple, where S is the state space, representing the decision-maker's observation of the environment, A represents the set of possible actions in state S, and P is P0. a (s, s') represents the probability of transitioning from the old state s∈S to the new state s'∈S by performing the action a∈A, where R is the probability of transitioning from the old state s∈S to the new state s'∈S. a (s, s') represents the immediate reward or penalty an agent receives after transitioning from state s to s' through action a; policy It is a mapping from state space to actions, that is... S→A; The goal of Dec-POMDP is to find the optimal policy to maximize the total reward function over a period of time. S102: The modeling of Dec-POMDP is concretized, with each node acting as an individual decision-making body. The node's overall observation of the channel state and the historical decisions of its neighboring nodes is represented as state s. t ,definition ,in This indicates the local state, which is the backoff window used in the previous transmission; This indicates the overall state, which is the CW level of the average CW received from the broadcast packets from neighboring nodes. S103: The action definition of a node is... That is, the node's decision is based on the backoff window of the previous transmission, either halving, keeping it unchanged, or doubling the window value; the update of the backoff window CW is defined as follows: ; S104: The reward r that a node receives after each transmission is completed. t The feedback obtained for transmission is related to the number of ACKs obtained within the time limit, the backoff delay, and the CW value of neighboring nodes, and is calculated in detail during the node's reception process; S105: After detecting that the channel is idle, the node invokes the Q-Learning algorithm. The input to the algorithm is the node's current observation of the environment, s. t The output is the node's action 'a'. t This refers to the backoff window value CW for this transmission, which is used as the initial value for the backoff counter to begin backoff.

2. The method for UAV ad hoc network channel access in a broadcast scenario according to claim 1, characterized in that, The broadcast scenario includes a drone swarm consisting of N nodes and one channel, wherein... After competing for access to the channel, nodes send data packets via broadcast to exchange information needed for network formation. Based on the characteristic of periodic broadcasting, each node can receive several data packets sent by all surrounding nodes within a certain period of time, and the node maintains a neighbor table according to the source of the data packets. Considering the highly dynamic characteristics of the UAV ad hoc network topology, the node updates the neighbor table according to the source of the broadcast packets received in the last second to ensure that the neighbors are within the communication range.

3. The method for unmanned aerial vehicle (UAV) ad hoc network channel access in a broadcast scenario according to claim 1, characterized in that, In step S1, the value function of Dec-POMDP is defined as follows: It represents each state s∈S in the policy The expected value obtained; the value function represents the quality of the strategy, and its expression is: ； Every state-action pair has a weight, or Q-value, and the optimal Q-value function is expressed as: Therefore, the optimal value function is expressed as Through continuous iterative solutions The maximum value; the elements contained in a single state transition are (s t a t r t s t+1 The update rule for the Q-value function is defined as follows: ； In equation (2), the discount factor γ determines the ratio of current reward to future reward. If γ=0, the decision-maker only focuses on maximizing immediate reward; γ=1 indicates that the decision-maker will strive for a longer-term reward value; α is the learning rate of the decision-maker. When α=1, the decision-maker only considers the information recently obtained, while α=0 means that the decision-maker will not learn any new information.

4. The method for unmanned aerial vehicle (UAV) ad hoc network channel access in a broadcast scenario according to claim 1, characterized in that, In step S3, if the source node receives ACKs from at least two forwarding nodes within the valid time after a transmission, then the source node has successfully received the data; the success rate p s Defined as the ratio of the number of times a node successfully receives data to the total number of data packets sent up to the current moment; the receive rate threshold p th It is a constant; when p s ≥p th When p, the node selects 3 forwarding nodes; s <p th When a node selects two forwarding nodes, it attaches the forwarding node's number to the data packet and broadcasts it.

5. The method for unmanned aerial vehicle (UAV) ad hoc network channel access in a broadcast scenario according to claim 1, characterized in that, In step S5, CW is defined. levels ={3,7,15,31,63,127,255} is the set of backoff window levels. CW is the neighboring CW queue; When a node receives a broadcast packet, it records the CW carried in the MAC header field and puts it into... In CW, the frequency of each CW level appearing in the count queue is used as the basis for CW ranking. levels Sort by frequency of occurrence in descending order; The CW is updated every 1 second to ensure that the queue only stores CWs received within 1 second.

6. The method for unmanned aerial vehicle (UAV) ad hoc network channel access in a broadcast scenario according to claim 5, characterized in that, In step S5, the fairness reward function is defined as follows: ,in Includes CW levels The corresponding 7 reward values, CW t exist The higher the frequency of a decision in the CW queue, the greater the reward for that decision. The delay reward function is defined as follows: ； In equation (3), N ACK The number of valid ACKs received by a node within a specified time limit after a single transmission; Combining the fairness reward function and the time-delay reward function yields the local reward function: ； In equation (4), w is the performance weight coefficient, representing the node's emphasis on latency and fairness; Define R Global (CW t ') represents the overall reward function: ； The local reward function and the global reward function are integrated into the total reward function in Dec-POMDP: ； In equation (5), K is the local weight coefficient.

7. A UAV ad hoc network channel access system for broadcast scenarios, used to implement the method of claim 1, characterized in that, include: The channel listening and parsing module is used to determine whether the channel is idle, whether the received signal power is sufficient to complete packet reception, and the type of packet. After completing the data packet reception, it parses the MAC frame and records the data. The neighbor node perception module maintains the neighbor node table and the neighbor CW table based on the MAC frame data parsed by the channel listening module, and updates them every 1 second. The parameter calculation module uses a node's observation of the environment as the state of the Q-Learning algorithm. This state includes a local state and a global state. The local state is the backoff window used in the previous transmission, and the global state is the CW level corresponding to the average CW in the neighboring nodes' broadcast packets. Node actions include halving, keeping the backoff window unchanged, or doubling it. Based on the state, the module calculates and updates the parameters of the Q-table using the Q-Learning algorithm and outputs the action. The window value corresponding to the action is used as the backoff window value CW. The module also sorts the neighboring CW table according to the frequency of each CW occurrence and calculates the success rate p. s ; The channel access module, based on the Q-table parameters and the successful reception rate p... s Determine the backoff and forwarding strategies, and access the channel for broadcast transmission.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

Images