An Adaptive Risk-Constrained CSMA Access Control Method for Unmanned Aerial Vehicle (UAV) Ad hoc Networks
By employing the Actor-Critic reinforcement learning method, the problem of coordinating service requirements and access risks in UAV ad hoc networks is solved. Adaptive CSMA access control parameters are generated, ensuring smooth parameter updates within the legal domain. This addresses the issues of candidate parameter out-of-bounds errors and window recovery obstacles, thereby improving network stability and efficiency.
Patent Information
- Application Number
- CN202611613609.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-19
- Publication Date
- 2026-11-17
AI Technical Summary
In drone ad hoc networks, existing technologies struggle to effectively coordinate business needs, access risks, and mobile link risks, leading to improper parameter adjustments, resulting in candidate parameter out-of-bounds errors, obstructed window recovery, and periodic update interference with backoff progress.
The Actor-Critic reinforcement learning method is adopted to model business requirements and access risks respectively. Continuous candidate parameters are generated through dynamic constraints. Combined with CSMA contention window and transmission probability, adaptive control of risk level is achieved to ensure that parameters are smoothly updated within the legal domain and backoff is executed.
It enables adaptive adjustment of parameters in UAV ad hoc networks, avoids candidate parameter out-of-bounds errors and window recovery obstruction, maintains the continuity and effectiveness of backoff progress, and improves network stability and efficiency.
Smart Images

Figure CN123124119A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of wireless communication and flight ad hoc network technology, and relates to a method that uses Actor-Critic reinforcement learning to generate candidate parameters, combines access congestion risk and actual next-hop mobile link risk, and jointly controls CSMA contention window and transmission probability. Background Technology
[0002] Unmanned aerial vehicle (UAV) ad hoc networks are characterized by rapid topology changes, limited link duration, and fluctuating loads. Adjusting access parameters solely based on traffic backlog or single-type link statistics makes it difficult to differentiate between transmission demand, channel contention, and mobile link risks. During periodic updates, the use of historical parameters for smoothing may cause results to exceed current boundaries, discrete window feedback may hinder window recovery, and resetting backoff may result in the loss of existing waiting progress. Therefore, it is necessary to coordinate risk constraints, continuous parameter updates, and MAC backoff execution, and to differentiate and utilize information at different rates of change. (Summary of the Invention)
[0003] This invention provides an adaptive risk-constrained CSMA access control method for UAV ad hoc networks, addressing issues such as candidate parameter and smoothing result out-of-bounds errors, quantization feedback hindering window recovery, and periodic updates of interference backoff progress. The specific technical solution is as follows:
[0004] 1. Separate modeling of business requirements and access risks
[0005] Collect queue occupancy rate, queue head waiting age, MAC access statistics, and actual next-hop link status to calculate business demand, access congestion risk, and mobile link risk, avoiding using business backlog as the basis for raising the lower bound of the risk window.
[0006] 2. Dynamic constraints based on hysteresis risk level
[0007] By integrating two types of risks, the risk level is determined by different upward and downward adjustment thresholds. The lower bound of the competition window and the upper bound of the transmission probability are selected, and a dynamic feasible region is constructed by combining the set of windows allowed by the protocol and the relationship between window sizes.
[0008] 3. Candidate parameter generation based on Actor-Critic
[0009] Local observations and control states are input into the Actor to generate a continuous 3D action, which is then mapped to candidate minimum windows, maximum windows, and transmission probabilities. The Critic evaluates the rewards during training and guides Actor updates; the training environment simulates subsequent parameter processing and the timing of their activation.
[0010] 4. Continuous state preservation and valid window quantization
[0011] Candidate parameters undergo boundary correction, are smoothed against historical unquantized continuous states, and then corrected again. The continuous states are saved for use in the next cycle. The same nearest valid value mapping is applied to both windows, and the larger value is taken when the intervals are equal. The quantization result does not replace the continuous state.
[0012] 5. Handover of parameters according to the retreat round.
[0013] Save the parameters to be activated and the active parameters separately. Load the latest effective version for each new backoff round. Normal periodic updates do not change the current active parameters or redraw count; backoff and send probability decisions are made based on the active parameters, and statistics are updated with the actual send results.
[0014] The above processing ensures that the generated parameters meet the corresponding risk boundaries and legal value requirements, avoids the additional window recovery being blocked due to quantitative feedback, and preserves the backoff progress during normal periodic updates. Attached Figure Description
[0015] Figure 1 This is the overall flowchart for periodic parameter control and MAC backoff execution.
[0016] Figure 2 A flowchart for constructing a dynamic feasible region using hysteresis risk levels.
[0017] Figure 3 The flowchart is for continuous state saving, valid quantization, and parameter round-by-round handover. Detailed Implementation
[0018] This embodiment employs a programmable MAC with a configurable contention window and the ability to execute transmission probability decisions, using data unicast with acknowledgment frames. The actual next-hop queue is the controlled object; each queue stores transmission statistics, risk level, and parameter status, shares busy observations of the interface, and a neighbor set. The existing scheduling mechanism selects the access queue.
[0019] The routing or forwarding module provides the next-hop identifier and link-layer destination address; the MAC layer provides access statistics; physical layer measurements and next-hop feedback provide link quality; and the positioning or flight control interface and neighbor maintenance messages provide location, speed, and timestamps. Status is associated with node identifiers and timestamps, and validity is checked. The access control module does not handle routing functions.
[0020] Step 1) Initialize parameters and control state
[0021] Set control cycle Queue capacity Waiting time (reference time) Neighbor information validity period Observation cache validity period Upper limit of prediction Link duration reference value and smoothing coefficient All time parameters are positive, and the set of valid windows is:
[0022] (1)
[0023] The set elements are positive integers and apply to the minimum window, the maximum window, and the actual backoff window. Indicates the number of optional backoff counts, the count is taken from... If the underlying field represents the maximum count value, then write... The initial risk level is 0, and the initial value for continuous state is... ,in , , The initial state must be corrected and quantized during the first effective cycle before it can be used for access.
[0024] Step 2) Collect service requirements and MAC access statistics
[0025] Set period The number of queue groups is Sampling time is The team leader's entry time is Define queue occupancy and waiting age:
[0026] (2)
[0027] (3)
[0028] The waiting period increases with the amount of uncompleted service; groups awaiting confirmation use a consistent in-service counting rule. Demand intensity is... The weights are non-negative and .
[0029] Effective carrier listening time Excluding data transmission and corresponding acknowledgment exchange at this node, the busy time is... The statistical period is determined by the transmission result; the actual number of transmission attempts is... The number of retransmission attempts was The number of attempts that did not receive valid confirmation was Calculations will be performed when the observation duration or sample size reaches the configured lower limit:
[0030] (4)
[0031] When no valid new observations are found, the readings do not exceed [number missing]. The most recent valid value; a preset initial estimate is used when the cache is missing or expired. Only the actual cache refresh time is observed; no sent sample does not equal successful transmission. Probabilistic rejections and pre-transmission cancellations are not included in the above count; retransmission identity is determined based on the failure history of the packet at the start of transmission. Access congestion risk is:
[0032] (5)
[0033] Reflecting access pressure such as busyness, retransmission, and failed confirmation does not equate to the probability of collision; business requirements participate in candidate parameter generation through Actors, which does not directly increase the lower bound of the risk window.
[0034] Step 3) Calculate the actual next-hop mobile link risk
[0035] Push the neighboring locations to the same sampling time; the state has not exceeded [a certain threshold]. And the predicted distance is less than the communication range. The nodes form an effective neighbor set. The relative position and velocity of this node to the actual next hop are respectively... , All use the same coordinate system.
[0036] Under the assumption of constant relative velocity over short time, let... , , , , The estimated link duration is:
[0037] (6)
[0038] make The second branch covers zero relative velocity, and the third branch... During numerical calculations, Available Calculate the root value of the third branch, rounding down any small negative discriminant value to 0. Define relative mobility intensity, neighbor change rate, and link quality:
[0039] (7)
[0040] (8)
[0041] (9)
[0042] in, This represents the symmetric difference between sets, where the previous set is initialized to equal the current set upon the first observation. The signal-to-interference-plus-noise ratio (SIR) of the local node's message fed back to the next hop at its receiving end. Units are consistent; for missing feedback, the caching rules of step 2 are followed and the validity flag is retained; the reverse reception quality is not directly regarded as the measured positive quality. Definition:
[0043] (10)
[0044] (11)
[0045] The four risk weights are non-negative and sum to 1. . Used to select parameter boundaries, it does not represent the probability of link outage as defined by probability. No backoff is initiated when the queue is empty; when the next hop fails, access to the corresponding queue is suspended, and the process is handled by neighbor maintenance and routing mechanisms. Furthermore, the two types of risks described above have different temporal characteristics in terms of their rate of change. Access congestion risk Ct consists of the effective listening busy ratio, retransmission attempt ratio, and failure ratio. These quantities can change significantly within adjacent control cycles as services arrive and channel contention occurs, thus primarily reflecting the rapid fluctuations in the current access process's operational status. Mobile link risk Mt consists of relative mobility intensity, the degree of change in the neighbor set, normalized link quality, and the expected duration of the link. These quantities depend on the node's motion state and topology evolution process, accumulating slowly over a longer period, thus primarily reflecting the slow evolutionary characteristics of the external scenario and link conditions. Therefore, the comprehensive access risk Ht can be considered a multi-rate representation of access pressure: the fast-changing component characterizes the short-term fluctuations in operational status, while the slow-changing component characterizes the long-term changes in scenario conditions. The two are combined by Equation (11) to jointly determine the current risk level, so that the feasible domain of the parameters can be tightened in time when business emergencies cause short-term intensification of competition, and can also be smoothly adjusted with the evolution of link conditions caused by node movement and topology changes, avoiding parameter decisions based solely on information of a single rate.
[0046] Step 4) Update the backlash level and construct the dynamic feasible region.
[0047] Set Level and strictly increasing upregulation and downregulation threshold sequences, satisfying .from In the beginning, if and Then the level will be raised by 1 level; otherwise, if and If the level is lowered by 1, repeat the judgment until no further adjustments are needed, allowing a cycle to span multiple levels.
[0048] grade Corresponding minimum window lower bound Maximum window lower bound and the upper bound of probability ,satisfy , , When the level increases No reduction No increase; the current feasible domain is:
[0049] (12)
[0050] The feasible region is not empty. Within the hysteresis interval, the same score may correspond to different boundaries due to different historical levels; this embodiment adopts a conservative configuration that limits access strength when the risk increases, and checks the validity of the next hop and the remaining exchange time before sending.
[0051] Step 5) Generate continuous candidate parameters using Actor
[0052] (13)
[0053] in, include ; This includes activity parameters, parameters to be activated and their existence or validity flags, remaining backoff count, actual number of packet failures, access phase and phase duration; This includes the observation validity indicator and the cache age. The continuous quantities are normalized according to the preset scale, and the levels and stages are categorized and coded. Missing slots are filled with fixed values and attached with valid labels.
[0054] Actor distribution based on conditions Sampling action Mapped to candidate parameters:
[0055] (14)
[0056] Action sampling and MAC transmission probability decision-making are performed separately. The Critic only evaluates the reward and guides Actor updates during the training phase; the training rules are described later.
[0057] Step 6) Correcting smoothing and valid window quantization
[0058] Candidate parameters are corrected according to the current boundary, and the maximum window correction uses the already corrected minimum window:
[0059] (15)
[0060] Let the above component-wise correction operator be denoted as The first correction result is Continuous state Save the unquantized real values and transmission probabilities of both windows, and update them as follows:
[0061] (16)
[0062] After smoothing, make further adjustments to prevent historical states from exceeding the tightened boundaries. Save. For use in the next cycle, not replaced by a quantization window or activity parameters. The same nearest valid value mapping is applied to both windows. When the distances are equal, select the larger one to generate:
[0063] (17)
[0064] because Furthermore, the mapping is non-subtractive, and the quantized result still satisfies the window boundary and size relationships. Finite-precision probabilities only exist when... Select a representable value within the interval, and configure the endpoints of the interval as representable values. Continuous states are used for subsequent smoothing, parameters to be activated are used for round handover, and active parameters are used for execution in this round; all three are saved separately.
[0065] Step 7) Load parameters according to round and execute access.
[0066] The complete set of parameters to be effective, along with their associated control cycle and the version identifier of the next hop, are published atomically. Upon initial team creation, change of the next hop, or restoration of validity, valid parameters are generated at the first subsequent control moment; no new backoff rounds are initiated before this. At any given moment, the published parameters are published first, followed by the loaded parameters; at other moments, the latest valid version is loaded.
[0067] The new round latches a complete parameter snapshot as the active parameter, which remains unchanged during the countdown, probability decision, and confirmation period of this round; normal periodic updates only replace parameters that are yet to take effect, without re-sampling the current count or changing the queue to which they belong. The continuous state of the overwritten version still participates in subsequent calculations, and the old round parameters correspond to the risk boundary at the time of their generation.
[0068] Let the actual number of packet transmission failures be . The initial value is 0, and the backoff window for this round is:
[0069] (18)
[0070] The largest window for the activity belongs to The above equation has a solution that does not exceed the window size. From The count is uniformly extracted, frozen during busy channel periods, and decremented according to the effective idle time slots after the underlying MAC idle interval is met. After the count reaches zero, the next-hop validity, carrier sensing, and transmission conditions are rechecked, and the remaining link time is predicted to cover data, inter-frame interval, acknowledgment exchange, and non-negative configuration margin.
[0071] When the conditions are met, a sending decision is made according to the activity probability. If the decision fails, no sending is made, no confirmation is waited for, no failure is added, or the count is restarted. The current round ends, and at least one new valid idle time slot must pass before entering the next round. Even if a zero count is drawn again, the decision must not be repeated at the original time.
[0072] Upon success, update statistics and end grouping processing; upon failure, [then...]. Add 1, Retreat again when the maximum number of attempts is reached. Discard or report as per protocol. The next group will be reassigned. Confirmation frames and necessary neighbor maintenance messages are not subject to data probability gating.
[0073] When the next hop changes, the old version awaiting activation becomes invalid, and the risk level, continuous status, and dedicated statistics of the new next hop are initialized. Busy observations shared by the interface continue to be used according to their validity period. Next hop failure, route change, or group cancellation can cancel rounds that have not yet been physically sent; cancellation does not count as a failure. Physical sending that has already started is determined based on the actual completion of the target next hop. Each node updates its local observations periodically, and repeats parameter generation when the queue is not empty and the next hop is valid.
[0074] Actor-Critic Strategy Training
[0075] Both Actor and Critic employ two fully connected hidden layers with 64 neurons each and ReLU activation. Actor has six outputs. through Transform and add 1 to obtain the shape parameters of the three Beta distributions. , Sampling actions are performed according to three distributions, and the Critic output value is estimated. The strategy distribution is as follows:
[0076] (19)
[0077] The training environment simulates carrier sensing, backoff, and data confirmation in time slots, and records the status, actions, reports, and next status according to the control cycle. Corrections, smoothing, quantization, parameter overwrite, and delayed implementation are consistent with deployment; at the start of a round, queues, buffers, risk levels, and control and MAC states are reset. Reports are defined as:
[0078] (20)
[0079] in, , , These are, in order, normalized values of the successfully confirmed payload, the actual number of failures, and the number of dropped packets; all reference values are positive. Delivery and dropping of the same packet are not counted repeatedly. The transmission result is assigned to the period based on the completion determination time; dropping follows rules based on overflow, number of attempts, or validity period. This represents the time average of the demand indicators within the period, with all weights being non-negative. Probabilistic rejection disregards actual failures, and zero-failure events do not alter the risk statistics rules for situations without samples.
[0080] ;(twenty one)
[0081] ;(twenty two)
[0082] ;(twenty three)
[0083] in, For network parameters, As a discount factor, This is the true end-of-round marker. Indicates whether the Actor is called in this cycle. This indicates that the gradient descent has stopped. First, calculate the target gradient. and advantages Then minimize the Critic loss separately. And Actor loss When the queue is empty or the next hop is invalid and no Actor is invoked, only the Critic is updated; sample batch truncation still preserves subsequent value estimates.
[0084] The current strategy employs periodic sampling and updates. Nodes sharing the same strategy first aggregate samples of the same version before updating uniformly. To improve sample utilization efficiency, a limited number of repeated updates are allowed for the same batch of samples collected under the current strategy: subsequent updates for that batch are stopped when the policy change exceeds a set threshold, keeping the gradient deviation within a controlled range. Possible implementations include near-end policy optimization methods that prune probability ratios. During deployment, the selected Actor parameters are fixed, and actions are output according to the mean of the Beta distribution; in implementations requiring preservation of randomness, sampling can continue according to the Beta distribution used during training.
Claims
1. An adaptive risk-constrained CSMA access control method for unmanned aerial vehicle (UAV) ad hoc networks, characterized in that, Includes the following steps: Step 1) Collect service demand information, MAC layer access statistics and actual next-hop link status of the queue to be sent, and determine the demand intensity, access congestion risk and mobile link risk respectively; Step 2) Integrate the two types of risks to construct a current dynamic feasible region that limits the minimum contention window, the maximum contention window, the transmission probability boundary, and the minimum window to be no larger than the maximum window; Step 3) Input the business requirement information, current observation and control status into the Actor policy network trained by Actor-Critic reinforcement learning to generate three-dimensional continuous candidate actions, which are mapped to the candidate minimum contention window, maximum contention window and transmission probability. Step 4) Correct the candidate parameters to the current feasible region, smooth them with the unquantized continuous state saved in the previous control cycle, correct them to the current feasible region again, and save them as the continuous state of this cycle for use in subsequent cycles. Step 5) Map the two windows in the continuous state to the set of discrete windows allowed by the protocol to form parameters to be activated that satisfy the current feasible domain boundary and window size relationship; Save the parameters to be activated and the active parameters separately. When a new backoff round begins in this queue, the latest valid parameters to be activated are loaded as the active parameters. Normal control cycle updates only replace the parameters to be activated, without changing the current active parameters or re-extracting the backoff count. Step 6) Execute random backoff and transmission probability decision based on the activity parameters, update the MAC layer access statistics according to the actual transmission results, and repeat the above steps.
2. The method according to claim 1, characterized in that, Demand intensity and access congestion risk are as follows: ;(1) ;(2) in, To control the cycle, These are queue occupancy rate and normalized queue head waiting age, respectively. Non-negative and sum to 1; These are the percentage of busy active listeners, the percentage of retransmission attempts during actual transmission, and the percentage of failures. Effective listeners exclude data from this node and corresponding acknowledgment exchanges. Transmission statistics are determined by the time period at which the transmission occurs, and probabilistic rejection excludes actual transmission samples. If there are no new effective observations, the most recent observation within the validity period is used; otherwise, a preset initial estimate is used. Reading from the cache does not extend the validity period.
3. The method according to claim 1, characterized in that, The severity of insufficient link duration, mobile link risk, and comprehensive access risk are as follows: ;(3) ;(4) in, This represents the normalized relative movement intensity between the node and the actual next hop. The degree of change in the neighbor set of a node. These are the normalized link quality and estimated duration of the actual next hop, respectively. A positive reference value for link duration; The four mobile risk weights are non-negative and sum to 1. ; For the aforementioned access congestion risk; The degree of change in the neighbor set is determined by the symmetric difference and union of the effective neighbor sets in adjacent periods; the expected duration is calculated based on relative position, relative speed, communication range, and prediction upper limit, under the assumption that the relative speed remains constant: it is 0 when the current distance is not less than the communication range, the prediction upper limit is taken when both the current position and the position at the prediction upper limit are within the range, and in other cases, the time of the first departure from the range is taken and limited to the prediction interval; before sending, the validity of the next hop information and the expected remaining link time are rechecked, and the sending probability decision is only made when the next hop is valid and the remaining time covers the data and acknowledgment exchange time and the configuration margin.
4. The method according to claim 1, characterized in that, Risk level The rising and falling thresholds are respectively ,satisfy The two threshold sequences are strictly increasing. Starting from the previous risk level, the risk level will be adjusted upwards when the overall access risk reaches the threshold for the next level, and downwards when it does not exceed the threshold for the current level, and this process will be repeated until the adjustment conditions are no longer met. grade Corresponding minimum window lower bound Maximum window lower bound and the upper bound of probability ,satisfy , , ; For a set of valid windows, When the level increases No reduction No increase; the current feasible domain is: ;(5) in, These are the minimum contention window, the maximum contention window, and the transmission probability, respectively.
5. The method according to claim 4, characterized in that, Actor inputs include business requirements, MAC access statistics, actual next-hop link status, current risk level, previous cycle continuous status, and MAC execution status; MAC execution status includes activity parameters, parameters to be activated, remaining backoff count, actual number of packet transmission failures, and access stage; Three bounded continuous action components Mapped to: ;(6) in, , The candidate minimum window, maximum window, and transmission probability are defined; each action component takes values from... The Critic estimates the expected cumulative reward of the discount and guides the Actor update. The reward includes delivery bonuses and penalties for backlog, waiting, actual failure, and discard. The training environment includes parameter correction, continuous smoothing, valid quantization, and round-based activation processes.
6. The method according to claim 4, characterized in that, Boundary correction for candidate parameters: ;(7) in, Let be the candidate minimum window, maximum window, and transmission probability. Let the component-wise correction be... The first correction result is Smooth and correct the formula as follows, then quantize: ;(8) ;(9) in, ; For the unquantized continuous states of the preceding and following periods, For two windows and probability components; Apply the same non-subtractive nearest valid value mapping to both windows, and take the larger value at equal intervals; save. Parameters pending activation and intended for use in the next cycle. A version identifier is provided for loading new backoff rounds.
