An entropy-driven step length self-adaptive diffusion reinforcement learning channel access method

Abstract

This invention discloses a diffusion reinforcement learning channel access method based on entropy-driven step-size adaptation, comprising: establishing a channel state transition probability matrix based on the channel state between an airborne base station and a ground edge aggregation server; grouping each channel, and randomly selecting one channel group for activation from the channel groups corresponding to the current macroscopic environment state according to a preset weight probability distribution; establishing a channel selection strategy and objective function; deploying a LADSAC model on the airborne base station side and interfacing with the existing ARQ protocol stack, wherein in each time slot, the LADSAC model sequentially performs observation acquisition, environmental perception, step number decision, diffusion denoising, action issuance, result receipt, and learning update to complete the training and update of the LADSAC model; this method solves the problem of balancing decision performance and computational efficiency in channel access under non-stationary emergency communication scenarios, reducing inference latency and energy consumption while maintaining decision quality.

Description

Technical Field

[0001] This invention relates to the field of information engineering, and specifically to a diffusion reinforcement learning channel access method based on entropy-driven step-size adaptive approach. Background Technology

[0002] In emergency scenarios where ground communication infrastructure fails, UAV-borne air base stations (AirBS) can help restore emergency communications by providing low-latency access and maintaining communication resilience. In such scenarios, the wireless environment is both non-stationary and highly time-varying, characteristics typically caused by factors such as continuous movement of network nodes, sudden interference, and frequent changes in network topology. Simultaneously, the onboard platform's computing resources are limited, and energy supply is constrained, making it difficult to balance decision-making performance and computational efficiency during channel access.

[0003] In non-stationary emergency network scenarios, research on airborne base station channel access control mainly follows two technical routes. Model-driven approaches rely on mathematical modeling of channel fading and apply optimization theory to solve problems such as resource allocation and energy efficiency. When the model is accurate, it can obtain better strategies, but its effectiveness decreases under model mismatch or strong non-stationary conditions, and its generalization ability is limited. Data-driven approaches, especially those based on deep reinforcement learning (DRL), learn access strategies through interaction with the environment and can adapt to non-stationary channels to a certain extent, but still have shortcomings in sample efficiency and training stability. To alleviate the above problems, generative diffusion models are introduced in the research to characterize complex data distributions and enhance the stability of strategy exploration and training. For example, diffusion-based soft actor-critic (D2SAC) shows high performance in edge AI-generated content (AIGC) service selection tasks; however, it uses a fixed number of denoising steps, which is difficult to adapt to dynamic non-stationary environments, and easily leads to increased computational overhead and high inference latency.

[0004] While diffusion-based strategies offer advantages in decision-making performance, their inference latency and energy consumption may exceed the limits of latency-sensitive services and endurance-constrained UAV platforms. Therefore, it is necessary to propose a lightweight and adaptive method that reduces computational requirements while maintaining decision quality. Summary of the Invention

[0005] The purpose of this invention is to address the difficulty in balancing decision performance and computational efficiency in channel access under non-stationary emergency communication scenarios, as well as the shortcomings of existing deep reinforcement learning methods based on diffusion models, such as high computational overhead, lack of adaptability with fixed step size, high inference latency, and high energy consumption. This invention proposes a diffusion reinforcement learning channel access method based on entropy-driven adaptive step size to reduce inference latency and energy consumption while maintaining decision quality.

[0006] To achieve the above functions, this invention designs a diffusion reinforcement learning channel access method based on entropy-driven step-size adaptation. For an emergency communication system including an airborne base station, a ground edge aggregation server, and ground users, the following steps S1-S4 are executed to optimize channel access for the non-stationary emergency communication system:

[0007] Step S1: The channel state is represented by a binary variable to characterize the success or failure of data packet transmission in each channel between the air base station and the ground edge aggregation server. Based on the macroscopic environment state, a channel state transition probability matrix is ​​established to characterize the channel state of each channel in the current time slot and the transition probability of the channel state.

[0008] Step S2: Group each channel and preset the corresponding weight probability distribution. From the channel groups corresponding to the current macro environment state, randomly select a channel group to activate according to the weight probability distribution.

[0009] Step S3: Establish a channel selection strategy that balances decision performance and computational efficiency, and construct the objective function of the channel selection strategy;

[0010] Step S4: Deploy the LADSAC model on the airborne base station side and interface it with the existing ARQ protocol stack. In each time slot, the LADSAC model sequentially performs observation and acquisition, environmental perception, step number decision, diffusion denoising, action issuance, result receipt and learning update to complete the training and update of the LADSAC model. Apply the LADSAC model to complete the channel access optimization for non-stationary emergency communication.

[0011] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0012] This invention utilizes a lightweight environmental perception and entropy-driven step-size adaptive mechanism to process historical action-observation sequences and infer the distribution of macroscopic environmental states. It allocates computational resources on demand based on environmental non-stationarity. By combining a diffusion-generated strategy decision-making module and a dual-value evaluation module, and learning a reward function constructed using success rewards and packet loss penalties, the method can quickly adapt to environmental changes under non-stationary channel conditions, achieving superior decision-making performance and computational efficiency. This method improves the algorithm's generalization and robustness while effectively reducing inference latency and energy consumption through on-demand computation, making it suitable for deployment in UAV emergency communication scenarios where computing power and energy are limited. Attached Figure Description

[0013] Figure 1 This is a diagram of a UAV emergency communication system architecture provided according to an embodiment of the present invention;

[0014] Figure 2 This is a structural diagram of the LADSAC model provided according to an embodiment of the present invention;

[0015] Figure 3 This is a learning curve of the long-term average packet loss rate of different algorithms provided in the embodiments of the present invention;

[0016] Figure 4 This is a comparison chart of the long-term average packet loss rate and average inference latency of different algorithms provided in the embodiments of the present invention. Detailed Implementation

[0017] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0018] This invention provides a diffusion reinforcement learning channel access method based on entropy-driven step-size adaptive approach, referring to... Figure 1 This is a diagram of the architecture of a UAV emergency communication system. The emergency communication system includes an air base station (AirBS), a ground edge traffic aggregation server (Edge Server), and ground users; the air base station is the UAV.

[0019] The airborne base station transmits the buffered data packets to the ground edge aggregation server, which then distributes the received services to multiple ground users on demand. Assuming the link between the ground edge aggregation server and the ground users is stable and reliable, this invention only models the downlink from the airborne base station to the ground edge aggregation server.

[0020] Perform the following steps S1-S4 to complete the channel access optimization for the non-stationary emergency communication system:

[0021] Step S1: The channel state is represented by a binary variable to characterize the success or failure of data packet transmission in each channel between the air base station and the ground edge aggregation server. Based on the macroscopic environment state, a channel state transition probability matrix is ​​established to characterize the channel state of each channel in the current time slot and the transition probability of the channel state.

[0022] The specific method for step S1 is as follows:

[0023] In any time slot t, the channel state of the nth channel is determined by a binary variable. It means that among them This indicates a data packet transmission failure in the channel, and is marked as a Bad state. This indicates that data packets were successfully transmitted in the channel, and is recorded as a "Good" state.

[0024] The generation probability of all channel states is determined by the macroscopic environment state of time slot t. Control, among which Indicates low interference mode. To represent the high-interference mode, establish the channel state transition probability matrix. As shown in the following formula:

[0025] ;

[0026] in, This indicates that the current macroeconomic environment is... At that time, the next time slot shifts to the macroscopic environmental state. The conditional probability, for any given... satisfy , This represents the macroscopic environmental state at time t+1. It represents probability.

[0027] Step S2: Group each channel and preset the corresponding weight probability distribution. From the channel groups corresponding to the current macro environment state, randomly select a channel group to activate according to the weight probability distribution.

[0028] To establish the correlation between macroscopic environmental conditions and channel conditions, a two-layer correlation modeling method is proposed, as follows:

[0029] For each macro-environmental state All are pre-set Each channel group and a corresponding weighted probability distribution The set consisting of each channel group is defined as... Each channel packet They are all subsets of the total channel set C; in any time slot t, from the current macroscopic environment state Corresponding channel packet set In, according to the weighted probability distribution Randomly select one channel group as the active channel group, denoted as . Activate channel group The channel state of channel n is determined. The conditional probability for the Good state is as follows:

[0030] ;

[0031] in, Indicates that channel n belongs to The conditional probability of the channel state being Good at that time. Indicates that channel n belongs to At that time, the conditional probability value of the channel state being Good. Indicates that channel n does not belong to The conditional probability value at time , and and satisfy .

[0032] Step S3: Establish a channel selection strategy that balances decision performance and computational efficiency, and construct the objective function of the channel selection strategy;

[0033] The specific method for step S3 is as follows:

[0034] In non-stationary emergency networks, airborne base stations need to select one downlink channel from multiple candidate channels to ensure reliable data packet delivery. Constrained by onboard computing resources, the channel access algorithm must balance decision performance with computational efficiency. Using long-term average packet loss rate and average inference delay as performance metrics, the channel access control problem under the channel selection strategy is expressed as follows: where Indicates the channel selection strategy for each time slot:

[0035] ;

[0036] in, Indicating in channel selection strategy The long-term average packet loss rate is as follows. Indicating in channel selection strategy Average inference latency under the following conditions This is a tradeoff coefficient between the long-term average packet loss rate and the average inference latency. In strategy The comprehensive cost function under, Indicates different channel selection strategies The optimization objective is to minimize the overall cost function; the two performance indicators are calculated as follows:

[0037] ;

[0038] ;

[0039] in, This represents the total number of time slots used for statistics. This indicates whether packet loss occurred in time slot t; it is set to 1 if packet loss occurred, and 0 otherwise. This represents the inference time of time slot t.

[0040] Step S4: Deploy the LADSAC model on the airborne base station side and interface it with the existing ARQ protocol stack. In each time slot, the LADSAC model sequentially performs observation and acquisition, environmental perception, step number decision, diffusion denoising, action issuance, result receipt and learning update to complete the training and update of the LADSAC model. Apply the LADSAC model to complete the channel access optimization for non-stationary emergency communication.

[0041] The airborne base station is configured with a capacity of The first-in, first-out (FIFO) buffer's number of data packets in time slot t is expressed as: For ease of explanation, this invention assumes that the arrival of data packets follows an independent Bernoulli process, that is, at the beginning of each time slot t, a new data packet arrives at... The probability of a packet being generated is determined at the airborne base station. If the generation event occurs and the buffer is not full, the packet enters the FIFO queue. To support downlink transmission, the system provides N mutually orthogonal channels, whose set is defined as... When the cache is not empty ( The task of the airborne base station is to select a channel from set C in the current time slot and send the head packet to the ground edge aggregation server.

[0042] To ensure communication reliability, the system employs the Automatic Repeat reQuest (ARQ) protocol with a limited number of retransmissions. In time slot t, the cumulative number of failures for the data packet at the head of the queue is recorded as follows: Where K is the maximum retransmission limit. If a data packet is successfully transmitted, it is removed from the buffer; if it fails, its retransmission count is updated. Packet loss will occur when the buffer is full when a new data packet arrives, or when the number of retransmissions of the first packet in the queue exceeds the limit K.

[0043] Unlike the traditional Gilbert-Elliott (GE) model, which assumes that channels are independent, this invention uses a Hidden Markov Model (HMM) to model the channels non-stationary, which better reflects the correlation fading caused by factors such as terrain occlusion.

[0044] like Figure 2 The diagram shows the structure of the LADSAC model, and the key parameter settings are shown in Table 1.

[0045] Table 1. Key parameter settings for the LADSAC model

[0046]

[0047] The specific steps of step S4 are as follows:

[0048] Step S4.1: Define the state as ,in This refers to the number of data packets in the buffer of the airborne base station; The number of retransmissions of the first data packet in the queue; This refers to the macro-environmental state, and Unobservable; Represents the set of available channels. ,in This represents the availability vector for each channel, where N is the total number of channels; since and Since it cannot be directly observed, the agent needs to infer the state of the environment through a finite-length sequence of historical actions and observations.

[0049] Step S4.2: Define the action space as Define the action of time slot t as ,when At that time, with This indicates that the nth channel is selected; Indicates idle, where N is the total number of channels;

[0050] Step S4.3: Define the observation space as The observation in time slot t+1 is defined as ,by This indicates that the transmission was successful. This indicates that the transmission failed.

[0051] Step S4.4: Set the observation function. Indicates the state in the previous time slot. and actions Under the given conditions, observations are obtained in the current time slot t. The probability distribution for performing an action; The situation depends on the channel state. With a fixed error rate Decide:

[0052] ;

[0053] ;

[0054] in, Indicates in time slot Channel When the actual state is 1, in the time slot Observation results The conditional probability of 1, i.e., the probability that the observation module correctly determines state 1, has the following value: ; Indicates in time slot Channel When the true state is 0, the observation results The conditional probability that the state is still 1 is the false alarm probability that the observation module misclassifies state 0 as 1; its value is the fixed observation error rate. .

[0055] To infer the state of the unobservable macroscopic environment Construct a historical action-observation sequence of length W in the control plane. And based on historical action-observation sequences Perform state estimation:

[0056] ;

[0057] Synchronization from business plane to strategy module With available channel set C; ARQ receipt with Write in the form .

[0058] Step S4.5: Set the instantaneous reward function The instantaneous reward function is designed to severely punish packet loss while providing positive incentives for successful transmission. As shown in the following formula:

[0059] ;

[0060] in, This indicates successful transmission. Indicates packet loss; , These are the reward and penalty coefficients, respectively. ;

[0061] Step S4.6: Employ a lightweight environment perception network with two layers of MLP. (64 neurons per layer), with As input, output the posterior distribution of the macroscopic environmental state. And calculate information entropy :

[0062] ;

[0063] in, , ,and , indicating in time slot Historical Actions – Observation Sequence At that time, the state of the macro environment In state The posterior probability;

[0064] To suppress short-term fluctuations, an exponential moving average (EMA) is applied to the information entropy:

[0065] ;

[0066] in, Information entropy The exponential moving average estimate, Let be the smoothing factor; initial value is . ;

[0067] To avoid singular values ​​for the logarithm at 0 or 1, the posterior distribution of the macroscopic environmental state is first pinched, as shown in the following formula:

[0068] ;

[0069] in, To make the posterior probability Cut off to interval The resulting value is used to avoid instability caused by values ​​close to 0 or 1 when calculating information entropy; This is the threshold constant;

[0070] Define instantaneous entropy with base 2 The specific formula is as follows:

[0071] ;

[0072] in, For time slots The instantaneous entropy is given by the truncated posterior probability. Calculated.

[0073] for instantaneous entropy Perform an exponential moving average:

[0074] ;

[0075] in, Instantaneous entropy The exponential moving average estimate, This is a smoothing factor used to control the weighting of historical entropy and current instantaneous entropy; its initial value is... .

[0076] Step S4.7: Set the number of diffusion reverse denoising steps based on the instantaneous entropy after exponential moving average, and map it to continuous values ​​using piecewise linear mapping, where... :

[0077] ;

[0078] in, , These are the lower and upper bounds of the entropy threshold, respectively. ; , These are the lower and upper limits of the number of noise reduction steps, respectively. ; The number of consecutive (real) steps;

[0079] The quantization and truncation are performed as follows:

[0080] ;

[0081] in, This indicates rounding down to the nearest integer (half-rounding: when the decimal part is 0.5, take the larger integer). The number of quantized integer steps;

[0082] To suppress jitter near the threshold, a hysteresis band is introduced, as shown in the following formula:

[0083] ;

[0084] Keep the current step number unchanged when the above conditions are met, that is, let the above formula satisfy the condition. ;in, For the hysteresis band half-width, when Keep Keep it unchanged to suppress jitter; This is the integer number of steps after quantization in the previous time slot;

[0085] Step S4.8: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require Combined with the original state to form an augmented state :

[0086] ;

[0087] in, Macro-environmental status The posterior distribution vector; This indicates the original state (including the number of data packets in the buffer). Number of retransmissions of the first data packet (etc.), augmented state As input to the diffusion strategy network;

[0088] Diffusion strategy network In consecutive steps Denoising and generating motion ;like Then the idle action is output directly. And skip the noise calculation; otherwise in Actions are generated under reverse denoising; to ensure that the results are reproducible within the same time slot, reverse denoising uses a pseudo-random number sequence initialized with a fixed seed;

[0089] Step S4.9: Employ a dual-critic network , augmented state-action pairs Value assessment is performed, and the target critic network generates a temporal difference (TD) target. The dual critic networks correspond to the target critic network and have identical structures. The target critic network, as a delayed copy of the corresponding critic network, is only used to generate the temporal difference target; its parameters are obtained from the corresponding critic network through soft updates to improve training stability. The replay samples obtained through interaction are... Write the format to playback storage. This represents the value of the instantaneous reward function;

[0090] The main network parameters are jointly updated using gradient descent:

[0091] ;

[0092] in, , , These are the parameter vectors for the policy network, the critic network, and the environment-aware network, respectively, and their values ​​during training are... The values ​​taken in the next iteration are denoted as follows: , , ; The learning rate; For joint losses; Represents the loss function For parameters , , The joint gradient vector.

[0093] To maintain consistency with the main network, a soft update is performed on the target network:

[0094] ;

[0095] in, This is a soft update coefficient; , , These are the parameter vectors for the target commentator network, the target policy network, and the target environment perception network, respectively, which are the parameters of the target commentator network, the target policy network, and the target environment perception network during training. The values ​​taken in the next iteration are denoted as follows: , , ; , , These are the corresponding main network parameters.

[0096] Step S4.10: Construct the joint loss using the replay samples :

[0097] ;

[0098] in, For the spread of loss, A loss for critics, To sense cross-entropy loss, The learning rate is used as the weighting factor. Gradient descent joint parameter update , , .

[0099] In the embodiment, constraints are imposed on the inference time for each time slot t. ;

[0100] when If the timeout occurs, a downgrade will be implemented, as shown in the following formula:

[0101] ;

[0102] in, Let t be the inference time of time slot t. The maximum allowable inference delay; The number of quantized integer steps. This is the lower limit of the number of steps;

[0103] After the downgrade is completed, skip the double commentator update in time slot t; if there are M consecutive timeouts, switch to a conservative strategy in the next time slot, as shown in the following formula:

[0104] ;

[0105] in, To be based on historical receipts The empirical success rate of channel n obtained through statistics This represents the empirically optimal channel. For a more conservative approach, you can... Replace with (idle).

[0106] like Figure 3 The diagram shows a comparison of the long-term average packet loss rates during training of the SAC algorithm, the D2SAC algorithm, and the LADSAC algorithm proposed in this invention under a non-stationary emergency communication environment. It can be seen that the classic SAC algorithm has shortcomings in both learning efficiency and final performance; its convergence speed is slow and its packet loss rate is high, making it difficult to adapt to complex non-stationary channel environments. In contrast, the D2SAC algorithm, relying on the advantages of the diffusion model in policy exploration, achieves better convergence results with a significantly lower packet loss rate than SAC, but its volatility during convergence remains relatively high. Furthermore, the LADSAC algorithm proposed in this invention demonstrates particularly outstanding performance, not only converging to the lowest packet loss rate at the fastest rate but also exhibiting a significantly narrowed standard deviation range, fully demonstrating the superior decision-making performance and high robustness of this method in non-stationary environments.

[0107] like Figure 4 The figure shows a performance comparison of SAC, D2SAC, and the LADSAC algorithm proposed in this invention under the final strategy. Figure 4 (a) and Figure 4 (b) shows the results of the three algorithms from the two dimensions of decision performance and computational efficiency. It can be seen that... Figure 4 In terms of the long-term average packet loss rate in (a), the SAC algorithm performed the worst, with a final long-term average packet loss rate of 25.9%; the D2SAC algorithm, relying on the diffusion mechanism, reduced the packet loss rate to some extent, resulting in 21.0%; while the LADSAC algorithm proposed in this invention performed the best, with a final long-term average packet loss rate of only 17.2%, which is about 18% and 34% higher than D2SAC and SAC respectively, significantly optimizing the system's decision-making performance. Meanwhile, in Figure 4 In (b) of the figure, the average inference latency of the D2SAC algorithm is 4.6 milliseconds, which is insufficient for latency-sensitive scenarios. In contrast, the LADSAC algorithm of this invention optimizes the inference latency to 2.4 milliseconds, an improvement of approximately 48% compared to D2SAC, demonstrating its significant advantage in computational efficiency. Furthermore, as can be seen from the error bars in the figure, the variance of LADSAC in both performance dimensions is significantly smaller than that of the comparative algorithms, indicating that its final strategy has higher consistency and robustness.

[0108] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A diffusion reinforcement learning channel access method based on entropy-driven step-size adaptive approach, characterized in that, For emergency communication systems including airborne base stations, ground edge aggregation servers, and ground users, perform the following steps S1-S4 to optimize channel system access for non-stationary emergency communication: Step S1: The channel state is represented by a binary variable to characterize the success or failure of data packet transmission in each channel between the air base station and the ground edge aggregation server. Based on the macroscopic environment state, a channel state transition probability matrix is ​​established to characterize the channel state of each channel in the current time slot and the transition probability of the channel state. Step S2: Group each channel and preset the corresponding weight probability distribution. From the channel groups corresponding to the current macro environment state, randomly select a channel group to activate according to the weight probability distribution. Step S3: Establish a channel selection strategy that balances decision performance and computational efficiency, and construct the objective function of the channel selection strategy; The specific method for step S3 is as follows: Using long-term average packet loss rate and average inference delay as performance metrics, the channel access control problem under the channel selection strategy is expressed as follows: where Indicates the channel selection strategy for each time slot: ； in, Indicating in channel selection strategy The long-term average packet loss rate is as follows. Indicating in channel selection strategy Average inference latency under the following conditions This is a tradeoff coefficient between the long-term average packet loss rate and the average inference latency. In strategy The comprehensive cost function under, Indicates different channel selection strategies The optimization objective is to minimize the overall cost function; the two performance indicators are calculated as follows: ； ； in, This represents the total number of time slots used for statistics. This indicates whether packet loss occurred in time slot t; it is set to 1 if packet loss occurred, and 0 otherwise. Indicates the inference time of time slot t; Step S4: Deploy the LADSAC model on the airborne base station side and interface it with the existing ARQ protocol stack. In each time slot, the LADSAC model sequentially performs observation and acquisition, environmental perception, step number decision, diffusion denoising, action issuance, result receipt and learning update to complete the training and update of the LADSAC model. Apply the LADSAC model to complete the channel access optimization for non-stationary emergency communication.

2. The method for channel access based on entropy-driven step-size adaptive diffusion reinforcement learning according to claim 1, characterized in that, The specific method for step S1 is as follows: In any time slot t, the channel state of the nth channel is determined by a binary variable. It means that among them This indicates a data packet transmission failure in the channel, and is marked as a Bad state. This indicates that data packets were successfully transmitted in the channel, and is recorded as a "Good" state. The generation probability of all channel states is determined by the macroscopic environment state of time slot t. Control, among which Indicates low interference mode. To represent the high-interference mode, establish the channel state transition probability matrix. As shown in the following formula: ； in, This indicates that the current macroeconomic environment is... At that time, the next time slot shifts to the macroscopic environmental state. The conditional probability, for any given... satisfy , This represents the macroscopic environmental state at time t+1. It represents probability.

3. The method for channel access based on entropy-driven step-size adaptive diffusion reinforcement learning according to claim 2, characterized in that, The specific method for step S2 is as follows: For each macro-environmental state All are pre-set Each channel group and a corresponding weighted probability distribution The set consisting of each channel group is defined as... Each channel packet They are all subsets of the total channel set C; in any time slot t, from the current macroscopic environment state Corresponding channel packet set In, according to the weighted probability distribution Randomly select one channel group as the active channel group, denoted as . Activate channel group The channel state of channel n is determined. The conditional probability for the Good state is as follows: ； in, Indicates that channel n belongs to The conditional probability of the channel state being Good at that time. Indicates that channel n belongs to At that time, the conditional probability value of the channel state being Good. Indicates that channel n does not belong to The conditional probability value at time , and and satisfy .

4. The diffusion reinforcement learning channel access method based on entropy-driven step-size adaptive according to claim 3, characterized in that, The specific steps of step S4 are as follows: Step S4.1: Define the state as ,in This refers to the number of data packets in the buffer of the airborne base station; The number of retransmissions of the first data packet in the queue; This refers to the macro-environmental state; Represents the set of available channels. ,in This represents the availability vector for each channel, where N is the total number of channels; Step S4.2: Define the action space as Define the action of time slot t as ,when At that time, with This indicates that the nth channel is selected; Indicates idle, where N is the total number of channels; Step S4.3: Define the observation space as The observation in time slot t+1 is defined as ,by This indicates that the transmission was successful. This indicates that the transmission failed. Step S4.4: Set the observation function. Indicates the state in the previous time slot. and actions Under the given conditions, observations are obtained in the current time slot t. The probability distribution for performing an action; The situation depends on the channel state. With a fixed error rate Decide: ； ； in, Indicates in time slot Channel When the actual state is 1, in the time slot Observation results The conditional probability that is 1; Indicates in time slot Channel When the true state is 0, the observation results The conditional probability is still 1; Construct a historical action-observation sequence of length W. And based on historical action-observation sequences Perform state estimation: ； Step S4.5: Set the instantaneous reward function As shown in the following formula: ； in, This indicates successful transmission. Indicates packet loss; , These are the reward and penalty coefficients, respectively. ; Step S4.6: Employ a lightweight environment perception network with two layers of MLP. ,by As input, output the posterior distribution of the macroscopic environmental state. And calculate information entropy : ； in, , ,and , indicating in time slot Historical Actions – Observation Sequence At that time, the state of the macro environment In state The posterior probability; Apply an exponential moving average to the information entropy: ； in, Information entropy The exponential moving average estimate, Let be the smoothing factor; initial value is . ; Step S4.7: Set the number of diffusion reverse denoising steps based on the information entropy estimated by the exponential moving average, where : ； in, , These are the lower and upper bounds of the entropy threshold, respectively. ; , These are the lower and upper limits of the number of noise reduction steps, respectively. ; The number of consecutive steps; The quantization and truncation are performed as follows: ； in, This means rounding down to the nearest integer. The number of quantized integer steps; Introducing the hysteresis band, specifically as follows: ； The above formula satisfies the time constraint. ;in, For the hysteresis band half-width, when Keep Keep it unchanged to suppress jitter; This is the integer number of steps after quantization in the previous time slot; Step S4.8: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require Combined with the original state to form an augmented state : ； in, Macro-environmental status The posterior distribution vector; Represents the original state and the augmented state. As input to the diffusion strategy network; by the diffusion strategy network In consecutive steps Denoising and generating motion ; Diffusion strategy network In consecutive steps Denoising and generating motion ;like Then the idle action is output directly. And skip the noise calculation; otherwise in Actions are generated under reverse denoising; to ensure that the results are reproducible within the same time slot, reverse denoising uses a pseudo-random number sequence initialized with a fixed seed; Step S4.9: Employ a dual-critic network , augmented state-action pairs Value assessment is performed, and a temporal difference objective is generated by the target critic network; the replay samples obtained through interaction are used as... Write the format to playback storage. This represents the value of the instantaneous reward function; The main network parameters are jointly updated using gradient descent: ； in, , , These are the parameter vectors for the policy network, the critic network, and the environment-aware network, respectively, and their values ​​during training are... The values ​​taken in the next iteration are denoted as follows: , , ; The learning rate; For joint losses; Represents the loss function For parameters , , The joint gradient vector; Perform a soft update on the target network: ； in, This is a soft update coefficient; , , These are the parameter vectors for the target commentator network, the target policy network, and the target environment perception network, respectively, which are the parameters of the target commentator network, the target policy network, and the target environment perception network during training. The values ​​taken in the next iteration are denoted as follows: , , ; , , These are the corresponding main network parameters; Step S4.10: Construct the joint loss using the replay samples : ； in, For the spread of loss, A loss for critics, To sense cross-entropy loss, The learning rate is used as the weighting factor. Gradient descent joint parameter update , , .

5. The diffusion reinforcement learning channel access method based on entropy-driven step-size adaptive according to claim 4, characterized in that, In step S4.6, to avoid singular values ​​for the logarithm at 0 or 1, the posterior distribution of the macroscopic environmental state is first pinched, as shown in the following formula: ； in, To make the posterior probability Cut off to interval The value obtained later; This is the threshold constant; Define instantaneous entropy with base 2 The specific formula is as follows: ； in, For time slots The instantaneous entropy is given by the truncated posterior probability. Calculated; for instantaneous entropy Perform an exponential moving average: ； in, Instantaneous entropy The exponential moving average estimate, This is a smoothing factor used to control the weighting of historical entropy and current instantaneous entropy; its initial value is... .

6. The diffusion reinforcement learning channel access method based on entropy-driven step-size adaptive according to claim 5, characterized in that, In step S4, constraints are imposed on the inference time for each time slot t. ; when If the timeout occurs, a downgrade will be implemented, as shown in the following formula: ； in, Let t be the inference time of time slot t. The maximum allowable inference delay; The number of quantized integer steps. This is the lower limit of the number of steps; After the downgrade is completed, skip the dual-commenter network update in time slot t; if there are M consecutive timeouts, switch to a conservative strategy in the next time slot, as shown in the following formula: ； in, Let n be the empirical success rate for channel n. This represents the empirically optimal channel.

7. The method for channel access based on entropy-driven step-size adaptive diffusion reinforcement learning according to claim 6, characterized in that, After the downgrade is completed, if there are M consecutive timeouts, the conservative strategy for switching in the next time slot is as follows: , indicating the action of selection Available.

Images