Improved Q-learning channel access method based on composite reward function

Abstract

The invention provides an improved Q-learning channel access method based on a composite reward function, and belongs to the technical field of communication. Comprising the following steps: an initialization algorithm: deploying sensor nodes and gateways, clustering and selecting cluster heads from nodes in clusters; locally initializing nodes, maintaining a Q table and a neighbor table by each node, and defining a state space, an action space, an initial parameter and a competition intensity threshold value; each node executes actions and observes in the current state, when a conflict does not pass in the data sending process, a composite reward participating in Q value updating is calculated, layered Q value updating based on the competitive strength level is firstly carried out after updating to the next state, a neighbor table is updated by the node in the Texchange period, and the neighbor table is updated by the node in the Texchange period; and in the Tcluster period, the weighted average Q value in the cluster is calculated by the cluster head and is broadcasted, and after the cluster member updates the local Q table, the action is executed and the observation is carried out in the next state until the conflict is solved and the node information is successfully sent. According to the method, the conflict probability in a high-competition environment can be effectively reduced, and the channel access efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to an improved Q-learning channel access method based on a composite reward function. Background Technology

[0002] In recent years, with the rapid development of Internet of Things (IoT) technology, the number of nodes in wireless sensor networks (WSNs) has grown exponentially, leading to increasingly prominent problems such as intensified competition for channel resources, communication conflicts, and decreased network efficiency. In traditional large-scale sensor network deployments, nodes typically employ a channel access mechanism based on Carrier Sense Multiple Access / Collision Avoidance (CSMA / CA). However, this method is prone to decreased throughput and increased latency in high-density node environments due to intensified competition, making it difficult to adapt to dynamically changing network loads.

[0003] To optimize channel access efficiency in distributed networks, existing technologies have proposed access methods based on random contention, such as Carrier Sense Multiple Access / Collision Avoidance (CSMA / CA). This method is simple and effective in low-density networks, relatively easy to implement, and does not require complex signaling interactions, making it suitable for large-scale deployment scenarios. However, it has some drawbacks, such as multiple terminals potentially choosing the same preamble, leading to collisions that require a contention resolution mechanism. The collision probability is positively correlated with the number of users, affecting the access success rate. Therefore, in high-contention environments, the increased collision probability can cause a sharp decline in performance. Existing technical solutions suffer from high collision probability and low channel access efficiency in high-contention environments. Summary of the Invention

[0004] In view of this, the present invention provides an improved Q-learning channel access method based on a composite reward function. The gateway selection problem is modeled as a Markov decision process. A composite reward function that integrates channel state, queue load and contention intensity is designed. A neighbor node information sharing mechanism is introduced to reduce the state space dimension. A hierarchical update strategy is adopted to accelerate convergence, enabling nodes to autonomously optimize access decisions, effectively reducing the probability of conflict in a high-contention environment and improving channel access efficiency.

[0005] The technical solution adopted by the embodiments of the present invention to solve its technical problem is as follows:

[0006] An improved Q-learning channel access method based on a composite reward function includes:

[0007] Step S1, Initialize the algorithm: Deploy sensor nodes and gateways, cluster them and select cluster heads from the nodes in the clusters; initialize nodes locally, each node maintains a Q table and a neighbor table, and defines the state space, action space, initial parameters and contention intensity threshold;

[0008] In step S2, each node performs actions and observes in the current state. If there is a conflict during the data transmission process, the composite reward for participating in the Q-value update is calculated. After updating to the next state, a hierarchical Q-value update based on the competition intensity level is performed first. During the T_exchange cycle, the node updates the neighbor table. During the T_cluster cycle, the cluster head calculates the weighted average Q-value within the cluster and broadcasts it, causing cluster members to update their local Q-tables. Then, in the next state, the node performs actions and observes until the conflict is resolved and the node information is successfully sent.

[0009] Preferably, step S1 includes:

[0010] Step S11, Network Topology Construction: Deploy N sensor nodes and several gateways. The nodes are connected wirelessly via multi-hop to form a self-organizing network. Define a communication radius R. If the distance between nodes is ≤ R, then a communication link exists.

[0011] Step S12, Cluster structure establishment: Divide the network into K clusters, select the node with the maximum selectivity from the clusters as the cluster head, and the cluster members maintain the routes to the cluster head;

[0012] Step S13, Node local initialization:

[0013] Q-table initialization: Each node maintains a Q-table, the dimension of which is state space × action space;

[0014] State space: S = (CQI_level, QL_level, CI_level) consists of discrete levels of channel quality indicator (CQI), discrete levels of queue load (QL), and discrete levels of contention intensity (CI).

[0015] Action space: A = {0, 1, 2, ..., C}, where 0 represents backoff and C represents channel number C;

[0016] The initial value of table Q is either all zeros or small random numbers;

[0017] Neighbor table initialization: Each node maintains a neighbor table, recording the node IDs, average recent Q-values, and access success rates of neighbors within k hops;

[0018] Parameter initialization: exploration rate ε, base learning rate η0, discount factor λ, reward weight α, latency β, fairness γ, competition intensity threshold CI threshold .

[0019] Preferably, in step S1, the values ​​of CQI, QL, and CI are discretized to define the discrete levels corresponding to the intervals:

[0020] The five levels of CQI, according to the RSSI / noise ratio from smallest to largest, correspond to the intervals [0,3], [4,7], [8,11], [12,15], [15,+∞];

[0021] The five levels of QL correspond to the following ranges in ascending order of the ratio of the number of data packets to be sent to the queue capacity: [0, 20%], [21%, 40%], [41%, 60%], [61%, 80%], and [81%, 100%].

[0022] The three levels of CI correspond to the intervals [0,4], [5,9], and [10,+∞] respectively, according to the number of nodes transmitting in the k-hop neighbors from smallest to largest.

[0023] Preferably, step S2 includes:

[0024] Step S21, at time t, sense the environmental state s t Get the current state s t =(CQI_level,QL_level,CI_level), specifically:

[0025] CQI_level: Monitor the channel, calculate the RSSI / noise ratio and map the ratio to 5 discrete level intervals to obtain the corresponding discrete level;

[0026] QL_level: Calculates the number of data packets to be sent / queue capacity and maps the ratio to 5 discrete level intervals to obtain the corresponding discrete level;

[0027] CI_level: Count the number of nodes currently transmitting in the k-hop neighbors and map them to three discrete level intervals to obtain the corresponding discrete level;

[0028] Step S22, select action a t Action a is randomly selected with probability ε. t Explore ∈A and choose the current state s with probability 1-ε. t The action with the largest Q value: a t =argmax_{a}Q(s t ,a);

[0029] Step S23, perform the action and observe:

[0030] If a t =0, set backoff timer, wait for random time slot;

[0031] If a t If the value is >0, access channel c and transmit data on channel c; detect whether a collision has occurred and record the transmission result, including success and failure, and the transmission delay D.t If the detection result is that the collision is passed, the signal is successfully sent and the process ends; if the detection result is that the collision is not passed, continue to execute step S24.

[0032] Step S24, calculate the compound reward R t :R t =α*T t +β*(1 / D t )+γ*F t , among which, T t = Number of bits successfully transmitted in this time slot / Maximum possible number of bits transmitted; Delay factor 1 / D t D t This is the transmission delay; F t To ensure fairness, the calculation is based on the success rate of the neighboring nodes' most recent 10 transmissions;

[0033]

[0034] x j Let be the transmission success rate of node j;

[0035] Step S25, observe the new state s t+1 At the start of the next time slot, re-sensing the channel, queue, and contention intensity, and obtaining s t+1 ;

[0036] Step S26, Layered Q-value update:

[0037] Calculate the dynamic learning rate η:

[0038]

[0039] In the formula, N neighbors Number of neighbors;

[0040] Then perform layered updates:

[0041] If CI_level <CI threshold This indicates relatively low competition intensity, and the update method is as follows:

[0042]

[0043] In the formula, a' represents the action;

[0044] If CI_level>=CI threshold This indicates high competition intensity, and the update method is as follows:

[0045] Calculate the mean Q-value of the neighbors. neighbor_avg ,

[0046] Q neighbor_avg = Neighbor nodes in the neighbor table in state st Action a t The mean Q value;

[0047] Calculate Q local Q local This represents the Q value of the local node;

[0048]

[0049] Then Q(s) t ,a t )=0.7*Q local +0.3*Q neighbor_avg

[0050] Furthermore, if the Q value exceeds 100, it is truncated to 100;

[0051] Step S27, Neighbor Information Exchange:

[0052] Every 10 time slots with T_exchange=10, this node performs a broadcast and neighbor table update operation:

[0053] Broadcast the average Q-value of this node and the success rate of the last 10 accesses. The average Q-value of this node refers to the average Q-value of all state-action pairs.

[0054] Receive broadcasts from neighboring nodes and update the neighbor table;

[0055] Every 100 time slots, perform a cluster member update operation on the Q table:

[0056] Cluster members send a local Q-table digest to the cluster head, the local Q-table digest including the mean Q-value;

[0057] The cluster head calculates the weighted average Q value within the cluster and adds Laplace noise. The weighted average Q value is weighted by the node degree in the calculation.

[0058] The global Q value after cluster head broadcast perturbation;

[0059] Cluster members update their local Q-table: Q_i = (1-ρ)*Q_i + ρ*Q_global; where ρ is the synchronization rate coefficient;

[0060] Step S28, parameter adaptation:

[0061] Exploration rate decay: ε = max(0.1, ε * 0.999);

[0062] When a conflict occurs, the backoff window value CW = min(1024, CW*2) is adjusted for competition window adjustment. If there is no conflict, then CW = 16.

[0063] Step S29: Return to step S21 and continue until the third step detects the conflict.

[0064] Preferably, in step S12, maximum selectivity refers to having the maximum number of neighboring nodes.

[0065] Preferably, in step S2, the parameter initialization values ​​are: exploration rate ε = 0.3, base learning rate η0 = 0.5, discount factor λ = 0.9, reward weight α = 0.5, latency β = 0.3, and fairness γ = 0.2; CI threshold =5; ρ=0.2.

[0066] As can be seen from the above technical solution, the improved Q-learning channel access algorithm based on a composite reward function provided in this embodiment of the invention first initializes the algorithm, deploys sensor nodes and gateways, clusters nodes, and selects a cluster head from the nodes in the cluster. Nodes initialize locally, each node maintains a Q-table and a neighbor table, and defines a state space, action space, initial parameters, and a contention strength threshold. Each node performs actions and observes in the current state. When a data transmission process encounters a conflict, the composite reward for participating in the Q-value update is calculated. After updating to the next state, a hierarchical Q-value update based on the contention strength level is performed. During the T_exchange cycle, the node updates its neighbor table. During the T_cluster cycle, the cluster head calculates the weighted average Q-value within the cluster and broadcasts it, causing cluster members to update their local Q-tables. Then, in the next state, actions are performed and observations are conducted until a conflict is resolved. This invention models the gateway selection problem as a Markov decision process, designs a composite reward function that integrates channel state, queue load, and contention strength, introduces a neighbor node information sharing mechanism to reduce the dimensionality of the state space, and adopts a hierarchical update strategy to accelerate convergence. This enables nodes to autonomously optimize access decisions, effectively reducing the probability of conflicts in high-contention environments and improving channel access efficiency. Attached Figure Description

[0067] Figure 1 This is a flowchart of the improved Q-learning channel access method based on a composite reward function according to the present invention.

[0068] Figure 2 The performance metrics are illustrated for different network sizes. Detailed Implementation

[0069] The technical solution and effects of the present invention will be further described in detail below with reference to the accompanying drawings.

[0070] This invention addresses the slow convergence speed of traditional Q-learning in ultra-large-scale networks (>1000 nodes) by proposing a novel reinforcement learning access algorithm that improves performance through the following innovations: designing a composite reward function to jointly optimize throughput, latency, and fairness; introducing a neighbor node information sharing mechanism to reduce the dimensionality of the state space; and employing a hierarchical update strategy to accelerate convergence. (Reference) Figure 2 As shown, the improved Q-learning channel access method based on a composite reward function of the present invention includes the following specific implementation steps:

[0071] Initialization algorithm:

[0072] 1. Network topology construction:

[0073] Deploy N sensor nodes (N≥1000) and several gateways. The nodes are connected wirelessly via multi-hop to form a self-organizing network. Define a communication radius R. If the distance between nodes is ≤R, then a communication link exists.

[0074] 2. Establishment of cluster structure:

[0075] The network is divided into K clusters, each containing approximately N / K nodes (e.g., K = N / 50). A cluster head (CH) is elected: the node with the highest degree (number of neighbors) is chosen as the cluster head, responsible for intra-cluster coordination. Cluster members (CMs) maintain routes to the cluster head.

[0076] 3. Node local initialization:

[0077] Q-table initialization: Each node maintains a Q-table, whose dimensions are state space × action space.

[0078] The state space S, consisting of triples (CQI, QL, CI), is discretized as follows:

[0079] CQI (Channel Quality Indicator): 5 levels (0-3dB, 4-7dB, 8-11dB, 12-15dB, >15dB)

[0080] QL (Queue Load): 5 levels (0-20%, 21-40%, 41-60%, 61-80%, 81-100%)

[0081] CI (Competition Intensity): Level 3 (Low: 0-4 active neighbors, Medium: 5-9, High: ≥10)

[0082] Action space A: {0 (backoff), 1 (channel 1), 2 (channel 2), ..., C (channel C)}, generally 3-5 channels are selected;

[0083] The initial value of table Q is either all 0s or a small random number.

[0084] Neighbor table initialization: Each node maintains a neighbor table, recording the node IDs, average recent Q values, and access success rates of neighbors within k hops.

[0085] Parameter initialization: exploration rate ε = 0.3, base learning rate η0 = 0.5, discount factor λ = 0.9, reward weight α = 0.5 (throughput), β = 0.3 (latency), γ = 0.2 (fairness); competition intensity threshold CI threshold =5 (used for tiered updates);

[0086] The algorithm execution steps include:

[0087] The first step is to perceive the environmental state at time t (s) t ), to obtain the current state s t =(CQI_level,QL_level,CI_level):

[0088] The five levels of CQI correspond to the following ranges in ascending order of RSSI / noise ratio: [0,3], [4,7], [8,11], [12,15], [15,+∞] (here, the ratios are rounded to integers).

[0089] The five levels of QL correspond to the following ranges in ascending order of the ratio of the number of data packets to be sent to the queue capacity: [0, 20%], [21%, 40%], [41%, 60%], [61%, 80%], and [81%, 100%]; (Here, the ratios are rounded to integers).

[0090] The three levels of CI correspond to the intervals [0,4], [5,9], and [10,+∞] respectively, according to the number of nodes transmitting in the k-hop neighbors from smallest to largest.

[0091] 2. Second step, action selection (a) t ):

[0092] Action a is randomly selected with probability ε. t ∈A (exploration), choose the current state s with probability 1-ε. t The action that maximizes Q (utilization): a t =argmax_{a}Q(s t ,a);

[0093] 3. The third step is to perform the action and observe.

[0094] If a t =0 (backoff): Sets a backoff timer, waiting for a random time slot. If a t If the value is >0, access channel c and transmit data on channel c. Detect whether a collision has occurred and record the transmission result, including success and failure, and the transmission delay D. t If the detection result is a successful collision, the signal is sent successfully and the process ends; if the detection result is a failed collision, the subsequent steps continue.

[0095] 4. Fourth step: Calculate the compound reward (R) t ):

[0096] R t =α*T t +β*(1 / D t )+γ*F t

[0097] Throughput normalization: T t = Number of bits successfully transmitted in this time slot / Maximum possible number of bits transmitted (0~1);

[0098] Delay factor 1 / D t D t This is the transmission delay, in milliseconds (ms).

[0099] Fairness F t Calculated based on the success rate of the last 10 transmissions of neighboring nodes.

[0100]

[0101] x j Let be the transmission success rate of node j.

[0102] 5. Fifth step, observe the new state (s) t+1 ):

[0103] At the start of the next time slot, re-sensitize the channel, queue, and contention intensity to obtain s. t+1 .

[0104] 6. Sixth step, hierarchical Q-value update:

[0105] First, calculate the dynamic learning rate:

[0106]

[0107] Where, N neighbors The number of neighbors.

[0108] Then perform layered updates:

[0109] If CI level <CI threshold This indicates relatively low competition intensity, and the update method is as follows:

[0110]

[0111] Where a' represents a temporary variable used in the calculation. Take the maximum, a' = a t a t+1 For each time step a = a', calculate an infinite number of Q values ​​and take the largest Q value;

[0112] If CI level >=CI threshold This indicates high competition. The following actions should be taken:

[0113] Calculate the mean Q-value of neighbors (for the same state-action pair)

[0114] Q neighbor_avg = Neighbor nodes in the neighbor table in state s t Action a t Mean Q value:

[0115]

[0116] Then Q(s) t ,a t )=0.7*Q local +0.3*Q neighbor_avg ;

[0117] If the Q value exceeds 100, it is truncated to 100.

[0118] 7. Seventh step, neighbor information exchange (cycle):

[0119] Perform the following operations every 10 time slots (T_exchange=10):

[0120] 1) Broadcast the average Q value of this node (the average Q value of all state-action pairs) and the success rate of the last 10 access attempts.

[0121] 2) Receive broadcasts from neighbors and update the neighbor table.

[0122] 3) Every 100 time slots (T_cluster=100):

[0123] 4) Cluster members send a local Q-table digest (such as the mean Q value) to the cluster head.

[0124] 5) The cluster head calculates the weighted average Q value within the cluster (weighted by node degree) and adds Laplace noise (differential privacy).

[0125] 6) Global Q value after cluster head broadcast perturbation.

[0126] 7) Cluster members update their local Q-table: Q_i = (1-ρ)*Q_i + ρ*Q_global, ρ = 0.2. ρ is the synchronization rate coefficient, representing the degree to which the Q-value of a local node converges to the global Q-value;

[0127] 8. Step 8, parameter adaptation:

[0128] Exploration rate decay: ε = max(0.1, ε * 0.999)

[0129] When a conflict occurs, the competition window adjustment is CW = min(1024, CW*2); if there is no conflict, then CW = 16.

[0130] Repeat steps 1-8 until the third step, which checks for conflicts, passes.

[0131] The performance verification of this algorithm was conducted under the following experimental conditions: The simulation platform used a workstation equipped with an Intel Core i9-12900K processor (base frequency 3.2GHz, 16 cores), 64GB DDR4 memory, and an NVIDIA RTX 3090 graphics card, running MATLAB R2022b (MathWorks, Inc.) and Communications Toolbox 8.7. The network topology was generated through random uniform distribution, with the node communication radius set to 100 meters, conforming to the λ / 4 free space path loss model. Specific parameter configurations are as follows:

[0132] The baseline scenario is set with 1000 sensor nodes deployed in a 1km×1km area. The scale of the expanded test ranges from 500 to 2000 nodes (in increments of 500). The number of gateways is configured in a ratio of N / 100 (where N is the total number of nodes). The bandwidth is set to 20MHz (compliant with the IEEE 802.11ax standard), using 5 orthogonal channels. The fading model is Rayleigh fading (multipath delay spread of 50ns), and the noise density is set to -174dBm / Hz. Packet generation uses a Poisson process (λ = 0.5-2 packets / ms adjustable), with a truncated normal distribution (mean 1500 bytes, σ = 300) for packet length distribution. The traffic type is 80% periodic data + 20% burst traffic. The algorithm's learning rate η∈[0.1,0.5], initial exploration rate ε of 0.3, exponential decay coefficient of 0.999, and reward weights determined as (α,β,γ)=(0.5,0.3,0.2).

[0133] All experiments were run 30 times per configuration (with different random seeds), and the results were taken from the 95% confidence interval. The simulation duration was set to 50,000 time slots (1ms in length), with the first 5,000 time slots being a pure exploration phase and not included in the performance statistics. To verify the robustness of the algorithm, the following interference factors were injected: 10% of nodes moved randomly (maximum speed 3m / s), channel parameters changed dynamically (updated every 500 time slots), and 5% of nodes went offline suddenly (duration 100-500 time slots).

[0134] The experimental comparison parameters are as follows:

[0135] 1. Throughput: Number of successfully transmitted bits / Total simulation time:

[0136]

[0137] 2. Latency: The average time from data packet generation to ACK confirmation;

[0138] 3. Fairness: Improved Jain Index;

[0139]

[0140] Based on experimental tests at different nodes, the performance metrics for different network sizes are as follows:

[0141] Table 1 Performance indicators for different network sizes

[0142]

[0143] Experimental results show that as the number of nodes increases in a gradient of 500, 1000, 1500, and 2000, the system throughput gradually decreases, the latency increases from 28.3ms to 45ms, and the fairness index also decreases in a stepwise manner. To more effectively compare the performance of the algorithm of this invention, the following comparative experiment was designed:

[0144] 1) In scenarios with large-scale node access, the performance of the proposed algorithm is compared with that of the standard Q-learning algorithm;

[0145] 2) To comprehensively evaluate the algorithm's performance, a comparative analysis was conducted with the traditional CSMA / CA protocol and the GT-MAC method based on game theory in four dimensions: convergence speed, throughput, latency, and number of convergence slots.

[0146] Table 2 Performance comparison between the improved algorithm and the standard Q-learning algorithm in the 1500-node scenario

[0147]

[0148] Table 3. Performance Indicator Comparison (1000-Node Scenario)

[0149] algorithm Throughput (Mbps) Delay (ms) Fairness Index Convergence time slots Proposed QL 22.2 32.0 0.93 3,800 Standard QL 18.7 45.3 0.88 12,500 CSMA / CA 15.4 58.6 0.82 - GT-MAC 19.5 41.2 0.85 8,200

[0150] Experimental results show that in the 1000-node scenario, the throughput of the proposed algorithm reaches 22.2 Mbps, which is about 21.6 percentage points higher than that of the CSMA / CA protocol (17.4 Mbps) and 15.7 percentage points higher than that of the standard Q-learning algorithm. This performance gain mainly stems from the optimization of channel utilization by the composite reward function. In terms of delay improvement, the median delay is: Proposed QL (34 ms) < GT-MAC (43 ms) < Standard QL (47 ms) < CSMA / CA (62 ms), and the 90% percentile delays are 38 ms (Proposed) vs 72 ms (CSMA / CA). In terms of fairness performance, the fairness index of this algorithm reaches 0.93, significantly higher than that of CSMA / CA (0.82), verifying the effectiveness of the fairness term in the reward function.

[0151] In response to the challenges of channel access in large-scale Internet of Things (IoT), this invention proposes an improved Q-learning algorithm based on a composite reward function. Through theoretical analysis and system simulation, it is proven that in a network with a scale of 1000 - 2000 nodes, the algorithm of this invention has a 21.6% increase in throughput and a 19.3% reduction in delay compared to traditional Q-learning, while maintaining a fairness index above 0.9. Especially in the 1500-node scenario, the convergence speed is increased by 2.29 times, verifying the efficiency of the algorithm. The composite reward function designed in this invention effectively balances the three key metrics of throughput, delay, and fairness; the proposed hierarchical update strategy and information sharing mechanism significantly reduce the state space dimension; the implemented dynamic parameter adjustment method enables the algorithm to adapt to changes in network scale. The algorithm can be applied in quasi-static industrial IoT scenarios and is suitable for application environments with high node density but low mobility requirements.

[0152] Subsequent research will focus on the fast convergence mechanism in mobile scenarios, joint optimization with physical layer security, and adaptive improvement in 6G terahertz communication. This invention provides a new solution for resource allocation in large-scale IoT, and its design concept can be extended to other distributed decision-making scenarios.

[0153] The above-disclosed are only the preferred embodiments of this invention. Of course, the scope of rights of this invention cannot be limited thereby. Those of ordinary skill in the art can understand the entire or partial processes of implementing the above embodiments, and the equivalent changes made according to the claims of this invention still fall within the scope covered by this invention.

Claims

1. An improved Q-learning channel access method based on a composite reward function, characterized in that, include: Step S1, Initialize the algorithm: Deploy sensor nodes and gateways, cluster nodes, and select cluster heads from the nodes in the clusters; Nodes are initialized locally, and each node maintains a Q table and a neighbor table, defining the state space, action space, initial parameters, and contention strength threshold. In step S2, each node performs actions and observes in the current state. If there is a conflict during the data transmission process, the composite reward for participating in the Q-value update is calculated. After updating to the next state, a hierarchical Q-value update based on the competition intensity level is performed first. During the T_exchange cycle, the node updates the neighbor table. During the T_cluster cycle, the cluster head calculates the weighted average Q-value within the cluster and broadcasts it, causing cluster members to update their local Q-tables. Then, in the next state, the node performs actions and observes until the conflict is resolved and the node information is successfully sent.

2. The improved Q-learning channel access method based on a composite reward function as described in claim 1, characterized in that, Step S1 includes: Step S11, Network Topology Construction: Deploy N sensor nodes and several gateways. The nodes are connected wirelessly via multi-hop to form a self-organizing network. Define a communication radius R. If the distance between nodes is ≤ R, then a communication link exists. Step S12, Cluster structure establishment: Divide the network into K clusters, select the node with the maximum selectivity from the clusters as the cluster head, and the cluster members maintain the routes to the cluster head; Step S13, Node local initialization: Q-table initialization: Each node maintains a Q-table, the dimension of which is state space × action space; State space: S = (CQI_level, QL_level, CI_level) consists of discrete levels of channel quality indicator (CQI), discrete levels of queue load (QL), and discrete levels of contention intensity (CI). Action space: A = {0, 1, 2, ..., C}, where 0 represents backoff and C represents channel number C; The initial value of table Q is either all zeros or small random numbers; Neighbor table initialization: Each node maintains a neighbor table, recording the node IDs, average recent Q-values, and access success rates of neighbors within k hops; Parameter initialization: exploration rate ε, base learning rate η0, discount factor λ, reward weight α, latency β, fairness γ, competition intensity threshold CI threshold .

3. The improved Q-learning channel access method based on a composite reward function as described in claim 2, characterized in that, In step S1, the values ​​of CQI, QL, and CI are discretized, and the discrete levels corresponding to the intervals are defined: The five levels of CQI, according to the RSSI / noise ratio from smallest to largest, correspond to the intervals [0,3], [4,7], [8,11], [12,15], [15,+∞]; The five levels of QL correspond to the following ranges in ascending order of the ratio of the number of data packets to be sent to the queue capacity: [0, 20%], [21%, 40%], [41%, 60%], [61%, 80%], and [81%, 100%]. The three levels of CI correspond to the intervals [0,4], [5,9], and [10,+∞] respectively, according to the number of nodes transmitting in the k-hop neighbors from smallest to largest.

4. The improved Q-learning channel access method based on a composite reward function as described in claim 3, characterized in that, Step S2 includes: Step S21, at time t, sense the environmental state s t Get the current state s t =(CQI_level,QL_level,CI_level), specifically: CQI_level: Monitor the channel, calculate the RSSI / noise ratio and map the ratio to 5 discrete level intervals to obtain the corresponding discrete level; QL_level: Calculates the number of data packets to be sent / queue capacity and maps the ratio to 5 discrete level intervals to obtain the corresponding discrete level; CI_level: Count the number of nodes currently transmitting in the k-hop neighbors and map them to three discrete level intervals to obtain the corresponding discrete level; Step S22, select action a t Action a is randomly selected with probability ε. t Explore ∈A and choose the current state s with probability 1-ε. t The action with the largest Q value: a t =argmax_{a}Q(s t ,a); Step S23, perform the action and observe: If a t =0, set backoff timer, wait for random time slot; If a t If the value is >0, access channel c and transmit data on channel c; detect whether a collision has occurred and record the transmission result, which includes success and failure, and the transmission delay D. t If the detection result is that the collision is passed, the signal is successfully sent and the process ends; if the detection result is that the collision is not passed, continue to execute step S24. Step S24, calculate the compound reward R t :R t =α*T t +β*(1 / D t )+γ*F t , among which, T t = Number of bits successfully transmitted in this time slot / Maximum possible number of bits transmitted; Delay factor 1 / D t D t This is the transmission delay; F t To ensure fairness, the calculation is based on the success rate of the neighboring nodes' most recent 10 transmissions; x j Let be the transmission success rate of node j; Step S25, observe the new state s t+1 At the start of the next time slot, re-sensing the channel, queue, and contention intensity, and obtaining s t+1 ; Step S26, Layered Q-value update: Calculate the dynamic learning rate η: In the formula, N neighbors Number of neighbors; Then perform layered updates: If CI_level <CI threshold This indicates relatively low competition intensity, and the update method is as follows: In the formula, a' represents the action; If CI_level>=CI threshold This indicates high competition intensity, and the update method is as follows: Calculate the mean Q-value of the neighbors. neighbor_avg , Q neighbor_avg = Neighbor nodes in the neighbor table in state s t Action a t The mean Q value; Calculate Q local Q local This represents the Q value of the local node; Then Q(s) t ,a t )=0.7*Q local +0.3*Q neighbor_avg Furthermore, if the Q value exceeds 100, it is truncated to 100; Step S27, Neighbor Information Exchange: Every 10 time slots with T_exchange=10, this node performs a broadcast and neighbor table update operation: Broadcast the average Q-value of this node and the success rate of the last 10 accesses. The average Q-value of this node refers to the average Q-value of all state-action pairs. Receive broadcasts from neighboring nodes and update the neighbor table; Every 100 time slots, perform a cluster member update operation on the Q table: Cluster members send a local Q-table digest to the cluster head, the local Q-table digest including the mean Q-value; The cluster head calculates the weighted average Q value within the cluster and adds Laplace noise. The weighted average Q value is weighted by the node degree in the calculation. The global Q value after cluster head broadcast perturbation; Cluster members update their local Q-table: Q_i = (1-ρ)*Q_i + ρ*Q_global; where ρ is the synchronization rate coefficient; Step S28, parameter adaptation: Exploration rate decay: ε = max(0.1, ε * 0.999); When a conflict occurs, the backoff window value CW = min(1024, CW*2) is adjusted for competition window adjustment. If there is no conflict, then CW = 16. Step S29: Return to step S21 and continue until the third step detects the conflict.

5. The improved Q-learning channel access method based on a composite reward function as described in claim 2, characterized in that, In step S12, maximum selectivity refers to having the maximum number of neighboring nodes.

6. The improved Q-learning channel access method based on a composite reward function as described in claim 3, characterized in that, In step S2, the parameters are initialized as follows: exploration rate ε = 0.3, base learning rate η0 = 0.5, discount factor λ = 0.9, reward weight α = 0.5, latency β = 0.3, and fairness γ = 0.2; CI threshold =5; ρ=0.2.

Images