Method and system for optimizing random access indefinite frame length based on reinforcement learning

Abstract

The invention discloses a random access variable frame length optimization method and system based on reinforcement learning, and the method comprises the steps: a base station transmits the number of currently available frame time slots and the number of lead codes contained in each time slot to all user equipment; selecting a time slot serial number in an equal probability mode, selecting a lead code from given lead code resources with the same probability, and initiating a random access operation; the base station carries out decoding operation, carries out statistical summary on the successful logarithm only selected by single user equipment and the idle logarithm not selected by any user equipment, and then inputs the current state into a Q-learning agent; and based on the state-action value function and the reward function, selecting the time slot number of the next frame, and carrying out loop iteration until the frame length is converged to a theoretical optimal value. According to the invention, the frame length parameter can be dynamically adjusted without depending on UE number estimation, the preamble competition conflict between UEs is effectively reduced, and the preamble resource utilization rate is improved, so that the random access performance of the system is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of wireless communication, and relates to frame length optimization technology in random access, in particular to a random access indefinite frame length optimization method and system based on reinforcement learning. BACKGROUND

[0002] Random access, as the core technology for user terminals and base stations to build initial connection in wireless communication system, has been widely used in many scenarios such as long-term evolution technology (LTE), fifth generation new radio (5G NR) and large-scale machine type communication (mMTC). The traditional random access mechanism adopts fixed frame length design, which divides a frame into a fixed number of access slots, and each terminal randomly sends a preamble in the slot. However, with the explosive growth of the number of UEs and the increasing diversity of business types, the fixed frame length mode is prone to cause slot conflict and significantly increase access delay in high-load environment, while in low-load state, it will cause resource waste and reduce the overall efficiency of the network.

[0003] To solve the above problems, the academic community has proposed a variety of access optimization schemes based on dynamic frame length. Literature (Hongguang D, Songpin L U, Lifei W, et al. Dynamic frame length selection algorithm for slotted access in LTE-Advanced[J]. Telecommunications Science, 2017) uses traffic estimation algorithm to realize frame length adaptive adjustment in LTE-Advanced system, and the simulation results show that this method can effectively improve the access success rate and reduce the preamble retransmission times. The research in the field of RFID focuses on dynamic frame length ALOHA (DFSA) algorithm: Literature (Zhang Y, Guan Y, Bo T, et al. Frame length adjustment method research of dynamic framed slotted ALOHA algorithm[C] / / 2010 International Conference on Future Information Technology and Management Engineering. IEEE, 2010, 2: 178-180) proposes to optimize the frame length selection by improving the tag number estimation technology, and literature (Salah H, Ahmed H A, Robert J, et al. A time and capture probability aware closed form frame slotted ALOHA frame length optimization[J]. IEEE communications letters, 2015, 19(11): 2009-2012) further gives the optimal frame length closed-form solution considering the time slot duration and capture probability, which verifies the key influence of frame length on system throughput and energy consumption.

[0004] In recent years, reinforcement learning has demonstrated strong adaptive capabilities in random access optimization. The paper (Sharma SK, Wang X. Collaborative distributed Q-learning for RACH congestion minimization in cellular IoT networks[J]. IEEE Communications Letters, 2019, 23(4): 600-603) proposes a collaborative distributed Q-Learning mechanism that enables resource-constrained MTC devices to select a unique RA slot to significantly reduce collisions; the paper (Bello LM, Mitchell P, Grace D, et al. Q-learning based random access with collision-free RACH interactions for cellular M2M[C] / / 20159th International Conference on Next Generation Mobile Applications, Services and Technologies. IEEE, 2015: 78-83) designs a FA-QL-RACH scheme, optimizes the frame structure for H2H and M2M users respectively, and uses Q-Learning to control M2M access to achieve collision-free access; the paper (Tubiana DA, Farhat J, Brante G, et al. Q-learning NOMA random access for IoT-satelliteterrestrial relay networks[J]. IEEE Wireless Communications Letters, 2022, 11(8): 1619-1623) combines Q-Learning with NOMA technology for dynamic resource scheduling of IoT satellite ground relay networks, effectively improving system throughput and access reliability.

[0005] Although the aforementioned literature studies have made progress in areas such as time slot allocation, access level restrictions, and multiple access technologies, they all rely on device number estimation. This dynamic adjustment based on device number estimation is susceptible to estimation errors, leading to inaccurate frame length settings. Summary of the Invention

[0006] Purpose of the Invention: To overcome the shortcomings of fixed frame lengths, which are difficult to adapt to drastic fluctuations in access traffic, high collision rates and large access delays in sudden high-concurrency scenarios, and the fact that dynamic adjustments based on device number estimation are easily affected by estimation errors, leading to inaccurate frame length settings, existing methods struggle to balance resource utilization and access latency, lack online adaptive capabilities, and cannot quickly respond to non-periodic and sudden changes in network load, this invention provides an optimization method and system for random access with variable frame lengths based on reinforcement learning. By continuously monitoring the number of successful and idle preambles within a frame, a state-action model is constructed, which autonomously makes decisions and updates the frame length online, achieving adaptive resource allocation in scenarios with drastic fluctuations in network load. This improves both the success rate of random access and the efficiency of network resource utilization.

[0007] Technical Solution: To achieve the above objectives, this invention provides an optimization method for random access with variable frame length based on reinforcement learning, comprising the following steps:

[0008] S1: The base station will determine the number of currently available frame time slots. and the number of preambles contained in each time slot. It is broadcast to all user equipment (UE).

[0009] S2: For each user equipment, the time slot number is selected with equal probability. and preamble Initiate a random access operation;

[0010] S3: All data received by the base station in this frame Perform decoding operations on the number of successful pairs selected only by a single User Equipment (UE). And the number of idle pairs not selected by any user equipment (UE) Perform statistical analysis and then summarize the current status. Input Q - Learn the agent;

[0011] S4: Q-learning agents are based on state-action value functions. and reward function, in the action set Select the next frame slot number Then, update the Q table according to the Bellman equation, and then jump to step S1, iterating repeatedly until the frame length is reached. Converging to the theoretical optimum .

[0012] Furthermore, in step S2, from the time slot set A time slot number is selected in a manner with equal probability. and from the given preamble resources A preamble is selected with equal probability. Subsequently, the user equipment will be in the... Each time slot, using the selected preamble Initiate a random access operation.

[0013] Furthermore, in step S4, the state-action value function Defined as Its update rules are as follows Where r is the immediate reward obtained after performing action a; γ∈[0,1] is the discount factor, which weighs the importance of current reward against future reward; The next state to transition to after performing action a; Indicates the next state The expected reward for taking the optimal action.

[0014] Furthermore, the reward function in step S4 is:

[0015]

[0016] in, It represents the average probability that the preamble is successfully selected without collision.

[0017] Furthermore, in step S4, the following is utilized: - Greedy strategies or heuristics in action sets Select the next frame slot number .

[0018] Furthermore, the theoretical optimal value in step S4 The calculation process includes:

[0019] when individual user equipment The average number of preambles successfully used in frame contention is:

[0020]

[0021] resource utilization rate Through calculation right The derivative is obtained as follows:

[0022]

[0023] The roots obtained by calculating the zeros of the above expression are: ; therefore when individual user equipment When competing for the preamble in a frame, ensure the utilization rate of preamble resources. Maximize the theoretical optimal frame length for:

[0024]

[0025] in, This indicates an integer that has been rounded to the nearest whole number.

[0026] This invention provides an optimization system for random access with variable frame length based on reinforcement learning, comprising:

[0027] The signal transmission module and the base station will transmit the number of currently available frame time slots. and the number of preambles contained in each time slot. It is broadcast to all user devices;

[0028] The random access module selects time slot numbers with equal probability for each user equipment. and preamble Initiate a random access operation;

[0029] The decoding module is responsible for decoding all data received by the base station in this frame. Perform decoding operations on the number of successful pairs selected by only a single user device. And the number of idle logs not selected by any user equipment Perform statistical analysis and then summarize the current status. Input Q - Learn the agent;

[0030] The iterative loop module, Q-learning agent based on state-action value function and reward function, in the action set Select the next frame slot number Then, update the Q table according to the Bellman equation, and then jump to step S1, iterating repeatedly until the frame length is reached. Converging to the theoretical optimum .

[0031] The technical approach of this invention is as follows: In each frame, by statistically analyzing the usage of preambles, the number of time slots in the frame is dynamically adjusted to expand available resources, reduce the probability of preamble collisions, and thus improve access efficiency. The specific operation process is as follows: First, an initial frame length is set. At the beginning of each frame, the base station broadcasts the number of time slots for this frame and the number of available preambles in each time slot to each user equipment. Subsequently, each user equipment selects a time slot and one preamble within that time slot with equal probability to initiate random access. Within the same time slot, if a preamble is selected by only one device, the device is considered to have successfully accessed, and the preamble is considered a successful preamble; if a preamble is not selected by any device, it is recorded as an idle preamble. After the base station summarizes the number of successful and idle preambles in this frame, the intelligent decision module determines the number of time slots to be used in the next frame based on these statistical results and historical access feedback. Through multiple iterations, the number of frame time slots quickly converges to the optimal range, enabling the system to provide more access opportunities under high load and avoid resource waste under low load. This invention does not impose any limit on the number of terminal devices participating in the access process, and the entire process does not require prior estimation of the total number of user devices, thus simplifying system complexity and reducing terminal power consumption.

[0032] This invention constructs a state-action model by continuously monitoring the number of successful preambles and idle preambles within a frame. This model can make autonomous decisions and update the frame length online, thereby achieving adaptive resource allocation in scenarios with drastic fluctuations in network load. This method not only improves the success rate of random access but also optimizes the utilization efficiency of network resources.

[0033] Beneficial Effects: In the development of IoT and 5G, the surge in random access demands from massive numbers of user devices (UEs) in sudden scenarios presents a challenge. Traditional fixed frame length mechanisms cannot dynamically adapt to fluctuations in access traffic, leading to intensified channel resource contention, increased preamble collision rates, and significantly increased access latency, thus hindering the performance of massive machine-type communication (mMTC). This invention utilizes reinforcement learning algorithms to autonomously perceive access traffic characteristics and dynamically adjust frame length parameters without relying on UE number estimation. It is suitable for dynamic network environments with non-periodic and bursty traffic. By optimizing the frame length, it effectively reduces preamble contention among UEs, improves preamble resource utilization, and thus significantly improves the system's random access performance. Attached Figure Description

[0034] Figure 1 This is a display image of a dynamic frame;

[0035] Figure 2 This is a flowchart of the method of the present invention;

[0036] Figure 3 This is a graph showing how the frame length (number of frame slots) changes with the number of learning rounds.

[0037] Figure 4 The optimized results are shown in the graph, which shows the frame length as the number of users and training rounds increases, with a comparison of the theoretically optimal frame length.

[0038] Figure 5 This is to measure the utilization rate of preamble resources and the cumulative average utilization rate as the number of users increases and training rounds increase;

[0039] Figure 6 This is a comparison chart showing the utilization rate of preamble resources between the invented method, the dynamic ACB parameter algorithm, and the DFSA algorithm as the number of user devices and training rounds increases. Detailed Implementation

[0040] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0041] Example 1:

[0042] In this embodiment, the system consists of one base station and a total of Preamble resources and The system consists of user equipment units. Among them, the number of frame time slots... and number of user devices It can change dynamically over time;

[0043] like Figure 1 As shown, this embodiment performs random access within a dynamically constructed frame. Each terminal is accessed by... Each time slot and From the matrix of preambles, a "time slot-preamble" combination is randomly selected. A device can successfully complete random access if and only if this combination is selected by only one device; if the same time slot-preamble combination is selected by two or more devices at the same time, the access requests of these devices collide and are all judged as access failures.

[0044] At the beginning of each frame, the base station broadcasts the number of currently available time slots. (Each time slot is a fixed time unit) and the number of preambles per time slot Each user device originates from Within a given time slot, a time slot is selected with equal probability. Then in the selected time slot A given number of preambles will select a particular preamble with equal probability. Subsequently at the Time slot transmission preamble Complete random access; the base station decodes all "slot-preamble" pairs received in this frame—if a certain pair... If a combination is selected by only one device, the device is considered to have successfully connected. However, if the combination is selected by two or more devices simultaneously, a conflict occurs, and all of these devices are considered to have failed to connect. The base station will count the number of successful preambles based on the decoding results. and the number of idle preambles and the current state Input Q-learning agent; agent based on state-action value function and - Greedy or experience-heuristic strategies, combined with a set of optional actions (Next frame slot number, such as) , , It makes its own decisions and adjusts the frame length of the next frame. ,make It converges rapidly and stabilizes at the theoretically optimal frame length through multiple iterations. This maximizes the utilization of preamble resources, reduces the probability of collisions, and improves the success rate and latency performance of random access.

[0045] Based on the above, such as Figure 2 As shown, this embodiment provides an optimization method for random access with variable frame length based on reinforcement learning, including the following steps:

[0046] S1: The base station will determine the number of currently available frame time slots. and the number of preambles contained in each time slot. It is broadcast to all user equipment (UE).

[0047] S2: For each user equipment, from the time slot set A time slot number is selected in a manner with equal probability. and from the given preamble resources A preamble is selected with equal probability. Subsequently, the user equipment will be in the... Each time slot, using the selected preamble Initiate a random access operation;

[0048] S3: All data received by the base station in this frame Perform decoding operations on the number of successful pairs selected only by a single User Equipment (UE). And the number of idle pairs not selected by any user equipment (UE) Perform statistical analysis and then summarize the current status. Input Q - Learn the agent;

[0049] S4: Q-learning agents are based on state-action value functions. and reward function, using - Greedy strategies or heuristics in action sets Select the next frame slot number Then, update the Q table according to the Bellman equation, and then jump to step S1, iterating repeatedly until the frame length is reached. Converging to the theoretical optimum This enables dual optimization of access efficiency and resource utilization in scenarios with sudden and dynamic traffic.

[0050] According to the Bellman expectation equation, the state-action value function Defined as Its update rules are as follows Where r is the immediate reward obtained after performing action a (which may be a random variable); The discount factor is used to weigh the importance of current rewards against future rewards. The next state to which the action is transitioned after action a is performed. Indicates the next state The expected reward for taking the optimal action.

[0051] The reward function is:

[0052]

[0053] in, It represents the average probability that the preamble is successfully selected without collision.

[0054] The state-action value function Q(S,A) is a function that evaluates the expected cumulative discounted reward that the agent can obtain after taking action A in a specific state S. In this embodiment, the Q table is used to store the expected cumulative reward for updating the next frame length under various random access congestion states.

[0055] Theoretical optimal value The calculation process includes:

[0056] when individual user equipment The average number of preambles successfully used in frame contention is:

[0057]

[0058] resource utilization rate Through calculation right The derivative is obtained as follows:

[0059]

[0060] The roots obtained by calculating the zeros of the above expression are: ; therefore when individual user equipment When competing for the preamble in a frame, ensure the utilization rate of preamble resources. Maximize the theoretical optimal frame length for:

[0061]

[0062] in, This indicates an integer that has been rounded to the nearest whole number.

[0063] Example 2:

[0064] This embodiment provides an optimization system for random access with variable frame length based on reinforcement learning, including:

[0065] The signal transmission module and the base station will transmit the number of currently available frame time slots. and the number of preambles contained in each time slot. It is broadcast to all user devices;

[0066] Random access module, used to access data from a set of time slots. A time slot number is selected in a manner with equal probability. and from the given preamble resources A preamble is selected with equal probability. The equipment will be in the first Each time slot, using the selected preamble Initiate a random access operation;

[0067] The decoding module is responsible for decoding all data received by the base station in this frame. Perform decoding operations on the number of successful pairs selected by only a single user device. And the number of idle logs not selected by any user equipment Perform statistical analysis and then summarize the current status. Input Q - Learn the agent;

[0068] The iterative loop module, Q-learning agent based on state-action value function and reward function, in the action set Select the next frame slot number Then, update the Q table according to the Bellman equation, and then jump to step S1, iterating repeatedly until the frame length is reached. Converging to the theoretical optimum .

[0069] Example 3:

[0070] To verify the effectiveness and efficacy of the method of the present invention, the following simulation experiments and analyses were conducted in this embodiment:

[0071] Number of user devices Number of preamble resources Initial frame length Under the condition of multiple time slots, the frame length is dynamically adjusted based on a reinforcement learning algorithm. The optimal frame length is obtained through theoretical calculation. .like Figure 3 As shown, after about 10 rounds of iteration, the frame length can converge to the theoretical optimal value and then fluctuate stably around this value.

[0072] This embodiment examines the frame length adjustment performance of reinforcement learning methods under a scenario of dynamically changing user numbers. To simulate a real-world access process, it is assumed that an additional number of users are added every 10 rounds. Obtain the parameter as The Poisson distribution is used; the initial frame length is set in the simulation. The number of preamble resources in the time slot is still set to [value]. As the number of user devices gradually increases over 100 iterations, such as Figure 4 The comparison between the optimized frame length (solid line) and the theoretical optimal frame length (dashed line) in three simulations shows that the frame length output by the reinforcement learning algorithm can track the theoretical optimal frame length in real time and always fluctuate slightly above and below it.

[0073] Figure 5 The figure shows the curve of preamble resource utilization as the number of user devices dynamically increases. Figure 5 The table shows the preamble resource utilization (solid line) and cumulative average utilization (dashed line) in three simulations, where the solid line represents the utilization in each round. Its value gradually tends to be between 0.35 and 0.40 and fluctuates slightly within this range; the dashed line represents the utilization rate. The cumulative average value eventually converges to the theoretical optimal value. .

[0074] This embodiment compares the preamble resource utilization of a reinforcement learning-based random access method with variable frame lengths, a probabilistic model-based algorithm for estimating Dynamic Access Class Limitation (ACB) parameters, and a dynamic frame length Slotted Aloha (DFSA) method. Figure 6 As shown, the preamble resource utilization (green solid line) is illustrated by the reinforcement learning-based variable frame length optimization algorithm (blue solid line), the dynamic ACB factor algorithm based on user equipment number estimation (yellow solid line), and the dynamic frame length Slotted Aloha (DFSA) algorithm. All three methods can quickly approach the theoretical optimal value. However, the fluctuation range of the ACB method is significantly greater than that of the variable frame length method, while the fluctuation range of the DFSA method is slightly higher than that of the variable frame length algorithm. Statistical results show that the utilization standard deviation of the variable frame length optimization algorithm is 0.04, while the standard deviation of the ACB method is 0.08 and the standard deviation of the DFSA method is 0.05, further verifying that the method of the present invention has higher stability in random access scenarios.

[0075] Therefore, the theoretically optimal frame length provided by this invention can be seen. The dynamic calculation method compared to the theoretical optimal frame length The static calculation method can converge to the optimal value in just a few iterations and continuously track the number of users in a dynamically increasing or decreasing environment. Compared with existing methods that require estimating the number of users and adjusting the access class (ACB) parameters, the method of this invention not only eliminates the complex estimation process, but also significantly reduces the utilization fluctuation through online learning, thereby achieving fast convergence, high resource utilization and excellent stability in large-scale, bursty IoT / 5G random access scenarios.

Claims

1. A method for optimizing random access variable frame length based on reinforcement learning, characterized in that, The method comprises the following steps: S1: The base station sends the number of frame slots currently available and the number of preambles contained in each slot to all user equipment in broadcast form; S2: for each user equipment, select a time slot number in an equiprobable manner and a preamble initiate a random access operation; S3: The base station counts all the successful pairs received in the current frame performs a decoding operation on the successful pairs selected by only a single user equipment and the idle pairs not selected by any user equipment performs a statistical aggregation and then inputs the current state to the Q-learning agent; S4: Q-learning agents are based on state-action value functions. and reward function, in the action set Select the next frame slot number Then, update the Q table according to the Bellman equation, and then jump to step S1, iterating repeatedly until the frame length is reached. Converging to the theoretical optimum .

2. The method of claim 1, wherein, The step S2 selects a time slot sequence number from the time slot set in an equiprobable manner , and selects a preamble from the given preamble resource in the same probability ; Subsequently, the user equipment will initiate a random access operation at the first time slot by means of the selected preamble .

3. The method of claim 1, wherein, The state-action value function in the step S4 is defined as with the update rule ; where r is the immediate reward obtained after performing action a; g e [0, 1] is a discount factor that weighs the importance of current rewards versus future rewards; s' is the next state transitioned to after performing action a; represents the expected reward of taking the optimal action a*in the next state s'; in the next state s'.

4. The method of claim 1, wherein, The reward function in the step S4 is: ； wherein, is the average probability that a preamble is successfully selected without collision.

5. The method of claim 1, wherein, The step S4 uses - a greedy policy or heuristic exploration to select the next number of time slots in the set of actions .​ 6. The method of claim 1, wherein, The theoretical optimum value in the step S4 The calculation process includes: when individual user equipment The average number of preambles successfully used in frame contention is: ； Resource utilization , by calculating the derivative of ​ ； The root obtained by calculating the zero point of the formula is ; therefore, when user equipment competes for a preamble in a frame , the guaranteed preamble resource utilization is maximized .

7. The method of claim 6, wherein, The theoretical optimum value in the step S4 is expressed as follows: ； wherein represents an integer rounded off.

8. An optimization system for random access variable frame length based on reinforcement learning, characterized in that, The system for implementing the method of claim 1 comprises: The signal transmission module and the base station will transmit the number of currently available frame time slots. and the number of preambles contained in each time slot. It is broadcast to all user devices; a random access module, for each user equipment, selects a time slot number in an equiprobable manner and a preamble initiates a random access operation; a decoding module, the base station carries out decoding operation on all the success logs selected by only single user equipment and idle logs not selected by any user equipment statistical summary is carried out, and then the current state is input into the Q-learning agent; The iterative loop module, Q-learning agent based on state-action value function and reward function, in the action set Select the next frame slot number Then, update the Q table according to the Bellman equation, and then jump to step S1, iterating repeatedly until the frame length is reached. Converging to the theoretical optimum .

Images