A wireless resource allocation optimization method and device
By introducing priority dynamic ACB mechanism and PPO algorithm in the wireless communication system, the problems of massive equipment access and MTC equipment overload control are solved, efficient resource allocation and delay optimization are achieved, and the overall performance and efficiency of the system are improved.
Patent Information
- Application Number
- CN202210899466.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-01-24
- Filing Date
- 2022-07-28
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-07-28
AI Technical Summary
The existing technology is difficult to effectively solve the problems of large-scale wireless device access and MTC device overload control in massive equipment access and service transmission, especially when the base station cannot understand the traffic status of each time slot in real time, resulting in low resource allocation efficiency. and delay increases.
Adopting a dynamic ACB mechanism and PPO algorithm based on priority, through intelligent models and deep learning technology, priority classification and access limit factor decisions are made for wireless devices, and ACB factors are dynamically adjusted to maximize the number of successful access and reduce delays. .
It realizes that the throughput and the time delay are reduced while ensuring priority access to high-priority devices, effectively alleviating the problem of large-scale equipment access, and improving resource utilization efficiency and system performance.
Smart Images

Figure CN115278908B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of wireless communication technology, and in particular to a method and device for optimizing wireless resource allocation. Background Art
[0002] With the explosive growth of the number of wireless devices, the access congestion problem of large-scale wireless devices is particularly prominent. Therefore, it is necessary to study an effective communication access mechanism to reasonably allocate access resources according to the heterogeneous needs of users. The present invention adds a priority-based dynamic ACB mechanism before the device performs a random access process, and adopts the PPO algorithm (Proximal Policy Optimization, enhanced learning algorithm) to select the optimal ACB factor (Access Class Barring, access restriction factor). Under the premise of ensuring the heterogeneous needs of various priority devices, the number of successful accesses is maximized and the delay is reduced, which effectively alleviates the access problem of large-scale devices, thereby supporting massive data transmission.
[0003] In M2M communication (Machine-to-Machine, machine-to-machine communication technology), the access control problem has always been one of the focuses of attention. Reasonable allocation of random access resources can greatly save system resources, meet the heterogeneous needs of various applications, and thus improve overall work efficiency. In actual scenarios, the base station cannot know the traffic status of each time slot, but only knows the number of devices that successfully accessed the previous time slot, the number of devices that conflicted, and the number of idle preambles. The base station can only dynamically adjust the future ACB factor through limited access information. Therefore, the access control problem can also be regarded as a partially observable Markov decision process. Deep reinforcement learning is an algorithm that converts the problem into a Markov decision process. Among them, reinforcement learning can quickly respond to traffic changes based on the observed local information, and deep neural networks can achieve complex nonlinear approximation through multi-layer arrangement of neurons and interconnection between layers. Therefore, deep reinforcement learning is one of the most effective methods to solve such dynamic decision problems. PPO, as a promising deep reinforcement learning algorithm, has good performance for resource allocation problems with large state space and action space, and has recently attracted widespread attention from the industry and academia. PPO proposes a novel objective function that can achieve small batch updates, solving the problem of difficult step size determination in the policy gradient algorithm, and is easier to solve than the TRPO algorithm that also solves the same problem.
[0004] In the prior art, the problem of massive device access and service transmission as well as overload control of supporting a large number of MTC (Machine Type Communication) devices still needs to be solved. Summary of the invention
[0005] In view of the problems of massive device access and service transmission as well as overload control problem of supporting a large number of machine type communication (MTC) devices in the prior art, the present invention proposes a wireless resource allocation optimization method and device.
[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0007] On the one hand, a method for optimizing wireless resource allocation is provided, comprising:
[0008] S1: Prioritize the wireless devices to obtain wireless devices with different priorities; set different access restriction factors for the wireless devices with different priorities;
[0009] S2: Establish an intelligent agent model through the base station to make decisions on various access restriction factors;
[0010] S3: Train the agent model through the proximal strategy optimization PPO algorithm to obtain immediate rewards and different states of the agent model; use the immediate rewards and different states of the agent model as transfer experience;
[0011] S4: Calculate the objective function, perform deep learning on the objective function through transfer experience, and complete the optimization of wireless resource allocation.
[0012] Optionally, in step S1, wireless devices are prioritized to obtain wireless devices of different priorities, and different access restriction factors are set for wireless devices of different priorities, including:
[0013] S11: Prioritize the wireless devices according to their sensitivity to delay, classify the wireless devices that are sensitive to delay as high-priority users, and classify the wireless devices that are tolerant to delay as low-priority users;
[0014] S12: Different access restriction factors are set for wireless devices of different priorities, so that devices with high priority have priority in accessing the restriction factor, and devices with low priority have access to the restriction factor later.
[0015] Optionally, the priorities from high to low are: emergency warning service, equipment control interaction service, and data collection service.
[0016] Optionally, in step S2, an intelligent agent model is established by the base station to make decisions on various access restriction factors, including:
[0017] S21: Setting the access restriction factor to When the kth wireless device attempts to access, a random number q is generated. When the wireless device enters the random access process, that is, it passes the access restriction ACB check; when , then we have to wait until a random time T barring , and then re-perform ACB examination; among them, T barring =(0.7+0.6u)T ACB ;
[0018] S22: The wireless device that has completed the ACB check randomly selects a preamble code from the preamble codes used for access contention and sends it to the base station for random access; when the preamble code selection of multiple wireless devices conflicts, the backoff mechanism is activated, and the process returns to step S21, and the ACB check is re-performed on all wireless devices that have conflicted, until the preamble code is selected by only one wireless device, and the wireless device successfully accesses the base station.
[0019] Optionally, in step S3, the agent model is trained by a proximal policy optimization (PPO) algorithm to obtain instant rewards and different states of the agent model; the instant rewards and different states of the agent model are transfer experiences, including:
[0020] S31: Initialize the wireless device, various parameters in the neural network in the PPO algorithm, and the environment information of the intelligent agent;
[0021] S32: The wireless device access result of the current time slot is used as the current state s t Input into the neural network, output the action with the highest probability, and use it as the current state s t The best action a under t ;
[0022] S33: The agent model performs the optimal action a t Interact with the agent environment and get instant rewards r t and the next state s t+1 ; Current status t , optimal action a t 、Instant Rewards t and the next state s t+1 Constitute the transfer experience, transfer experience (s t ,a t ,r t ,s t+1 ) is stored in the buffer;
[0023] S34: Repeat steps S32-33 until the optimal action selection in all states is completed, and all the constructed transfer experiences are stored in the buffer.
[0024] Optionally, the action space is the access restriction factor of the base station in each time slot The state space consists of the access results for each time slot known to the base station, including the number of devices that successfully accessed Number of conflicting devices and the number of idle preambles
[0025] Optionally, an instant reward r t It includes main line rewards and auxiliary rewards; the main line reward is the main goal that should be completed by the task, and the main goal is set as the number of devices that successfully access; the main goal is decomposed into multiple sub-goals, and the reward corresponding to each sub-goal is the auxiliary reward, and the auxiliary reward is set as the absolute value of the difference between the number of devices that pass the ACB check and the number of preambles F. Then the instant reward r for the kth type of wireless device t The expression is:
[0026]
[0027] where α k is the coefficient, is the number of successful accesses of the kth type of wireless devices, The number of devices of category k that passed the ACB check.
[0028] Optionally, in step S4, calculating the objective function and performing deep learning on the objective function by transferring experience includes:
[0029] S41: Calculate the advantage function A in each state according to the following formula (3): πθ (s t ,a t );
[0030]
[0031] Where V πθ (s t ) is the base value, γ is the discount factor, and T is the total number of steps in one round;
[0032] S42: The agent randomly selects transfer experience from the buffer for deep learning and calculates the objective function according to the following formula (4): Update network parameters θ:
[0033]
[0034] Among them, c1 and c2 are coefficients, Objective Function By clipping objective function Error term in value function and Entropy Reward The clipping objective function consists of three parts; It can be expressed as:
[0035]
[0036] in, is the long-term reward function, representing the new strategy π θ (a t |s t ) and the old strategy ratio; when , then the current state s is encouraged t The best action a under t , σ t (θ) will show an increasing trend; otherwise, the current state s will be punished t The best action a under t , σ t (θ) will show a decreasing trend;
[0037] S43: Determine whether the preset number of learning times has been reached. If the preset number of learning times has been reached, use the network parameters θ obtained by deep learning to replace the original network parameters θ old ; If the preset number of learning times is not reached, return to step S41;
[0038] S44: The network parameter θ is trained and iterated multiple times by the gradient ascent method. The long-term reward function is maintained within a certain range and does not rise any more. This means that the number of successful accesses of each category reaches a maximum, and the latency reaches a minimum on the basis of satisfying the access success rate of each category. The resource allocation optimization process is terminated, and the wireless resource allocation optimization is completed.
[0039] On the one hand, a wireless resource allocation optimization device is provided, the device comprising:
[0040] A priority classification module is used to classify the wireless devices into priorities to obtain wireless devices of different priorities; and to set different access restriction factors for wireless devices of different priorities;
[0041] The decision module is used to establish an intelligent agent model through the base station and make decisions on various access restriction factors;
[0042] A model training module is used to train the agent model through the proximal policy optimization PPO algorithm to obtain instant rewards and different states of the agent model; and use the instant rewards and different states of the agent model as transfer experience;
[0043] The deep learning module is used to calculate the target function, perform deep learning on the target function by transferring experience, and complete the allocation optimization of wireless resources.
[0044] Optionally, a prioritization module is used to:
[0045] Prioritize devices based on their sensitivity to latency. Devices that are sensitive to latency are classified as high-priority users, while devices that are latency-tolerant are classified as low-priority users.
[0046] Different access restriction factors are set for the different priorities, so that the devices with high priority have priority in access restriction factors, and the devices with low priority have access restriction factors later.
[0047] On the one hand, an electronic device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned method for optimizing wireless resource allocation.
[0048] On the one hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned method for optimizing wireless resource allocation.
[0049] The above technical solution of the embodiment of the present invention has at least the following beneficial effects:
[0050] In the above scheme, the present invention proposes an effective priority-based dynamic access category restriction mechanism for the problem of massive device access and service transmission, reasonably allocates access resources according to the heterogeneous needs of users, and focuses on overload control to support a large number of machine-type communication MTC devices. In order to meet the heterogeneous needs of different applications, MTC devices are divided into multiple categories according to their characteristics, different ACB factors are used for different categories of devices, and dynamic adjustments are made according to real-time traffic conditions to maximize the utilization of random access resources. By modeling the random resource optimization problem as a partially observable Markov decision process, the intelligent agent is trained with the help of the PPO algorithm to obtain corresponding actions and strategies to dynamically adjust the ACB factors of various types of devices, and maximize the throughput while ensuring that high-priority devices have priority access. In the PPO framework of the present invention, by introducing a clipping alternative objective function, the problem of difficult step size determination in the PG algorithm (Policy Optimization, policy gradient algorithm) is solved. The uniqueness of the random resource allocation optimization method and device based on PPO will be proved through iterative cycles, and the overall economic benefits of random access resource allocation will be finally achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0052] Figure 1 is a flow chart of a wireless resource allocation optimization method provided by an embodiment of the present invention;
[0053] Figure 2 is a flow chart of a wireless resource allocation optimization method provided by an embodiment of the present invention;
[0054] Figure 3 It is a block diagram of a wireless resource allocation optimization device provided by an embodiment of the present invention;
[0055] Figure 4 A large-scale wireless device access network scenario diagram of a wireless resource allocation optimization device provided by an embodiment of the present invention;
[0056] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0057] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0058] The embodiment of the present invention provides a method for optimizing wireless resource allocation, which can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The process flow of the method for optimizing wireless resource allocation shown in the figure may include the following steps:
[0059] S101: Prioritize wireless devices to obtain wireless devices with different priorities; set different access restriction factors for wireless devices with different priorities;
[0060] S102: Establishing an intelligent agent model through the base station to make decisions on various access restriction factors;
[0061] S103: Train the agent model through the proximal strategy optimization PPO algorithm to obtain instant rewards and different states of the agent model; use the instant rewards and different states of the agent model as transfer experience;
[0062] S104: Calculate the objective function, perform deep learning on the objective function through transfer experience, and complete the optimization of wireless resource allocation.
[0063] Optionally, in step S101, wireless devices are prioritized to obtain wireless devices of different priorities, and different access restriction factors are set for wireless devices of different priorities, including:
[0064] S111: Prioritize the wireless devices according to their sensitivity to delay, classify the wireless devices that are sensitive to delay as high-priority users, and classify the wireless devices that are tolerant to delay as low-priority users;
[0065] S112: Different access restriction factors are set for wireless devices of different priorities, so that devices with high priority have priority in accessing the restriction factor, and devices with low priority have access to the restriction factor later.
[0066] Optionally, the priorities from high to low are: emergency warning service, equipment control interaction service, and data collection service.
[0067] Optionally, in step S102, an intelligent agent model is established by the base station to make decisions on various access restriction factors, including:
[0068] S121: Setting the access restriction factor to When the kth wireless device attempts to access, a random number q is generated. When the wireless device enters the random access process, that is, it passes the access restriction ACB check; when When the ACB check is repeated, it is necessary to wait for a random time Tbarring, and then re-check the ACB. barring =(0.7+0.6u)T ACB ;
[0069] S122: The wireless device that has completed the ACB check randomly selects a preamble code from the preamble codes used for access contention and sends it to the base station for random access; when a conflict occurs in the preamble code selection of multiple wireless devices, the backoff mechanism is started, and the process returns to step S121, and the ACB check is re-performed on all wireless devices that have conflicted, until the preamble code is selected by only one wireless device, and the wireless device successfully accesses the base station.
[0070] Optionally, in step S103, the agent model is trained by a proximal policy optimization (PPO) algorithm to obtain instant rewards and different states of the agent model; the instant rewards and different states of the agent model are transfer experiences, including:
[0071] S131: Initialize the wireless device, various parameters in the neural network in the PPO algorithm, and the environment information of the intelligent agent;
[0072] S132: Use the wireless device access result of the current time slot as the current state s of the intelligent agent t Input into the neural network, output the action with the highest probability, and use it as the current state s t The best action a under t ;
[0073] S133: Agent model performs optimal action a t Interact with the agent environment and get instant rewards r t and the next state s t+1 ; Current status t , optimal action a t , Instant Rewards t and the next state s t+1 Constitute the transfer experience, transfer experience (s t ,a t ,r t ,s t+1 ) is stored in the buffer;
[0074] S134: Repeat steps S132-133 until the optimal action selection in all states is completed, and all the constructed transfer experiences are stored in the buffer.
[0075] Optionally, the action space is the access restriction factor of the base station in each time slot The state space consists of the access results for each time slot known to the base station, including the number of devices that successfully accessed Number of conflicting devices and the number of idle preambles
[0076] Optionally, an instant reward r t It includes main line rewards and auxiliary rewards; the main line reward is the main goal that should be completed by the task, and the main goal is set as the number of devices that successfully access; the main goal is decomposed into multiple sub-goals, and the reward corresponding to each sub-goal is the auxiliary reward, and the auxiliary reward is set as the absolute value of the difference between the number of devices that pass the ACB check and the number of preambles F. Then the instant reward r for the kth type of wireless device t The expression is:
[0077]
[0078] where α k is the coefficient, is the number of successful accesses of the kth type of wireless devices, The number of devices of category k that passed the ACB check.
[0079] Optionally, in step S104, calculating the objective function and performing deep learning on the objective function by transferring experience includes:
[0080] S141: Calculate the advantage function A in each state according to the following formula (3): πθ (s t ,a t );
[0081]
[0082] Where V πθ (s t ) is the base value, γ is the discount factor, and T is the total number of steps in one round;
[0083] S142: The agent randomly selects transfer experience from the buffer for deep learning and calculates the objective function according to the following formula (4): Update network parameters θ:
[0084]
[0085] Among them, c1 and c2 are coefficients, Objective Function By clipping objective function Error term in value function and Entropy Reward The clipping objective function consists of three parts; It can be expressed as:
[0086]
[0087] in, is the long-term reward function, representing the new strategy π θ (a t |s t ) and the old strategy ratio; when , then the current state s is encouraged t The best action a under t , σ t (θ) will show an increasing trend; otherwise, the current state s will be punished t The best action a under t , σ t (θ) will show a decreasing trend;
[0088] S143: Determine whether the preset number of learning times has been reached. If the preset number of learning times has been reached, use the network parameters θ obtained by deep learning to replace the original network parameters θ old ; If the preset number of learning times is not reached, return to step S141;
[0089] S144: The network parameter θ is iteratively updated for multiple trainings through the gradient ascent method. The long-term reward function is maintained within a certain range and does not rise any more. This means that the number of successful accesses of each category reaches a maximum, and the latency reaches a minimum on the basis of satisfying the access success rate of each category. The resource allocation optimization process is terminated, and the wireless resource allocation optimization is completed.
[0090] In the embodiment of the present invention, the present invention proposes an effective priority-based dynamic access category restriction mechanism for the problem of massive device access and service transmission, reasonably allocates access resources according to the heterogeneous needs of users, and focuses on overload control to support a large number of machine-type communication MTC devices. In order to meet the heterogeneous needs of different applications, MTC devices are divided into multiple categories according to their characteristics, different ACB factors are used for different categories of devices, and dynamic adjustments are made according to real-time traffic conditions to maximize the utilization of random access resources. By modeling the random resource optimization problem as a partially observable Markov decision process, the intelligent agent is trained with the help of the PPO algorithm to obtain corresponding actions and strategies to dynamically adjust the ACB factors of various types of devices, and maximize the throughput while ensuring that high-priority devices have priority access. In the PPO framework of the invention, by introducing a clipping alternative objective function, the problem of difficult step size determination in the PG algorithm (Policy Optimization, policy gradient algorithm) is solved. The uniqueness of the random resource allocation optimization method and device based on PPO will be proved through iterative cycles, and the overall economic benefits of random access resource allocation will be finally achieved.
[0091] The embodiment of the present invention provides a method for optimizing wireless resource allocation, which can be implemented by an electronic device, which can be a terminal or a server. Figure 2 The process flow of the method for optimizing wireless resource allocation shown in the figure may include the following steps:
[0092] S201: Prioritize the wireless devices according to their sensitivity to delay, classify the wireless devices that are sensitive to delay as high-priority users, and classify the wireless devices that are tolerant to delay as low-priority users;
[0093] In a feasible implementation, the present invention optimizes random access resources for various service requirements such as equipment monitoring, environmental monitoring, and emergency warning. In order to meet the heterogeneous QoS (Quality of Service) requirements of users, the present invention divides access devices into multiple priorities, from high to low: emergency warning service, equipment control interaction service, and data collection service.
[0094] S202: Different access restriction factors are set for wireless devices of different priorities, so that devices with high priority have priority in accessing the restriction factor, and devices with low priority have access to the restriction factor later.
[0095] In an embodiment of the present invention, by dividing the services into multiple priorities, different access restriction factors are adopted for different priorities to meet the heterogeneous needs of different services; and because the adopted PPO method is a deep reinforcement learning algorithm based on continuous action selection, the priority can be expanded to more than 2 levels.
[0096] S203: Setting the access restriction factor to When the kth wireless device attempts to access, a random number q is generated. When the wireless device enters the random access process, that is, it passes the access restriction ACB check; when Then you have to wait until a random time T barring , and then re-perform ACB examination; among them, T barring =(0.7+0.6u)T ACB .
[0097] In a feasible implementation, the intelligent agent model based on the PPO framework deployed by the base station outputs the ACB factor p of the base station in each time slot, and the user attempting to access also generates a random number q in [0,1]. If q≤p, the UE (User Equipment) passes the ACB check and can proceed to the next random access process, otherwise it needs to wait for a random time to try again.
[0098] S204: The wireless device that has completed the ACB check randomly selects a preamble code from the preamble codes used for access contention and sends it to the base station for random access; when the preamble code selection of multiple wireless devices conflicts, the backoff mechanism is started, and the process returns to step S203, and the ACB check is re-performed on all wireless devices that have conflicted, until the preamble code is selected by only one wireless device, and the wireless device successfully accesses the base station.
[0099] In a feasible implementation, after the ACB check, a random access process is performed. The BO mechanism is started for the device after the conflict occurs, that is, a backoff time T is required. BO After that, restart the ACB inspection steps, where T BO =U[0,T b ]. The device that passes the ACB check randomly selects a preamble from the preambles used for competition and sends it to the base station to perform a random access process. When multiple users select different preambles, a conflict will occur. At this time, the backoff mechanism needs to be started, and step 203 is performed again after a random backoff time. When the preamble is only selected by one user, the user can successfully access the base station and perform data transmission.
[0100] The access status of the access device and the base station is modeled as a partially observable Markov decision model. Since the overall system of the present invention is a discrete time slot system, the system state does not change at the same time, and the system state at the next time is converted according to the state transition probability. The action space is set to the ACB factor p (0≤p≤1) broadcast by the base station in each time slot. The state space is composed of the access results of each time slot known to the base station, including the number of devices that successfully accessed Number of conflicting devices and the number of idle preambles
[0101] S205: Initialize the wireless device, various parameters in the neural network in the PPO algorithm, and the environment information of the intelligent agent.
[0102] In a feasible implementation, various parameters in the wireless device, the user terminal device and the neural network and the agent environment information are initialized. The agent environment information includes: the agent's optimal action a t , current status t , buffer B, etc. Optimal action a t Including access restriction factors for each priority level, Current Status t Includes the access results of each category in the previous time slot: the number of successful accesses, the difference between the number of ACB checks and the number of preamble codes, and the number of idle codes.
[0103] S206: Using the wireless device access result of the current time slot as the current state s t Input into the neural network, output the action with the highest probability, and use it as the current state s t The best action a under t .
[0104] In one feasible implementation, the action is selected based on the current old strategy Select the action with the highest probability as the action decision in the current state.
[0105] S207: The agent model executes the optimal action a t Interact with the agent environment and get instant rewards r t and the next state s t+1 ; Current status t , optimal action a t , Instant Rewards t and the next state s t+1 Constitute the transfer experience, transfer experience (s t ,a t ,r t ,st+1 ) is stored in the buffer;
[0106] S208: Repeat steps S206-S207 until the optimal action selection in all states is completed, and all the constructed transfer experiences are stored in the buffer.
[0107] In one feasible implementation, the agent performs action a t Interact with the environment and get immediate rewards t and the next state s t+1 . And will transfer experience (s t ,a t ,r t ,s t+1 ) is stored in buffer B for subsequent learning of the agent.
[0108] In a feasible implementation, the action space is set to the access restriction factor of the base station in each time slot. The state space consists of the access results for each time slot known to the base station, including the number of devices that successfully accessed Number of conflicting devices and the number of idle preambles
[0109] In one possible implementation, the action is the access restriction factor for the kth type of equipment. in
[0110] In a feasible implementation, the instant reward r t It includes main line rewards and auxiliary rewards; the main line reward is the main goal that should be completed by the task, and the main goal is set as the number of devices that successfully access; the main goal is decomposed into multiple sub-goals, and the reward corresponding to each sub-goal is the auxiliary reward, and the auxiliary reward is set as the absolute value of the difference between the number of devices that pass the ACB check and the number of preambles F. Then the instant reward r for the kth type of wireless device t The expression is:
[0111]
[0112] where α k is the coefficient, is the number of successful accesses of the kth type of wireless devices, The number of devices of category k that passed the ACB check.
[0113] If you want to get instant rewards, you must meet the respective latency requirements, namely:
[0114]
[0115]
[0116] C2:0≤P ACB ≤1
[0117] The constraint C1 It can be written as:
[0118]
[0119] in and They are the number of time slots that each device needs to wait for after the k-th type of device fails the ACB check and conflicts in the t-th time slot. T ACB To limit the time, T s The length of a time slot. T b is the backoff index.
[0120] According to the environment, the immediate reward of the k-th device can be expressed as:
[0121]
[0122] Therefore, the total reward can be expressed as:
[0123]
[0124] S209: Calculate the advantage function A in each state according to the following formula (3): πθ (s t ,a t );
[0125]
[0126] Where V πθ (s t ) is the base value, γ is the discount factor, and T is the total number of steps in one round;
[0127] In one possible implementation, the advantage function It is used to measure the advantage of executing action a in state s. It represents the increment of value, which helps to improve the stability of the algorithm and avoid overfitting. It can be expressed as:
[0128]
[0129] in is the benchmark value, γ is the discount factor, and T is the total number of steps in one round.
[0130] S210: The agent randomly selects transfer experience from the buffer for deep learning and calculates the objective function according to the following formula (4): Update network parameters θ:
[0131]
[0132] Among them, c1 and c2 are coefficients, Objective Function By clipping objective function Error term in value function and Entropy Reward The clipping objective function consists of three parts; It can be expressed as:
[0133]
[0134] in, is the long-term reward function, representing the new strategy π θ (a t |s t ) and the old strategy ratio; when , then the current state s is encouraged t The best action a under t , σ t (θ) will show an increasing trend; otherwise, the current state s will be punished t The best action a under t , σ t (θ) will show a decreasing trend.
[0135] In one feasible implementation, the objective function By clipping objective function Error term in value function and Entropy Reward It consists of three parts. Clipping objective function It can be expressed as:
[0136]
[0137] in is the ratio of the new strategy to the old strategy. , then the current state s is encouraged t The next action a t , σ t (θ) will show an increasing trend. Otherwise, the current state s is punished t The next action a t , σ t (θ) will show a decreasing trend. In order to prevent the new strategy πθ (a t ∣s t ) and the old strategy The gap is too large, PPO uses clip(σ t (θ),1-ε,1+ε) will r t (θ) is limited to [1-ε, 1+ε]. The objective function of the PPO algorithm It can be expressed as:
[0138]
[0139] Where c1 and c2 are coefficients,
[0140] S211: Determine whether the preset number of learning times has been reached. If the preset number of learning times has been reached, use the network parameters θ obtained by deep learning to replace the original network parameters θ old ; If the preset number of learning times is not reached, return to step S141;
[0141] S212: The network parameter θ is iteratively updated for multiple trainings through the gradient ascent method. The long-term reward function is maintained within a certain range and does not rise any more. This means that the number of successful accesses of each category reaches a maximum, and the latency reaches a minimum on the basis of satisfying the access success rate of each category. The resource allocation optimization process is terminated, and the wireless resource allocation optimization is completed.
[0142] In the embodiment of the present invention, the present invention proposes an effective priority-based dynamic access category restriction mechanism for the problem of massive device access and service transmission, reasonably allocates access resources according to the heterogeneous needs of users, and focuses on overload control to support a large number of machine-type communication MTC devices. In order to meet the heterogeneous needs of different applications, MTC devices are divided into multiple categories according to their characteristics, different ACB factors are used for different categories of devices, and dynamic adjustments are made according to real-time traffic conditions to maximize the utilization of random access resources. By modeling the random resource optimization problem as a partially observable Markov decision process, the intelligent agent is trained with the help of the PPO algorithm to obtain corresponding actions and strategies to dynamically adjust the ACB factors of various types of devices, and maximize the throughput while ensuring that high-priority devices have priority access. In the PPO framework of the invention, by introducing a clipping alternative objective function, the problem of difficult step size determination in the PG algorithm (Policy Optimization, policy gradient algorithm) is solved. The uniqueness of the random resource allocation optimization method and device based on PPO will be proved through iterative cycles, and the overall economic benefits of random access resource allocation will be finally achieved.
[0143] Figure 3is a block diagram of a wireless resource allocation optimization device according to an exemplary embodiment. Figure 3 , the device 300 comprises:
[0144] The priority classification module 301 is used to classify the wireless devices into priorities to obtain wireless devices of different priorities; and to set different access restriction factors for the wireless devices of different priorities;
[0145] A decision module 302 is used to establish an intelligent agent model through a base station to make decisions on various access restriction factors;
[0146] The model training module 303 is used to train the agent model through the proximal policy optimization (PPO) algorithm to obtain instant rewards and different states of the agent model; and use the instant rewards and different states of the agent model as transfer experience;
[0147] The deep learning module 304 is used to calculate the target function, perform deep learning on the target function by transferring experience, and complete the allocation optimization of wireless resources.
[0148] Optionally, the priority division module 301 is used to:
[0149] Prioritize devices based on their sensitivity to latency. Devices that are sensitive to latency are classified as high-priority users, while devices that are latency-tolerant are classified as low-priority users.
[0150] Different access restriction factors are set for the different priorities, so that the devices with high priority have priority in access restriction factors, and the devices with low priority have access restriction factors later.
[0151] Optionally, the priorities from high to low are: emergency warning service, equipment control interaction service, and data collection service.
[0152] Optionally, the decision module 302 is further configured to set the access restriction factor to When the kth wireless device attempts to access, a random number q is generated. When the wireless device enters the random access process, that is, it passes the access restriction ACB check; when Then you have to wait until a random time T barring , and then re-perform ACB examination; among them, T barring =(0.7+0.6u)T ACB ;
[0153] The wireless device that has completed the ACB check randomly selects a preamble code from the preamble codes used for access contention and sends it to the base station for random access; when the preamble code selection of multiple wireless devices conflicts, the backoff mechanism is activated, and the process returns to step S21, and the ACB check is re-performed on all wireless devices that have conflicted, until the preamble code is selected by only one wireless device, and the wireless device successfully accesses the base station.
[0154] Optionally, a model training module 303 is used to initialize the wireless device, various parameters in the neural network in the PPO algorithm, and the environment information of the agent;
[0155] The wireless device access result of the current time slot is used as the current state s of the agent t Input into the neural network, output the action with the highest probability, and use it as the current state s t The best action a under t ;
[0156] The agent model performs the optimal action a t Interact with the agent environment and get instant rewards r t and the next state s t+1 ; The current state s t , optimal action a t , Instant Rewards t and the next state s t+1 Constitute the transfer experience, transfer experience (s t ,a t ,r t ,s t+1 ) is stored in the buffer;
[0157] Repeat the process until the optimal action selection is completed in all states, and store all the transfer experience in the buffer.
[0158] Optionally, the action space is set to the access restriction factor of the base station in each time slot The state space consists of the access results for each time slot known to the base station, including the number of devices that successfully accessed Number of conflicting devices and the number of idle preambles
[0159] Optionally, the deep learning module 304 is used to instantly reward r t It includes main line rewards and auxiliary rewards; the main line reward is the main goal that should be completed by the task, and the main goal is set as the number of devices that successfully access; the main goal is decomposed into multiple sub-goals, and the reward corresponding to each sub-goal is the auxiliary reward, and the auxiliary reward is set as the absolute value of the difference between the number of devices that pass the ACB check and the number of preambles F. Then the instant reward r for the kth type of wireless devicet The expression is:
[0160]
[0161] where α k is the coefficient, is the number of successful accesses of the kth type of wireless devices, The number of devices of category k that passed the ACB check.
[0162] Optionally, the advantage function A in each state is calculated according to the following formula (3): πθ (s t ,a t );
[0163]
[0164] Where V πθ (s t ) is the base value, γ is the discount factor, and T is the total number of steps in one round;
[0165] The agent randomly selects transfer experience from the buffer for deep learning and calculates the objective function according to the following formula (4): Update network parameters θ:
[0166]
[0167] Among them, c1 and c2 are coefficients, Objective Function By clipping objective function Error term in value function and Entropy Reward The clipping objective function consists of three parts; It can be expressed as:
[0168]
[0169] in, is the long-term reward function, representing the new strategy π θ (a t |s t ) and the old strategy ratio; when , then the current state s is encouraged t The best action a under t , σ t (θ) will show an increasing trend; otherwise, the current state s will be punished t The best action a under t , σ t (θ) will show a decreasing trend;
[0170] Determine whether the preset number of learning times has been reached. If the preset number of learning times has been reached, the network parameters θ obtained by deep learning are used to replace the original network parameters θ old ; If the preset number of learning times is not reached, return to step S41;
[0171] The network parameter θ is trained and iterated multiple times by the gradient ascent method. The long-term reward function is maintained within a certain range and no longer increases, which means that the number of successful accesses of each category reaches the maximum, and the delay reaches the minimum on the basis of satisfying the access success rate of each category. The resource allocation optimization process is terminated and the allocation optimization of wireless resources is completed.
[0172] In the embodiment of the present invention, the present invention proposes an effective priority-based dynamic access category restriction mechanism to solve the problem of massive device access and service transmission, reasonably allocate access resources according to the heterogeneous needs of users, and focus on overload control to support a large number of machine type communication MTC devices. Figure 4 As shown, it is a diagram of a large-scale wireless device access network scenario. In order to meet the heterogeneous needs of different applications, MTC devices are divided into multiple categories according to their characteristics, different ACB factors are used for different categories of devices, and dynamic adjustments are made according to real-time traffic conditions to maximize the utilization of random access resources. By modeling the random resource optimization problem as a partially observable Markov decision process, the intelligent agent is trained with the help of the PPO algorithm to obtain the corresponding actions and strategies to dynamically adjust the ACB factors of various types of devices, thereby maximizing the throughput while ensuring that high-priority devices have priority access. In the PPO framework of the invention, the problem of difficult step size determination in the PG algorithm (Policy Optimization, policy gradient algorithm) is solved by introducing a clipping alternative objective function. The uniqueness of the random resource allocation optimization method and device based on PPO will be proved through iterative cycles, and the overall economic benefits of random access resource allocation will eventually be achieved.
[0173] Figure 5 4 is a schematic diagram of the structure of an electronic device 400 provided in an embodiment of the present invention. The electronic device 400 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 401 and one or more memories 402, wherein the memory 402 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 401 to implement the following steps of the wireless resource allocation optimization method:
[0174] S1: Prioritize the wireless devices to obtain wireless devices of different priorities; set different access restriction factors for the wireless devices of different priorities;
[0175] S2: Establish an intelligent agent model through the base station to make decisions on various access restriction factors;
[0176] S3: Train the agent model through the proximal policy optimization PPO algorithm to obtain instant rewards and different states of the agent model; use the instant rewards and different states of the agent model as transfer experience;
[0177] S4: Calculate the objective function, perform deep learning on the objective function through transfer experience, and complete the optimization of wireless resource allocation.
[0178] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, and the instructions can be executed by a processor in a terminal to complete the above-mentioned wireless resource allocation optimization method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0179] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0180] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for optimizing wireless resource allocation, characterized in that: include: S1: Prioritize the wireless devices to obtain wireless devices of different priorities; set different access restriction factors for the wireless devices of different priorities; S2: Establish an intelligent agent model through the base station to make decisions on various access restriction factors; In S2, an intelligent agent model is established through the base station to make decisions on various access restriction factors, including: S21: Setting the access restriction factor to When the kth wireless device attempts to access, a random number q is generated. When the wireless device enters the random access process, that is, it passes the access restriction ACB check; when , then we have to wait until a random time T barring , and then re-perform ACB examination; among them, T barring =(0.7+0.6u)T ACB ; S22: The wireless device that has completed the ACB check randomly selects a preamble from the preambles used for access contention and sends it to the base station for random access; when a conflict occurs in the preamble selection of multiple wireless devices, the backoff mechanism is started, and the process returns to step S21, and the ACB check is performed again on all wireless devices that have conflicted, until the preamble is selected by only one wireless device, and the wireless device successfully accesses the base station; S3: Train the agent model through the proximal policy optimization PPO algorithm to obtain instant rewards and different states of the agent model; use the instant rewards and different states of the agent model as transfer experience; In S3, the agent model is trained by the proximal policy optimization PPO algorithm to obtain instant rewards and different states of the agent model; the instant rewards and different states of the agent model are transfer experiences, including: S31: Initialize the wireless device, various parameters in the neural network in the PPO algorithm, and the environment information of the intelligent agent; S32: The wireless device access result of the current time slot is used as the current state s of the intelligent agent t Input into the neural network, output the action with the highest probability, and use it as the current state s t The best action a under t ; S33: The agent model performs the optimal action a t Interact with the agent environment and get instant rewards r t and the next state s t+1 ; The current state s t , optimal action a t , Instant Rewards t and the next state s t+1 Constitute the transfer experience, transfer experience (s t ,a t ,r t ,s t+1 ) is stored in the buffer; S34: Repeat steps S32-33 until the optimal action selection in all states is completed, and all the formed transfer experiences are stored in the buffer; S4: Calculate the objective function, perform deep learning on the objective function through transfer experience, and complete the allocation optimization of wireless resources; In S4, the objective function is calculated, and deep learning of the objective function is performed by transferring experience, including: S41: Calculate the advantage function A in each state according to the following formula (3): πθ (s t ,a t ); Where V πθ (s t ) is the base value, γ is the discount factor, and T is the total number of steps in one round; S42: The agent randomly selects transfer experience from the buffer for deep learning and calculates the objective function according to the following formula (4): Update network parameters θ: Among them, c1 and c2 are coefficients, Objective Function By clipping objective function Error term in value function and Entropy Reward The clipping objective function consists of three parts: It is expressed as: in, is the long-term reward function, representing the new strategy π θ (a t |s t ) and the old strategy ratio; when , then the current state s is encouraged t The best action a under t , σ t (θ) will show an increasing trend; otherwise, the current state s will be punished t The best action a under t , σ t (θ) will show a decreasing trend; S43: Determine whether the preset number of learning times has been reached. If the preset number of learning times has been reached, use the network parameters θ obtained by deep learning to replace the original network parameters θ old ; If the preset number of learning times is not reached, return to step S41; S44: The network parameter θ is iteratively updated for multiple trainings through the gradient ascent method. The long-term reward function is maintained within a preset range and does not rise any more. This means that the number of successful accesses of each category reaches a maximum, and the latency reaches a minimum on the basis of satisfying the access success rate of each category. The resource allocation optimization process is terminated, and the wireless resource allocation optimization is completed.
2. The method for optimizing wireless resource allocation according to claim 1, characterized in that: In S1, wireless devices are prioritized to obtain wireless devices with different priorities, and different access restriction factors are set for the wireless devices with different priorities, including: S11: Prioritize the wireless devices according to their sensitivity to delay, classify the wireless devices that are sensitive to delay as high-priority users, and classify the wireless devices that are tolerant to delay as low-priority users; S12: Different access restriction factors are set for wireless devices of different priorities, so that devices with high priority have priority in accessing the restriction factor, and devices with low priority have access to the restriction factor later.
3. The method for optimizing wireless resource allocation according to claim 2, characterized in that: The priorities from high to low are: emergency warning service, equipment control interaction service and data collection service.
4. The method for optimizing wireless resource allocation according to claim 3, characterized in that: Set the action space to the access restriction factor of the base station in each time slot The state space consists of the access results for each time slot known to the base station, including the number of devices that successfully accessed Number of conflicting devices and the number of idle preambles 5. The method for optimizing wireless resource allocation according to claim 4, characterized in that: The instant reward t It includes main line rewards and auxiliary rewards; the main line reward is the main goal that should be completed by the task, and the main goal is set as the number of devices that successfully access; the main goal is decomposed into multiple sub-goals, and the reward corresponding to each sub-goal is the auxiliary reward, and the auxiliary reward is set as the absolute value of the difference between the number of devices that pass the ACB check and the number of preambles F. Then the instant reward r for the kth type of wireless device t The expression is: where α k is the coefficient, is the number of successful accesses of the kth type of wireless devices, The number of devices of category k that passed the ACB check.
6. A wireless resource allocation optimization device, characterized in that: The device is used for the method described in any one of claims 1 to 5, and the device comprises: A priority classification module, used to classify the wireless devices into priorities to obtain wireless devices of different priorities; and to set different access restriction factors for the wireless devices of different priorities; The decision module is used to establish an intelligent agent model through the base station and make decisions on various access restriction factors; A model training module is used to train the agent model through the proximal policy optimization PPO algorithm to obtain instant rewards and different states of the agent model; and use the instant rewards and different states of the agent model as transfer experience; The deep learning module is used to calculate the target function, perform deep learning on the target function by transferring experience, and complete the allocation optimization of wireless resources.
7. The wireless resource allocation optimization device according to claim 6, characterized in that: The priority division module is used to: Prioritize devices based on their sensitivity to latency. Devices that are sensitive to latency are classified as high-priority users, while devices that are latency-tolerant are classified as low-priority users. Different access restriction factors are set for the different priorities, so that the devices with high priority have priority in access restriction factors, and the devices with low priority have access restriction factors later.
