5G mixed slice resource optimal allocation method based on prediction and reinforcement learning

Abstract

The invention provides a 5G mixed slice resource optimal allocation method based on prediction and reinforcement learning, and the method comprises the steps: obtaining historical state data based on the interaction between an intelligent agent and a system environment, and the intelligent agent is constructed based on a Gaussian kernel reinforcement learning algorithm; using a prediction model to predict the historical state data to obtain prediction data including a plurality of SLA indexes, the prediction model being constructed based on a stacked long short-term memory network prediction algorithm; and taking data meeting a preset condition in the prediction data as new state data, obtaining action data by the intelligent agent based on the new state data, and determining a resource allocation scheme based on the action data. According to the invention, the time-frequency resource can be efficiently utilized on the premise of ensuring the SLA index.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a method for optimizing the allocation of 5G hybrid slice resources based on prediction and reinforcement learning. Background Technology

[0002] Fifth-generation (5G) mobile communication technology is commonly used in smart manufacturing scenarios. Due to the increasing demands of numerous mobile terminals on communication systems in terms of latency, flexibility, and bandwidth, traditional wired networks are no longer sufficient to meet their needs for widespread connectivity and high mobility. Therefore, 5G hybrid slicing technology can be used to address the frequently changing communication requirements of numerous terminals.

[0003] Existing resource allocation efforts typically rely on model-free reinforcement learning to design resource allocation algorithms, such as TD3 and NAF. However, their traditional static resource allocation strategies are insufficient to meet the performance requirements of mixed scenarios, and suffer from problems such as poor algorithm interpretability, low sample efficiency, redundant policy space exploration, and difficulty in handling multi-objective conflicts. Furthermore, most algorithms are highly complex and lack systematic consideration of agent learning frequency and training overhead; simultaneously, in computationally limited deployment environments, this can lead to excessively long system learning cycles, affecting their deployability and real-time response capabilities in practical scenarios. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a 5G hybrid slice resource optimization allocation method based on prediction and reinforcement learning.

[0005] To achieve the above objectives, the present invention adopts the following technical solution.

[0006] In a first aspect, the present invention provides a 5G hybrid slice resource optimization allocation method based on prediction and reinforcement learning, comprising:

[0007] Historical state data is obtained based on the interaction between the intelligent agent and the system environment. The intelligent agent is constructed based on the Gaussian kernel reinforcement learning algorithm.

[0008] The historical state data is used to make predictions using a prediction model to obtain predicted data including multiple SLA indicators. The prediction model is constructed based on a stacked long short-term memory network prediction algorithm.

[0009] The data in the predicted data that meet the preset conditions are taken as new state data. The agent obtains action data based on the new state data and determines a resource allocation scheme based on the action data.

[0010] In some embodiments of this application, the preset condition includes that the predicted data for Y consecutive rounds all meet the SLA index, and the step of using the data in the predicted data that meets the preset condition as new state data includes:

[0011] Based on the data in the predicted data that meet the preset conditions, obtain the signal-to-noise ratio sequence, the queuing buffer capacity sequence, the number of connected devices sequence, and the delay sequence;

[0012] The worst-performing quantity is selected from the signal-to-noise ratio sequence, the queuing buffer capacity sequence, the number of connected devices sequence, and the delay sequence, respectively, to obtain the target signal-to-noise ratio, the target queuing buffer capacity, the target number of connected devices, and the target delay, thus forming the new state data.

[0013] In some embodiments of this application, obtaining the signal-to-noise ratio sequence, the queuing buffer capacity sequence, the number of connected devices sequence, and the delay sequence based on data satisfying preset conditions in the predicted data includes:

[0014] The intelligent agent scans and detects the prediction data. If the scan results indicate that the prediction data for Y consecutive rounds after the starting point all meet the SLA index, it is determined that the prediction data for Y consecutive rounds meets the preset condition.

[0015] The signal-to-noise ratio sequence, queuing buffer capacity sequence, number of connected devices sequence, and delay sequence are obtained from the prediction data of the Y consecutive rounds.

[0016] In some embodiments of this application, the intelligent agent obtains action data based on the new state data, including:

[0017] In each round, the agent performs an action based on the new state data. Then, randomly select any number;

[0018] If the randomly selected number is less than the random start probability threshold, the agent will stop executing actions in the next round. And perform actions based on the previous round. Reinforcement learning is initiated based on the observed system environment state.

[0019] In some embodiments of this application, the prediction data includes eMBB slice SLA prediction data, URLLC slice SLA prediction data, and mMTC slice SLA prediction data.

[0020] Secondly, the present invention also provides a 5G hybrid slice resource optimization and allocation device based on prediction and reinforcement learning, comprising:

[0021] An interaction module is used to obtain historical state data based on the interaction between the intelligent agent and the system environment. The intelligent agent is constructed based on the Gaussian kernel reinforcement learning algorithm.

[0022] The prediction module is used to predict the historical state data using a prediction model to obtain predicted data including multiple SLA indicators. The prediction model is constructed based on a stacked long short-term memory network prediction algorithm.

[0023] The resource allocation module is used to take the data in the prediction data that meets the preset conditions as new state data. The agent obtains action data based on the new state data and determines a resource allocation scheme based on the action data.

[0024] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described above.

[0025] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0026] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0027] The beneficial effects of this invention are as follows: The 5G hybrid slice resource optimization allocation method based on prediction and reinforcement learning provided by this invention, by considering the constraints such as the probability of service level agreement (SLA) violation in the mixed service scenarios of eMBB, URLLC, and mMTC, and the limited total amount of 5G air interface time-frequency resources, defines the state, action, and reward functions of Gaussian kernel reinforcement learning, establishes the objective of minimizing the amount of time-frequency resources used, and utilizes the advantage of Gaussian kernels in handling highly nonlinear data to help the agent select appropriate resource allocation strategies. The key steps of Gaussian kernel reinforcement learning training and decision-making are also given. The Gaussian kernel reinforcement learning (GKBRL) algorithm provided in this application introduces the idea of ​​machine learning, optimizes the design of model-free reinforcement learning algorithms to avoid overexploration of the policy space, and solves problems such as low resource utilization efficiency and frequent violation of service level agreements, providing a new approach for the design of 5G hybrid slice time-frequency resource optimization allocation algorithms. Furthermore, the 5G hybrid slice resource optimization allocation method based on prediction and reinforcement learning provided in the embodiments of this application also utilizes Stacked-LSTM networks to predict key SLA indicators such as signal-to-noise ratio, queuing buffer capacity, latency, and number of device connections in eMBB, URLLC, and mMTC scenarios. In addition, during the process of the agent obtaining action data based on the new state data, randomness is also considered. To increase the randomness of the Stacked-LSTM-GKBRL model, a random start probability threshold is set. Finally, Gaussian kernel reinforcement learning uses the SLA index prediction value and the random start probability threshold to execute the learning algorithm to obtain a near-optimal allocation strategy, while effectively reducing the frequency and complexity of machine learning.

[0028] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the pipelined 5G network architecture provided in an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of the system network structure provided in an embodiment of the present invention;

[0032] Figure 3This is one of the flowcharts illustrating the 5G hybrid slice resource optimization allocation method based on prediction and reinforcement learning provided in this embodiment of the invention.

[0033] Figure 4 The second flowchart illustrates the 5G hybrid slice resource optimization allocation method based on prediction and reinforcement learning provided in this embodiment of the invention.

[0034] Figure 5 This is a schematic diagram of the structure of the Stacked-LSTM model provided in an embodiment of the present invention;

[0035] Figure 6 A schematic diagram illustrating the process by which an intelligent agent obtains new state data from prediction data, as provided in an embodiment of the present invention.

[0036] Figure 7 This is a schematic diagram of the predictive data processing flow provided in an embodiment of the present invention;

[0037] Figure 8 Figures (a) to (d) show the prediction results of the Stacked-LSTM model provided in the embodiments of the present invention for signal-to-noise ratio, queue buffer capacity, latency, and number of connected devices, respectively.

[0038] Figure 9 The loss function curve of the Stacked-LSTM algorithm provided in the embodiments of the present invention;

[0039] Figure 10 The predicted residual distribution of the Stacked-LSTM model provided in this embodiment of the invention;

[0040] Figure 11 MSE plot of the prediction results of the Stacked-LSTM model provided in this embodiment of the invention;

[0041] Figure 12 Resource allocation diagrams for four algorithms provided in embodiments of the present invention;

[0042] Figure 13 This is a schematic diagram illustrating the cumulative SLA violation situation provided in an embodiment of the present invention. Detailed Implementation

[0043] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0044] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0045] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0046] Before explaining the 5G hybrid slicing resource optimization allocation method based on prediction and reinforcement learning provided in the embodiments of this application, we will first introduce the 5G private network design of the carrier communication module detection pipeline and the 5G network slicing system model.

[0047] The carrier communication module testing pipeline contains some testing terminals and transfer equipment, such as AGVs, information acquisition equipment, and performance testing units. Since traditional 4G networks cannot meet the requirements of these terminals, such as high bandwidth, massive device access, and low latency data transmission, this application embodiment designs a 5G private network for the carrier communication module testing pipeline based on the 5G network architecture.

[0048] like Figure 1As shown, in this 5G network architecture for the production line, the access network consists of a 5G Industrial IoT gateway (CPE) and 5G base stations, while the core network consists of a MEC (Multi-access Edge Computing) edge service platform and a production management system. The 5G Industrial IoT gateway (CPE) is primarily responsible for connecting 5G service terminals and is a crucial link in 5G communication. Regarding the communication process of terminal devices accessing the 5G private network, taking an AGV as an example: when performing a transportation task, the AGV sends its own location data to the 5G Industrial IoT gateway (CPE) in real time; the 5G Industrial IoT gateway (CPE) then forwards the location data to the MEC edge service platform via the 5G base station; finally, the production management system uniformly arranges tasks and performs scheduling optimization.

[0049] Since multiple terminals accessing a 5G private network can easily lead to contention and occupation of time and frequency resources, 5G network slicing technology is considered to isolate resources between different 5G services and ensure data transmission security.

[0050] Consider a multi-slice downlink transmission scenario in a production line factory, consisting of a 5G base station and multiple terminals. Figure 2 This is a schematic diagram of the system network. Terminals of different service types have different data packet sizes, packet arrival time intervals, and different SLA requirements. The base station includes... Each slice shares all the radio resources available to the base station. The radio interface between the base station and the terminal is constructed as a frame, with each frame divided into time and frequency dimensions: in the time dimension, the frame is divided into Transmission Time Intervals (TTIs), assuming a TTI duration of 1ms; in the frequency dimension, the frame is divided into frequency subcarriers, assuming a subcarrier spacing of 15kHz. The smallest time-frequency combination unit is called a resource block (RB), consisting of one TTI and 12 frequency subcarriers. These basic resource blocks collectively constitute the entire radio spectrum resources. This application embodiment considers allocating corresponding time-frequency resources to each slice, and these resources are used exclusively by that slice, thereby ensuring the isolation of resources between slices. For coexisting radio frequency interfaces A set of network slices, in In a system scenario with individual slices The total number of resource blocks allocated among these slices, and the number of SLA (Service Level Agreement) violations. This is the upper bound. Therefore, the optimal resource allocation strategy objective of the 5G hybrid slice resource optimization allocation model proposed in this application embodiment is:

[0051] (1)

[0052] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0053] Example 1

[0054] See Figure 3 as well as Figure 4 A method for optimizing the allocation of 5G hybrid slice resources based on prediction and reinforcement learning includes the following steps:

[0055] S101 obtains historical state data based on the interaction between the intelligent agent and the system environment.

[0056] The intelligent agent is constructed based on the Gaussian-Kernel-Based Reinforcement Learning (GKBRL) algorithm.

[0057] In this step, the agent first interacts with the system environment for a preset number of rounds (e.g., 200 rounds), and then records the historical state data output by the agent in each round.

[0058] S102, the historical state data is predicted using a prediction model to obtain predicted data including multiple SLA indicators.

[0059] The prediction model is constructed based on the Stacked-LSTM prediction algorithm. The prediction data includes eMBB slice SLA prediction data, URLLC slice SLA prediction data, and mMTC slice SLA prediction data.

[0060] Stacked-LSTM consists of three gates: a forget gate, an input gate, and an output gate, such as... Figure 5 This is a layer in a Stacked-LSTM structure. Each gate also contains a separate memory cell for recording long-term information. The forget gate indicates that some information is discarded from the memory cell. The input gate is responsible for storing new information in the memory cell. The third is the output gate, which provides the final output to the LSTM module at the transmission interval. The output of each round serves as the input for the next round, continuing the prediction iteration until the 200th round is completed. Finally, the result is processed into a state sequence, which serves as the input to the agent.

[0061] The Stacked-LSTM algorithm also uses mean squared error (MSE) as a loss function, which measures the difference between the predicted value and the true value. It calculates the average of the squares of the prediction error, as shown in formula (2).

[0062] (2)

[0063] in The total number of samples, For the first The true value of each sample The model predicts the first The values ​​of each sample. Furthermore, in this embodiment, Adam (Adaptive Moment Estimation) will be used as the optimizer to continuously optimize the loss function, helping the model to output the most accurate results possible.

[0064] S103, the data in the predicted data that meets the preset conditions is taken as new state data, the agent obtains action data based on the new state data, and determines a resource allocation scheme based on the action data.

[0065] The preset conditions include that the predicted data for Y consecutive rounds all meet the SLA index, and the step of using the predicted data that meets the preset conditions as new state data includes:

[0066] Based on the predicted data that meet the preset conditions, obtain the signal-to-noise ratio sequence, the queuing buffer capacity sequence, the number of connected devices sequence, and the delay sequence.

[0067] Schematic, the intelligent agent scans and detects the prediction data. If the scan result indicates that the prediction data of the Y consecutive rounds after the starting point all meet the SLA index, it is determined that the prediction data of the Y consecutive rounds meet the preset condition.

[0068] The signal-to-noise ratio sequence, queuing buffer capacity sequence, number of connected devices sequence, and delay sequence are obtained from the prediction data of the Y consecutive rounds.

[0069] The worst-performing quantity is selected from the signal-to-noise ratio sequence, the queuing buffer capacity sequence, the number of connected devices sequence, and the delay sequence, respectively, to obtain the target signal-to-noise ratio, the target queuing buffer capacity, the target number of connected devices, and the target delay, thus forming the new state data.

[0070] In some embodiments of this application, the intelligent agent obtains action data based on the new state data, including:

[0071] In each round, the agent performs an action based on the new state data. Then, randomly select any number;

[0072] If the randomly selected number is less than the random start probability threshold, the agent will stop executing actions in the next round. And perform actions based on the previous round. Reinforcement learning is initiated based on the observed system environment state.

[0073] like Figure 6 As shown, the process by which an agent obtains new state data from prediction data includes the following steps:

[0074] After receiving the prediction data from the Stacked-LSTM prediction module, the agent first scans and checks the various indicator data. If the following situation exists: the same round is selected as the starting point, subsequent rounds... The data from each consecutive round satisfies the SLA. The agent then bases its actions on this... The state variables of this interval are composed of data from consecutive rounds, and the starting round of this interval is recorded. and the end of the round Secondly, regarding If the data from consecutive rounds all meet the SLA condition, the prediction data is divided into four sequences: signal-to-noise ratio. Queue buffer capacity Number of connected devices and latency Each sequence All data points meet the SLA, and the quantity with the worst relative performance is selected from each set of sequences: signal-to-noise ratio. Queue buffer capacity Delay and the number of connected devices Then, these four quantities with the worst relative performance are combined to form a new state. .

[0075] like Figure 7 As shown, the agent is based on the new state Execute the GKBRL algorithm to obtain actions Used as this Resource allocation in consecutive rounds. Among them, in this... The agent will no longer perform reinforcement learning within consecutive rounds, but will instead execute actions in each round. and record the actions performed. The algorithm then analyzes the subsequent system environment status to provide data for subsequent Stacked-LSTM model predictions. Simultaneously, it considers randomness factors, setting a random start probability threshold to increase the randomness of the Stacked-LSTM-GKBRL model. Perform actions in each round. A random number is then drawn. If the number is less than the random start probability threshold, the agent stops executing actions in the next round. and based on the previously executed actions Reinforcement learning is initiated based on the observed system environment state.

[0076] When the random number is less than the random start probability threshold, or for cases where there is no random number... If the data from consecutive rounds all satisfy the SLA, the agent will not use the predicted data to form new state variables. Instead, before the start of each round, it will first observe the system environment to obtain the observed state, then perform reinforcement learning to configure actions based on the observed state, and then proceed to the next round, and so on.

[0077] In some embodiments of this application, the 5G slice time-frequency resource allocation problem is modeled using a Markov Decision Process (MDP), which includes the following mathematical definitions: The goal is to find a suitable strategy that minimizes the amount of resources allocated. It represents the set of all possible states, i.e., the state space; It represents the set of all possible behaviors or actions that an intelligent agent can choose to perform; Indicates starting from the current state Execute action Then transition to the next state The probability distribution; Indicates the execution of an action Afterwards, the agent changes state. Transferred to The reward received at that time; It is a value between 0 and 1, representing a discount factor that determines the degree of importance placed on future rewards. In the 5G slicing time-frequency resource allocation problem, the focus will be on the state. ,action and rewards The specific definition is as follows:

[0078] 1. State Space

[0079] Because the Stacked-LSTM-GKBRL algorithm calculates and makes the optimal action based on the current observations, it ensures the SLA of the terminal device while allocating RBs as efficiently as possible. Therefore, the state It needs to reflect the current status of metrics for each slice in the network. For each slice... In each round ,Will Defined as an agent in round Receive slice The observed values. Represented as system observations or state-space values, i.e., the first... The set of observations for each slice in each round. The main observation variables include signal-to-noise ratio, queuing buffer capacity, latency, and number of connected devices. In the round After preprocessing, it reflects the traffic requests of each slice in the current network.

[0080] 2. Motion space

[0081] In this embodiment of the application, the action of allocating slice resources is defined as allocating corresponding time-frequency resources to each service type slice. Therefore, Indicating in round Agents assigned to slices The number of RBs. Indicating in round The set of time-frequency resources allocated to each slice, i.e., action space values.

[0082] 3. Reward Function

[0083] To measure the SLA violation of a slice, an indicator function is first defined. It includes key metrics defined in the SLA, and in each round, it feeds back to the agent the SLA implementation status of each slice. Indicating in round Violation of Article The SLA for each slice is set; a value of 0 indicates that the SLA has not been violated. When the SLA remains unviolated for an extended period, the agent may get trapped in a local optimum. Therefore, another indicator function is still needed. , express All consecutive rounds of data satisfy the SLA; otherwise The reward function is defined as follows:

[0084] (3)

[0085] Indicating in round The reward for the observed action pair, among which and These are discount factors. If the slice's SLA is violated, it means the actions configured in the current round cannot meet business needs, requiring more time-frequency resources to be allocated. Conversely, if the slice's SLA is met, it means the actions configured in the current round already meet business needs, leading to an attempt to reduce the amount of allocated resources and explore a near-optimal time-frequency resource allocation strategy, thereby saving resources. In this case, appropriately reducing rewards reduces the allocated time-frequency resources, thus stimulating the agent to explore outwards and gradually approach the near-optimal time-frequency resource allocation strategy, thereby achieving resource conservation.

[0086] 4. Optimization Objectives

[0087] Since the indicator function directly affects the reward function, to ensure the accuracy of the indicator function, the hypothesis function will be... Defined as the observations received at the end of the previous round. To make assumptions and judgments about the next round Does it meet the SLA? However, inaccurate assumptions may lead to issues in the next round. This would violate the SLA too much, so an error function also needs to be set. The error function... Defined as by In observation-action pairs The assumption given above is the probability of actually violating the SLA even though it was not violated, i.e. The probability of misjudgment. Therefore, the error function is expressed as the error probability. To find the optimal resource allocation strategy, the objective function in equation (1) is optimized as follows:

[0088] (4)

[0089] In some embodiments of this application, the agent constructed based on the Gaussian kernel reinforcement learning (GKBRL) algorithm mainly consists of a hypothesis module, an error module, and a decision module.

[0090] The hypothesis module's role is to assume whether each network slice can meet its SLA under a given time-frequency resource allocation. In actual detection pipelines, facing complex and diverse environments, simple linear online learning cannot accurately simulate the numerous nonlinear detection scenarios in real-world scenes. To address this shortcoming, a relatively effective strategy is to use kernel functions. The hypothesis function... The kernel expansion is shown below:

[0091] (5)

[0092] Among them, the core Indicates new sample and existing samples in the sample set Similarity between them; It is an existing sample in the sample set. Corresponding to their respective indicator functions; This is the sample set for which the assumption is incorrect.

[0093] Regarding kernel functions Since the Gaussian kernel is suitable for highly nonlinear data, it can be used to... Written as:

[0094] (6)

[0095] in To control the width of the kernel function, The Euclidean distance between two vectors:

[0096] (7)

[0097] in Indicates the dimension of the vector. When and When very close, Approaching 1; when and When they are far apart, Close to 0.

[0098] If the assumption is incorrect, a new sample will be used. Add to sample set In, it becomes the new sample set And simultaneously update the hypothesis function:

[0099] (8)

[0100] The updated hypothesis function As the input for the next round of the agent.

[0101] The error module is used to evaluate the accuracy of the hypothesis module, given a slice. Observations Assuming that slicing can be satisfied... In the SLA action, if If it is the smallest action, then the action will be... security boundary Defined as:

[0102] (9)

[0103] Under the assumption that the SLA is satisfied, the larger the safety margin, the larger the error function. The smaller the value, the lower the probability of a false assumption. The closer the safety margin is to 0, the higher the probability of a false assumption, and the easier it is to violate the SLA. This is due to the error function... Defined as The probability of misjudgment, so only Updated only when an assumption error occurs. Then, based on the actual SLA fulfillment situation Therefore, the error function is precisely defined. for:

[0104] (10)

[0105] in This is the learning rate. At the end of each round, the error function corresponding to each slice is output. And use it as the input for the agent in the next round.

[0106] The decision module receives the observations at the end of the previous round. round hypothesis function set Sum of error functions The prediction can satisfy the slice. The smallest movement in SLA. The corresponding error function may not necessarily satisfy the condition of being less than that in formula (4). Given the bounded error probability, we need to further search for the optimal action. That is, to find the minimum action that satisfies the following two conditions:

[0107] 1) This action can satisfy the slicing requirement. SLA;

[0108] 2) The error function corresponding to this action is less than the error probability. .

[0109] For each slice, the decision module uses the error function corresponding to that slice to... Given the condition, select the minimum safety boundary for each slice. Simultaneously, select the smallest action that satisfies the SLA. Then and Add them together to get the number of rounds. Optimal action for each slice The specific process is as follows:

[0110] 1. Input: Total number of resource blocks Error probability The maximum number of resource blocks obtained from each slice ;

[0111] 2. Initialize the hypothesis function set Error function set Starting round End of round ;

[0112] 3. Obtain initial state observations from the environment. ;

[0113] 4. for rounds do;

[0114] 5.if ;

[0115] 6. Stacked-LSTM predicts the next 200 rounds based on the observations of the past 200 rounds, and inputs the prediction results into the agent;

[0116] 7. The agent combines new states based on the prediction results. and update the starting round. End of round ;

[0117] 8.if ;

[0118] 9. The decision-making module adjusts according to the new state. The hypothesis function of the previous round Error function Generate and execute actions Record observations Obtain reward function ;continue;

[0119] 10. if ;

[0120] 11. The agent continues to perform the action. Record observations Obtain reward function ;

[0121] 12. If the random number is less than the random start probability threshold;

[0122] 13. ;continue;

[0123] 14. Update the hypothesis function Error function ;

[0124] 15. Decision module execution actions Obtain and record the observations. Obtain reward function , end for.

[0125] In some embodiments of this application, to verify the advantages of the Stacked-LSTM-GKBRL model in prediction accuracy, relevant simulation experiments were built in the PyCharm simulation platform to simulate the allocation of time-frequency resources between RAN slices in a 5G hybrid scenario. This scenario involves a base station serving three different service types of slices: eMBB, URLLC, and mMTC. The specific physical parameter settings for the network slicing environment are shown in Table 1.

[0126] Table 1 Wireless Network Environment Simulation Parameter Settings

[0127]

[0128] To verify the advantages of the Stacked-LSTM-GKBRL model in prediction accuracy and resource allocation, a simulation scenario was designed consisting of 2 eMBB slices, 2 URLLC slices, and 1 mMTC slice, with each subframe using a maximum of 150 RBs. The specific settings of the service models for each slice are shown in Table 2, where the transmission period and number of packet retransmissions for the mMTC service model are randomly selected from the set shown.

[0129] Table 2 Key Parameter Settings for RAN Slice Flow

[0130]

[0131] Depend on Figure 8 (a) to Figure 8 (d) shows a relatively intuitive comparison between the predicted results (including the predicted signal-to-noise ratio, queuing buffer capacity, latency, and number of connected devices) and the actual values ​​obtained by the Stacked-LSTM prediction model. It is evident that the predicted data curves generally follow the same trend as the actual curves, and are quite close in most cases. Even if the actual data fluctuates, the Stacked-LSTM prediction model can still roughly fit the distribution of the actual data.

[0132] like Figure 9As shown, the training loss curve drops rapidly in the first few epochs and then gradually levels off, indicating that the Stacked-LSTM prediction model is still learning, gradually learning the features of the data, and reducing errors. The test loss curve does not show a significant upward trend; instead, it gradually decreases and levels off as training progresses, without significant fluctuations. This indicates that the Stacked-LSTM prediction model's error on the test set is stable, its generalization ability has improved, and there is no obvious overfitting or underfitting.

[0133] To compare the prediction accuracy performance of the Stacked-LSTM model, a control experiment was conducted with four other prediction models: Recurrent Neural Network (RNN), Gated Recurrent Unit (GRU), eXtreme Gradient Boosting (XGBoost), and Long Short-Term Memory (LSTM) network. Figure 10 The distribution of residuals (differences between predicted and actual values) for signal-to-noise ratio predictions using five algorithms is shown in the figure. Overall, the residual distribution of the Stacked-LSTM prediction model approximates a normal distribution, indicating that the residuals are randomly distributed without significant systematic bias. It can be seen that most of the residuals from the Stacked-LSTM prediction model are concentrated between [-0.2, 0.2] dB, with fewer extreme residuals. The residuals roughly conform to a unimodal bell-shaped curve, indicating that the prediction error of the Stacked-LSTM prediction model is symmetrical and has small bias. Compared to the residual distributions of the RNN, GRU, XGBoost, and LSTM models, the Stacked-LSTM model's residual distribution is more concentrated, while the other four models are relatively dispersed, indicating that the Stacked-LSTM prediction model has smaller bias and better prediction performance.

[0134] Figure 11The graph shows the mean squared error (MSE) of five prediction algorithms—Stacked-LSTM, RNN, XGBoost, GRU, and LSTM—regarding the signal-to-noise ratio (SNR) prediction results. It can be seen that the MSE value of the Stacked-LSTM prediction model is lower than that of the RNN, XGBoost, GRU, and LSTM algorithms. Furthermore, the MSE value of the Stacked-LSTM prediction model remains relatively stable with increasing epochs, without significant fluctuations. This indicates that the trained Stacked-LSTM prediction model performs well in actual predictions, without significant prediction bias. The MSE value of the Stacked-LSTM prediction model increases slightly between 150 and 200 epochs. Since the Stacked-LSTM prediction model builds upon the previous prediction result, this suggests that the prediction error inevitably accumulates gradually with increasing epochs.

[0135] To fully evaluate the performance of the Stacked-LSTM-GKBRL model, this embodiment also compares it with four classic reinforcement learning algorithms: TD3 (Twin Delayed Deep Deterministic Policy Gradient), NAF (Normalized Advantage Function), SAC (Soft Actor Critic), and A2C (Synchronous Advantage Actor Critic), as well as GKBRL without prediction. Additionally, [further details are needed]. , and Three control scenarios.

[0136] exist Figure 12In the resource allocation diagram, the average curve and confidence interval at a 90% confidence level were plotted for each algorithm. It can be seen that among the other four algorithms, A2C and NAF converge faster; while the curves of NAF, GKBRL, and Stacked-LSTM-GKBRL series show an upward trend in the first 3000 rounds, gradually converging in subsequent rounds; the TD3 curve only gradually converges around 6000 rounds. Regarding resource allocation, the NAF and A2C curves are basically stable at around 120 RBs, indicating over-allocation of resources in the early rounds; while the TD3 and SAC curves gradually rise in the range of 120-145 RBs, allocating resources close to the upper limit of 150 RBs while trying to meet the SLA, indicating over-allocation of resources. In contrast, during the initial 3000 rounds, the curves of GKBRL and Stacked-LSTM-GKBRL series rise rapidly, subsequently showing a gradual convergence trend and stabilizing resource allocation. Stacked-LSTM-GKBRL prevents over-allocation of resources in the early rounds, thereby optimizing resource utilization. In comparison, although the convergence speed of the control algorithm is not significantly different, it wastes more resources in resource allocation, and its resource allocation accuracy performance is far inferior to the Stacked-LSTM-GKBRL algorithm proposed in this application.

[0137] As can be seen from the curves of GKBRL and Stacked-LSTM-GKBRL series algorithms, the overall time-frequency resources allocated by the GKBRL algorithm are less than those of the Stacked-LSTM-GKBRL series algorithms, and the overall resource allocation of Stacked-LSTM-GKBRL decreases with... The increased number of rounds leads to a decrease in resource allocation accuracy and performance. This is because the Stacked-LSTM prediction model requires reserving additional resources to ensure SLA compliance, resulting in a trade-off between resource allocation accuracy and SLA compliance. Furthermore, the increased number of rounds necessitates considering more relatively poor-performing samples, thus increasing the overall resource allocation required. Figure 12 The curves of the Stacked-LSTM-GKBRL series are relatively close, indicating that the additional resources required are not particularly large, and they do not consume excessive resources. Furthermore, the curves of Stacked-LSTM-GKBRL-50 and Stacked-LSTM-GKBRL-80 are quite similar, indicating that when... Even after it exceeds 50, No matter how many intervals there are, the resources that need to be allocated do not increase significantly.

[0138] Figure 13For cumulative SLA violations, from Figure 13 It is evident that the NAF, TD3, SAC, and A2C curves are significantly higher than those of the Stacked-LSTM-GKBRL series curves, indicating that Stacked-LSTM-GKBRL performs excellently in avoiding SLA violations. Meanwhile, the Stacked-LSTM-GKBRL series curves are generally lower than the GKBRL curves, demonstrating that using the Stacked-LSTM prediction model effectively reduces SLA violations, further improving performance.

[0139] Furthermore, since the agent does not need to learn frequently after using the Stacked-LSTM prediction model, it not only saves computing resources and reduces algorithm complexity, but also maintains a SLA satisfaction rate close to that of the GKBRL algorithm, and even outperforms it in some rounds. Combining resource allocation and the number of SLA violations as two indicators, A value less than or equal to 50 is considered the relatively optimal solution.

[0140] This invention provides a 5G hybrid slice resource optimization allocation method based on prediction and reinforcement learning. By considering constraints such as the probability of Service Level Agreement (SLA) violation in mixed service scenarios (eMBB, URLLC, and mMTC) and the limited total amount of 5G air interface time-frequency resources, it defines the state, action, and reward functions of Gaussian kernel reinforcement learning. It establishes an objective of minimizing the amount of time-frequency resources used, leveraging the Gaussian kernel's suitability for handling highly nonlinear data to help the agent select appropriate resource allocation strategies. The key steps of Gaussian kernel reinforcement learning training and decision-making are also provided. The Gaussian kernel reinforcement learning (GKBRL) algorithm provided in this application introduces machine learning ideas, optimizes the design of model-free reinforcement learning algorithms to avoid overexploration of the policy space, and solves problems such as low resource utilization efficiency and frequent SLA violations. This provides a new approach for the design of 5G hybrid slice time-frequency resource optimization allocation algorithms.

[0141] In addition, the 5G hybrid slice resource optimization allocation method based on prediction and reinforcement learning provided in this application also uses Stacked-LSTM network to predict key SLA indicators such as signal-to-noise ratio, queuing buffer capacity, latency, and number of device connections in eMBB, URLLC, and mMTC scenarios.

[0142] In addition, during the process of the agent obtaining action data based on the new state data, randomness is also considered. To increase the randomness of the Stacked-LSTM-GKBRL model, a random start probability threshold is set. Finally, Gaussian kernel reinforcement learning uses the SLA index prediction value and the random start probability threshold to execute the learning algorithm to obtain a near-optimal allocation strategy, while effectively reducing the frequency and complexity of machine learning.

[0143] In summary, the 5G hybrid slice resource optimization allocation method based on prediction and reinforcement learning provided in this application demonstrates good performance in prediction accuracy, convergence speed, and resource allocation while ensuring basic performance such as SLA violation probability constraints and the quantity of time-frequency resources. It effectively reduces the frequency and complexity of machine learning and ultimately verifies the performance of the 5G hybrid slice resource optimization allocation algorithm.

[0144] Example 2

[0145] Based on Embodiment 1, Embodiment 2 provides a 5G hybrid slice resource optimization and allocation device based on prediction and reinforcement learning. This 5G hybrid slice resource optimization and allocation device based on prediction and reinforcement learning corresponds to the aforementioned 5G hybrid slice resource optimization and allocation method based on prediction and reinforcement learning, and specifically includes:

[0146] An interaction module is used to obtain historical state data based on the interaction between the intelligent agent and the system environment. The intelligent agent is constructed based on the Gaussian kernel reinforcement learning algorithm.

[0147] The prediction module is used to predict the historical state data using a prediction model to obtain predicted data including multiple SLA indicators. The prediction model is constructed based on a stacked long short-term memory network prediction algorithm.

[0148] The resource allocation module is used to take the data in the prediction data that meets the preset conditions as new state data. The agent obtains action data based on the new state data and determines a resource allocation scheme based on the action data.

[0149] For specific details, please refer to the description in the section on 5G hybrid slicing resource optimization allocation method based on prediction and reinforcement learning, which will not be repeated here.

[0150] Example 3

[0151] Embodiment 3 of the present invention provides an electronic device, including a memory and a processor, which communicate with each other. The memory stores program instructions that can be executed by the processor. The processor calls the program instructions to execute a 5G hybrid slice resource optimization allocation method based on prediction and reinforcement learning. The method includes the following steps:

[0152] Historical state data is obtained based on the interaction between the intelligent agent and the system environment. The intelligent agent is constructed based on the Gaussian kernel reinforcement learning algorithm.

[0153] The historical state data is used to make predictions using a prediction model to obtain predicted data including multiple SLA indicators. The prediction model is constructed based on a stacked long short-term memory network prediction algorithm.

[0154] The data in the predicted data that meet the preset conditions are taken as new state data. The agent obtains action data based on the new state data and determines a resource allocation scheme based on the action data.

[0155] Example 4

[0156] Embodiment 4 of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements a 5G hybrid slice resource optimization allocation method based on prediction and reinforcement learning. The method includes the following steps:

[0157] Historical state data is obtained based on the interaction between the intelligent agent and the system environment. The intelligent agent is constructed based on the Gaussian kernel reinforcement learning algorithm.

[0158] The historical state data is used to make predictions using a prediction model to obtain predicted data including multiple SLA indicators. The prediction model is constructed based on a stacked long short-term memory network prediction algorithm.

[0159] The data in the predicted data that meet the preset conditions are taken as new state data. The agent obtains action data based on the new state data and determines a resource allocation scheme based on the action data.

[0160] Example 5

[0161] Embodiment 5 of the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements a 5G hybrid slice resource optimization allocation method based on prediction and reinforcement learning. The method includes the following steps:

[0162] Historical state data is obtained based on the interaction between the intelligent agent and the system environment. The intelligent agent is constructed based on the Gaussian kernel reinforcement learning algorithm.

[0163] The historical state data is used to make predictions using a prediction model to obtain predicted data including multiple SLA indicators. The prediction model is constructed based on a stacked long short-term memory network prediction algorithm.

[0164] The data in the predicted data that meet the preset conditions are taken as new state data. The agent obtains action data based on the new state data and determines a resource allocation scheme based on the action data.

[0165] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0166] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for method or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the description of the method embodiments. The method and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0167] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A 5G hybrid slice resource optimization allocation method based on prediction and reinforcement learning, characterized in that, include: Historical state data is obtained based on the interaction between the intelligent agent and the system environment. The intelligent agent is constructed based on the Gaussian kernel reinforcement learning algorithm. The historical state data is used to make predictions using a prediction model to obtain predicted data including multiple SLA indicators. The prediction model is constructed based on a stacked long short-term memory network prediction algorithm. The data in the predicted data that meet the preset conditions are taken as new state data. The agent obtains action data based on the new state data and determines a resource allocation scheme based on the action data.

2. The method according to claim 1, characterized in that, The preset conditions include that the predicted data for Y consecutive rounds all meet the SLA index, and the step of using the predicted data that meets the preset conditions as new state data includes: Based on the data in the predicted data that meet the preset conditions, obtain the signal-to-noise ratio sequence, the queuing buffer capacity sequence, the number of connected devices sequence, and the time delay sequence; The worst-performing quantity is selected from the signal-to-noise ratio sequence, the queuing buffer capacity sequence, the number of connected devices sequence, and the delay sequence, respectively, to obtain the target signal-to-noise ratio, the target queuing buffer capacity, the target number of connected devices, and the target delay, thus forming the new state data.

3. The method according to claim 2, characterized in that, The step of obtaining the signal-to-noise ratio sequence, queuing buffer capacity sequence, number of connected devices sequence, and delay sequence based on data in the predicted data that meet preset conditions includes: The intelligent agent scans and detects the prediction data. If the scan results indicate that the prediction data for Y consecutive rounds after the starting point all meet the SLA index, it is determined that the prediction data for Y consecutive rounds meets the preset condition. The signal-to-noise ratio sequence, queuing buffer capacity sequence, number of connected devices sequence, and delay sequence are obtained from the prediction data of the Y consecutive rounds.

4. The method according to claim 1, characterized in that, The agent obtains action data based on the new state data, including: In each round, the agent performs an action based on the new state data. Then, randomly select any number; If the randomly selected number is less than the random start probability threshold, the agent will stop executing actions in the next round. And perform actions based on the previous round. Reinforcement learning is initiated based on the observed system environment state.

5. The method according to any one of claims 1 to 4, characterized in that, The prediction data includes eMBB slice SLA prediction data, URLLC slice SLA prediction data, and mMTC slice SLA prediction data.

6. A 5G hybrid slice resource optimization and allocation device based on prediction and reinforcement learning, characterized in that, include: An interaction module is used to obtain historical state data based on the interaction between the intelligent agent and the system environment. The intelligent agent is constructed based on the Gaussian kernel reinforcement learning algorithm. The prediction module is used to predict the historical state data using a prediction model to obtain predicted data including multiple SLA indicators. The prediction model is constructed based on a stacked long short-term memory network prediction algorithm. The resource allocation module is used to take the data in the prediction data that meets the preset conditions as new state data. The agent obtains action data based on the new state data and determines a resource allocation scheme based on the action data.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-5.

Images