Network slice arrangement method based on user experience quality

Abstract

The invention is applicable to the technical field of communication, and provides a network slice arrangement method based on quality of experience (QoS), which comprises the following steps of: converting QoS into QoE (Quality of Experience), combining the QoE with a cost factor to provide a corresponding optimization target, and solving by adopting a VMGMDQN algorithm; the input of the VMGMDQN algorithm is a reinforcement learning related parameter set Omega, the maximum number of iterations maxiter and a network resource set Sigma, and the output of the VMGMDQN algorithm is an optimal mapping strategy Xbest. Simulation results prove that the method can effectively improve the overall benefit of the network, realize more reasonable VNF mapping and greatly improve the user experience quality. In short, the method obtains an optimal deployment scheme from the virtual nodes to the physical nodes, so that resources can be saved, and a better effect can be achieved.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and in particular to a network slicing orchestration method based on user experience quality. Background Art

[0002] Network slicing orchestration is the process of creating and managing multiple virtual slices on the same physical network to meet different business needs and provide customized services. Ensure that each slice meets its service standards to support applications such as the Internet of Things, high-speed data and low-latency communications, while improving resource efficiency and user experience.

[0003] At present, a large number of scholars have conducted extensive research and analysis on 5G network slicing orchestration strategies. In their research, there are both traditional research methods, such as creating an orchestration system or making new divisions in the orchestration process, and machine learning methods, such as using various deep learning networks (DQN, LSTM, DDPG) to perform network slicing orchestration. However, most of these studies focus on improving the user service quality (QoS) indicator, while relatively ignoring the importance of user experience quality (QoE). However, for 5G and even future super-5G network environments, user experience quality, as an evaluation indicator centered on user perception, is becoming increasingly important. Different from service quality based only on technical parameters, user experience quality pays more attention to the actual feelings and satisfaction of users, and is a more comprehensive and intuitive standard for evaluating the superiority of network services. Therefore, future network slicing orchestration solutions should consider user experience quality more to truly meet user needs and expectations and promote the development and optimization of 5G and future network technologies. To this end, a network slicing orchestration method based on user experience quality is proposed. Summary of the invention

[0004] The purpose of the present invention is to provide a network slicing orchestration method based on user experience quality, aiming to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A network slice orchestration method based on user experience quality is proposed. The method converts QoS into QoE, combines QoE with cost factors to propose corresponding optimization goals, and adopts VMGMDQN algorithm to solve. The input of the VMGMDQN algorithm is a reinforcement learning related parameter set Ω, a maximum number of iterations max_iter and a network resource set Σ, and the output is an optimal mapping strategy X. best , the specific steps of the VMGMDQN algorithm are as follows:

[0007] Step 1: Initialization, including initialization of the agent set and the reinforcement learning environment;

[0008] Step 2: Iterative learning. The algorithm enters a double-layer iteration loop. The outer loop controls the number of iterations, and the inner loop represents the number of elements in the agent set.

[0009] Step 3: Evolution mechanism, including scoring and ranking, strategy parameter crossover, strategy parameter mutation, cumulative rewards, and new set generation;

[0010] Step 4, End and Output: After all iterations are completed, the algorithm ends and returns the best mapping strategy.

[0011] Furthermore, the specific process of step 1 is as follows:

[0012] Initialize the agent set: Create an agent set Agents, each of which executes a mapping strategy from VNF to physical network;

[0013] Reinforcement learning environment initialization: define the state space St, action space Ac and reward function Re.

[0014] Furthermore, the specific process of step 2 is as follows:

[0015] Observation and reset: At the beginning of each iteration, observe the initial state s, which belongs to the state set St; and reset the environment to prepare for a new round of mapping process;

[0016] Mapping process: If the mapping process is not completed, the ε-greedy strategy is used to select action a, which belongs to the action set Ac; after executing action a, the new state s' and reward r are obtained; the agent calculates the expected benefit of executing the action through equations 1 and 2, and updates the current state;

[0017] Formula 1 is as follows:

[0018]

[0019] Formula 2 is as follows:

[0020]

[0021] Where π is the strategy adopted by the agent; Pro is the state transition probability function, and the transition probability Pro(s'|s,a) of executing action a in state s, entering the new state s', and generating immediate reward Re(s,a); V π is the state value function, which represents the benefit of following strategy π starting from a given state s; Q π is the action value function, which represents the expected return of taking action a in a given state s and following the strategy π;

[0022] Record and evaluate: Record the reward score finally obtained by agent j.

[0023] Furthermore, the specific process of step 3 is as follows:

[0024] Scoring and sorting: At a specific iteration interval, score and sort each agent according to its reward score to select the elite set E agent ;

[0025] Strategy parameter crossover: Randomly select E agent A pair of agents in c,1 ,agent c,2 , according to probability aP c Exchange Agent c,1 ,agent c,2 strategy parameters; add the generated new agent to the new agent set newAgents;

[0026] Strategy parameter variation: Randomly select E agent An agent in m , with a certain probability aP m Randomly select policy parameters for mutation, i.e. apply small random perturbations to the selected parameters; add the generated new agents to newAgents;

[0027] Cumulative rewards and new set generation: Calculate the cumulative rewards newRe of the corresponding strategies of each agent in the new agent set newAgents; Based on newRe, accept bad individuals with a certain probability and generate a new set of Agents.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] In view of the common problems of low network resource utilization and poor user experience in the process of virtual network function (VNF) orchestration, the present invention takes user experience quality (QoE) as the core and proposes a network slice orchestration method. This method converts the traditional network quality of service (QoS) indicator into a more intuitive user experience indicator QoE, and combines it with the cost factor to propose the corresponding optimization goal. Subsequently, an evolutionary reinforcement learning algorithm (VMGMDQN) is proposed to solve it. Simulation results show that this method can effectively improve the overall efficiency of the network, achieve more reasonable VNF mapping and greatly improve the user experience quality. In short, this method obtains an optimal deployment plan from virtual nodes to physical nodes, which can save resources and achieve better results. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of SFC mapping of the present invention.

[0031] Figure 2It is the cost under different demand numbers in the present invention.

[0032] Figure 3 It is the total profit under different demand numbers of the present invention.

[0033] Figure 4 is the user access rate under different demand numbers of the present invention.

[0034] Figure 5 is the average cost of the present invention under different delays.

[0035] Figure 6 is the number of access requirements under different delays of the present invention.

[0036] Figure 7 is the total throughput of the system of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0038] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0039] like Figure 1 As shown, it is a schematic diagram of the SFC (Service Function Chaining) mapping of the present invention, which embodies the best deployment scheme from virtual nodes to physical nodes, which can save resources and achieve better results.

[0040] Example 1. This embodiment provides a network slice orchestration method based on user experience quality. First, a mapping mechanism is constructed to upgrade the traditional quality of service (QoS) indicator to a QoE indicator that directly reflects the user experience. Then, based on this QoE indicator and combined with network resource consumption, a corresponding optimization goal is proposed. Finally, an advanced evolutionary reinforcement learning algorithm (VMGMDQN) is proposed. A network slice orchestration strategy corresponds to an intelligent agent, which is also a single individual of the evolutionary algorithm. The intelligent agent continuously interacts with the environment (network resource conditions), optimizes the structure, and rewards performance. After multiple iterations, the intelligent agent obtains the best fitness, thereby obtaining the best solution and forming the best set of network slices.

[0041] The input of the VMGMDQN algorithm is the reinforcement learning related parameter set Ω, the maximum number of iterations max_iter and the network resource set Σ, and the output is the optimal mapping strategy X best , the specific steps of the VMGMDQN algorithm are as follows:

[0042] Step 1: Initialization, including:

[0043] Initialization of the agent set: Create a set of agents, each of which executes a mapping strategy from VNF (virtual network function) to physical network; these agents will continuously adjust their strategies in the subsequent learning process to maximize the overall network performance.

[0044] Reinforcement learning environment initialization: define the state space St (all situations that the agent may encounter), the action space Ac (all actions that the agent can take), and the reward function Re (the evaluation criterion for the agent's actions).

[0045] Step 2: Iterative learning. The algorithm enters a double-layer iteration loop. The outer loop controls the number of iterations, and the inner loop represents the number of elements in the agent set. Specifically, it includes:

[0046] Observation and reset: At the beginning of each iteration, observe the initial state s, which belongs to the state set St; and reset the environment to prepare for a new round of mapping process;

[0047] Mapping process: If the mapping process is not completed, the ε-greedy strategy is used to select action a, which belongs to the action set Ac; after executing action a, the new state s' and reward r are obtained; the agent calculates the expected benefit of executing the action through equations 1 and 2, and updates the current state;

[0048] Formula 1 is as follows:

[0049]

[0050] Formula 2 is as follows:

[0051]

[0052] Where π is the strategy adopted by the agent; Pro is the state transition probability function, and the transition probability Pro(s'|s,a) of executing action a in state s, entering the new state s', and generating immediate reward Re(s,a); V π is the state value function, which represents the benefit of following strategy π starting from a given state s; Q π is the action value function, which represents the expected return of taking action a in a given state s and following the strategy π;

[0053] Record and evaluate: Record the reward score finally obtained by agent j.

[0054] Step 3: Evolutionary mechanism, including:

[0055] Scoring and sorting: At a specific iteration interval (e.g., once every several generations), score and sort each agent according to its reward score to select the elite set E. agent ;

[0056] Strategy parameter crossover: Randomly select E agent A pair of agents in c,1 ,agent c,2 , according to probability aP c Exchange Agent c,1 ,agent c,2 The strategy parameters are used to realize the crossover and mutation of the strategy. The generated new agents are added to the new agent set newAgents;

[0057] Strategy parameter variation: Randomly select E agent An agent in m , with a certain probability aP m Randomly select policy parameters for mutation, i.e. apply small random perturbations to the selected parameters; add the generated new agents to newAgents;

[0058] Cumulative rewards and new set generation: Calculate the cumulative rewards newRe of the corresponding strategies of each agent in the new agent set newAgents; Based on newRe, accept bad individuals with a certain probability and generate a new set of Agents;

[0059] Step 4: End and output, including:

[0060] After all iterations are completed, the algorithm ends and returns the best mapping strategy, which is the optimal strategy learned in all iterations and can achieve the best resource allocation and network performance for given network requirements and resource constraints.

[0061] The description of VMGMDQN algorithm is shown in Table 1:

[0062] Table 1 VMGMDQN algorithm description

[0063]

[0064] About the conversion from QoS to QoE:

[0065] Traditional QoS-QoE mapping methods and strategies are usually designed based on traditional network architecture and application scenarios, and are difficult to directly apply to the new scenario of multiple slices in 5G networks. First, users connecting to multiple slices may affect each other. For example, the load of one slice may affect the performance of another slice. Traditional methods may not be able to consider this interactive impact. In addition, the allocation and access of slice resources in 5G networks are dynamic and adjusted according to user needs and network conditions. Traditional methods may not be able to adapt to this dynamic slice allocation in a timely manner. Therefore, the conversion from QoS to QoE will be discussed below, and the specific conversion calculation method obtained will be used to calculate the optimization target value later. The main QoS indicators include bandwidth, delay, availability, jitter and packet loss rate. The present invention takes eMBB, URLLC, mMTC, and V2X slices in the Internet of Vehicles as the basis for QoE calculation, and uses the TOPSIS method for analysis, which is a multi-attribute decision analysis tool based on the distance between each solution and the ideal solution and the negative ideal solution, used to identify the optimal decision solution.

[0066] First, based on the importance of each indicator to different network slices, an evaluation matrix for each network slice is constructed, as shown in Equation 3, where the columns represent four example network slices: eMBB, uRLLC, mMTC, and V2X; the rows represent different QoS indicators: bandwidth, latency, availability, jitter, and packet loss rate.

[0067] The evaluation matrix is ​​normalized by applying Equation 4 and Equation 5. Equation 4 is used to normalize positive indicators, such as bandwidth and availability. For other indicators, Equation 5 is used for calculation, where min({E i,j}) represents the minimum value in the jth column. This method converts smaller original values ​​into larger standardized values ​​to reflect that smaller values ​​in the cost indicator are more optimal. i,j , R i,j are the elements of the i-th row and j-th column of the original matrix and the result matrix, respectively. mr represents the number of rows, and MaxR represents the maximum number of rows. This process actually calculates the Euclidean norm of the j-th column.

[0068] Formula 3:

[0069]

[0070] Formula 4:

[0071]

[0072] Formula 5:

[0073]

[0074] The standardized evaluation matrix can be obtained as shown in Formula 6. In the vehicle-to-everything (V2X) environment, different network slices serve different application scenarios, and their importance depends on the contribution of these applications to the overall performance of the vehicle network. Based on this, a weight can be assigned to each slice as Formula 7.

[0075] Formula 6:

[0076]

[0077] Formula 7:

[0078] W = [0.25, 0.35, 0.15, 0.25]

[0079] Among them, eMBB and V2X are very important for providing high-quality media transmission and basic IoV communications, so they receive relatively high weights. URLLC is crucial to ensuring security and real-time performance in IoVs, so it receives the highest weight. mMTC is also important in IoVs, especially for large-scale sensor and device communications, but its urgency and impact may be smaller than other scenarios, so it receives a lower weight. Thus, the weighted normalized decision matrix in Equation 8 can be obtained:

[0080] Formula 8:

[0081]

[0082] The maximum and minimum values ​​in the weighted standardized decision matrix V are taken as the ideal optimal solution V + and the worst solution V - Based on the weighted normalization matrix, the ideal optimal solution V can be determined + and the worst solution V - :

[0083] Ideal optimal solution V + :

[0084] Formula 9:

[0085] V + =[0.2439,0.35,0.35,0.35,0.1755]

[0086] Worst Solution V - :

[0087] Formula 10:

[0088] V - =[0.0029,0.0015,0.0075,0.0015,0.0745]

[0089] For each QoS indicator, use Equation 11 and Equation 12 to calculate its ideal optimal solution and the worst solution The Euclidean distance of .

[0090] Formula 11:

[0091]

[0092] Formula 12:

[0093]

[0094] where v ij It is the element in the i-th row and j-th column of the weighted normalized decision matrix V, representing the weighted performance of the i-th scheme (QoS indicator) under the j-th evaluation criterion (network slice). is the ideal optimal solution vector V + The jth element in represents the best performance of all solutions under the jth evaluation criterion. is the worst solution vector V - The jth element in represents the worst performance of all solutions under the jth evaluation criterion. ne is the total number of evaluation criteria (network slices), which is 4 here.

[0095] Therefore, we can get the relative proximity of the five QoS indicators, that is, the Euclidean distance between the indicator and the ideal optimal solution or the worst solution:

[0096] Formula 13:

[0097]

[0098] Formula 14:

[0099]

[0100] In the TOPSIS method, relative proximity is used to evaluate the closeness of each solution to the ideal solution. In the present invention, it is the closeness of each QoS indicator to the ideal solution. The formula for calculating relative proximity is:

[0101] Formula 15:

[0102]

[0103] According to the calculation, the relative closeness of each QoS indicator is as follows:

[0104] Formula 16:

[0105] RP=[0.3978,0.3851,0.3909,0.4524,0.5008]

[0106] The higher the relative closeness, the closer the performance of the QoS indicator is to the ideal optimal solution, that is, the better the performance in the Internet of Vehicles environment. Therefore, according to formula 16, the mapping formula for converting QoS indicators to QoE indicators can be obtained:

[0107] Formula 17:

[0108] QoE=RP×QoS

[0109] Among them, QoS is a column vector, each column represents the detailed value of bandwidth, delay, jitter, packet loss rate and availability indicators. The cross product of this column vector and RP can obtain the quality of experience QoE.

[0110] Example 2: MATLAB is used for simulation, and each set of experimental data is obtained by repeating 50 independent experiments and taking the average value. Consider an IoT system consisting of 10 base stations, 8 slices, and multiple vehicle users. Each base station maintains a certain amount of slices. Users travel from one end of the base station to the other end. The number of users continues to increase, and users have random QoS requirements.

[0111] The experimental results are as follows:

[0112] like Figure 2 and Figure 3 As shown ( Figure 2 The vertical axis in is a dimensionless value, and the specific value is used to determine whether the deployment consumption is greater or not. Figure 3 Similarly), as the number of users increases, changes in system costs and profits can be observed. Figure 2 It shows that the VMGMDQN algorithm maintains a low resource consumption rate as user demand increases, showing efficient resource allocation. In contrast, the resource consumption of the A2C algorithm increases significantly as the number of users increases, reflecting lower efficiency. Figure 3 It shows that the VMGMDQN algorithm provides higher system benefits as the number of users increases, indicating that it is more effective in meeting user needs and predicting long-term rewards. In comparison, the A2C algorithm has high resource consumption but relatively stable system benefits, and its benefits grow slowly.

[0113] exist Figure 4 In the figure, as user demand increases, the network system approaches its maximum load, resulting in a decrease in the access rate of user demand. In this process, the access rate of the VMGMDQN algorithm decreases slowly, while the other two algorithms (DQN and VNS algorithms), especially the VNS algorithm, show a faster rate of decline. This may be because VNS, as a heuristic algorithm, allocates a large amount of resources to users in the early stage to improve efficiency, but in the case of limited resources, as time goes by, the number of users who can access the system decreases, resulting in a rapid decrease in the access rate.

[0114] Figure 5 and Figure 6 The average cost consumption and access demand of the VMGMDQN algorithm under different delay requirements are shown. Through experimental observations of three delay requirements of 1ms, 5ms and 8ms, it is found that when the network demand is low, as the demand gradually increases, the average cost of the network shows an upward trend. However, when the average cost reaches a certain peak, as the demand further increases, the cost begins to decline. This is because more network demands are met and absorbed by the system. It is worth noting that stricter delay requirements (such as 1ms) seem to bring lower average costs. This may be because when resources are tight and fast delay requirements cannot be met, some user demands cannot be connected to the system, and thus no corresponding costs are incurred. Therefore, Figure 6 It shows that as the latency requirement increases, the number of user demands that can be met decreases. Especially under the latency requirement of 1ms, the resource shortage becomes more obvious in the later stage of the experiment, resulting in a slowdown in the growth trend of the number of demands that can be connected.

[0115] Figure 7 It shows that although the performance of each algorithm was similar at the beginning, the VMGMDQN algorithm showed significant advantages as demand increased, especially in the high-demand range, where its performance significantly exceeded the VNS algorithm and the other two reinforcement learning algorithms (DQN and A2C algorithms). This is due to the efficient resource mapping strategy of the VMGMDQN algorithm, which can optimize user access under resource constraints and achieve optimal throughput. Although the VNS algorithm has a higher throughput in the early stage, its performance is limited as the system pressure increases. On the other hand, the reinforcement learning algorithm, although not as good as VNS in the early stage, gradually adapts to the growth in demand through continuous optimization, and its performance gradually improves.

[0116] In general, the VMGMDQN algorithm provided by the present invention shows excellent and stable performance through a detailed resource allocation strategy under the consideration of resource constraints and user needs, especially in high-load scenarios, which is significantly better than other algorithms, proving its superiority in network systems that handle large-scale user access.

[0117] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A network slice orchestration method based on user experience quality, characterized in that: The method converts QoS into QoE, combines QoE with cost factors to propose corresponding optimization goals, and uses the VMGMDQN algorithm to solve; the input of the VMGMDQN algorithm is the reinforcement learning related parameter set Ω, the maximum number of iterations max_iter and the network resource set Σ, and the output is the optimal mapping strategy X best , the specific steps of the VMGMDQN algorithm are as follows: Step 1: Initialization, including initialization of the agent set and reinforcement learning environment; Step 2: Iterative learning. The algorithm enters a double-layer iteration loop. The outer loop controls the number of iterations, and the inner loop represents the number of elements in the agent set. Step 3: Evolution mechanism, including scoring and ranking, strategy parameter crossover, strategy parameter mutation, cumulative rewards, and new set generation; Step 4, End and Output: After all iterations are completed, the algorithm ends and returns the best mapping strategy.

2. The network slice arrangement method based on user experience quality according to claim 1 is characterized in that: The specific process of step 1 is as follows: Initialize the agent set: Create an agent set Agents, each of which executes a mapping strategy from VNF to physical network; Reinforcement learning environment initialization: define the state space St, action space Ac and reward function Re.

3. The network slice arrangement method based on user experience quality according to claim 1, characterized in that: The specific process of step 2 is as follows: Observation and reset: At the beginning of each iteration, observe the initial state s, which belongs to the state set St; and reset the environment to prepare for a new round of mapping process; Mapping process: If the mapping process is not completed, the ε-greedy strategy is used to select action a, which belongs to the action set Ac; after executing action a, the new state s' and reward r are obtained; the agent calculates the expected benefit of executing the action through equations 1 and 2, and updates the current state; Formula 1 is as follows: Formula 2 is as follows: Where π is the strategy adopted by the agent; Pro is the state transition probability function, and the transition probability Pro(s'|s,a) of executing action a in state s, entering the new state s', and generating immediate reward Re(s,a); V π is the state value function, which represents the benefit of following strategy π starting from a given state s; Q π is the action value function, which represents the expected return of taking action a in a given state s and following the strategy π; Record and evaluate: Record the reward score finally obtained by agent j.

4. The network slice arrangement method based on user experience quality according to claim 1, characterized in that: The specific process of step 3 is as follows: Scoring and sorting: At a specific iteration interval, score and sort each agent according to its reward score to select the elite set E agent ; Strategy parameter crossover: Randomly select E agent A pair of agents in c,1 ,agent c,2 , according to probability aP c Exchange Agent c,1 ,agent c,2 strategy parameters; add the generated new agent to the new agent set newAgents; Strategy parameter variation: Randomly select E agent An agent in m , with a certain probability aP m Randomly select policy parameters for mutation, i.e. apply small random perturbations to the selected parameters; add the generated new agents to newAgents; Cumulative rewards and new set generation: Calculate the cumulative rewards newRe of the corresponding strategies of each agent in the new agent set newAgents; Based on newRe, accept bad individuals with a certain probability and generate a new set of Agents.

Images