A network slice resource optimization method and device based on deep reinforcement learning

Abstract

The present invention discloses a method and device for optimizing network slice resources based on deep reinforcement learning, wherein the method includes: setting a network slice framework of a soft-hard hybrid strategy; based on the network slice framework, on the basis of guaranteeing the network slice SLA within the network slice window, maximizing the spectrum efficiency to obtain a network slice optimization problem; based on the network slice optimization problem, under the condition of a given initial resource allocation, using a DQN network to perform online network slice resource optimization and adjustment processing to obtain a network slice resource optimization result. In an embodiment of the present invention, resources can be allocated to network slices with arbitrary QoS requirements based on a unified optimization framework; spectrum efficiency can be improved on the basis of guaranteeing different slice SLAs.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and in particular to a network slice resource optimization method and device based on deep reinforcement learning. Background Art

[0002] In recent years, with the implementation and development of 5G technology, vertical industry applications such as industrial Internet of Things, cellular Internet of Vehicles, and augmented / virtual reality have flourished. Different applications bring differentiated performance requirements. For example, Internet of Vehicles has extremely high requirements for latency and reliability, and augmented / virtual reality requires high speed and low latency. Different vertical application scenarios have prompted 5G networks to provide differentiated services on the same physical network, and have put forward higher requirements for the scalability, availability, and cost of 5G networks. Network slicing technology has been widely studied in 3GPP and is considered to be a disruptive technology that enables 5G to provide flexible and customizable heterogeneous services on the same physical network.

[0003] Network slicing enables multiple network slices to complete different transmission tasks through virtualization, isolation and sharing of wireless resources on top of the common physical network infrastructure. At present, standard organizations, such as 3GPP, are also widely discussing how to design the Radio Access Network (RAN) to support network slicing, and conducting research on the service level agreement (SLA) guarantee based on slice resource orchestration and management on the RAN side. The network slicing technology of the core network is relatively mature and can be implemented through advanced computing, caching and virtualized container technologies. In comparison, RAN-side slicing is more challenging due to limited radio resources, heterogeneous service requirements and dynamic wireless network environment. From the perspective of Radio Resource Management (RRM), RRM between multiple network slices is crucial to guarantee the SLA agreement of network slicing. The traditional MAC scheduler directly schedules each user in each Transmission Time Interval (TTI). Due to the existence of multiple slices, the traditional MAC scheduling problem has actually become a multi-objective optimization problem, which is too complex to be solved in actual deployment. In contrast, multi-slice RRM under the SLA framework is decomposed into inter-slice and intra-slice RRM, which can be executed separately on different time scales. Intra-slice RRM is responsible for allocating intra-slice dedicated resources to corresponding intra-slice users, focusing on TTI-level packet performance, corresponding to the slice customized MAC scheduler. Inter-slice radio resource allocation is responsible for allocating total radio resources to each slice at the beginning of each slice cycle window, focusing on the global guarantee of the slice SLA protocol. Therefore, how to optimize the inter-slice radio resource allocation to ensure the global SLA of the network has become a new technical issue.

[0004] Prior art 1 provides a resource allocation method and system for improving network quality in a multi-slice network that can make effective use of wireless resources. To achieve the above purpose, the invention adopts the following technical scheme for a resource allocation method for improving network quality in a multi-slice network, including the following contents: 1) Slicing the heterogeneous network to be tested, and modeling the resource allocation problem that affects the network quality of each slice, and establishing a resource allocation model for each slice; 2) Processing the non-convex function in each resource allocation model to obtain an approximate convex function of each resource allocation model; 3) Under the condition that the tenants corresponding to each slice divide the total subcarriers equally, using a continuous convex approximation algorithm, according to the number of subcarriers of each slice, solve the approximate convex objective function of the resource allocation model of each slice, and determine the optimal slice resource allocation strategy and scoring criteria for each slice; 4) Under the condition that the tenants corresponding to each slice do not divide the total subcarriers equally, using a non-cooperative game method, according to the number of subcarriers of each slice, solve the approximate convex objective function of the resource allocation model of each slice, and determine the optimal slice resource allocation strategy and scoring criteria for each slice.

[0005] The second prior art proposes a 5G network slicing resource allocation method, which takes the user delay requirement as the main constraint to meet the service needs of different users under different delay requirements and improve spectrum utilization.

[0006] The main contents of the invention include: 1) Acquisition of user communication delay sensitivity: Acquisition of the sensitivity of mobile users to communication delay in 5G wireless network slices, and data quantification thereof; 2) Allocation of user service requests: Allocation of service requests proposed by mobile users to different network slices according to business types; 3) Allocation of spectrum resources between slices: According to the generalized Kelly algorithm and the channel state of the current user, a cost function based on user transmission delay is defined, and under the goal of minimizing the cost function, the network parameters for base station spectrum resource allocation in the cost function are iteratively solved, and after obtaining the parameters, the spectrum resources of the base station are allocated to different network slices based on this; 4) User spectrum resource allocation: The base station spectrum resources allocated to different network slices are further divided using the Lyapunov algorithm, so that each mobile user obtains the required spectrum resources; 5) Data transmission: The user completes data transmission based on the obtained spectrum resources.

[0007] The above-mentioned existing technologies have improved and enhanced the slice network resource allocation in certain aspects for certain performance indicators. However, in the scenario where multiple heterogeneous service slices coexist, the existing technologies have the following problems: 1. Since the existence of multiple slices brings about diverse quality of service (QoS) indicators, the traditional resource allocation problem has actually become a multi-objective optimization problem, which is too complex to be solved in actual deployment. The network module designed based on QoS guarantee makes the network module and QoS indicators highly coupled, which violates the design principle of flat and scalable 5G network; 2. Performance isolation and spectrum efficiency between slices is a tricky trade-off problem. Existing technologies rarely consider how to balance the relationship between performance isolation and spectrum efficiency between slices. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art. The present invention provides a network slice resource optimization method and device based on deep reinforcement learning, which can allocate resources for network slices with arbitrary QoS requirements based on a unified optimization framework; it can improve spectrum efficiency while ensuring the SLA of different slices.

[0009] In order to solve the above technical problems, an embodiment of the present invention provides a network slice resource optimization method based on deep reinforcement learning, the method comprising:

[0010] Set up a network slicing framework with a mix of hard and soft strategies;

[0011] Based on the network slicing framework, the network slicing SLA is guaranteed within the network slicing window, and the spectrum efficiency is maximized to obtain the network slicing optimization problem;

[0012] Based on the network slicing optimization problem and under the condition of given initial resource allocation, the DQN network is used to perform online network slicing resource optimization and adjustment processing to obtain the network slicing resource optimization result.

[0013] Optionally, the network slicing framework for setting a soft and hard hybrid strategy includes:

[0014] Divide the total bandwidth network slice resources into two parts: soft policy and hard policy;

[0015] For the hard policy, dedicated bandwidth resources are allocated to each network slice in the network slice window;

[0016] For soft policies, public network slice resources are occupied in each network slice according to the preset scheduling priority and traffic demand;

[0017] Among them, the total bandwidth network slice resources are the sum of dedicated bandwidth resources and public network slice resources, and the formula is as follows:

[0018]

[0019] Among them, w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; represents a set of network slices, w c，k Represents the resources allocated to the public network slice c in the network slice window k.

[0020] Optionally, the network slicing framework for setting a soft and hard hybrid strategy further includes:

[0021] For the network slice isolation, it is the isolation based on resource occupancy; for the network slice m, the isolation of the network slice m in a network slice window k is defined as follows:

[0022]

[0023] Among them, m，k represents the isolation degree of the network slice m in a network slice window k; w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; w c，m，k Indicates that network slice m is derived from the public network slice resource w c，k The amount of resources occupied in c，k Represents the resources allocated to the public network slice c in the network slice window k.

[0024] Optionally, the expression for maximizing spectrum efficiency based on guaranteeing the network slice SLA within the network slice window based on the network slice framework is:

[0025]

[0026] QUR m，k =f(d m，k , w m，k , w c，m，k );

[0027]

[0028]

[0029]

[0030] Among them, Q m，k represents the SLA satisfaction rate of each network slice m in the network slice window k; S k represents the spectrum efficiency in the network slice window k; d m，krepresents the traffic demand of network slice m in network slice window k; f(.) represents the SLA satisfaction rate Q m，k The complex mathematical relationship between traffic demand and network slice resource allocation and the internal scheduling algorithm of network slices; m represents the utility factor of network slice m; β represents the utility factor corresponding to spectrum efficiency; Represents an indicator function, which is used to indicate whether the network slice SLA satisfaction rate meets the requirements; represents the isolation required by network slice m; w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; represents a set of network slices, w c，k represents the resources allocated to the public network slice c in the network slice window k; W represents the network slice resources with total bandwidth; o m，k represents the isolation degree of the network slice m in a network slice window k; w c，m，k Indicates that network slice m is derived from public network slice resource w c，k The amount of resources occupied in n，t It represents the data packet rate of user n at time t.

[0031] Optionally, the formula for the rate of the data packets of the user n at time t is expressed as follows:

[0032]

[0033]

[0034] Among them, W n，t represents the allocated bandwidth of user n at the tth TTI; ∈ represents the transmission error probability; Q -1 (.) represents the inverse function of Gaussian Q function; l n，t represents the length of the codeword block in units of γ; n，t represents the signal-to-interference-to-noise ratio of the received signal of user n associated with base station b at time t; C n，t represents the channel dispersion of user n at time t.

[0035] Optionally, the signal to interference plus noise ratio of the received signal of the user n associated with the base station b at time t is expressed in the following formula:

[0036]

[0037] Among them, P b represents the transmission power of base station b; H b，n，t represents the channel gain between user n and base station b at time t; N 0Represents the power spectral density of Gaussian white noise; W b The network slice resource representing the total bandwidth of base station b.

[0038] Optionally, the SLA satisfaction rate includes:

[0039] For throughput, within a network slice window, the SLA satisfaction rate of network slice m is defined as the ratio of the rate achieved by users in the slice to the minimum transmission speed requirement, as follows:

[0040]

[0041] For the delay and reliability indicators, within a network slice window, the SLA satisfaction rate of network slice m is defined as the proportion of data packets successfully transmitted within the specified maximum delay, as follows:

[0042]

[0043] in, represents the user set of the mth network slice; represents the minimum data rate required by network slice m; r n，t represents the rate of data packets of user n at time t; It represents the transmission reliability of user n in the kth network slice window under the maximum transmission delay constraint.

[0044] Optionally, the using the DQN network to perform online network slice resource optimization and adjustment processing includes:

[0045] The DQN network is defined as:

[0046]

[0047] Among them, μ m，k represents the resource utilization of network slice m in network slice window k; w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; represents a set of network slices, o m，k represents the isolation degree of the network slice m in a network slice window k; Q m，k represents the SLA satisfaction rate of each network slice m in the network slice window k;

[0048] The action of a network slice is defined as increasing, maintaining or decreasing the amount of resources allocated to it in the previous network slice window. The object of the network slice action interaction is the public network slice. In the kth network slice window, network slice m returns additional resources to the public network slice, and network slice m+1 requests resources from the public network slice to increase its dedicated resources.

[0049] For a network slice, its action space is defined as: Where 0<a 1 <…<a i , i is an integer;

[0050] Assume that the action of network slice m in the kth network slice window is Then in the next network slice window, the amount of dedicated resources is: w m，k+1 =w m，k +a m，k ;

[0051] After a network slice action definition is given, the definition of the agent action space is the permutation and combination of all network slice actions, which is mathematically expressed as:

[0052]

[0053] Given the system utility, the reward is defined directly in terms of the system utility.

[0054] Optionally, the definition of directly giving rewards based on system utility is as follows:

[0055]

[0056] Where [x] + =max(0,x), ρ represents the penalty factor for not meeting the isolation requirement, ρ>0; S k / S max represents the normalized spectral efficiency, and the exponential reward function is to achieve the SLA satisfaction rate Q m，k When it is close to 1, the network is trained more effectively; Q m，k represents the SLA satisfaction rate of each network slice m in the network slice window k; α m represents the utility factor of network slice m; β represents the utility factor corresponding to spectrum efficiency; o m，k represents the isolation degree of the network slice m in a network slice window k; represents a set of network slices, S k represents the spectrum efficiency in the network slice window k; S max Indicates the maximum spectral efficiency; Represents an indicator function, which is used to indicate whether the network slice SLA satisfaction rate meets the requirements.

[0057] In addition, an embodiment of the present invention further provides a network slice resource optimization device based on deep reinforcement learning, the device comprising:

[0058] Setting module: a network slicing framework for setting soft and hard hybrid strategies;

[0059] Optimization problem module: used to maximize spectrum efficiency and obtain network slicing optimization problems based on the network slicing framework guaranteeing the network slicing SLA within the network slicing window;

[0060] Optimization and adjustment module: used to use the DQN network to perform online network slice resource optimization and adjustment processing based on the network slice optimization problem and under the condition of given initial resource allocation, so as to obtain the network slice resource optimization result.

[0061] In an embodiment of the present invention, resources can be allocated to network slices with arbitrary QoS requirements based on a unified optimization framework; spectrum efficiency can be improved while ensuring SLAs for different slices.

[0062] Other beneficial effects are correspondingly embodied in the specific embodiments in the specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0064] Figure 1 is a flowchart of a network slice resource optimization method based on deep reinforcement learning in an embodiment of the present invention;

[0065] Figure 2 Schematic diagram of the structure of a network slice resource optimization device based on deep reinforcement learning in an embodiment of the present invention;

[0066] Figure 3 This is a network slicing framework diagram of a soft-hard hybrid strategy in an embodiment of the present invention;

[0067] Figure 4 4 is a diagram of the DQN network structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] Embodiment 1

[0070] See also Figure 1 , Figure 1 It is a flow chart of a network slice resource optimization method based on deep reinforcement learning in an embodiment of the present invention.

[0071] like Figure 1 As shown, a network slice resource optimization method based on deep reinforcement learning, the method comprising:

[0072] S11: Network slicing framework for setting soft and hard hybrid strategies;

[0073] In the specific implementation process of the present invention, the network slicing framework for setting the soft and hard hybrid strategy includes:

[0074] Divide the total bandwidth network slice resources into two parts: soft policy and hard policy;

[0075] For the hard policy, dedicated bandwidth resources are allocated to each network slice in the network slice window;

[0076] For the soft policy, public network slice resources are occupied in each network slice according to the preset scheduling priority and traffic demand; among them, the total bandwidth network slice resources are the sum of dedicated bandwidth resources and public network slice resources, and the formula is as follows:

[0077]

[0078] Among them, w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; represents a set of network slices, w c，k Represents the resources allocated to the public network slice c in the network slice window k.

[0079] Furthermore, the network slicing framework for setting the soft and hard hybrid strategy also includes:

[0080] For the network slice isolation, it is the isolation based on resource occupancy; for the network slice m, the isolation of the network slice m in a network slice window k is defined as follows:

[0081]

[0082] Among them, m，k represents the isolation degree of the network slice m in a network slice window k; w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; w c，m，k Indicates that network slice m is derived from public network slice resource w c，k The amount of resources occupied in c，k Represents the resources allocated to the public network slice c in the network slice window k.

[0083] Specifically, a resource slicing framework that mixes soft slicing and hard slicing is proposed in the 5G slicing network, and the DQN network is used to allocate slice resources, so as to achieve the purpose of full-army guarantee slicing SLA and maximize spectrum efficiency on this basis. Below we first receive the slicing framework, wireless communication model and SLA performance indicators involved in the soft and hard hybrid strategy.

[0084] In a pure hard slicing strategy, each network slice can only occupy the resources allocated to it, which can ensure complete isolation between network slices, but reduces spectrum efficiency. In contrast, the soft network slicing method driven by a flexible resource sharing strategy can maximize spectrum efficiency, but the performance isolation between network slices requires complex algorithm design, which is even difficult to guarantee in actual systems. Therefore, a novel slicing framework of a soft-hard hybrid strategy is proposed, which can simultaneously take advantage of the advantages of hard slicing strategy and soft slicing strategy to maximize spectrum efficiency while ensuring slice SLA and certain performance isolation. Specifically, the hard decision part in the hybrid slicing framework ensures a certain degree of isolation between slices; the soft strategy, that is, the introduction of the setting of common slices, is used to ensure SLA and improve spectrum efficiency. In general, the slicing framework of the soft-hard hybrid strategy can be understood from two aspects: common slice setting and periodic slice resource adjustment.

[0085] Public network slice settings: Figure 3 The network slice framework diagram of the soft-hard hybrid strategy shown in the figure; the resources are divided into two parts corresponding to the hard strategy and the soft strategy: the hard strategy corresponds to the dedicated resources allocated to each network slice; the soft strategy corresponds to the resources allocated to the public network slice. For hard network slices, the resources allocated to each network slice can only be occupied by users in the network slice. For soft network slices, that is, public network slices, users of all network slices can occupy the resources of the public network slice according to traffic demand and priority.

[0086] In a network slice resource allocation window, consider the worst case, that is, the maximum amount of resources required by the network slice greatly exceeds its average required amount of resources. For example, when a user moves to the edge of the cell or a new user joins the cell, the RB resources required by the corresponding network slice at this time are greater than the average RB resources required by the entire network slice window. To illustrate with a more specific numerical example: Assume that a network slice window contains 1000 TTIs. In the first 900 TTIs, the number of RBs required for network slice 1 is 30. In the next 100 TTIs, because of user movement, the number of RBs required for network slice 1 increases to 40. In order to ensure the service performance of network slice 1 users within the entire network slice window, the hard slicing strategy is to reserve resources according to the worst case. According to the above example, before the network slice window starts, the hard network slicing strategy allocates 40R base stations to slice 1. Obviously, a pure hard slicing strategy will reduce the spectrum efficiency, because in the first 900 TTIs, C network slice 1 only needs 30RBs, resulting in a waste of 10RB resources, corresponding to a resource ratio of 25%. However, the soft-hard hybrid strategy can maximize spectrum efficiency while satisfying the SLA of slice 1 through reasonable resource allocation, with only a small sacrifice in performance isolation. For example, for the above example, the hybrid strategy allocates 30RB of dedicated resources to slice 1 and 15RB of resources to the public network slice. This resource configuration enables the resources of the hard slice part to achieve SLA guarantees and performance isolation in most TTIs of the network slice window. In addition, when the resources of the hard network slice cannot meet the SLA satisfaction rate of network slice 1, the slice can occupy the RB resources of the public network slice according to the scheduling priority. Figure 3 The right part of the figure shows the optimal resource configuration diagram for the soft-hard hybrid strategy. Under reasonable configuration, the soft-hard hybrid strategy can achieve the guarantee of slice SLA and maximize spectrum efficiency under the constraint of ensuring certain isolation.

[0087] Periodic network slice resource adjustment: e.g. Figure 3 As shown in Figure 1, radio resource allocation is periodically adjusted according to the network slice window to sense and adapt to the dynamic wireless environment. For example, in the initial stage of perception, in order to ensure the SLA of the network slice, more resources can be allocated to the public network slice, and the isolation of the corresponding network slice is low. As the network's perception of the wireless environment increases, the network slice resource configuration will converge to an accurate solution that meets the isolation requirements, while ensuring the SLA and maximizing the spectrum efficiency, corresponding to Figure 3 Slice resource configuration in the rightmost convergence case.

[0088] Consider a classic cellular area network based on Orthogonal Frequency Division Multiple Access (OFDMA) technology. Assume that the area network contains a total of B base stations, which are represented by a set: The user set is represented as The network slice set is represented as Each user can belong to one or more slices according to different services, and the set of users belonging to slice m is defined as Each base station provides wireless traffic services of corresponding quality to users in the corresponding slice according to the service type and SLA requirements of the slice. Radio resources are divided in units of TTI in the time domain, expressed as t∈{1, 2, …}; similarly, they are divided in units of RB in the frequency domain. Assume that the bandwidth of base station b can be divided into W b RBs, considering the average power allocation scheme, the signal-to-interference-noise ratio of the received signal of user n associated with base station b at time t can be expressed as:

[0089]

[0090] Among them, P b represents the transmission power of base station b; H b，n，t represents the channel gain between user n and base station b at time t; N 0 Represents the power spectral density of Gaussian white noise; W b The network slice resource representing the total bandwidth of base station b.

[0091] For classic large data packet traffic, such as enhanced mobile broadband (eMBB) service traffic, the wireless field has been studying the achievable rates of such traffic for a long time, and the wireless rate that user n can obtain from base station b can be directly estimated based on the Shannon formula. For services such as massive machine type communication (mMTC) and ultra-reliable low-latency communication (uRLLC), the data packet services corresponding to these services usually have the characteristics of small data packet size, and the traditional Shannon formula cannot accurately estimate the corresponding achievable rate. Therefore, the achievable rate theory of small-sized data packets is obtained using the theory of finite block length channel coding systems. We give the achievable rates for large-sized and small-sized data packets in wireless channel scenarios as follows:

[0092]

[0093]

[0094] Among them, W n，t represents the allocated bandwidth of user n at the tth TTI; ∈ represents the transmission error probability; Q -1 (.) represents the inverse function of Gaussian Q function; l n，t represents the length of the codeword block in units of γ; n，t represents the signal-to-interference-to-noise ratio of the received signal of user n associated with base station b at time t; C n，t represents the channel dispersion of user n at time t.

[0095] Generally speaking, the classic performance indicators in network slicing SLA include throughput, latency, and transmission reliability. For throughput, it can be directly obtained by accumulating the amount of data successfully transmitted over a period of time. However, for the latency of data packets, it is necessary to clarify the user's data packet queuing and service model.

[0096] In the present invention, the arrival distribution of slice traffic packets is determined by the slice service type, and the network has no prior knowledge of the traffic arrival and traffic distribution of slice users. Assume that each user corresponds to a cache data queue in the base station. When the user's data packet arrives, the data packet is cached in the corresponding buffer and served according to the first-come-first-served (FCFS) queuing strategy. Therefore, the data packet delay consists of two parts, namely the queuing delay and the transmission delay. The former is affected by the scheduling strategy, and the latter is determined by the instantaneous data rate. For example, the delay D of the i-th data packet of user n is n，i It can be calculated by the following formula:

[0097] D n，i =W n，i +δ n，i ;

[0098] Among them, W n，i represents the queuing delay of the i-th data packet of user n; δ n，i represents the transmission delay of the i-th data packet of user n.

[0099] For transmission reliability, from the perspective of network transmission, if the delay of a data packet exceeds the preset maximum delay, the data packet will be discarded. Transmission reliability is the proportion of successfully transmitted data packets, so the transmission reliability of user n in slice m can be expressed as:

[0100]

[0101] in, Represents the maximum transmission delay allowed by network slice m to its users.

[0102] For throughput, within a network slice window, the SLA satisfaction rate of network slice m is defined as the ratio of the rate achieved by users in the slice to the minimum transmission speed requirement, as follows:

[0103]

[0104] For the delay and reliability indicators, within a network slice window, the SLA satisfaction rate of network slice m is defined as the proportion of data packets successfully transmitted within the specified maximum delay, as follows:

[0105]

[0106] in, represents the user set of the mth network slice; represents the minimum data rate required by network slice m; r n，t represents the rate of data packets of user n at time t; It represents the transmission reliability of user n in the kth network slice window under the maximum transmission delay constraint.

[0107] Setting up a network slicing framework with a soft and hard hybrid strategy: The network resources with a total bandwidth of W are divided into two parts: soft and hard. For the hard strategy, a dedicated bandwidth resource w is allocated to each slice m in the network slice window k. m，k ; For the soft policy, the amount of resources allocated to the public slice is w c，k , each network slice can occupy the resources of the public network slice according to the preset scheduling priority and traffic demand. The sum of the dedicated resources and public resources is the total bandwidth, that is, Among them, w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; represents a set of network slices, w c，k Represents the resources allocated to the public network slice c in the network slice window k.

[0108] For network slice isolation, an isolation based on resource usage is defined: for slice m, its isolation in a slice window k is defined as follows:

[0109]

[0110] Among them, m，k represents the isolation degree of the network slice m in a network slice window k; w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; wc，m，k Indicates that network slice m is derived from the public network slice resource w c，k The amount of resources occupied in c，k Represents the resources allocated to the public network slice c in the network slice window k.

[0111] S12: Based on the network slicing framework, on the basis of ensuring the network slicing SLA within the network slicing window, maximizing the spectrum efficiency, and obtaining the network slicing optimization problem;

[0112] In the specific implementation process of the present invention, the expression for maximizing spectrum efficiency based on the network slicing framework guaranteeing the network slicing SLA within the network slicing window is:

[0113]

[0114] QUR m，k =f(d m，k , w m，k , w c，m，k );

[0115]

[0116]

[0117]

[0118] Among them, Q m，k represents the SLA satisfaction rate of each network slice m in the network slice window k; S k represents the spectrum efficiency in the network slice window k; d m，k represents the traffic demand of network slice m in network slice window k; f(.) represents the SLA satisfaction rate Q m，k The complex mathematical relationship between traffic demand and network slice resource allocation and the internal scheduling algorithm of network slices; m represents the utility factor of network slice m; β represents the utility factor corresponding to spectrum efficiency; Represents an indicator function, which is used to indicate whether the network slice SLA satisfaction rate meets the requirements; represents the isolation required by network slice m; w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; represents a set of network slices, w c，k represents the resources allocated to the public network slice c in the network slice window k; W represents the network slice resources with total bandwidth; o m，k represents the isolation degree of the network slice m in a network slice window k; w c，m，kIndicates that network slice m is derived from the public network slice resource w c，k The amount of resources occupied in n，t It represents the data packet rate of user n at time t.

[0119] Furthermore, the formula for the rate of the data packet of the user n at time t is expressed as follows:

[0120]

[0121]

[0122] Among them, W n，t represents the allocated bandwidth of user n at the tth TTI; ∈ represents the transmission error probability; Q -1 (.) represents the inverse function of Gaussian Q function; l n，t represents the length of the codeword block in units of γ; n，t represents the signal-to-interference-to-noise ratio of the received signal of user n associated with base station b at time t; C n，t represents the channel dispersion of user n at time t.

[0123] Furthermore, the signal to interference and noise ratio of the received signal of the user n associated with the base station b at time t is expressed in the following formula:

[0124]

[0125] Among them, P b represents the transmission power of base station b; H b，n，t represents the channel gain between user n and base station b at time t; N 0 Represents the power spectral density of Gaussian white noise; W b The network slice resource representing the total bandwidth of base station b.

[0126] Furthermore, the SLA satisfaction rate includes:

[0127] For throughput, within a network slice window, the SLA satisfaction rate of network slice m is defined as the ratio of the rate achieved by users in the slice to the minimum transmission speed requirement, as follows:

[0128]

[0129] For the delay and reliability indicators, within a network slice window, the SLA satisfaction rate of network slice m is defined as the proportion of data packets successfully transmitted within the specified maximum delay, as follows:

[0130]

[0131] in, represents the user set of the mth network slice; r represents the minimum data rate required by network slice m; n，t represents the rate of data packets of user n at time t; It represents the transmission reliability of user n in the kth network slice window under the maximum transmission delay constraint.

[0132] S13: Based on the network slice optimization problem and given the initial resource allocation, the DQN network is used to perform online network slice resource optimization and adjustment processing to obtain the network slice resource optimization result.

[0133] In the specific implementation process of the present invention, the online network slice resource optimization and adjustment processing using the DQN network includes:

[0134] The DQN network is defined as:

[0135]

[0136] Among them, μ m，k represents the resource utilization of network slice m in network slice window k; w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; represents a set of network slices, o m，k represents the isolation degree of the network slice m in a network slice window k; Q m，k represents the SLA satisfaction rate of each network slice m in the network slice window k; the action in the network slice is defined as increasing, maintaining or decreasing the amount of resources allocated to it in the previous network slice window, and the object of the network slice action interaction is the public network slice. In the kth network slice window, the network slice m returns the additional resources to the public network slice, and the network slice m+1 requests resources from the public network slice to increase its dedicated resources; for a network slice, its action space is defined as: Where 0<a 1 <…<a i , i is an integer; assuming that the action of network slice m in the kth network slice window is Then in the next network slice window, the amount of dedicated resources is: w m，k+1 =w m，k +a m，k ; After giving the action definition of a network slice, the definition of the agent action space is the permutation and combination of all network slice actions, which is mathematically expressed as:

[0137]

[0138] Given the system utility, the reward is defined directly in terms of the system utility.

[0139] Furthermore, the definition of the reward given directly according to the system utility is as follows:

[0140]

[0141] Where [x] + =max(0,x), ρ represents the penalty factor for not meeting the isolation requirement, ρ>0; S k / S max represents the normalized spectral efficiency, and the exponential reward function is to achieve the SLA satisfaction rate Q m，k When it is close to 1, the network is trained more effectively; Q m，k represents the SLA satisfaction rate of each network slice m in the network slice window k; α m represents the utility factor of network slice m; β represents the utility factor corresponding to spectrum efficiency; o m，k represents the isolation degree of the network slice m in a network slice window k; represents a set of network slices, S k represents the spectrum efficiency in the network slice window k; S max Indicates the maximum spectral efficiency; Represents an indicator function, which is used to indicate whether the network slice SLA satisfaction rate meets the requirements.

[0142] According to the state, action, and reward designed by deep reinforcement learning, the first two hidden layers of DQN are fully connected layers consisting of 64 neurons, using the ReLU activation function. This is followed by two ResNet blocks, each containing two fully connected hidden layers, plus a direct path from the input to the output of the ResNet block, such as Figure 4 Shown as dotted box.

[0143] In the embodiments of the present invention, different QoS indicators are unified into the SLA satisfaction rate, and resources can be allocated to network slices with arbitrary QoS requirements based on a unified optimization framework; the system has high utility performance and is close to the optimal algorithm performance, and can improve spectrum efficiency while ensuring the SLA of different slices; the SLA performance of network slices can be guaranteed globally, even during deep reinforcement learning training.

[0144] Embodiment 2

[0145] See also Figure 2 , Figure 2 It is a schematic diagram of the structural composition of a network slice resource optimization device based on deep reinforcement learning in an embodiment of the present invention.

[0146] like Figure 2 As shown, a network slice resource optimization device based on deep reinforcement learning, the device includes:

[0147] Setting module 21: a network slicing framework for setting a soft and hard hybrid strategy;

[0148] Optimization problem module 22: used to maximize spectrum efficiency and obtain network slicing optimization problem based on the network slicing framework guaranteeing the network slicing SLA within the network slicing window;

[0149] Optimization and adjustment module 23: used to use the DQN network to perform online network slice resource optimization and adjustment processing based on the network slice optimization problem and under the condition of given initial resource allocation, so as to obtain the network slice resource optimization result.

[0150] In the specific implementation process of the present invention, the implementation method of the device part of the present invention can refer to the embodiments of the method part, which will not be repeated here.

[0151] In the embodiments of the present invention, different QoS indicators are unified into the SLA satisfaction rate, and resources can be allocated to network slices with arbitrary QoS requirements based on a unified optimization framework; the system has high utility performance and is close to the optimal algorithm performance, and can improve spectrum efficiency while ensuring the SLA of different slices; the SLA performance of network slices can be guaranteed globally, even during deep reinforcement learning training.

[0152] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0153] In addition, the above is a detailed introduction to a network slice resource optimization method and device based on deep reinforcement learning provided by an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A network slice resource optimization method based on deep reinforcement learning, characterized in that: The method comprises: Set up a network slicing framework with a soft and hard hybrid strategy; Based on the network slicing framework, the network slicing SLA is guaranteed within the network slicing window, and the spectrum efficiency is maximized to obtain the network slicing optimization problem; Based on the network slice optimization problem, under the condition of given initial resource allocation, the DQN network is used to perform online network slice resource optimization and adjustment processing to obtain the network slice resource optimization result; The network slicing framework for setting a soft and hard hybrid strategy includes: Divide the total bandwidth network slice resources into two parts: soft policy and hard policy; For the hard policy, dedicated bandwidth resources are allocated to each network slice in the network slice window; For soft policies, public network slice resources are occupied in each network slice according to the preset scheduling priority and traffic demand; Among them, the total bandwidth network slice resources are the sum of dedicated bandwidth resources and public network slice resources, and the formula is as follows: Among them, w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; represents a set of network slices, w c，k represents the common network slice resources allocated to each network slice c in the network slice window k; The expression for maximizing spectrum efficiency based on the network slicing framework guaranteeing the network slicing SLA within the network slicing window is: s.t.Q m,k =f(d m,k ,w m,k ,w c,m，k ); Among them, Q m,k represents the SLA satisfaction rate of each network slice m in the network slice window k; S k represents the spectrum efficiency in the network slice window k; d m，k represents the traffic demand of network slice m in network slice window k; f(.) represents the SLA satisfaction rate Q m,k The complex mathematical relationship between traffic demand and network slice resource allocation and the internal scheduling algorithm of network slices; m represents the utility factor of network slice m; β represents the utility factor corresponding to spectrum efficiency; Represents an indicator function, used to indicate whether the network slice SLA satisfaction rate meets the requirements; represents the isolation required by network slice m; w m,k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; represents a set of network slices, w c,k represents the resources allocated to the public network slice c in the network slice window k; W represents the network slice resources with total bandwidth; o m,k represents the isolation degree of the network slice m in a network slice window k; w c,m，k Indicates that network slice m is derived from public network slice resource w c，k The amount of resources occupied in n，t represents the rate of data packets of user n at time t; The online network slice resource optimization and adjustment processing using the DQN network includes: The DQN network is defined as: Among them, μ m，k represents the resource utilization of network slice m in network slice window k; w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; represents a set of network slices, o m，k represents the isolation degree of the network slice m in a network slice window k; Q m,k represents the SLA satisfaction rate of each network slice m in the network slice window k; The action of a network slice is defined as increasing, maintaining or decreasing the amount of resources allocated to it in the previous network slice window. The object of the network slice action interaction is the public network slice. In the kth network slice window, network slice m returns additional resources to the public network slice, and network slice m+1 requests resources from the public network slice to increase its dedicated resources. For a network slice, its action space is defined as: Where 0 1 <… i , i is an integer;​​ Assume that the action of network slice m in the kth network slice window is Then in the next network slice window, the amount of dedicated resources is: w m,k+1 =w m,k +a m,k ; After a network slice action definition is given, the definition of the agent action space is the permutation and combination of all network slice actions, which is mathematically expressed as: Given the system utility, the reward is defined directly in terms of the system utility.

2. The network slice resource optimization method according to claim 1, characterized in that: The network slicing framework for setting the soft and hard hybrid strategy also includes: For the network slice isolation, it is the isolation based on resource occupancy; for the network slice m, the isolation of the network slice m in a network slice window k is defined as follows: Among them, m，k represents the isolation degree of the network slice m in a network slice window k; w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; w c，m，k Indicates that network slice m is derived from public network slice resource w c，k The amount of resources occupied.

3. The network slice resource optimization method according to claim 1, characterized in that: The formula for the rate of the data packet of user n at time t is as follows: Among them, W n，t represents the allocated bandwidth of user n at the tth TTI; ∈ represents the transmission error probability; Q -1 (.) represents the inverse function of Gaussian Q function; l n，t represents the length of the codeword block in units of γ; n，t represents the signal-to-interference-to-noise ratio of the received signal of user n associated with base station b at time t; C n，t represents the channel dispersion of user n at time t.

4. The network slice resource optimization method according to claim 3, characterized in that: The formula for the signal to interference noise ratio of the received signal of the user n associated with the base station b at time t is expressed as follows: Among them, P b represents the transmission power of base station b; H b，n，t represents the channel gain between user n and base station b at time t; N0 represents the power spectral density of Gaussian white noise; W b The network slice resource representing the total bandwidth of base station b.

5. The network slice resource optimization method according to claim 1, characterized in that: The SLA satisfaction rate includes: For throughput, within a network slice window, the SLA satisfaction rate of network slice m is defined as the ratio of the rate achieved by users in the slice to the minimum transmission speed requirement, as follows: For the delay and reliability indicators, within a network slice window, the SLA satisfaction rate of network slice m is defined as the proportion of data packets successfully transmitted within the specified maximum delay, as follows: in, represents the user set of the mth network slice; represents the minimum data rate required by network slice m; r n，t represents the rate of data packets of user n at time t; It represents the transmission reliability of user n in the kth network slice window under the maximum transmission delay constraint.

6. The network slice resource optimization method according to claim 1, characterized in that: The definition of rewards given directly based on system utility is as follows: Where [x] + =max(0,x), ρ represents the penalty factor for not meeting the isolation requirement, ρ>0; S k / S max represents the normalized spectral efficiency, The exponential reward function is to achieve the SLA satisfaction rate Q m,k When it is close to 1, the network is trained more effectively; Q m,k represents the SLA satisfaction rate of each network slice m in the network slice window k; α m represents the utility factor of network slice m; β represents the utility factor corresponding to spectrum efficiency; o m,k represents the isolation degree of the network slice m in a network slice window k; represents a set of network slices, S k represents the spectrum efficiency in the network slice window k; S max Indicates the maximum spectrum efficiency; Represents an indicator function, which is used to indicate whether the network slice SLA satisfaction rate meets the requirements.

7. A network slice resource optimization device based on deep reinforcement learning, characterized in that: The device comprises: Setting module: a network slicing framework for setting soft and hard hybrid strategies; Optimization problem module: used to maximize spectrum efficiency and obtain network slicing optimization problems based on the network slicing framework guaranteeing the network slicing SLA within the network slicing window; Optimization and adjustment module: used to perform online network slice resource optimization and adjustment processing using the DQN network based on the network slice optimization problem and under the condition of given initial resource allocation, to obtain network slice resource optimization results; The network slicing framework for setting a soft and hard hybrid strategy includes: Divide the total bandwidth network slice resources into two parts: soft policy and hard policy; For the hard policy, dedicated bandwidth resources are allocated to each network slice in the network slice window; For soft policies, public network slice resources are occupied in each network slice according to the preset scheduling priority and traffic demand; Among them, the total bandwidth network slice resources are the sum of dedicated bandwidth resources and public network slice resources, and the formula is as follows: Among them, w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; represents a set of network slices, w c，k represents the common network slice resources allocated to each network slice c in the network slice window k; The expression for maximizing spectrum efficiency based on the network slicing framework guaranteeing the network slicing SLA within the network slicing window is: s.t.Q m,k =f(d m,k ,w m,k ,w c,m,k ); Among them, Q m , k represents the SLA satisfaction rate of each network slice m in the network slice window k; S k represents the spectrum efficiency in the network slice window k; d m，k represents the traffic demand of network slice m in network slice window k; f(.) represents the SLA satisfaction rate Q m，k The complex mathematical relationship between traffic demand and network slice resource allocation and the internal scheduling algorithm of network slices; m represents the utility factor of network slice m; β represents the utility factor corresponding to spectrum efficiency; Represents an indicator function, used to indicate whether the network slice SLA satisfaction rate meets the requirements; represents the isolation required by network slice m; w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; represents a set of network slices, w c,k represents the resources allocated to the public network slice c in the network slice window k; W represents the network slice resources with total bandwidth; o m,k represents the isolation degree of the network slice m in a network slice window k; w c,m，k Indicates that network slice m is derived from public network slice resource w c，k The amount of resources occupied in n，t represents the rate of data packets of user n at time t; The online network slice resource optimization and adjustment processing using the DQN network includes: The DQN network is defined as: Among them, μ m，k represents the resource utilization of network slice m in network slice window k; w m，k represents the dedicated bandwidth resources allocated to each network slice m in the network slice window k; represents a set of network slices, o m，k represents the isolation degree of the network slice m in a network slice window k; Q m,k represents the SLA satisfaction rate of each network slice m in the network slice window k; The action of a network slice is defined as increasing, maintaining or decreasing the amount of resources allocated to it in the previous network slice window. The object of the network slice action interaction is the public network slice. In the kth network slice window, network slice m returns additional resources to the public network slice, and network slice m+1 requests resources from the public network slice to increase its dedicated resources. For a network slice, its action space is defined as: Where 0 1 <… i , i is an integer;​​ Assume that the action of network slice m in the kth network slice window is Then in the next network slice window, the amount of dedicated resources is: w m,k+1 =w m,k +a m,k ; After a network slice action definition is given, the definition of the agent action space is the permutation and combination of all network slice actions, which is mathematically expressed as: Given the system utility, the reward is defined directly in terms of the system utility.

Images