A 5G smart grid slice allocation method based on deep reinforcement learning

Abstract

The present invention discloses a 5G smart grid slice allocation method based on deep reinforcement learning, including: S1, dividing the grid slice type according to the smart grid task requirements, and constructing a slice request; S2, normalizing the resources required for the slice request and the currently allocated resource status; S3, constructing the input tensor O, action space and reward function required for deep reinforcement learning and training; S4, outputting the 5G grid slice resource allocation result for the slice request after the training is completed. The present invention integrates the current use status, idle status and slice resource request of the resource into a unified tensor, and designs the corresponding action space and reward function, which effectively improves the use efficiency and utilization rate of resources and reduces the operating costs of power companies.

Description

Technical Field

[0001] The present invention relates to the technical field of resource allocation of power networks, and in particular to a 5G smart grid slice allocation method based on deep reinforcement learning. Background Art

[0002] Compared with the fourth-generation communication system, 5G has significant advantages in terms of peak and average data transmission rates, terminal connection quantity and density, and latency. With the continuous improvement of 5G coverage, 5G applications are becoming more and more extensive. As a key infrastructure of the new infrastructure, 5G not only serves individual users, but also needs to meet the needs of many vertical industries, such as smart cities, smart manufacturing, and smart grids. However, the services of different industries have different requirements for network bandwidth, latency, connectivity, mobility, security, and reliability. In order to solve the contradiction between differentiated requirements and network construction costs, network slicing has become an ideal choice. Network slicing uses SDN / NFV technology to divide a single physical network into multiple virtual networks. Each slice represents an independent, isolated, and virtualized end-to-end network from the wireless access network to the bearer network and the core network. The resource sharing of network slicing makes resource utilization high, thereby reducing operating costs; the logical isolation of network slicing makes data transmission safe and reliable; network slicing also supports elastic scaling, allowing 5G operators to customize according to user needs to meet different business needs.

[0003] The "5G network slicing method for smart grid" disclosed in the Chinese patent literature has a publication number of CN113395169A and a publication date of 2021.09.14, which includes the following steps: S1: First, the access, transmission, and core network domain slicing enabling system performs network slicing on the network of the external terminal; S2: Then the network slice identifier and the access system map the network slicing instance and the terminal service type, and register the terminal to the correct network slicing instance. The beneficial effects of the present invention are: through the access, transmission, and core network domain slicing enabling system as the basic supporting technology, the network slicing instance of the access, transmission, and core network is realized; the mapping of network slicing instance and terminal service type is realized through the network slicing identifier and the access system, and the terminal is registered to the correct network slicing instance; the end-to-end network slicing orchestration and management are realized through the network slicing end-to-end management system; the network slicing end-to-end SLA guarantee system can collect, analyze and process the network performance indicators of each domain in quasi-real time to ensure that the system performance meets the user's SLA requirements. This technology focuses on ensuring the performance of the power network and user requirements, but the management of resources of different types and states in the power grid is still relatively extensive, and there is room for further improvement in utilization. Summary of the invention

[0004] The present invention aims to overcome the problem in the prior art of lacking integrated management of resources of different types and states in the power grid, resulting in low resource utilization. A 5G smart grid slice allocation method based on deep reinforcement learning is provided, which integrates the current usage status, idle status and slice resource request of resources into a unified tensor, and designs the corresponding action space and reward function, which effectively improves the efficiency and utilization of resource use and reduces the operating costs of power companies.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A 5G smart grid slice allocation method based on deep reinforcement learning, comprising:

[0007] S1. Divide the grid slice types according to the smart grid task requirements and construct a slice request;

[0008] S2. Normalize the resources required by the slice request and the current allocated resource status;

[0009] S3, construct the input tensor O, action space and reward function required for deep reinforcement learning and train it;

[0010] S4. After completing the training, the 5G power grid slice resource allocation result is output for the slice request.

[0011] The present invention utilizes the significant advantages of 5G in peak and average data transmission rates, terminal connection quantity and density, latency, etc. to receive business requests of smart grids; in response to the real-time resource management problem of network slicing, the current usage status, idle status and slice resource requests of resources are innovatively integrated into a unified tensor, and the corresponding action space and reward function are designed. After training the deep reinforcement learning model, the 5G power grid slice resource allocation results are output. The output results aim to maximize the long-term return results, which can effectively improve the efficiency and utilization of power grid resources and reduce the operating costs of power companies.

[0012] Preferably, S3 comprises the following steps:

[0013] S31, constructing an input tensor O according to the normalized current resource status data and the resource data required by the slice request;

[0014] S32, extracting input features of the input tensor O using a convolutional neural network;

[0015] S33. Construct the action space and reward function required for deep reinforcement learning;

[0016] S34. Use the A3C algorithm based on the actor-critic framework for parallel training.

[0017] The slice request processing process of the 5G smart grid in the present invention includes data normalization, construction of the input tensor O, extraction of input features, construction of the action space and reward function, and deep reinforcement learning training, so as to obtain a trained deep reinforcement learning model to process the slice request data and output the resource allocation result that can maximize the long-term reward result; the A3C algorithm is used, which utilizes a multi-threaded method to interact and learn with the environment in multiple threads at the same time, thereby accelerating the convergence speed.

[0018] Preferably, the grid task requirements in S1 include bandwidth, real-time and reliability, and the classification of grid slices includes control services, collection services and mobile services; the smart grid slice construction request includes the structure, configuration and workflow-related information required for slice instantiation.

[0019] The three major application scenarios of 5G in the present invention include eMBB enhanced mobile broadband, which belongs to mobile services, mMTC massive Internet of Things communications, which belong to collection services, and URLLC high reliability and low latency communications, which belong to control services. The service that requires a slice request is determined based on parameters such as bandwidth, real-time performance and reliability of the task, and then a slice request is constructed for resource allocation processing.

[0020] Preferably, the resources in S2 include wireless resources, storage resources and computing resources;

[0021] Normalize the total amount of wireless resources W, the total amount of storage resources S, the total amount of computing resources C, and the minimum allocation unit of each resource to obtain the total amount of wireless, storage, and computing resources W', S', and C' relative to the minimum allocation unit;

[0022] According to the grid slice type, the wireless, storage and computing resources required for the slice request are constructed relative to the minimum allocation unit to form a three-dimensional vector [w′ i ,s′ i ,c′ i ]', where i represents the grid slice type.

[0023] In the present invention, the types of resources are divided into three categories: wireless resources, storage resources and computing resources. Different types of grid slices require different types and quantities of resources. Therefore, each grid slice type corresponds to a vector constructed by the number of resources required by the type. At the same time, normalizing all resource data and minimum allocation units can ensure the unification of the minimum allocation unit, so as to facilitate the allocation calculation of resource data.

[0024] Preferably, the construction process of the input tensor O includes:

[0025] The current wireless, storage and computing resources in the power grid are represented by a 3×l two-dimensional matrix M, where the first dimension represents the three types of resources: wireless, storage and computing, and l is the normalized total length of resources;

[0026] The resources required for the slice request corresponding to the grid slice type i are vector [w′ i ,s′ i ,c′ i ]', and replicate it to form a 3×l two-dimensional matrix Q;

[0027] Merge the channels corresponding to the matrices M and Q to form the input tensor O.

[0028] In the present invention, the input tensor O is synthesized by the current resource matrix M and the slice request resource matrix Q, which contains the current resource status information and the information of the resources required for the grid slice request. Each type of resource request on the matrix Q can be allocated at any position of the matrix M. The shape of the final tensor O is 3×l×2, which is determined by the matrix M and the matrix Q.

[0029] Preferably, the action space is represented by a∈{0, 1}, where action a=1 indicates accepting the slice request and allocating resources, and action a=0 indicates rejecting the slice request and not allocating resources. The action space in the present invention is a discretized space, and there are only two situations: accepting the slice request and rejecting the slice request.

[0030] Preferably, the reward function is R=∑ t γ t r(s t ,a t ), where γ is the return discount factor, r(s t ,a t ) represents the state s at time t t Take action a t The reward obtained; when the action is 1, a reward r is generated, which is determined by the grid slice type i i , when the action is 0, the reward is 0.

[0031] The reward function in the present invention represents the long-term reward. The ultimate training goal is to maximize the long-term reward. When the resource allocation for the slice request is the optimal result, the maximized long-term reward can be obtained. The training process of deep reinforcement learning is to make the final output resource allocation result converge to the optimal result, thereby maximizing the long-term reward and improving resource efficiency and utilization.

[0032] Preferably, the A3C algorithm based on the actor-critic framework is used for parallel training; the actor subnetwork includes a convolutional layer, a fully connected layer, and a softmax layer and outputs a two-dimensional vector, which corresponds to the probability of accepting a slice request action and the probability of rejecting a slice request action respectively; the critic subnetwork includes a convolutional layer and a fully connected layer and outputs a value evaluation scalar value, that is, the value of each state.

[0033] The present invention uses the actor-critic algorithm for parallel training. In the deep reinforcement learning process, the main discussion is about the agent maximizing the rewards that can be obtained after slicing resources in the power grid in a complex and uncertain environment. The actor is used to predict the probability of behavioral actions, and the critic is used to predict the value in the corresponding state. The actor selects behavioral actions based on probability, and the critic estimates the value of each state, that is, the value evaluation scalar value, and subtracts the value of the next state from the value of the current state. If the obtained value is positive, the actor's next action will increase the update amplitude, otherwise it will reduce the update amplitude.

[0034] The present invention has the following beneficial effects: the current usage status, idle status and slice resource request of resources are integrated into a unified tensor, and the corresponding action space and reward function are designed, which effectively improves the efficiency and utilization of resource use and reduces the operating costs of power companies; the A3C algorithm is used, and by using a multi-threaded method, multiple threads interact with the environment respectively at the same time, each thread summarizes and saves the learning results in a common location, and then regularly exports the summarized results from the common location to guide subsequent environmental learning interactions, thereby accelerating the convergence of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the smart grid slice allocation method of the present invention;

[0036] Figure 2 Schematic diagram of three types of services of the smart grid of the present invention;

[0037] Figure 3 It is a schematic diagram of deep reinforcement learning for 5G smart grid slice allocation in the present invention. DETAILED DESCRIPTION

[0038] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0039] like Figure 1 As shown, a 5G smart grid slice allocation method based on deep reinforcement learning includes:

[0040] S1. Divide the grid slice types according to the smart grid task requirements and construct a slice request; the grid task requirements in S1 include bandwidth, real-time and reliability, and the classification of grid slices includes control services, collection services and mobile services; the smart grid constructs a slice request containing the structure, configuration and workflow-related information required for slice instantiation.

[0041] S2. Normalize the resources required for the slice request and the current allocated resource status; the resources in S2 include wireless resources, storage resources and computing resources; the total amount of wireless resources W, the total amount of storage resources S, the total amount of computing resources C and the minimum allocation unit of each resource are all normalized.

[0042] S3, construct the input tensor O, action space and reward function required for deep reinforcement learning and train it; S3 includes the following steps:

[0043] S31, constructing an input tensor O according to the normalized current resource status data and the resource data required by the slice request;

[0044] S32, extracting input features of the input tensor O using a convolutional neural network;

[0045] S33. Construct the action space and reward function required for deep reinforcement learning;

[0046] S34. Use the A3C algorithm based on the actor-critic framework for parallel training.

[0047] S4. After completing the training, the 5G power grid slice resource allocation result is output for the slice request.

[0048] The A3C algorithm based on the actor-critic framework is used for parallel training; the actor subnetwork includes convolutional layers, fully connected layers, and softmax layers and outputs a two-dimensional vector, which corresponds to the probability of accepting a slice request action and the probability of rejecting a slice request action respectively; the critic subnetwork includes convolutional layers and fully connected layers and outputs a scalar value of value evaluation, that is, the value of each state.

[0049] The present invention utilizes the significant advantages of 5G in peak and average data transmission rates, terminal connection quantity and density, latency, etc. to receive business requests of smart grids; in response to the real-time resource management problem of network slicing, the current usage status, idle status and slice resource requests of resources are innovatively integrated into a unified tensor, and the corresponding action space and reward function are designed. After training the deep reinforcement learning model, the 5G power grid slice resource allocation results are output. The output results aim to maximize the long-term return results, which can effectively improve the efficiency and utilization of power grid resources and reduce the operating costs of power companies.

[0050] The slice request processing process of the 5G smart grid in the present invention includes data normalization, construction of the input tensor O, extraction of input features, construction of the action space and reward function, and deep reinforcement learning training, so as to obtain a trained deep reinforcement learning model to process the slice request data and output the resource allocation result that can maximize the long-term reward result; the A3C algorithm is used, which utilizes a multi-threaded method to interact and learn with the environment in multiple threads at the same time, thereby accelerating the convergence speed.

[0051] The three major application scenarios of 5G in the present invention include eMBB enhanced mobile broadband, which belongs to mobile services, mMTC massive Internet of Things communications, which belong to collection services, and URLLC high reliability and low latency communications, which belong to control services. The service that requires a slice request is determined based on parameters such as bandwidth, real-time performance and reliability of the task, and then a slice request is constructed for resource allocation processing.

[0052] In the present invention, the types of resources are divided into three categories: wireless resources, storage resources and computing resources. Different types of grid slices require different types and quantities of resources. Therefore, each grid slice type corresponds to a vector constructed by the number of resources required by the type. At the same time, normalizing all resource data and minimum allocation units can ensure the unification of the minimum allocation unit, so as to facilitate the allocation calculation of resource data.

[0053] In the present invention, the input tensor O is synthesized by the current resource matrix M and the slice request resource matrix Q, which contains the current resource status information and the information of the resources required for the grid slice request. Each type of resource request on the matrix Q can be allocated at any position of the matrix M. The shape of the final tensor O is 3×l×2, which is determined by the matrix M and the matrix Q.

[0054] The action space in the present invention is a discretized space, and there are only two situations: accepting slice requests and rejecting slice requests. The reward function in the present invention represents the long-term return. The ultimate training goal is to maximize the long-term return. When the resource allocation for the slice request is the optimal result, the maximized long-term return can be obtained. The training process of deep reinforcement learning is to make the final output resource allocation result converge to the optimal result, thereby maximizing the long-term return and improving resource efficiency and utilization.

[0055] The present invention uses the actor-critic algorithm for parallel training. In the deep reinforcement learning process, the main discussion is about the agent maximizing the rewards that can be obtained after slicing resources in the power grid in a complex and uncertain environment. The actor is used to predict the probability of behavioral actions, and the critic is used to predict the value in the corresponding state. The actor selects behavioral actions based on probability, and the critic estimates the value of each state, that is, the value evaluation scalar value, and subtracts the value of the next state from the value of the current state. If the obtained value is positive, the actor's next action will increase the update amplitude, otherwise it will reduce the update amplitude.

[0056] In an embodiment of the present invention, the 5G smart grid slice allocation method specifically includes the following steps.

[0057] Step 1: Set the constraints required for smart grid slice allocation;

[0058] Step 1.1: Set up the resources in the smart grid system to be wireless, storage, and computing resources, with the total amount of each resource being W, S, and C respectively;

[0059] Step 1.2: The set of grid slice types is {1, 2, ..., I}. For smart grids, the typical value of I is 3, corresponding to control services, collection services, and mobile services, respectively. The slice type i∈{1, 2, ..., I}, where the wireless, storage, and computing resources to be allocated constitute a three-dimensional vector, expressed as [w i ,s i ,c i ]';

[0060] Step 1.3: The vector of the number of each grid slice type at time t is [n t,1 ,n t,2 ,......,n t,I ], then the constraints that need to be satisfied are:

[0061] Step 2: The smart grid determines which type of grid slice the service needs to request belongs to, uRLLC, mMTC, or eMBB, based on the service's requirements for bandwidth, real-time performance, reliability, and other parameters. The smart grid then constructs a slice request, which contains the structure, configuration, and workflow information required for slice instantiation. eMBB is enhanced mobile broadband, which belongs to the mobile category; mMTC is massive IoT communication, which belongs to the collection category; and URLLC is high-reliability, low-latency communication, which belongs to the control category; Figure 2 Shown is a schematic diagram of three types of business.

[0062] Step 3: Normalize the unused and used states of three different grid slice resources: wireless, storage, and computing;

[0063] Step 3.1: Normalize the total amount of wireless, storage and computing resources and the minimum allocation unit of each resource, normalize the minimum allocation unit of the three resources to 1, and obtain the total amount of wireless, storage and computing resources relative to the minimum allocation unit W', S', C';

[0064] Step 3.2: Take l = max(W', S', C') as the uniform total resource length, and scale the minimum allocation unit of each resource according to the ratio of the total resource length l to the total resource W', S', C'. The corresponding grid slice type i∈{1,2,...,I}, the three-dimensional vector consisting of the wireless, storage and computing resources to be allocated in the slice request is represented as [w′ i ,s′ i ,c′ i ]'.

[0065] Step 4: Design the input tensor O for training;

[0066] Step 4.1: After normalization, the wireless, storage and computing resources in the power grid are represented by a two-dimensional matrix M, which has a shape of 3×l, where the first dimension represents the three types of resources: wireless, storage and computing, and l is the normalized total length of the resources;

[0067] Step 4.2: The resource requirement corresponding to the grid slice type i making the slice request is [w′ i ,s′ i ,c′ i ]′, for the vector [w′ i ,s′ i ,c′ i ]' is copied and expanded to form a 3×l two-dimensional matrix Q, in which the element values ​​of each row are the same, namely w′ i ,s′ i ,c′ i ; Thus, each type of resource request can be allocated at any position on the matrix M;

[0068] Step 4.3: Merge the channels corresponding to the matrices M and Q to form a tensor O with a shape of 3×l×2.

[0069] Step 5: Figure 3As shown in the figure, a convolutional neural network designed to extract input features uses 5 convolutional layers to extract input features from the input tensor O. The size of the convolution kernel of the 5 convolutional layers is 3×3, the weight parameter of the first layer is 64×2×3×3, and the offset parameter is 64, where 64 is the number of defined feature maps, 2 represents two channels, 3×3 is the size of the convolution kernel, and the size of the offset parameter is consistent with the number of feature maps; the weight parameters and offset parameters of the second to fifth layers are the same, which are 64×64×3×3 and 64 respectively, where the stride of the convolution kernel for each calculation is 1 and the padding is 1.

[0070] Step 6: Design the action space and reward function required for deep reinforcement learning training;

[0071] Step 6.1: Design the response of the intelligent agent to accept the slice request and allocate resources, in which case the action a is 1; or reject the slice request, in which case the action a is 0, thus forming a discretized space, so the action space is represented as a∈{0,1}; the operating state of the power grid intelligent terminal, the slice request arrival state, and the current slice resource allocation state constitute the current environment, and the intelligent agent is used to interact with the environment, and respond to the terminal's slice resource allocation request by observing the current environment and the interaction history, so as to obtain long-term rewards;

[0072] Step 6.2: After taking an action, the agent interacts with the environment to generate rewards and states, and the agent uses this state to output actions and decisions. Under the determined state s, the smart grid slice request arriving is q i , i∈{1,2,......,I}, if the agent decides to accept the slice request through the policy function, it will allocate the required resources [w′] for the slice request from the total amount of wireless, storage and computing resources W,S,C i ,s′ i ,c′ i ]', and will receive a reward r determined by slice type i i ; If the decision is to reject the slice request, the reward obtained by the system is 0, that is:

[0073]

[0074] r(s t ,a t ) represents the state s at time t t Take action a t The reward obtained is a temporary reward. Since the ultimate goal of resource slice allocation is to maximize the long-term reward, let γ be the reward discount factor. The reward function corresponding to the long-term reward is R = ∑ t γ t r(st ,a t ); The strategy function is the judgment of what action the agent will take in a given state. According to the distribution of the arriving slice requests, it can accept the slice request when the long-term reward is maximized, otherwise it will be rejected.

[0075] Step 7: Figure 3 As shown in the figure, the A3C algorithm based on the actor-critic framework is used for parallel training, wherein, for the actor sub-network, one convolution layer is used, the size of the convolution kernel is 1×1, the number of feature maps is 8, the stride is 1, and its weight parameter is 8×64×1×1; after the convolution layer, there are two fully connected layers, the weight parameters of the two fully connected layers are 8×3l×256 and 256×2, respectively, and the offset parameters are 256 and 2, respectively. Then, a softmax layer is used to output a two-dimensional vector, corresponding to the probability of accepting a slice request and the probability of rejecting a slice request, respectively; for the critic sub-network, one convolution layer is used, the size of the convolution kernel is 1×1, the number of feature maps is 4, the stride is 1, and its weight parameter is 4×64×1×1. After the convolution layer, there are two fully connected layers, the weight parameters of the two fully connected layers are 4×3l×256 and 256, respectively, and the offset parameters are both 256. The final output is a scalar value for value assessment. The multi-threaded method of the A3C algorithm is used for training, and interactive learning with the environment is performed in multiple threads at the same time to speed up the convergence.

[0076] Step 8: After the training is completed, the 5G grid slice resource allocation result is output for the slice request; for a slice resource allocation result, if its corresponding long-term return is the largest, it means that the allocation strategy is the optimal strategy, otherwise it is not. Since the training of deep reinforcement learning is an exploration process, by exploring new strategies, the output resource allocation action converges to the optimal strategy, thereby maximizing the long-term return.

[0077] In this embodiment, there are three types of slices, corresponding to the control services, collection services and mobile application services of the smart grid, respectively, denoted as s-control, s-collect, and s-mobile. The arrival rates of the three slice resource requests follow the Poisson distribution, which are 40 requests / hour, 20 requests / hour, and 30 requests / hour respectively. The resource usage time of the three slices follows the exponential distribution, and its mean is 2, that is, an average of 2 slices can be completed per hour. The wireless resources required by the slice s-control are 100Mbps, the storage resources are 100GB, and the computing resources are 2CPUs; the wireless resources required by the slice s-collect are 200Mbps, the storage resources are 100GB, and the computing resources are 1CPUs; the wireless resources required by the slice s-mobile are 400Mbps, the storage resources are 200GB, and the computing resources are 2CPUs. Consider that the total resources W, S, and C of the power grid are 3Gbps, 3TB, and 60CPUs respectively.

[0078] In order to verify the effectiveness of the method proposed in the present invention in solving the long-term return maximization goal, the rewards for accepting slice requests for the two slice types s-control and s-collect are fixed to 1, and then the slice rewards for the s-mobile type are increased successively. Under each s-mobile slice reward value, the average number of slices of different service types that can run simultaneously is observed. In addition, in order to verify the impact of different algorithms on the results, the Greedy algorithm, Deep Dueling algorithm and Q-Learning algorithm are used as controls, and the experimental results are shown in the following table. The results in the table show the number of slices for the three types of services running under different algorithms when the slice reward value of s-mobile is 6.

[0079]

[0080] It can be seen from the above table that when the slice reward of s-mobile is 6, the number of s-mobile slices that can run simultaneously reaches 6.19, while the number of slices of the other two types is about 1.50. This shows that the A3C algorithm adopted by the present invention is more sensitive to changes in returns and rewards than the other three algorithms. Experiments show that the resource utilization rate of the A3C algorithm adopted by the present invention as a slice resource allocation algorithm reaches about 93%, which is significantly improved compared with the first three methods.

[0081] In addition, in terms of average reward, it is defined as a period of time when many slices apply for resource allocation. Some of these slices apply successfully, while others fail. Rewards are generated, and then these rewards are averaged to obtain the average reward. When the slice reward of s-mobile is 6, the average reward of the Greedy method is about 0.62, and the average reward of Q-Learning is about 0.70. Deep Dueling achieves better results than the Q-Learning method, and its average reward value reaches about 0.80. The A3C algorithm used in the present invention achieves the best results, and its average reward value reaches 0.89. The above experimental results show that compared with the Greedy, Q-Learning and Deep Dueling slice resource allocation algorithms, the A3C algorithm used in the present invention can obtain better long-term returns, and can effectively help power companies improve resource utilization efficiency and utilization, and reduce operating costs.

[0082] The above embodiments are further elaborations and illustrations of the present invention for ease of understanding, and are not limitations of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A 5G smart grid slice allocation method based on deep reinforcement learning, characterized in that: include: S1. Divide the grid slice types according to the smart grid task requirements and construct a slice request; S2. Normalize the resources required by the slice request and the current allocated resource status; S3, construct the input tensor O, action space and reward function required for deep reinforcement learning and train it; S31, constructing an input tensor O according to the normalized current resource status data and the resource data required by the slice request; S32, extracting input features of the input tensor O using a convolutional neural network; S33. Construct the action space and reward function required for deep reinforcement learning; S34, use the A3C algorithm based on the actor-critic framework for parallel training; S4. After completing the training, the 5G power grid slice resource allocation result is output for the slice request.

2. According to a 5G smart grid slice allocation method based on deep reinforcement learning according to claim 1, it is characterized in that: The grid task requirements in S1 include bandwidth, real-time and reliability. The classification of grid slices includes control services, collection services and mobile services. The smart grid construction slice request contains the structure, configuration and workflow-related information required for slice instantiation.

3. According to a 5G smart grid slice allocation method based on deep reinforcement learning according to claim 1, it is characterized in that: The resources in S2 include wireless resources, storage resources and computing resources; Normalize the total amount of wireless resources W, the total amount of storage resources S, the total amount of computing resources C, and the minimum allocation unit of each resource to obtain the total amount of wireless, storage, and computing resources W', S', and C' relative to the minimum allocation unit; According to the grid slice type, the wireless, storage and computing resources required for the slice request are constructed relative to the minimum allocation unit to form a three-dimensional vector [w i ',s i ',c i ']', where i represents the grid slice type.

4. A 5G smart grid slice allocation method based on deep reinforcement learning according to claim 3, characterized in that: The construction process of the input tensor O includes: The current wireless, storage and computing resources in the power grid are represented by a 3×l two-dimensional matrix M, where the first dimension represents the three types of resources: wireless, storage and computing, and l is the normalized total length of resources; The resources required for the slice request corresponding to the grid slice type i are vector [w i ',s i ',c i ']', replicate it to form a 3×l two-dimensional matrix Q; Merge the channels corresponding to the matrices M and Q to form the input tensor O.

5. A 5G smart grid slice allocation method based on deep reinforcement learning according to claim 1 or 4, characterized in that: The action space is represented as a∈{0, 1}, where action a=1 indicates that the slice request is accepted and resources are allocated, and action a=0 indicates that the slice request is rejected and no resources are allocated.

6. A 5G smart grid slice allocation method based on deep reinforcement learning according to claim 5, characterized in that: The reward function is R = ∑ t γ t r(s t ,a t ), where γ is the return discount factor, r(s t ,a t ) represents the state s at time t t Take action a t The reward obtained; when the action is 1, a reward r is generated, which is determined by the grid slice type i i , when the action is 0, the reward is 0.

7. A 5G smart grid slice allocation method based on deep reinforcement learning according to claim 1, 4 or 6, characterized in that: The A3C algorithm based on the actor-critic framework is used for parallel training; the actor subnetwork includes convolutional layers, fully connected layers, and softmax layers and outputs a two-dimensional vector, which corresponds to the probability of accepting a slice request action and the probability of rejecting a slice request action respectively; the critic subnetwork includes convolutional layers and fully connected layers and outputs a scalar value of value evaluation, that is, the value of each state.

Images