Resource management method and system based on 5G power virtual private network

Abstract

The invention discloses a resource management method and system based on a 5G electric power virtual private network, and the method comprises the steps: building a 5G electric power virtual private network-slice system which comprises a plurality of base stations, a plurality of electric power terminal users and a plurality of slices, building an objective function to maximize the system benefit, and building a constraint condition; splitting the target function into an upper-layer target function and a lower-layer target function, and allocating bandwidth resources for the slices in a long time domain through a base station according to resource requirements based on the upper-layer target function by adopting a DDQN algorithm, based on a lower-layer objective function, multi-agent priority experience playback is combined with an action actor commentator algorithm to allocate resource blocks and transmitting power for power terminal users in a short-time domain by utilizing slices, and adaptive hierarchical resource management is adopted, so that the network structure is more stable, a better overall resource allocation strategy is made, and the overall resource allocation efficiency is improved. According to the method, fine granularity can be considered and distributed, and dynamic resource scheduling is realized, so that the service quality requirement of a user is met, and the service performance is ensured to be stable.

Description

Technical Field

[0001] The present invention relates to the field of power virtual private networks, and in particular to a resource management method and system based on a 5G power virtual private network. Background Art

[0002] 5G power virtual private network refers to the use of software-defined network technology in 5G networks to divide physical facilities into multiple logically isolated soft slices. In addition, hard slices can be formed through physical isolation, thereby virtualizing a dedicated network for the power industry in the wireless access network, transmission network and core network. With the development of new services such as enhanced mobile broadband, low-latency and high-reliability communications, and massive machine-type communications, power virtual private optical network communication technology has gradually failed to meet the increasingly differentiated business needs of emerging services. On this basis, it is urgent to explore new communication technologies to meet the complex and diverse business needs of new power grids. 5G power virtual private network includes two types of virtual private networks, namely local virtual private network and wide area virtual private network. Among them, local virtual private network is mainly for local tasks in closed scenarios, and the needs of power business are realized through 5G power virtual private network, base stations, and mobile edge computing. Wide area virtual private network is for tasks in open scenarios, and end-to-end differentiated services are usually realized through power slice private network.

[0003] Network slicing is one of the key technologies of 5G. It is an on-demand networking method that separates multiple virtual end-to-end networks on a unified infrastructure. Each network slice is logically isolated on the wireless access network, bearer network and core network to adapt to various types of applications. With the expansion of the power system and the introduction of new applications, different applications have different requirements for the network. Network slicing can provide dedicated network resources for each application, ensure its service quality and performance, provide ultra-low latency communication services for tasks, and ensure timely response and operation.

[0004] However, existing work only allocates resources based on real-time network information to ensure the QoS (Quality of Service) requirements of the service, ignoring the impact of the dynamic nature of the network on service performance. Simply allocating resources in real time based on the current service requests cannot guarantee stable service performance within a certain time range. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a resource management method and system based on a 5G power virtual private network, which can ensure stable service performance while meeting the service quality requirements of users.

[0006] In order to solve the above technical problems, a technical solution adopted by the present invention is:

[0007] A resource management method based on a 5G power virtual private network, comprising the steps of:

[0008] Establishing a 5G power virtual private network-slice system, wherein the 5G power virtual private network-slice system includes multiple base stations, multiple power terminal users and multiple slices;

[0009] Establishing an objective function to maximize system benefits and establishing constraints of the objective function;

[0010] Splitting the objective function into an upper objective function and a lower objective function;

[0011] Based on the upper-level objective function, the base station uses a dual deep Q network algorithm in the long time domain to allocate bandwidth resources to the slice according to resource requirements to obtain an overall resource allocation strategy, and based on the lower-level objective function, the slice is used in the short time domain to allocate resource blocks and transmission power to the power terminal user using a multi-agent priority experience replay combined with an actor-critic algorithm to obtain a fine-grained resource partitioning strategy.

[0012] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0013] A resource management system based on a 5G power virtual private network includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0014] Establishing a 5G power virtual private network-slice system, wherein the 5G power virtual private network-slice system includes multiple base stations, multiple power terminal users and multiple slices;

[0015] Establishing an objective function to maximize system benefits and establishing constraints of the objective function;

[0016] Splitting the objective function into an upper objective function and a lower objective function;

[0017] Based on the upper-level objective function, the base station uses a dual deep Q network algorithm in the long time domain to allocate bandwidth resources to the slice according to resource requirements to obtain an overall resource allocation strategy, and based on the lower-level objective function, the slice is used in the short time domain to allocate resource blocks and transmission power to the power terminal user using a multi-agent priority experience replay combined with an actor-critic algorithm to obtain a fine-grained resource partitioning strategy.

[0018] The beneficial effects of the present invention are as follows: a 5G power virtual private network-slice system including multiple base stations, multiple power terminal users and multiple slices is established, an objective function is established to maximize the system benefit, and constraints of the objective function are established, the objective function is split into an upper objective function and a lower objective function, based on the upper objective function, the base station uses a dual deep Q network algorithm in the long time domain to allocate bandwidth resources to the slice according to resource requirements, and an overall resource allocation strategy is obtained, and based on the lower objective function, the slice is used in the short time domain to allocate resource blocks and transmission power to the power terminal users by using multi-agent priority experience playback combined with an actor critic algorithm, and a detailed resource allocation strategy is obtained. Granular resource partitioning strategy. The present invention adopts adaptive hierarchical resource management. In the upper-level strategy, the DDQN algorithm uses the dual-network structure formed by the target network and the training network to solve the problem that the DQN estimated reward is higher than the actual reward, and makes the network structure more stable, making a better overall resource allocation strategy. In the lower-level strategy, since it involves both discrete and continuous actions, the multi-agent priority experience playback combined with the action actor critic algorithm is used to solve the problem of both discrete and continuous actions. It can perform fine-grained consideration and allocation, and realize dynamic scheduling of resources, thereby ensuring stable service performance while meeting the user's service quality requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flowchart of a resource management method based on a 5G power virtual private network according to an embodiment of the present invention;

[0020] Figure 2 This is a structural diagram of a resource management system based on a 5G power virtual private network according to an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of a 5G power virtual private network-slicing system in a resource management method based on a 5G power virtual private network according to an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of the dual deep Q network algorithm and multi-agent priority experience playback combined with the action actor critic algorithm in the resource management method based on the 5G power virtual private network in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0024] Please refer to Figure 1 , a resource management method based on 5G power virtual private network, comprising the steps of:

[0025] Establishing a 5G power virtual private network-slice system, wherein the 5G power virtual private network-slice system includes multiple base stations, multiple power terminal users and multiple slices;

[0026] Establishing an objective function to maximize system benefits and establishing constraints of the objective function;

[0027] Splitting the objective function into an upper objective function and a lower objective function;

[0028] Based on the upper-level objective function, the base station uses a dual deep Q network algorithm in the long time domain to allocate bandwidth resources to the slice according to resource requirements to obtain an overall resource allocation strategy, and based on the lower-level objective function, the slice is used in the short time domain to allocate resource blocks and transmission power to the power terminal user using a multi-agent priority experience replay combined with an actor-critic algorithm to obtain a fine-grained resource partitioning strategy.

[0029] From the above description, it can be seen that the beneficial effects of the present invention are: establishing a 5G power virtual private network-slice system including multiple base stations, multiple power terminal users and multiple slices, establishing an objective function to maximize the system benefit, and establishing the constraint conditions of the objective function, splitting the objective function into an upper-level objective function and a lower-level objective function, and based on the upper-level objective function, using the base station to allocate bandwidth resources to the slices in the long time domain according to resource requirements using a dual deep Q network algorithm to obtain an overall resource allocation strategy, and based on the lower-level objective function, using the slices in the short time domain using a multi-agent priority experience playback combined with an actor critic algorithm to allocate resource blocks and transmission power to the power terminal users. , and obtain a fine-grained resource partitioning strategy. The present invention adopts adaptive hierarchical resource management. In the upper-level strategy, the DDQN algorithm uses the dual-network structure formed by the target network and the training network to solve the problem that the DQN estimated reward is higher than the actual reward, and makes the network structure more stable, making a better overall resource allocation strategy. In the lower-level strategy, since it involves both discrete and continuous actions, the multi-agent priority experience replay combined with the action actor critic algorithm is used to solve the problem of both discrete and continuous actions. It can perform fine-grained consideration and allocation, and realize dynamic scheduling of resources, thereby ensuring stable service performance while meeting the user's service quality requirements.

[0030] Furthermore, establishing an objective function to maximize system benefits includes:

[0031]

[0032] In the formula, represents the long-term bandwidth reservation set, represents the short-time domain resource block allocation set, represents the user power allocation set, T represents the number of long-term domains, and U lRepresents the system benefit in the lth long time domain.

[0033] From the above description, it can be seen that optimizing resource allocation with the goal of maximizing system benefits can reduce system costs while providing the best service performance.

[0034] Furthermore, the constraint conditions for establishing the objective function include:

[0035] Establish resource block allocation constraints, transmission queue constraints, allocation constraints of each resource block within a single time slot, data packet transmission constraints, constraints on the sum of slice occupied frequency band resources, constraints on the sum of power terminal user bandwidths, service quality requirement constraints, power terminal user transmission rate constraints, and power terminal user transmission power constraints.

[0036] From the above description, it can be seen that establishing resource block allocation constraints, transmission queue constraints, allocation constraints of each resource block in a single time slot, data packet transmission constraints, constraints on the sum of slice occupied frequency band resources, constraints on the sum of power terminal user bandwidths, service quality requirement constraints, power terminal user transmission rate constraints, and power terminal user transmission power constraints ensures the feasibility of the final resource management strategy.

[0037] Furthermore, the upper layer objective function is specifically:

[0038]

[0039] In the formula, represents the user power allocation set after converting continuous variables into discrete variables, ω2 represents the payment resource cost function weight, ω3 represents the overload penalty weight, It represents the cost function of resources that the slice needs to pay in the lth long time domain, represents the overload penalty for slice n in the lth long time domain, Represents a collection of slices;

[0040] The lower layer objective function is specifically:

[0041]

[0042] In the formula, ω1 represents the total system revenue weight, ΔT represents the multiple relationship of the time span, and REtotal(t) represents the total system revenue.

[0043] From the above description, it can be seen that due to the coupling between different levels, the objective function is split into two-level sub-problems, namely the upper-level objective function and the lower-level objective function, so as to meet the dynamic traffic demand and the differentiated service needs of power terminal users, ensure the stability of service performance and meet QoS.

[0044] Further, the base station allocates bandwidth resources to the slices in a long-term domain according to resource requirements based on the upper-layer objective function by using a dual-depth Q network algorithm to obtain an overall resource allocation strategy, and the slices are used in a short-term domain based on the lower-layer objective function to allocate resource blocks and transmission power to the power terminal users by using a multi-agent priority experience playback combined with an actor-critic algorithm to obtain a fine-grained resource partitioning strategy, including:

[0045] Determine a state set, an action set, and a reward function based on the upper layer objective function and the lower layer objective function;

[0046] Define the loss function;

[0047] The Adam algorithm is used to minimize the gradient of the loss function, and the Q network parameters of the current Q network are updated based on the state set, action set and reward function to obtain the target Q network;

[0048] Allocating bandwidth resources to the slice in a long time domain according to resource requirements through the base station using the target Q network to obtain an overall resource allocation strategy;

[0049] Taking each of the base stations as an intelligent agent, and determining a discrete action of the intelligent agent that maximizes the Q function value in the current state;

[0050] For training the actor-home network, the agent combines the state and the overall resource allocation strategy, uses the state and the overall resource allocation strategy as inputs to the actor-home network, and the actor-home network outputs a selection of power and observation frequency based on the inputs;

[0051] The agent combines the state, the selected continuous actions and the made fine-grained resource partitioning strategy as input of the critic network, and the critic network outputs an estimated Q-value function of the fine-grained resource partitioning strategy according to the input, and determines the optimal fine-grained resource partitioning strategy according to the obtained estimated Q-value function;

[0052] The agent takes corresponding actions according to the maximized expected reward, and updates the actor network according to the policy gradient and the priority experience replay mechanism;

[0053] For the training of the critic network, the error between the target Q value and the actual Q value of the critic network is minimized, and the parameters of the critic network are updated according to the priority experience replay mechanism by the gradient descent method;

[0054] The actor network after training is used to allocate resource blocks and transmission power to the power terminal user in a short time domain using the slices, thereby obtaining a fine-grained resource partitioning strategy.

[0055] From the above description, it can be seen that the DDQN algorithm utilizes the dual-network structure formed by the target network and the training network, solves the problem that the DQN estimated reward is higher than the actual reward, and makes the network structure more stable. However, the single DDQN algorithm is not suitable for resource scheduling in the short time domain, so the multi-agent priority experience replay combined with the actor-critic algorithm is used in the lower layer to dynamically schedule resources, support diversified power services with different QoS requirements and ensure fine-grained division of resources, meet dynamic traffic requirements and differentiated service requirements of power terminal users.

[0056] Further, the determining of the state set, action set and reward function based on the upper layer objective function and the lower layer objective function includes:

[0057] Determine a first state set consisting of remaining communication resources, resource block allocation status, and slice set based on the upper layer objective function, and determine a second state set consisting of reserved slice resources, number of users, and user arrival rate based on the lower layer objective function;

[0058] Determine, based on the upper layer objective function, a first action set to complete slice reserved communication resource partitioning at the lth long time domain moment, and determine, based on the lower layer objective function, a second action set consisting of t time slot slice frequency band resource block scheduling and user equipment transmit power allocation;

[0059] The target value of the upper layer target function is determined as a first reward function, and the target value of the lower layer target function is determined as a second reward function.

[0060] From the above description, it can be seen that by determining the state set, action set and reward function to construct a Markov process, it can effectively simplify the complexity of the optimization problem, improve resource management efficiency, and allocate resources in a timely manner.

[0061] Furthermore, the definition of the loss function includes:

[0062]

[0063] In the formula, L(μ) represents the loss function, Q(s,a,μ) represents the current Q network, s represents the state, a represents the selected action, μ represents the neural network parameters of the training network, r represents the reward obtained, and γ S represents the weight, represents the target Q network, s' represents the state of the target Q network, and μ' represents the neural network parameters of the target Q network.

[0064] From the above description, we can see that the loss function designed with dual networks can measure the quality of the strategy, allow the strategy to be trained in the direction of the optimal strategy, and ensure the reliability of the overall resource allocation strategy.

[0065] Further, the determining of the discrete action of the agent that maximizes the Q function value in the current state includes:

[0066]

[0067] In the formula, a m (t) represents the discrete action of agent m that maximizes the Q function value in the current state s(t), represents the best overall resource allocation strategy, a s,m (t) represents the action of agent m in state s, μ m represents the parameters of the actor-family network, represents the parameters of the critic network, Indicates the maximum Q value.

[0068] From the above description, we can see that determining the discrete action that maximizes the Q function value of the agent in the current state can maximize the expected return of the agent, optimize the policy function, and improve decision-making efficiency.

[0069] Furthermore, it also includes:

[0070] In the priority experience replay mechanism, the importance of the rth experience is measured by the absolute value of TD-error, specifically:

[0071]

[0072] In the formula, δ r represents the importance of the r-th experience, represents the target Q value, represents the actual Q value, represents the rth state of agent m, represents the r-th action of agent m, represents the parameters of the critic network;

[0073] The sampling probability is determined according to the importance of the r-th experience, specifically:

[0074]

[0075] p r =|δ r |+ζ;

[0076] In the formula, P(r) represents the sampling probability, α represents the execution priority decision parameter, and p r represents the priority weight of the r-th experience, and ζ represents a preset positive value.

[0077] From the above description, we can see that by introducing the priority experience replay mechanism to increase the sampling probability of important experiences, these experiences can be used more frequently during training. The intelligent agent can focus more on those experiences that have an important impact on its decision-making, thereby improving learning efficiency. In addition, sampling according to the priority of experience makes the data distribution more stable, which helps to improve the stability and effect of training.

[0078] Please refer to Figure 2 Another embodiment of the present invention provides a resource management system based on a 5G power virtual private network, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of the above-mentioned resource management method based on the 5G power virtual private network is implemented.

[0079] The above-mentioned resource management method and system based on 5G power virtual private network of the present invention can be applied to the 5G power virtual private network resource management scenario, which is described below through specific implementation methods:

[0080] Please refer to Figure 1 , Figure 3 and Figure 4 , Embodiment 1 of the present invention is:

[0081] A resource management method based on a 5G power virtual private network, comprising the steps of:

[0082] S1. Establish a 5G power virtual private network-slice system, which includes multiple base stations, multiple power terminal users and multiple slices.

[0083] Specifically, multiple power terminal users and multiple base stations are determined. The base station is responsible for communicating with the power terminal users within its coverage area. The power terminal users have their own specific service requirements. The basic network facilities are divided into multiple virtual networks, namely slices, to support different user services. Multiple groups of associated power terminal users are determined, and a base station and a slice are allocated to each group of associated power terminal users. A 5G power virtual private network-slice system is established based on multiple base stations, multiple power terminal users and multiple slices. Figure 3 shown.

[0084] Among them, multiple base stations constitute a base station set, which is expressed as m represents the mth base station, and M represents the total number of base stations. Multiple power terminal users constitute a power terminal set, which is represented by i represents the i-th power terminal user, I represents the total number of power terminal users, each power terminal user has a base station to provide services, and each base station has a group of associated power terminal users, Multiple slices constitute a slice set, which is represented by n represents the nth slice, N represents the total number of slices, and each slice has a group of associated power end users. express.

[0085] S2. Establish an objective function to maximize system benefits, and establish constraints for the objective function.

[0086] Wherein, establishing an objective function to maximize system benefits includes:

[0087]

[0088] In the formula, represents the long-term bandwidth reservation set, represents the short-time domain resource block allocation set, represents the user power allocation set, T represents the number of long-term domains, and U l Represents the system benefit in the lth long time domain.

[0089] The constraint conditions for establishing the objective function include:

[0090] Establish resource block allocation constraints, transmission queue constraints, allocation constraints of each resource block within a single time slot, data packet transmission constraints, constraints on the sum of slice occupied frequency band resources, constraints on the sum of power terminal user bandwidths, service quality requirement constraints, power terminal user transmission rate constraints, and power terminal user transmission power constraints.

[0091] The resource block allocation constraint indicates that the resource block allocation variable is a binary variable, specifically:

[0092]

[0093] The transmission queue constraints ensure the stability of the transmission queue, specifically:

[0094]

[0095] The allocation constraint of each resource block in a single time slot ensures that each resource block is allocated to only one power terminal user in a single time slot, specifically:

[0096]

[0097] The packet transmission constraints ensure that the packet transmission process meets the AoI requirements, specifically:

[0098]

[0099] The constraint on the total amount of frequency band resources occupied by the slices ensures that the total amount of frequency band resources occupied by the slices does not exceed the total frequency band resources, specifically:

[0100]

[0101] In the formula, Indicates the total bandwidth resources.

[0102] The power terminal user bandwidth sum constraint indicates that the sum of user bandwidth cannot be higher than the bandwidth resource of the slice, specifically:

[0103]

[0104] The service quality requirement constraints ensure that all devices meet the QoS requirements, specifically:

[0105]

[0106] The power terminal user transmission rate constraint is specifically:

[0107]

[0108] The power terminal user transmission power constraint is specifically:

[0109]

[0110] Where P max Indicates the upper limit of the transmission power of the power terminal user.

[0111] Due to the coupling between different levels, the optimization problem of the above objective function is divided into two-level sub-problems. In the upper-level strategy, the system makes a slice resource allocation strategy in the long-term domain. For such problems of optimizing both discrete and continuous variables, it is necessary to simplify them before using deep reinforcement learning to solve them. The focus is on the overall resource allocation strategy. It only needs to reserve certain resources for slices without fine-grained consideration and allocation. It can accept the slight loss caused by converting continuous variables into discrete variables and convert continuous variables into discrete variables. Convert to discrete variables Its collection is Therefore, the above objective function is split into an upper objective function and a lower objective function, as described in S3.

[0112] S3. Split the objective function into an upper layer objective function and a lower layer objective function.

[0113] Among them, the upper layer objective function is specifically:

[0114]

[0115] In the formula, represents the user power allocation set after converting continuous variables into discrete variables, ω2 represents the payment resource cost function weight, ω3 represents the overload penalty weight, It represents the cost function of resources that the slice needs to pay in the lth long time domain, represents the overload penalty for slice n in the lth long time domain, Represents a collection of slices;

[0116] The lower layer objective function is specifically:

[0117]

[0118] In the formula, ω1 represents the total system revenue weight, ΔT represents the multiple relationship of the time span, and REtotal(t) represents the total system revenue.

[0119] The resource management process is divided into two levels: inter-slice and intra-slice. At the inter-slice level, bandwidth resources are allocated to slices by the base station based on resource requirements. At the intra-slice level, the slice performs the resource block (RB) allocation process to allocate necessary RBs to each final power terminal user. Each slice allocates a certain number of RBs from the pre-allocated bandwidth resources to each power terminal user on the slice to meet the various QoS requirements of each final power terminal user in terms of data rate and latency.

[0120] In order to adapt to the dynamic topology changes of the network, fine-grained resource scheduling is required, and to avoid the high cost and resource waste of frequent slice reconfiguration, the present invention performs resource management under the dual time domain framework, such as Figure 4 As shown. The short time domain is defined as STI, represented by t∈{1,2,...}, and the fixed duration between time slot t and the next time t+1 is τ; the long time domain is defined as LTI, represented by l∈{1,2,...}, and the duration is ΔT STIs. In each LTI, the system redefines the association strategy between the power terminal user and the base station, as well as the bandwidth resources and computing resources reserved for each slice. In each STI, in order to ensure that the power terminal user meets the stability and dynamics in each time slot t, the system will schedule the resources within the slice according to the terminal user status.

[0121] In the long time domain, is the bandwidth resource reserved by the system for slice n in the lth LTI, and the bandwidth resources reserved for different slices are obtained Then the number of RBs in slice n can be calculated, that is, In the short time domain, the remaining resource blocks in slice n are expressed as R n ; Indicates the allocation of RBs and defines a binary variable It is used to indicate the allocation of the k∈Kth RB in time slot t slice n. If it is allocated to the i-th device, then otherwise

[0122] Assuming that the communication between the power terminal user i and the base station m includes uplink communication for transmitting the user's dynamic service request to the base station and downlink communication for sending the result to the user, the present invention only considers the uplink communication delay in the communication model, then the SINR (Signal to Interference plus Noise Ratio) that can be achieved between the power terminal user i and the base station m on the time slot t slice n is:

[0123]

[0124] Where, SINR i (t) represents the SINR achievable between power terminal user i and base station m in time slot t slice n, p i (t) represents the transmission power of power terminal user i, represents the channel power gain of power end user i, represents the path loss, α represents the exponent of the path loss, represents the interference noise, σ 2 represents the additive white Gaussian noise power, d i,m (t) represents the distance between power terminal user i and base station m, x u The horizontal axis of the electricity terminal user i, y u represents the ordinate of the power terminal user i, x m represents the horizontal coordinate of base station m, y m Represents the vertical coordinate of base station m.

[0125] B represents the bandwidth of one RB. The transmission rate that a user equipment can achieve on one resource block is r. B , expressed as:

[0126] r B (t) = Blog2(1 + SINR i (t));

[0127] Therefore, the electricity end users Wireless communication rate It is expressed as:

[0128]

[0129] Where K represents the total number of resource blocks owned by slice n.

[0130] The transmission rate requirement for the i-th power terminal user connected to slice n is expressed as Then the number of RBs required by the i-th power terminal user is In order to satisfy the requirement that the number of RBs is an integer, define It is expressed as a ceiling function, that is, if x is an integer, then is x itself, if If is a decimal, then

[0131] Assume that the size of the data packet consisting of application and data sent to the base station is D i (t), the base station receives D i (t) and then returns a feedback signal ACK. After receiving the feedback signal, the power terminal user device transmits the next data packet. If the received data packet has a bit error, the data packet will be discarded and the error data packet will be fed back to the terminal to resend the error data packet. If the SINR is less than the threshold Γ th , then use the binary variable γ i (t) = 1 means that the information sent by the power terminal user device i in time slot t can withstand interference, and the data can be successfully decoded at the base station side, otherwise γ i (t) = 0, expressed as:

[0132]

[0133] When a data packet is being transmitted, the transmission rate also directly affects whether the data packet can successfully reach the base station. When the transmission rate is too low, the data cannot be completely transmitted within the unit time interval τ, which may cause data loss or the need to split the data packet for transmission. Therefore, this situation will affect whether the data packet can successfully reach the base station. Using binary variables i (t) = 1 means that the sensing data is successfully sent within the unit time slot, otherwise i (t) = 0, expressed as:

[0134]

[0135] Assume that the tasks between power terminal users in the virtual power network are independent, and the task arrival rate of power terminal user i in time slot t follows a Poisson process, expressed as λ i (t). Electric power terminal users have different task arrival rates in the present invention, so the overall arrival rate can be expressed as In order to evaluate the average transmission delay, the process of power end-user equipment transmitting information to the base station is modeled as an M / M / 1 queue. Due to the randomness of the wireless channel, it is assumed that the average service rate follows an exponential distribution and can be expressed as:

[0136]

[0137] According to the M / M / 1 queuing theory, the average transmission delay can be defined as: Tup(t) = 1 / (μ(t)-λ(t)). In order to ensure the stability of the transmission queue, the following conditions must be met:

[0138]

[0139] AoI is a metric for information timeliness and is defined as the time elapsed since the last data packet received by the target node was generated. i (t) measures the AoI of the data packet generated by the power terminal user device i. When the power terminal user device i transmits a data packet to the base station, the AoI value of the power terminal user device at time slot t is Δ i (t) is expressed as:

[0140]

[0141] Define the average power terminal user device i at DT synchronization for: The AoI of all users should meet the target constraints: In the formula, Indicates user AoI, Indicates the AoI upper limit of the user on slice n.

[0142] Assuming that the system's revenue is related to the transmission AoI, the revenue is proportional to the difference between the system's required and obtained AoI. As the AoI difference increases, the revenue also increases accordingly, expressed as:

[0143]

[0144] In the formula, RE age (t) represents the system benefit related to transmission AoI, χ in represents the system's unit revenue with respect to delay, ξ max Indicates the AoI upper limit for all users.

[0145] Similarly, the system's transmission error benefit is related to the user data packet transmission process. Assume that η in and ε in They are the unit revenues of data decoding and successful data packet transmission during the system transmission process, specifically:

[0146]

[0147] In the formula, RE tr (t) represents the system's error benefit regarding transmission, D i (t) indicates a data packet.

[0148] Therefore, the total revenue of the system is: REtotal(t)=β1REage(t)+β2REtr(t), where β1 represents the first weight coefficient, β2 represents the second weight coefficient, and β1+β2=1.

[0149] The cost function of resources that a slice needs to pay in the lth long time domain, that is, the resource allocation cost is: In the formula, c sl Indicates the usage cost of unit bandwidth, l sl Indicates the unit cost of tile scaling.

[0150] Assumptions is the bandwidth resource reservation rate of slice n at the lth LTI, and the calculation formula is:

[0151]

[0152] Assume α E Indicates the slice resource reservation warning value. The resource reservation rate that is less than the threshold is penalized. The greater the difference with the warning value, the greater the penalty. The overload penalty of slice n at the lth LTI is Specifically:

[0153]

[0154] In the formula, ε E represents the unit penalty for insufficient resource reservation rate, α B Indicates the minimum threshold of slice resource remaining rate. Not less than α B , when α B When it is 0, it means that the resources in the slice are allowed to be allocated without reservation without considering future situations.

[0155] Networked power terminal users transmit tasks by accessing base stations within the coverage area. In the upper-layer strategy, the base station allocates bandwidth resources to slices in the long time domain according to resource requirements, dynamically adjusts the slice resources reserved for the business and establishes resource constraints; in the lower-layer strategy, the age of information (Age of Information, AoI) is used to ensure the certainty of transmission delay, and slices allocate a certain number of resource blocks to users in the short time domain, and further provide users with reliable communication through more flexible resource scheduling, as described below.

[0156] S4, based on the upper objective function, the base station uses a dual deep Q network algorithm in the long time domain to allocate bandwidth resources to the slice according to resource requirements to obtain an overall resource allocation strategy, and based on the lower objective function, the slice is used in the short time domain to use a multi-agent priority experience playback combined with an actor-critic algorithm to allocate resource blocks and transmission power to the power terminal user to obtain a fine-grained resource partitioning strategy, such as Figure 4 As shown, specifically including S4.1 to S4.10:

[0157] S4.1, based on the upper layer objective function and the lower layer objective function, determine the state set, action set and reward function, specifically including S4.1.1 to S4.1.3:

[0158] S4.1.1. Based on the upper-layer objective function, a first state set consisting of the remaining communication resources, resource block allocation status and slice set is determined, and based on the lower-layer objective function, a second state set consisting of the slice reserved resource status, the number of users and the user arrival rate is determined.

[0159] Among them, the first state set for The second state set for

[0160]

[0161] S4.1.2. Based on the upper layer objective function, determine the first action set to complete the slice reserved communication resource division at the lth long time domain moment, and based on the lower layer objective function, determine the second action set consisting of t time slot slice frequency band resource block scheduling and user equipment transmit power allocation.

[0162] Among them, the first action set for The second action set for

[0163]

[0164] S4.1.3, determining the target value of the upper layer objective function as the first reward function R l ,Right now The target value of the lower layer objective function is determined as the second reward function R s , that is, R s =ω1RE total (t).

[0165] S4.2. Define the loss function. DDQN uses the dual-network structure formed by the target network and the training network to solve the problem that the estimated reward of DQN is higher than the actual reward, and makes the network structure more stable. Therefore, the loss function designed by the dual network can measure the quality of the strategy and train the strategy towards the optimal strategy. Specifically:

[0166]

[0167] In the formula, L(μ) represents the loss function, Q(s,a,μ) represents the current Q network, s represents the state, a represents the selected action, μ represents the neural network parameters of the training network, r represents the reward obtained, and γ S represents the weight, represents the target Q network, s' represents the state of the target Q network, and μ' represents the neural network parameters of the target Q network.

[0168] S4.3. Use the Adam algorithm to minimize the gradient of the loss function, and update the Q network parameters of the current Q network based on the state set, action set and reward function to obtain the target Q network.

[0169] Among them, the gradient of the loss function is specifically:

[0170]

[0171] In the formula, represents the gradient of the loss function, Represents the gradient of the current Q network.

[0172] S4.4. Allocate bandwidth resources to the slice in the long time domain according to resource requirements through the base station using the target Q network to obtain an overall resource allocation strategy.

[0173] Specifically, the target Q network is used to control the base station through the SDN controller to allocate bandwidth resources to the slice in the long time domain according to resource requirements, so as to obtain an overall resource allocation strategy.

[0174] A single DDQN algorithm is no longer suitable for resource scheduling in the short-term domain, so a multi-agent priority experience replay combined with the actor-critic algorithm PER-MACA2C is adopted, as described below.

[0175] S4.5. Treat each of the base stations as an intelligent agent, and determine the discrete action of the intelligent agent that maximizes the Q function value in the current state.

[0176] Wherein, the step of determining the discrete action of the agent that maximizes the Q function value in the current state includes:

[0177]

[0178] In the formula, a m (t) represents the discrete action of agent m that maximizes the Q function value in the current state s(t), represents the best overall resource allocation strategy, a s,m (t) represents the action of agent m in state s, μ m represents the parameters of the actor-family network, represents the parameters of the critic network, Indicates the maximum Q value.

[0179] S4.6. For the training of the actor network, the agent m combines the state s(t) and the overall resource allocation strategy, and uses the state and the overall resource allocation strategy as inputs of the actor network, and the actor network selects output power and observation frequency based on the input.

[0180] S4.7. The intelligent agent m combines the state, the selected continuous actions and the made fine-grained resource partitioning strategy as the input of the critic network. The critic network outputs the estimated Q-value function of the fine-grained resource partitioning strategy based on the input, and determines the optimal fine-grained resource partitioning strategy based on the obtained estimated Q-value function.

[0181] S4.8. The agent m takes corresponding actions according to the maximized expected reward, and updates the actor network according to the policy gradient and the priority experience replay mechanism.

[0182] The strategy gradient is specifically:

[0183]

[0184] represents the policy gradient, r represents the index of experience, represents the policy function gradient, represents the action value function Q with respect to action a m gradient.

[0185] S4.9. For the training of the critic network, the error between the target Q value and the actual Q value of the critic network is minimized, and the parameters of the critic network are updated by the gradient descent method according to the priority experience replay mechanism.

[0186] The output y of the target critic network m Specifically:

[0187]

[0188] In the formula, r mrepresents the reward of agent m, ε represents the weight value, represents the Q-value function of the target network of agent m, represents the target network parameters of the actor network, represents the target network parameters of the critic network to reduce the tendency of overestimating Q-values.

[0189] The error between the target Q-value and the actual Q-value of the critic network Specifically:

[0190]

[0191] In the formula, represents the target Q value, Indicates the actual Q value.

[0192] Among them, it also includes:

[0193] In the priority experience replay mechanism, the importance of the rth experience is measured by the absolute value of TD-error, specifically:

[0194]

[0195] In the formula, δ r represents the importance of the r-th experience, represents the rth state of agent m, represents the r-th action of agent m, represents the parameters of the critic network;

[0196] The larger the TD-error, the greater the possibility of experience replay. The sampling probability is determined according to the importance of the r-th experience, specifically:

[0197]

[0198] p r =|δ r |+ζ;

[0199] In the formula, P(r) represents the sampling probability, α represents the execution priority decision parameter, and p r represents the priority weight of the r-th experience, ζ represents a preset positive value, that is, a preset positive number, which is used to prevent the probability of sample r being drawn being 0.

[0200] S4.10. According to the trained actor network, the slice is used to allocate resource blocks and transmission power to the power terminal user in a short time domain to obtain a fine-grained resource partitioning strategy.

[0201] Please refer to Figure 2 , Embodiment 2 of the present invention is:

[0202] A resource management system based on a 5G power virtual private network includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of the above-mentioned resource management method based on a 5G power virtual private network is implemented.

[0203] In summary, the present invention provides a resource management method and system based on a 5G power virtual private network, establishes a 5G power virtual private network-slice system including multiple base stations, multiple power terminal users and multiple slices, establishes an objective function to maximize the system benefit, and establishes constraints for the objective function, splits the objective function into an upper-level objective function and a lower-level objective function, and based on the upper-level objective function, uses a dual deep Q network algorithm in a long-term domain to allocate bandwidth resources to slices according to resource requirements through the base station to obtain an overall resource allocation strategy, and based on the lower-level objective function, uses slices in a short-term domain to allocate resource blocks and transmission power to power terminal users using a multi-agent priority experience playback combined with an actor critic algorithm to obtain a fine-grained resource partitioning strategy. The present invention adopts adaptive hierarchical resource management. In the upper-level strategy DDQN algorithm, the dual-network structure formed by the target network and the training network is used to solve the problem that the DQN estimated reward is higher than the actual reward, and allows the network to The network structure is more stable and a better overall resource allocation strategy can be made. In the lower-level strategy, since both discrete and continuous actions are involved, the multi-agent priority experience replay combined with the actor-critic algorithm is used to solve the problem of both discrete and continuous actions, which can be considered and allocated in a fine-grained manner, and realize dynamic scheduling of resources, thereby ensuring stable service performance while meeting the user's service quality requirements; in addition, the DDQN algorithm utilizes the dual-network structure formed by the target network and the training network, which solves the problem that the DQN estimated reward is higher than the actual reward, and makes the network structure more stable. The single DDQN algorithm is not suitable for resource scheduling in the short time domain, so the multi-agent priority experience replay combined with the actor-critic algorithm is used in the lower layer to dynamically schedule resources, support diversified power services with different QoS requirements and ensure fine-grained division of resources, meet dynamic traffic requirements and differentiated service requirements of power terminal users.

[0204] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A resource management method based on 5G power virtual private network, characterized in that: Includes steps: Establishing a 5G power virtual private network-slice system, wherein the 5G power virtual private network-slice system includes multiple base stations, multiple power terminal users and multiple slices; Establishing an objective function to maximize system benefits and establishing constraints of the objective function; Splitting the objective function into an upper objective function and a lower objective function; Based on the upper-level objective function, the base station uses a dual deep Q network algorithm in the long time domain to allocate bandwidth resources to the slice according to resource requirements to obtain an overall resource allocation strategy, and based on the lower-level objective function, the slice is used in the short time domain to allocate resource blocks and transmission power to the power terminal user using a multi-agent priority experience replay combined with an actor-critic algorithm to obtain a fine-grained resource partitioning strategy.

2. According to claim 1, a resource management method based on 5G power virtual private network is characterized in that: The objective function established to maximize system benefits includes: In the formula, represents the long-term bandwidth reservation set, represents the short-time domain resource block allocation set, represents the user power allocation set, T represents the number of long-term domains, and U l Represents the system benefit in the lth long time domain.

3. According to a resource management method based on 5G power virtual private network according to claim 1, it is characterized in that: The constraint conditions for establishing the objective function include: Establish resource block allocation constraints, transmission queue constraints, allocation constraints of each resource block within a single time slot, data packet transmission constraints, constraints on the sum of slice occupied frequency band resources, constraints on the sum of power terminal user bandwidths, service quality requirement constraints, power terminal user transmission rate constraints, and power terminal user transmission power constraints.

4. According to claim 2, a resource management method based on 5G power virtual private network is characterized in that: The upper layer objective function is specifically: In the formula, represents the user power allocation set after converting continuous variables into discrete variables, ω2 represents the payment resource cost function weight, ω3 represents the overload penalty weight, It represents the cost function of resources that the slice needs to pay in the lth long time domain, represents the overload penalty for slice n in the lth long time domain, Represents a collection of slices; The lower layer objective function is specifically: In the formula, ω1 represents the total system revenue weight, ΔT represents the multiple relationship of the time span, and REtotal(t) represents the total system revenue.

5. According to a resource management method based on 5G power virtual private network according to claim 1, it is characterized in that: The method of allocating bandwidth resources to the slices based on the upper objective function by the base station according to resource requirements in the long time domain using a dual deep Q network algorithm to obtain an overall resource allocation strategy, and allocating resource blocks and transmission power to the power terminal users based on the lower objective function by using the slices in the short time domain using a multi-agent priority experience playback combined with an actor critic algorithm to obtain a fine-grained resource partitioning strategy includes: Determine a state set, an action set, and a reward function based on the upper layer objective function and the lower layer objective function; Define the loss function; The Adam algorithm is used to minimize the gradient of the loss function, and the Q network parameters of the current Q network are updated based on the state set, action set and reward function to obtain the target Q network; Allocating bandwidth resources to the slice in a long time domain according to resource requirements through the base station using the target Q network to obtain an overall resource allocation strategy; Taking each of the base stations as an intelligent agent, and determining a discrete action of the intelligent agent that maximizes the Q function value in the current state; For training the actor-home network, the agent combines the state and the overall resource allocation strategy, uses the state and the overall resource allocation strategy as inputs to the actor-home network, and the actor-home network outputs a selection of power and observation frequency based on the inputs; The agent combines the state, the selected continuous actions and the made fine-grained resource partitioning strategy as input of the critic network, and the critic network outputs an estimated Q-value function of the fine-grained resource partitioning strategy according to the input, and determines the optimal fine-grained resource partitioning strategy according to the obtained estimated Q-value function; The agent takes corresponding actions according to the maximized expected reward, and updates the actor network according to the policy gradient and the priority experience replay mechanism; For the training of the critic network, the error between the target Q value and the actual Q value of the critic network is minimized, and the parameters of the critic network are updated according to the priority experience replay mechanism by the gradient descent method; The actor network after training is used to allocate resource blocks and transmission power to the power terminal user in a short time domain using the slices, thereby obtaining a fine-grained resource partitioning strategy.

6. A resource management method based on 5G power virtual private network according to claim 5, characterized in that: The determining of a state set, an action set, and a reward function based on the upper layer objective function and the lower layer objective function comprises: Determine a first state set consisting of remaining communication resources, resource block allocation status, and slice set based on the upper layer objective function, and determine a second state set consisting of reserved slice resources, number of users, and user arrival rate based on the lower layer objective function; Determine, based on the upper layer objective function, a first action set to complete slice reserved communication resource partitioning at the lth long time domain moment, and determine, based on the lower layer objective function, a second action set consisting of t time slot slice frequency band resource block scheduling and user equipment transmit power allocation; The target value of the upper layer target function is determined as a first reward function, and the target value of the lower layer target function is determined as a second reward function.

7. A resource management method based on 5G power virtual private network according to claim 5, characterized in that: Defining the loss function includes: In the formula, L(μ) represents the loss function, Q(s,a,μ) represents the current Q network, s represents the state, a represents the selected action, μ represents the neural network parameters of the training network, r represents the reward obtained, and γ S represents the weight, represents the target Q network, s' represents the state of the target Q network, and μ' represents the neural network parameters of the target Q network.

8. According to a resource management method based on 5G power virtual private network according to claim 5, it is characterized in that: Determining the discrete action of the agent to maximize the Q function value in the current state includes: In the formula, a m (t) represents the discrete action of agent m that maximizes the Q function value in the current state s(t), represents the best overall resource allocation strategy, a s,m (t) represents the action of agent m in state s, μ m represents the parameters of the actor-family network, represents the parameters of the critic network, Indicates the maximum Q value.

9. A resource management method based on 5G power virtual private network according to claim 5, characterized in that: Also includes: In the priority experience replay mechanism, the importance of the rth experience is measured by the absolute value of TD-error, specifically: In the formula, δ r represents the importance of the r-th experience, represents the target Q value, represents the actual Q value, represents the rth state of agent m, represents the r-th action of agent m, represents the parameters of the critic network; The sampling probability is determined according to the importance of the r-th experience, specifically: p r =|δ r |+ζ; In the formula, P(r) represents the sampling probability, α represents the execution priority decision parameter, and p r represents the priority weight of the r-th experience, and ζ represents a preset positive value.

10. A resource management system based on a 5G power virtual private network, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements each step in a resource management method based on a 5G power virtual private network as described in any one of claims 1 to 9.

Images