Power distribution network cloud side-end cooperative scheduling method and device based on reinforcement learning

Abstract

The invention provides a reinforcement learning-based distribution network cloud side end cooperative scheduling method and device, and relates to the technical field of intelligent distribution networks, and the method comprises the steps: building a distribution network model of cloud side end cooperative scheduling, the distribution network model of cloud side end cooperative scheduling comprises a basic distribution network model and an electric energy flow unloading model, the basic power distribution network model comprises network nodes in the power distribution network and a topological structure formed by network links among the network nodes, and the electric energy flow unloading model is used for representing service request states of the nodes; based on the electric energy flow unloading model, establishing a target function by taking electric energy flow unloading maximization as a target, and determining constraint conditions of the target function; and based on the basic power distribution network model, the objective function and the constraint condition, performing optimization solution on the objective function through reinforcement learning to obtain an optimal electric energy flow scheduling scheme. According to the invention, the pressure of distribution network cloud side-end cooperative computing can be effectively reduced, and the operation efficiency of the distribution network is improved.

Description

Technical Field

[0001] The present application relates to the technical field of smart distribution networks, and specifically to a distribution network cloud-edge collaborative scheduling method based on reinforcement learning, a distribution network cloud-edge collaborative scheduling device based on reinforcement learning, a machine-readable storage medium and a terminal device. Background Art

[0002] At present, the main stations of traditional distribution network services are mostly built based on the cloud computing framework, and the cloud computing resources are sufficient to effectively meet the processing needs of business data. However, the long-distance link transmission capacity between the cloud and the distribution network terminal is unstable, and the adaptability to high-frequency acquisition scenarios with high signal acquisition frequency and fast business scale growth is insufficient. Therefore, some cloud center functions can be sunk to the vicinity of the local side through edge computing, and the data processing pressure of the cloud center can be reduced by cutting large-scale computing services. It has the advantages of short transmission distance and rapid business response, and can form an effective complement to cloud computing. In addition, intelligent terminals with certain computing power are increasingly widely used in distribution networks. New intelligent terminals have functions such as data collection, access, processing, and upload. Distributed intelligent computing enables cloud-edge-end collaboration to effectively realize the coordination of cross-level and multi-scale heterogeneous computing resources. By building a cloud-edge-end collaborative data processing mechanism for high-frequency acquisition of distribution networks, it will help improve the transmission and processing capabilities of distribution network business data and support the improvement of the level of intelligent decision-making in distribution networks. However, the current business processing requirements of edge computing in the distribution network in the power system are relatively high, and the pressure of cloud-edge collaborative computing is high. Therefore, optimizing the business processing requirements of edge computing in the distribution network can effectively reduce the computing pressure of cloud-edge collaborative computing in the distribution network, thereby improving the operating efficiency of the distribution network. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a distribution network cloud-edge collaborative scheduling method based on reinforcement learning, a distribution network cloud-edge collaborative scheduling device based on reinforcement learning, a machine-readable storage medium and a terminal device to solve the above problems.

[0004] In order to achieve the above objectives, the first aspect of the present application provides a distribution network cloud-edge collaborative scheduling method based on reinforcement learning, comprising:

[0005] Establish a distribution network model for cloud-edge-end collaborative scheduling, wherein the distribution network model for cloud-edge-end collaborative scheduling includes a basic distribution network model and an electric energy flow unloading model, wherein the basic distribution network model includes a topological structure consisting of network nodes in the distribution network and network links between the network nodes, and the electric energy flow unloading model is used to represent the service request status of each node;

[0006] Based on the electric energy flow unloading model, an objective function is established with the goal of maximizing the electric energy flow unloading, and constraints of the objective function are determined;

[0007] Based on the basic distribution network model, the objective function and the constraints, the objective function is optimized and solved through reinforcement learning to obtain an optimal electric energy flow scheduling solution.

[0008] Optionally, the basic power distribution network model includes a network topology of the power distribution network:

[0009] G=(V,L)

[0010] Among them, G represents the network topology structure of the distribution network, V represents each network node in the distribution network, L represents the network link between each node, and the network nodes include edge terminals, end terminals and mobile terminals.

[0011] Optionally, the electric energy flow unloading model includes:

[0012] Local traffic offloading model and remote traffic offloading model;

[0013] The local traffic offloading model is:

[0014]

[0015] The remote traffic offloading model is:

[0016]

[0017] in, Indicates the amount of local traffic offload, represents the traffic unloading amount of service request f in the local mobile terminal set at time t, represents the traffic unloading amount of service request f in the local terminal set at time t, s f represents the weight coefficient of the service request f, M i (t) represents the local mobile terminal set, Indicates the amount of remote traffic offload, represents the traffic unloading amount of service request f in the edge terminal set at time t, represents the traffic unloading amount of service request f in the local terminal set at time t, represents the traffic unloading amount of service request f in the remote side terminal set at time t, B i Indicates the local terminal set, B j Indicates the remote side terminal set.

[0018] Optionally, the objective function includes:

[0019]

[0020] Where T represents the time interval, F represents the request set of business request f, Represents the weight coefficient of the mth mobile terminal at time t.

[0021] Optionally, the constraint conditions include:

[0022]

[0023] Among them, constraint C1 means A is a Boolean variable with a value of 0 or 1. i，m (t) indicates whether the service request f is processed by the local terminal, and C2 indicates that the amount of power unloaded on link l cannot exceed its bandwidth B. l , C3 indicates that the amount of power cached by a network node cannot exceed the storage limit of the device, and represents the amount of offloading of service request f in the local terminal, s k The feature vector representing the service request f, C i Indicates the maximum unloading capacity of the local terminal, C j represents the maximum unloading capacity of the side terminal, C m Indicates the maximum uninstall volume of a mobile terminal.

[0024] Optionally, optimizing and solving the objective function through reinforcement learning includes:

[0025] S1. Construct a DQN deep neural network including an evaluation network and a target network, and construct the state space, action space and reward function of reinforcement learning;

[0026] S2, initialize the target network and evaluation network;

[0027] S3, initialize the experience replay pool and current state S t , let the initial stage t=1;

[0028] S4. Use the ω-greedy algorithm to select the scheduling strategy with the largest Q value. The scheduling strategy includes action a t ;

[0029] S5. Execute action a t , get reward r t and the new state S t+1 , the current state S t 、Action a t , Reward t and the next state action S t+1 Stored in the experience replay pool;

[0030] S6. Determine whether the end state has been reached. If yes, end the process. If not, randomly select sample data from the experience replay pool (S i , A i , R i , S i+1 ), calculate the target Q value through the target network, calculate the evaluation Q value through the evaluation network, and calculate the loss function based on the target Q value and the evaluation Q value;

[0031] Update the weights of the evaluation network through back propagation and gradient descent algorithms to determine whether stage t is nc, where c is a constant and n = 1, 2, ...N;

[0032] If so, use the weights of the evaluation network to update the weights of the target network, and set t = t + 1; otherwise, set t = t + 1 and return to step S4.

[0033] Optionally, the state space includes:

[0034]

[0035] Among them, S t is the current state, G t is the network topology including time slot t, f t is the number of service requests, indicating the power demand in time slot t, n t is a network node, R nt is the link bandwidth status, indicating the network bandwidth of time slot t, T nt is the network node n t The node type flag set of the adjacent nodes, P nt is the network node n t The demand queue state set;

[0036] The action space includes:

[0037]

[0038] Among them, use X nt Represents network node n t Whether to process cached business requests, n t+1 Represents network node n t The number of next-hop routing nodes;

[0039] The reward function includes:

[0040]

[0041] Where γ is the discount factor, K is the maximum number of time slots, is the reward for the mth time slot.

[0042] In a second aspect of the present application, a distribution network cloud-edge collaborative scheduling device based on reinforcement learning is provided, comprising:

[0043] A model building module is configured to establish a distribution network model for cloud-edge-end collaborative scheduling, wherein the distribution network model for cloud-edge-end collaborative scheduling includes a basic distribution network model and an electric energy flow unloading model, wherein the basic distribution network model includes a topological structure consisting of network nodes in the distribution network and network links between the network nodes, and the electric energy flow unloading model is used to represent the service request status of each node;

[0044] An objective function building module is configured to establish an objective function based on the power flow unloading model with the power flow unloading maximization as the goal, and determine the constraint conditions of the objective function;

[0045] The scheduling optimization module is configured to optimize and solve the objective function through reinforcement learning based on the basic distribution network model, the objective function and the constraints to obtain an optimal power flow scheduling solution.

[0046] According to a third aspect of the present application, a machine-readable storage medium is provided, on which instructions are stored. When the instructions are executed by a processor, the processor is configured to execute the method as described above.

[0047] In a fourth aspect of the present application, a terminal device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0048] This application can reasonably coordinate the business processing needs of edge computing in the distribution network in the power system, reduce the computing pressure of cloud-edge collaborative computing in the distribution network, and effectively improve the efficiency of distribution network operation.

[0049] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0051] Figure 1 A method flow chart of a method for cloud-edge-end collaborative scheduling of a power distribution network based on reinforcement learning provided in a preferred embodiment of the present application;

[0052] Figure 2 A schematic diagram of a reinforcement learning model provided for a preferred embodiment of the present application;

[0053] Figure 3 A schematic block diagram of a distribution network cloud-edge collaborative scheduling device based on reinforcement learning provided in a preferred embodiment of the present application;

[0054] Figure 4 A schematic diagram of a terminal device provided for a preferred embodiment of the present application.

[0055] Description of Reference Numerals

[0056] 10-terminal device, 100-processor, 101-memory, 102-computer program. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0058] It should be noted that the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0059] like Figure 1 As shown, the first aspect of the present application provides a distribution network cloud-edge collaborative scheduling method based on reinforcement learning, comprising:

[0060] S100. Establish a distribution network model for cloud-edge-end collaborative scheduling. The distribution network model for cloud-edge-end collaborative scheduling includes a basic distribution network model and an electric energy flow unloading model. The basic distribution network model includes a topological structure consisting of network nodes in the distribution network and network links between network nodes. The electric energy flow unloading model is used to represent the service request status of each node.

[0061] S200, based on the power flow unloading model, establishing an objective function with the goal of maximizing the power flow unloading, and determining the constraint conditions of the objective function;

[0062] S300, based on the basic distribution network model, objective function and constraints, the objective function is optimized and solved through reinforcement learning to obtain the optimal power flow scheduling solution.

[0063] In this way, the application can reasonably coordinate the business processing needs of edge computing in the distribution network of the power system, reduce the computing pressure of the cloud-edge collaborative computing of the distribution network, and effectively improve the efficiency of the distribution network operation.

[0064] In step S100, a distribution network model for cloud-edge-end collaborative scheduling is first established, wherein the distribution network model for cloud-edge-end collaborative scheduling includes a basic distribution network model and an electric energy flow unloading model. It can be understood that the method of the present application can be applied to cloud-side terminals such as cloud servers, and the scheduling control of electric energy service requests is performed by the cloud-side controller of the cloud-side terminal, that is, it is used to decide the distribution control of edge device traffic. The network nodes of the basic distribution network model also include edge terminals such as macro base stations MBSs, end terminals such as small base stations SBSs and mobile terminals MCs, wherein MBS represents a macro base station, MBSs represents a macro base station set, SBS represents a small base station, SBSs represents a small base station set, MC represents a mobile terminal, and MCs represents a mobile terminal set. Specifically, the basic distribution network model includes a network topology structure of the distribution network: G = (V, L), wherein G represents the network topology structure of the distribution network, V represents each network node in the distribution network, L represents the network link between each node, and the set of all base stations is set to B, and the time slot is t.

[0065] Among them, according to the cache status of network data and the validity of network resources, the request sent by MC will be routed to the MBS it accesses, the adjacent MCs, and other MBSs adjacent to the MBS it accesses in sequence. For SBSs behind MCs, their requests will be routed to SBS, adjacent MCs, SBS's adjacent SBS, SBS's MBS, MBS adjacent to MBS, and finally connected to the cloud server to search for the corresponding content.

[0066] In this application, the power flow unloading model includes: a local flow unloading model and a remote flow unloading model; wherein the local flow unloading model is:

[0067]

[0068] The remote traffic offloading model is:

[0069]

[0070] in, Indicates the amount of local traffic offload, represents the traffic unloading amount of service request f in the local mobile terminal set at time t, represents the traffic unloading amount of service request f in the local terminal set at time t, s f represents the weight coefficient of the service request f, M i(t) represents the local mobile terminal set, Indicates the amount of remote traffic offload, represents the traffic unloading amount of service request f in the edge terminal set at time t, represents the traffic unloading amount of service request f in the local terminal set at time t, represents the traffic unloading amount of service request f in the remote side terminal set at time t, B i Indicates the local terminal set, B j Indicates the remote side terminal set.

[0071] The local traffic offloading model is for requests that can be processed by the local neighboring MC set and SBS, and the remote traffic offloading model is for requests that cannot be processed by the local neighboring MC set and SBS and need to be processed by the MBS.

[0072] For example, in a request process, the mobile terminal MC sends a service request for electric energy flow, i.e., the current flow of the power distribution, such as uploading data or power distribution requirements. The request first reaches the nearest macro base station MBS, which receives and records the request and determines whether it can be processed locally. If the request can be processed locally in the MBS, for example, the local MBS has sufficient computing resources, the request will be processed inside the MBS; if the request cannot be processed inside the MBS, the request will be forwarded to the adjacent small base station SBS. Similarly, the SBS determines whether to process the request inside the SBS by checking whether there are sufficient resources to process the request. If the SBS can process the request, the request will be processed inside the SBS, otherwise, the request will continue to be forwarded to the adjacent MC. If the adjacent MC can process the request, the request will be processed inside the MC, otherwise, the request will be returned to the SBS. If the SBS still cannot process the request, the request will be forwarded to another SBS until a node that can process the request is found. If all SBSs cannot process the request, the request will be forwarded back to the MBS, which will try to process the request. If successful, the request will be processed within the MBS. If the MBS cannot process the request, the request will be forwarded to other MBSs until a node that can process the request is found. If all MBSs cannot process the request, the request will be forwarded to the cloud server for processing. In this way, by finding the most suitable processing node at different levels of the network, communication delay and energy consumption can be minimized, thereby achieving efficient resource utilization and fast service response.

[0073] In step S200, the present application constructs an objective function based on the problem of maximizing the power flow unloading, wherein the objective function includes:

[0074]

[0075] Where T represents the time interval, F represents the request set of business request f, Represents the weight coefficient of the mth mobile terminal at time t.

[0076] Optionally, constraints include:

[0077]

[0078] Among them, the constraint C1 means A is a Boolean variable with a value of 0 or 1. i，m (t) indicates whether the service request f is processed by the local terminal. C2 indicates the link. The amount of power unloaded on it cannot exceed its bandwidth B. l , C3 indicates that the amount of power cached by a network node cannot exceed the storage limit of the device, and represents the amount of offloading of service request f in the local terminal, s k The feature vector representing the service request f, C i Indicates the maximum unloading capacity of the local terminal, C j represents the maximum unloading capacity of the side terminal, C m In other words, set constraint C1 to be a Boolean variable with all values ​​of 0 or 1, and set constraint C2 to be that the traffic unloaded on link l cannot exceed its bandwidth B. l ,Set constraint C3 indicates that the service requests cached by a network node cannot exceed the storage limit of the device.

[0079] like Figure 2 As shown, this application optimizes and solves the objective function through reinforcement learning, including:

[0080] S1. Construct a DQN deep neural network including an evaluation network and a target network, and construct the state space, action space and reward function of reinforcement learning.

[0081] Among them, the state space includes:

[0082]

[0083] Among them, S t is the current state, G t is the network topology including time slot t, f t is the number of service requests, indicating the power demand in time slot t, n t is a network node, R nt is the link bandwidth status, indicating the network bandwidth of time slot t, T nt is the network node n t The node type flag set of the adjacent nodes, P nt is the network node nt The set of demand queue states. t The adjacent nodes refer to n t For other adjacent edge computing nodes, the demand queue refers to the queue composed of the demand for power flow of each node. The node type flag can be determined in advance and each node can be marked in advance; the state of the demand queue refers to the current comprehensive state of the demand queue, such as how many nodes request power, the queue status of business requests, etc.; the number of down-hop routing nodes refers to the number of nodes that are connected to n t The number of directly connected nodes.

[0084] Action space, including:

[0085]

[0086] Among them, use X nt Represents network node n t Whether to process cached business requests, n t+1 Represents network node n t The number of next-hop routing nodes.

[0087] Reward function, including:

[0088]

[0089] Where γ is the discount factor, K is the maximum number of time slots, k is the current number of time slots, is the reward for the mth time slot.

[0090] S2. Initialize the target network and the evaluation network, and initialize the network parameters of the target network and the evaluation network.

[0091] S3, initialize the experience replay pool and current state S t , let the initial stage t=1.

[0092] S4. Use the ω-greedy algorithm to select the scheduling strategy with the largest Q value. The scheduling strategy includes the action a of the cloud controller. t This application uses a reinforcement learning method to develop a power distribution scheduling strategy for the power distribution network according to the priority of different business requests in each time slot, thereby ensuring the efficiency, stability and reliability of power distribution.

[0093] S5. Execute action a t , get reward r l and the new state S t+1 , the current state S t 、Action a t , Reward t and the next state action S t+1This application uses the self-exploration characteristics of DQN to enable the cloud controller to select the scheduling strategy with the largest Q value according to the ε-greedy strategy with a probability of ε, randomly select the scheduling strategy with a probability of 1-ε, and count the rewards obtained for each scheduling, and store the current state S t , action a t , reward r t and the next state action S t|1 The experience is stored in the experience replay pool for subsequent updates, and the optimal strategy for exploring the environment is learned through continuous interaction with the environment.

[0094] S6. Determine whether the end state has been reached. If yes, end the process. If not, randomly select sample data from the experience replay pool (S i , A i , R i , S i+1 ), calculate the target Q value through the target network, calculate the evaluation Q value through the evaluation network, and calculate the loss function based on the target Q value and the evaluation Q value, wherein the end state can be that the performance of the cloud controller meets a specific standard or reaches a limit on the number of iterations.

[0095] The weights of the evaluation network are updated by back propagation and gradient descent algorithms to determine whether stage t is nc, where c is a constant and n = 1, 2, ... N. For example, it can be updated every 100 iterations;

[0096] If so, use the weights of the evaluation network to update the weights of the target network, and set t = t + 1; otherwise, set t = t + 1 and return to step S4.

[0097] In this application, the cloud controller executes the power distribution scheduling decision for the business request based on the reinforcement learning algorithm, where the input of the DQN neural network is a tuple consisting of state S and action A, and the output is the recommended action A' in the next state. The action value function can be expressed as Qπ(S, A), which is also called Q value, where π represents the strategy of the cloud controller when selecting an action. The optimal action strategy is the one with the largest Q value of the action value function under all strategies. The target network outputs the target Q value r t +Q π (s t+1 ,π(s t+1 )), the evaluation network outputs the evaluation Q value Q π (s t , a t ), where the target Q value is fixed and updated as the parameters of the target network are updated. During the optimization process, the evaluation Q value output by the evaluation network is made to continuously approach the target Q value, thereby obtaining a trained neural network model. It can be understood that the training process of reinforcement learning DQN is a prior art and is not limited here.

[0098] like Figure 3 As shown, in a second aspect of the present application, a distribution network cloud-edge collaborative scheduling device based on reinforcement learning is provided, comprising:

[0099] A model building module is configured to establish a distribution network model for cloud-edge-end collaborative scheduling, wherein the distribution network model for cloud-edge-end collaborative scheduling includes a basic distribution network model and an electric energy flow unloading model, wherein the basic distribution network model includes a topological structure consisting of network nodes in the distribution network and network links between the network nodes, and the electric energy flow unloading model is used to represent the service request status of each node;

[0100] The objective function building module is configured to build an objective function based on the electric energy flow unloading model with the goal of maximizing the electric energy flow unloading, and determine the constraint conditions of the objective function;

[0101] The scheduling optimization module is configured to optimize and solve the objective function through reinforcement learning based on the basic distribution network model, objective function and constraints to obtain the optimal power flow scheduling plan.

[0102] It is understandable that the technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0103] According to a third aspect of the present application, a machine-readable storage medium is provided, on which instructions are stored. When the instructions are executed by a processor, the processor is configured to execute the method as described above.

[0104] In a fourth aspect of the present application, a terminal device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0105] like Figure 4FIG. 1 is a schematic diagram of a terminal device provided in an embodiment of the present application. Figure 4 As shown, the terminal device 10 of this embodiment includes: a processor 100, a memory 101, and a computer program 102 stored in the memory 101 and executable on the processor 100. When the processor 100 executes the computer program 102, the steps in the above method embodiment are implemented. Alternatively, when the processor 100 executes the computer program 102, the functions of each module / unit in the above device embodiments are implemented.

[0106] Exemplarily, the computer program 102 may be divided into one or more modules / units, one or more modules / units are stored in the memory 101, and are executed by the processor 100 to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 102 in the terminal device 10.

[0107] The terminal device 10 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device 10 may include, but is not limited to, a processor 100 and a memory 101. Those skilled in the art will appreciate that Figure 4 It is only an example of the terminal device 10 and does not constitute a limitation of the terminal device 10. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0108] The processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0109] The memory 101 may be an internal storage unit of the terminal device 10, such as a hard disk or memory of the terminal device 10. The memory 101 may also be an external storage device of the terminal device 10, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 10. Further, the memory 101 may also include both an internal storage unit of the terminal device 10 and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal device 10. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0110] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0111] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0112] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A cloud-edge-end collaborative scheduling method for distribution network based on reinforcement learning, characterized in that: include: Establish a distribution network model for cloud-edge-end collaborative scheduling, wherein the distribution network model for cloud-edge-end collaborative scheduling includes a basic distribution network model and an electric energy flow unloading model, wherein the basic distribution network model includes a topological structure consisting of network nodes in the distribution network and network links between the network nodes, and the electric energy flow unloading model is used to represent the service request status of each node; Based on the electric energy flow unloading model, an objective function is established with the goal of maximizing the electric energy flow unloading, and constraints of the objective function are determined; Based on the basic distribution network model, the objective function and the constraints, the objective function is optimized and solved through reinforcement learning to obtain an optimal electric energy flow scheduling solution.

2. The method for cloud-edge-end collaborative scheduling of power distribution networks based on reinforcement learning according to claim 1 is characterized in that: The basic power distribution network model includes the network topology of the power distribution network: G=(V,L) Among them, G represents the network topology structure of the distribution network, V represents each network node in the distribution network, L represents the network link between each node, and the network nodes include edge terminals, end terminals and mobile terminals.

3. The method for cloud-edge-end collaborative scheduling of power distribution networks based on reinforcement learning according to claim 1 is characterized in that: The electric energy flow unloading model includes: Local traffic offloading model and remote traffic offloading model; The local traffic offloading model is: The remote traffic offloading model is: in, Indicates the amount of local traffic offload, represents the traffic unloading amount of service request f in the local mobile terminal set at time t, represents the traffic unloading amount of service request f in the local terminal set at time t, s f represents the weight coefficient of the service request f, Mi(t) represents the local mobile terminal set, Indicates the amount of remote traffic offload, represents the traffic unloading amount of service request f in the edge terminal set at time t, represents the traffic unloading amount of service request f in the local terminal set at time t, represents the traffic unloading amount of service request f in the remote side terminal set at time t, B i Indicates the local terminal set, B j Indicates the remote side terminal set.

4. The method for cloud-edge-end collaborative scheduling of distribution networks based on reinforcement learning according to claim 3 is characterized in that: The objective function includes: Where T represents the time interval, F represents the request set of business request f, Represents the weight coefficient of the mth mobile terminal at time t.

5. The method for cloud-edge-end collaborative scheduling of power distribution networks based on reinforcement learning according to claim 4 is characterized in that: The constraints include: C2: C3: Among them, constraint C1 means A is a Boolean variable with a value of 0 or 1. i，m (t) indicates whether the service request f is processed by the local terminal, and C2 indicates that the amount of power unloaded on link l cannot exceed its bandwidth B. l , C3 indicates that the amount of power cached by a network node cannot exceed the storage limit of the device, and represents the amount of offloading of service request f in the local terminal, s k The feature vector representing the service request f, C i Indicates the maximum unloading capacity of the local terminal, C j represents the maximum unloading capacity of the side terminal, C m Indicates the maximum uninstall volume of a mobile terminal.

6. The method for cloud-edge-end collaborative scheduling of power distribution networks based on reinforcement learning according to claim 5 is characterized in that: The objective function is optimized and solved by reinforcement learning, including: S1. Construct a DQN deep neural network including an evaluation network and a target network, and construct the state space, action space and reward function of reinforcement learning; S2, initialize the target network and evaluation network; S3, initialize the experience replay pool and current state S t , let the initial stage t = 1; S4. Use the ω-greedy algorithm to select the scheduling strategy with the largest Q value. The scheduling strategy includes action a t ; S5. Execute action a t , get reward r t and the new state S t+1 , the current state S t 、Action a t , Reward t and the next state action S t+1 Stored in the experience replay pool; S6. Determine whether the end state has been reached. If yes, end the process. If not, randomly select sample data from the experience replay pool (S i ,A i ,R i ,S i+1 ), calculate the target Q value through the target network, calculate the evaluation Q value through the evaluation network, and calculate the loss function based on the target Q value and the evaluation Q value; Update the weights of the evaluation network through back propagation and gradient descent algorithms to determine whether stage t is nc, where c is a constant and n = 1, 2, ... N; If so, use the weights of the evaluation network to update the weights of the target network, and set t = t + 1; otherwise, set t = t + 1 and return to step S4.

7. The method for cloud-edge-end collaborative scheduling of power distribution networks based on reinforcement learning according to claim 6 is characterized in that: The state space includes: Among them, S t is the current state, G t is the network topology including time slot t, f t is the number of service requests, indicating the power demand in time slot t, n t is a network node, R nt is the link bandwidth status, indicating the network bandwidth of time slot t, T nt is the network node n t The node type flag set of the adjacent nodes, P nt is the network node n t The demand queue state set; The action space includes: Among them, use X nt Represents network node n t Whether to process cached business requests, n t+1 Represents network node n t The number of next-hop routing nodes; The reward function includes: Where γ is the discount factor, K is the maximum number of time slots, is the reward for the mth time slot.

8. A cloud-edge-end collaborative dispatching device for a power distribution network based on reinforcement learning, characterized in that: include: A model building module is configured to establish a distribution network model for cloud-edge-end collaborative scheduling, wherein the distribution network model for cloud-edge-end collaborative scheduling includes a basic distribution network model and an electric energy flow unloading model, wherein the basic distribution network model includes a topological structure consisting of network nodes in the distribution network and network links between the network nodes, and the electric energy flow unloading model is used to represent the service request status of each node; An objective function building module is configured to establish an objective function based on the power flow unloading model with the power flow unloading maximization as the goal, and determine the constraint conditions of the objective function; The scheduling optimization module is configured to optimize and solve the objective function through reinforcement learning based on the basic distribution network model, the objective function and the constraints to obtain an optimal power flow scheduling solution.

9. A machine-readable storage medium having instructions stored thereon, characterized in that: When the instruction is executed by a processor, the processor is configured to perform the method according to any one of claims 1 to 8.

10. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images