Method and device for determining resource scheduling scheme of power distribution network and electronic equipment

Abstract

The invention discloses a method and device for determining a resource scheduling scheme of a power distribution network and electronic equipment. The method comprises the steps that state information of a user is acquired, and target state information is obtained; processing the target state information through an inverse reinforcement learning model to obtain an action corresponding to the target state information; the power grid state information of the target power distribution network and the action corresponding to the target state information are processed through a target discriminator, a resource scheduling scheme of the target power distribution network is obtained, and the target discriminator adopts a deep Q network algorithm. According to the method and the device, the technical problem that the accuracy of the determined resource scheduling scheme is low due to the fact that the user behavior is guided and predicted based on the load control strategy or the electricity price in the power distribution network with high user side energy storage permeability in the related technology is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a method and device for determining a resource scheduling scheme of a power distribution network and an electronic device. BACKGROUND

[0002] With the rapid development of energy internet and smart grid technology, user-side energy storage systems are gradually popularized and play an increasingly important role in the power system. User-side energy storage, such as electric vehicles, home energy storage devices, etc., has flexible charging and discharging characteristics, can effectively participate in demand-side response, improve the flexibility of power grid operation, promote renewable energy consumption, and provide economic benefits such as reducing electricity costs for users. However, in a regional power grid with high user-side energy storage penetration, the autonomous charging and discharging behavior of users, if lacking effective guidance and management, may instead pose new challenges and risks to the safe and stable operation of the power distribution network. When a large number of users perform unorderly and irrational energy storage charging and discharging, it will have adverse effects on voltage stability, power flow distribution, safe operation of equipment, etc. of the power distribution network. For example, user centralized charging may cause local voltage to drop sharply, and user centralized discharging may cause reverse power flow or even line overload, in addition, user unorderly charging and discharging also increases power loss of the power distribution network, reduces system operation efficiency, and may compete with distributed photovoltaic resources, leading to difficulty in renewable energy consumption, which seriously threatens the reliability and power quality of the power distribution network. Therefore, in a power distribution network with high user-side energy storage penetration, it is necessary to dynamically determine the resource scheduling scheme of the power distribution network according to user behavior to effectively protect the power distribution network.

[0003] To cope with the challenges brought by user-side energy storage access, existing technologies mainly focus on direct load control, price-guided demand-side response, and user behavior modeling and scheduling strategies based on traditional game theory. Direct load control strategies can achieve centralized optimization and scheduling, but have poor user autonomy, are difficult to be widely accepted by users, and have high implementation costs. Price-guided demand-side response strategies rely on price signals to indirectly guide user behavior, but the influence of price signals on user behavior is uncertain and lagging, making it difficult to achieve precise control, and users' responses to prices may not be completely rational, resulting in limited effectiveness of price guidance. Traditional game theory-based methods attempt to model and predict user behavior, but such methods usually require pre-assumptions of user behavior motivation and utility function, while actual user behavior motivation is complex and diverse, making it difficult to accurately model, and deviations in preset models will directly affect the effectiveness of scheduling strategies. Therefore, related technologies have low prediction accuracy of user behavior, resulting in low accuracy of the determined resource scheduling scheme of the power distribution network when determining the resource scheduling scheme of the power distribution network according to user behavior.

[0004] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0005] The embodiment of the present application provides a method and device for determining resource scheduling scheme of power distribution network, and electronic equipment, so as to at least solve the technical problem that the related art determines the resource scheduling scheme of the power distribution network based on the load control strategy or the electricity price to predict the user behavior, and the accuracy of the determined resource scheduling scheme is low.

[0006] According to an aspect of the embodiment of the present application, a method for determining resource scheduling scheme of power distribution network is provided, comprising: obtaining state information of a user to obtain target state information, wherein the target state information comprises state of charge of energy storage equipment held by the user at a current time, electricity price of a target power distribution network at the current time, and the target power distribution network is used for charging and discharging operation of the energy storage equipment; processing the target state information by an inverse reinforcement learning model to obtain an action corresponding to the target state information, wherein the action refers to charging and discharging power of the energy storage equipment held by the user at the current time; processing power grid state information of the target power distribution network and the action corresponding to the target state information by a target discriminator to obtain a resource scheduling scheme of the target power distribution network, wherein the target discriminator adopts a deep Q network algorithm.

[0007] Further, the method for determining resource scheduling scheme of power distribution network further comprises: training an upper framework of an initial inverse reinforcement learning model according to historical charging and discharging data of the user to obtain a trained upper framework and a charging and discharging reward function, wherein the charging and discharging reward function is learned by the upper framework during the training process; training a lower framework of the initial inverse reinforcement learning model according to the charging and discharging reward function, user action space and user state space to obtain a trained lower framework; and determining the inverse reinforcement learning model according to the trained upper framework and the trained lower framework.

[0008] Further, the method for determining the resource scheduling scheme of the power distribution network further comprises: determining a user state sequence and a user action sequence according to historical charging and discharging data of the user, wherein the user state sequence comprises state of charge of the energy storage device held by the user at a plurality of historical time points and electricity price of the target power distribution network at the plurality of historical time points, and the user action sequence comprises charging and discharging power of the energy storage device held by the user at the plurality of historical time points; taking the user state sequence and the user action sequence as a user trajectory, calculating a feature expectation of the user trajectory through the upper layer framework to obtain a first feature expectation, and determining a sample trajectory through the agent strategy in the upper layer framework to calculate a feature expectation of the sample trajectory to obtain a second feature expectation; updating a weight vector in the charging and discharging reward function based on a difference between the first feature expectation and the second feature expectation, and repeating the steps of determining the sample trajectory through the agent strategy in the upper layer framework to update the weight vector until a preset iteration condition of the upper layer framework is reached, to obtain the trained upper layer framework and the charging and discharging reward function, if the preset iteration condition is not reached.

[0009] Further, the method for determining the resource scheduling scheme of the power distribution network further comprises: constructing a user state space according to state of charge of the energy storage device held by the user and electricity price of the target power distribution network; constructing a user action space according to charging and discharging power of the energy storage device held by the user; and training the lower layer framework of the initial inverse reinforcement learning model according to the charging and discharging reward function, the user action space and the user state space to obtain the trained lower layer framework.

[0010] Further, the method for determining the resource scheduling scheme of the power distribution network further comprises: generating sample actions of the user according to sample state information of the user through the lower layer framework in the inverse reinforcement learning model; constructing a power grid state space based on the sample actions of the user and a power grid state of a simulation environment corresponding to the target power distribution network; constructing a power grid action space based on the resource scheduling scheme of the target power distribution network; and training an initial discriminator based on the power grid reward function, the power grid state space and the power grid action space to obtain the target discriminator.

[0011] Further, the method for determining the resource scheduling scheme of the power distribution network further comprises: obtaining a current power grid state of a simulation environment of the target power distribution network according to a power grid state space; generating an experience tuple according to the current power grid state, a power grid action space and the simulation environment through an initial discriminator, wherein the experience tuple comprises the current power grid state, a current power grid action, a reward obtained by executing the current power grid action on the current power grid state, and a next power grid state obtained by executing the current power grid action on the current power grid state; storing the experience tuple to an experience replay pool, sampling the updated experience replay pool to obtain a target experience tuple; determining a loss function value according to the target experience tuple and a power grid reward function, and updating parameters in the initial discriminator according to the loss function value; in the case where a preset iteration condition of the initial discriminator is not reached, determining the next power grid state in the experience tuple as an updated current power grid state of the simulation environment, and repeatedly executing the step of generating the experience tuple according to the updated current power grid state until the preset iteration condition of the initial discriminator is reached, to obtain a target discriminator.

[0012] Further, the method for determining the resource scheduling scheme of the power distribution network further comprises: determining a behavior characteristic of a user according to historical load data of a storage device held by the user and charge-discharge records of the storage device; determining an adjustment demand characteristic of a target power distribution network according to a power grid voltage deviation, a line overload degree value and a photovoltaic output prediction deviation of the target power distribution network at a target time; processing the behavior characteristic and the adjustment demand characteristic through a target neural network model to obtain a sample resource scheduling scheme of the target power distribution network; determining a sample power grid state of the target power distribution network based on the behavior characteristic and the adjustment demand characteristic, and performing expansion processing on an experience replay pool of the initial discriminator based on the sample resource scheduling scheme and the sample power grid state.

[0013] According to another aspect of the embodiment of the present application, a device for determining a resource scheduling scheme of a power distribution network is also provided, comprising: an obtaining module, configured to obtain state information of a user to obtain target state information, wherein the target state information comprises a state of charge of a storage device held by the user at a current time and a price of electricity of a target power distribution network at the current time, and the target power distribution network is used for charge-discharge operation of the storage device; a first processing module, configured to process the target state information through an inverse reinforcement learning model to obtain an action corresponding to the target state information, wherein the action refers to charge-discharge power of the storage device held by the user at the current time; and a second processing module, configured to process power grid state information of the target power distribution network and the action corresponding to the target state information through a target discriminator to obtain a resource scheduling scheme of the target power distribution network, wherein the target discriminator adopts a deep Q network algorithm.

[0014] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program, wherein the computer program is configured to execute the method for determining the resource scheduling scheme of the power distribution network when executed.

[0015] According to another aspect of the embodiments of the present application, an electronic device is also provided, and the electronic device comprises one or more processors, and a memory configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement a program configured to execute the method for determining the resource scheduling scheme of the power distribution network.

[0016] In the embodiments of the present application, the user behavior is predicted based on the inverse reinforcement learning model, and the resource scheduling scheme of the target power distribution network is determined based on the predicted user behavior. The state information of the user is obtained to obtain target state information, and then the target state information is processed by the inverse reinforcement learning model to obtain the action corresponding to the target state information. The power grid state information of the target power distribution network and the action corresponding to the target state information are processed by the target discriminator to obtain the resource scheduling scheme of the target power distribution network. The target state information includes the state of charge of the energy storage device held by the user at the current time and the electricity price of the target power distribution network at the current time. The action refers to the charging and discharging power of the energy storage device held by the user at the current time. The target discriminator adopts a deep Q network algorithm.

[0017] In the above process, the inverse reinforcement learning model can inversely deduce the potential behavior motivation of the user from the historical data of the user in the training process to realize unsupervised mining of the behavior motivation of the user. Therefore, the target state information is processed by the inverse reinforcement learning model to obtain the action corresponding to the target state information, which can realize accurate prediction of the user behavior. The power grid state information of the target power distribution network and the action corresponding to the target state information are processed by the target discriminator to obtain the resource scheduling scheme of the target power distribution network, which realizes determination of the resource scheduling scheme of the power distribution network based on the predicted user behavior with high accuracy, thereby improving the accuracy of the determined resource scheduling scheme.

[0018] Therefore, the scheme provided in the present application achieves the purpose of predicting the user behavior based on the inverse reinforcement learning model and determining the resource scheduling scheme of the target power distribution network based on the predicted user behavior, thereby realizing the technical effect of improving the accuracy of the determined resource scheduling scheme, and further solving the technical problem that the related art predicts the user behavior based on the load control strategy or the electricity price guide in the power distribution network with high user-side energy storage penetration, determines the resource scheduling scheme of the power distribution network according to the user behavior, and the accuracy of the determined resource scheduling scheme is low. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0020] Figure 1 is a schematic diagram of an optional method for determining a resource scheduling scheme of a power distribution network according to an embodiment of the application;

[0021] Figure 2 is a schematic diagram of an optional training of a reinforcement learning model according to an embodiment of the application;

[0022] Figure 3 is a schematic diagram of an optional device for determining a resource scheduling scheme of a power distribution network according to an embodiment of the application;

[0023] Figure 4 is a schematic diagram of an optional electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0024] In order to make the person skilled in the art better understand the application scheme, the technical scheme in the embodiment of the application will be described clearly and completely below in conjunction with the drawings in the embodiment of the application. Obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily mean a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of the relevant region, and provide corresponding operation portal for user to choose authorization or refusal.

[0027] Embodiment 1

[0028] According to the embodiment of the present application, an embodiment of a method for determining a resource scheduling scheme of a power distribution network is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0029] Figure 1 is a schematic diagram of an optional method for determining a resource scheduling scheme of a power distribution network according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0030] Step S101, obtaining the state information of the user to obtain the target state information, wherein the target state information includes the state of charge of the energy storage device held by the user at the current time, the electricity price of the target power distribution network at the current time, and the target power distribution network for charging and discharging operation of the energy storage device.

[0031] Optionally, electronic devices, application systems, servers and other devices can be used as the execution subject of the present application. In the present embodiment, the target processing system is used as the execution subject of the present application.

[0032] Optionally, the target state information can be the state information of a single user, or the statistical value of the state information of all users in a certain region corresponding to the target power distribution network, for example, the average value of the state of charge of all users in the region is calculated to obtain the average value as the final state information to be used, or the target state information can be the statistical value of the state information of all users corresponding to the target power distribution network.

[0033] Step S102, processing the target state information through an inverse reinforcement learning model to obtain an action corresponding to the target state information, wherein the action refers to the charging and discharging power of the energy storage device held by the user at the current time.

[0034] Optionally, the inverse reinforcement learning is a machine learning method for inferring a potential reward function from exemplary expert behavior (in this application, the historical charging and discharging behavior of the user) in the training process, and then guiding the production behavior strategy (i.e., the charging and discharging power of the energy storage device). In the actual application process, the inverse reinforcement learning model can process the target state information based on the learned knowledge to obtain the action corresponding to the target state information, that is, the charging and discharging power of the energy storage device held by the user at the current time.

[0035] Optionally, if the target state information is the state information of a single user, the output action refers to the charging and discharging power of the energy storage device held by the single user at the current time, and if the target state information is determined based on the state information of multiple users, the output action can refer to the sum of the charging and discharging power of the energy storage devices held by the multiple users at the current time.

[0036] In step S103, the target discriminator is used to process the grid state information of the target power distribution network and the action corresponding to the target state information to obtain a resource scheduling scheme of the target power distribution network, wherein the target discriminator adopts a deep Q network algorithm.

[0037] Optionally, the target discriminator adopts a deep Q network algorithm, which is used to learn and generate a resource scheduling scheme on the grid side.

[0038] Optionally, the grid state information includes similar data of the real-time running state of the target power distribution network, such as including ordinary load data, photovoltaic output data, voltage level, etc.

[0039] Optionally, the target discriminator generates a resource scheduling scheme based on the user behavior prediction and the current state of the grid to balance the grid load, ensure voltage stability, and smooth power fluctuations, etc. For example, the resource scheduling scheme can include the charging and discharging power of each energy storage device in the target power distribution network.

[0040] Optionally, after the resource scheduling scheme is determined, the target power distribution network is scheduled according to the resource scheduling scheme. If the target state information is the state information of a single user, the charging and discharging power of the energy storage device in the resource scheduling scheme corresponding to each user is summarized to obtain a target resource scheduling scheme. For example, for each energy storage device in each resource scheduling scheme, the difference between the current charging and discharging power of the energy storage device and the charging and discharging power of the energy storage device in the resource scheduling scheme is calculated to obtain the corresponding difference value of the energy storage device in the scheduling scheme. Then, the difference values of the energy storage device in all scheduling schemes are summed, and the sum is added to the current charging and discharging power of the energy storage device, so that the charging and discharging power of the energy storage device in the target resource scheduling scheme is obtained, and the resource scheduling is performed based on the target resource scheduling scheme. If the target state information is determined based on the state information of multiple users, the target processing system can directly schedule the target power distribution network based on the resource scheduling scheme of the target power distribution network.

[0041] Based on the schemes defined in steps S101 to S103, it can be known that in the embodiments of the present application, the user behavior is predicted based on the inverse reinforcement learning model, and the resource scheduling scheme of the target power distribution network is determined based on the predicted user behavior. The state information of the user is obtained to obtain the target state information, and then the target state information is processed by the inverse reinforcement learning model to obtain the action corresponding to the target state information. The power grid state information of the target power distribution network and the action corresponding to the target state information are processed by the target discriminator to obtain the resource scheduling scheme of the target power distribution network. The target state information includes the state of charge of the energy storage device held by the user at the current time, and the price of the target power distribution network at the current time. The action refers to the charging and discharging power of the energy storage device held by the user at the current time. The target discriminator uses a deep Q network algorithm.

[0042] It is easy to note that in the above process, since the inverse reinforcement learning model can unsupervisedly infer the potential behavior motivation of the user from the historical data of the user in the training process, the unsupervised mining of the behavior motivation of the user is realized. Therefore, by designing to process the target state information using the inverse reinforcement learning model to obtain the action corresponding to the target state information, the accurate prediction of the user behavior can be realized. By processing the power grid state information of the target power distribution network and the action corresponding to the target state information using the target discriminator to obtain the resource scheduling scheme of the target power distribution network, the resource scheduling scheme of the power distribution network is determined based on the predicted behavior of the user with high accuracy, thereby improving the accuracy of the determined resource scheduling scheme.

[0043] It can be seen that the scheme provided in the application achieves the purpose of predicting user behavior based on the inverse reinforcement learning model and determining the resource scheduling scheme of the target power distribution network based on the predicted user behavior, thereby achieving the technical effect of improving the accuracy of the determined resource scheduling scheme, and further solving the technical problem that in the related art, in a power distribution network with high user-side energy storage penetration, user behavior is predicted based on a load control strategy or a price guide, and a resource scheduling scheme of the power distribution network is determined according to the user behavior, resulting in low accuracy of the determined resource scheduling scheme.

[0044] In an optional embodiment, the inverse reinforcement learning model adopts a double-layer inverse reinforcement learning framework, wherein, Figure 2 is a training schematic diagram of an optional inverse reinforcement learning model according to an embodiment of the application, as Figure 2 shown, the target processing system can obtain the inverse reinforcement learning model through the following steps:

[0045] In step S201, the upper layer framework of the initial inverse reinforcement learning model is trained according to the user historical charging and discharging data, and a trained upper layer framework and a charging and discharging reward function are obtained, wherein the charging and discharging reward function is learned by the upper layer framework during the training process.

[0046] Optionally, before training, the target processing system can first collect data. For example, collect power distribution network topology data, including node information (number, type, geographic location, voltage level), line information (first and last nodes, line length, line parameters: resistance, reactance, susceptance), transformer information (first and last nodes, transformation ratio, impedance), etc.; collect historical load, photovoltaic output data information of nodes; collect user-side energy storage historical charging and discharging data, which includes but is not limited to the state of charge of the energy storage equipment held by the user at multiple historical time points, the price of the target power distribution network at multiple historical time points, and the charging and discharging power of the energy storage equipment held by the user at multiple historical time points. Among them, the aforementioned node can refer to a power generation or power consumption area, for example, the node can be a certain transformer substation, power plant, etc., and the transformer can be a line between nodes or a node.

[0047] After data collection is completed, data processing is performed, such as data cleaning, standardization, format conversion, etc. using common data processing methods. Then, data set division is performed. For example, the data set is divided into a training set, a validation set and a test set in chronological order, and the proportion can be set to 70%, 15% and 15%. The inverse reinforcement learning model is trained based on the training set, the validation set and the test set obtained by division.

[0048] Optionally, in the embodiment, reverse modeling and prediction of user-side energy storage behavior logic are implemented through a double-layer inverse reinforcement learning framework. The upper-layer framework of the initial inverse reinforcement learning refers to a model based on a maximum entropy inverse reinforcement learning algorithm, which is used to deduce a potential reward function (i.e., a charging and discharging reward function) behind the user charging and discharging behavior. The charging and discharging reward function can reflect the internal motivation and preference of the user charging and discharging behavior.

[0049] In step S202, the lower-layer framework of the initial inverse reinforcement learning model is trained according to the charging and discharging reward function, a user action space, and a user state space, to obtain a trained lower-layer framework.

[0050] Optionally, the lower-layer framework of the initial inverse reinforcement learning model is a user behavior dynamic modeling framework based on Markov game theory. The user action space can be constructed according to the charging and discharging power of the energy storage equipment held by the user, and the user state space can be constructed according to the state of charge of the energy storage equipment held by the user and the electricity price of the target power grid. The charging and discharging reward function used in the training of the lower-layer framework is the result obtained after the training of the upper-layer framework, which is used to guide the learning of the lower-layer framework.

[0051] In step S203, the inverse reinforcement learning model is determined according to the trained upper-layer framework and the trained lower-layer framework.

[0052] Optionally, after the upper-layer framework and the lower-layer framework in the initial inverse reinforcement learning model are trained, the current initial inverse reinforcement learning model is determined as the inverse reinforcement learning model, i.e., the inverse reinforcement learning model is determined according to the trained upper-layer framework and the trained lower-layer framework.

[0053] It should be noted that the upper-layer of the double-layer inverse reinforcement learning framework adopts the maximum entropy inverse reinforcement learning algorithm to deduce the potential reward function of the user from the historical charging and discharging data of the user. The reward function can effectively reflect the behavior motivation and preference of the user, and provide a basis for subsequent user behavior prediction. The lower-layer of the framework is based on Markov game theory, which models the interaction process between the user and the power grid as a Markov game, fully considers the mutual influence between the user behavior and the power grid state, and realizes dynamic modeling and prediction of the user behavior, so as to realize accurate modeling and prediction of the user-side energy storage behavior.

[0054] In one optional embodiment, during the process of training the upper-layer framework of the initial inverse reinforcement learning model based on the user's historical charging and discharging data to obtain the trained upper-layer framework and the charging and discharging reward function, the target processing system can determine the user state sequence and user action sequence based on the user's historical charging and discharging data. The user state sequence includes the state of charge of the user's energy storage device at multiple historical moments and the electricity price of the target distribution network at multiple historical moments. The user action sequence includes the charging and discharging power of the user's energy storage device at multiple historical moments. The user state sequence and user action sequence are used as the user trajectory. The upper-layer framework calculates the feature expectation of the user trajectory to obtain a first feature expectation. The intelligent agent strategy in the upper-layer framework determines the sampling trajectory, and the feature expectation of the sampling trajectory is calculated to obtain a second feature expectation. The weight vector in the charging and discharging reward function is updated based on the difference between the first and second feature expectations. If the preset iteration condition of the upper-layer framework is not met, the step of determining the sampling trajectory through the intelligent agent strategy in the upper-layer framework to update the weight vector is repeated until the preset iteration condition is met, resulting in the trained upper-layer framework and the charging and discharging reward function.

[0055] Optionally, at the upper layer, a maximum entropy inverse reinforcement learning algorithm is used to deduce the user's potential charging and discharging reward function R from the user's historical charging and discharging data. u (s u ,a u Optionally, the upper-level framework can be trained through the following steps:

[0056] Step S31: Define user behavior data, including user state sequences. and user action sequence User status includes: time t, electricity price P at time t. price,t The State of Charge (SOC) of the user's energy storage device at time t u,t User actions include the charging and discharging power P of the user's energy storage device at time t. ES,u,t The user state sequence and user action sequence are determined based on the user's historical charging and discharging data, with time t corresponding to a historical time.

[0057] Step S32: Define the form of the reward function. Assume the reward function is a linear form of the state and action. Among them, w u Let s be the weight vector of the reward function, and state s be... u and action a u eigenvectors φ(s) u ,a u = [φ1, φ2, φ3] TThe feature vector can be defined as an electricity cost item, an energy storage degradation cost, and a discharging reward item in a high SOC state, and the calculation manner of the feature vector is learned by the model during a training process.

[0058] Step S33: maximum entropy inverse reinforcement learning algorithm process

[0059] Initialization: randomly initialize the reward function weight vector w u .

[0060] Policy learning: based on the current reward function R u , learn the user's optimal policy using the Soft Actor-Critic (SAC) algorithm, that is, learn a policy similar to the user's historical charging and discharging behavior. This process is to update the agent policy u (a u |s u ). The agent policy will be adjusted according to the current reward function weight vector to maximize the cumulative reward. The SAC algorithm aims to maximize the expected cumulative reward and policy entropy.

[0061] Gradient update: calculate the maximum entropy IRL gradient and update the reward function weight vector w u :

[0062]

[0063] wherein, represents the feature expectation of the user trajectory, that is, the first feature expectation, represents the feature expectation of the sampled trajectory, that is, the second feature expectation, wherein the sampled trajectory is obtained by the interaction between the agent policy and the environment, and the environment refers to the environment simulating the state of the user's energy storage device, represents the difference between the first feature expectation and the second feature expectation, that is, the gradient. After obtaining the gradient, the upper framework can multiply the gradient by a preset learning rate to obtain a first value, and then add the first value to the current weight vector in the charging and discharging reward function to obtain an updated weight vector, thereby updating the weight vector in the charging and discharging reward function.

[0064] Iteration: repeat the steps of policy learning and gradient update until the preset iteration condition of the upper framework is reached. The preset iteration condition of the upper framework can be algorithm convergence or reaching a preset number of iterations.

[0065] It should be noted that by the above method, the upper framework is effectively trained, and the accuracy of the obtained charging and discharging reward function is improved.

[0066] In an optional embodiment, the lower framework of the initial inverse reinforcement learning model adopts a deep Q network algorithm, wherein in the process of training the lower framework of the initial inverse reinforcement learning model according to the charging and discharging reward function, the user action space and the user state space, the target processing system can construct the user state space according to the state of charge of the energy storage device held by the user and the electricity price of the target power distribution network; construct the user action space according to the charging and discharging power of the energy storage device held by the user; and train the lower framework of the initial inverse reinforcement learning model according to the charging and discharging reward function, the user action space and the user state space to obtain the trained lower framework.

[0067] Optionally, the lower framework is based on Markov game theory to model the interaction process of the user and the power grid, and the lower framework uses a DQN (Deep Q-Network) algorithm to solve the optimal strategy of the user. The target processing system can construct the user state space according to the state of charge of the energy storage device held by the user and the electricity price of the target power distribution network, and construct the user action space according to the charging and discharging power of the energy storage device held by the user, so as to train the lower framework of the initial inverse reinforcement learning model according to the charging and discharging reward function output by the upper framework and the user action space and the user state space constructed as described above based on the traditional DQN algorithm, to obtain the trained lower framework. In the training process, the lower framework combines the charging and discharging reward function, the user action space and the user state space to learn the Q value of the optimal charging and discharging strategy of the user in different states. Through repeated iteration, the parameters of the Q network in the lower framework are continuously adjusted, so that the lower framework can predict the optimal action of the user in a specific state, i.e., the charging and discharging power.

[0068] It should be noted that by the above method, the lower framework is effectively trained, so that the trained lower framework can accurately predict the charging and discharging strategy of the user's energy storage device, i.e., the accuracy of predicting the user's behavior is improved.

[0069] In an optional embodiment, the target processing system can obtain the target discriminator by the following method: generating a sample action of a user by a lower framework in an inverse reinforcement learning model according to sample state information of the user; constructing a power grid state space based on the sample action of the user and the state of the power grid of a simulation environment corresponding to the target power distribution network; constructing a power grid action space based on a resource scheduling scheme of the target power distribution network; and training an initial discriminator based on a power grid reward function, the power grid state space and the power grid action space to obtain the target discriminator.

[0070] Optionally, the lower framework in the inverse reinforcement learning model utilizes the input sample state information of the user (the sample state information is in the same data form as the aforementioned state information) to generate a set of representative user sample actions, i.e., the charging and discharging power prediction of the user energy storage device in these states. This process simulates the decision-making behavior of the user under specific environmental conditions, providing necessary user behavior data for subsequent construction of the power grid state space and training of the discriminator.

[0071] Optionally, the simulation environment is used to simulate the operating characteristics of the target power distribution network. In the process of constructing the simulation environment for simulating the operating characteristics of the target power distribution network, the target processing system can construct graph structure data according to the connection relationship between nodes in the target power distribution network; simulate real-time changes of the load of the target power distribution network according to historical load data or reference daily load curve of the target power distribution network to obtain load fluctuation simulation data; simulate real-time output power of renewable resources in the target power distribution network according to output characteristics of the renewable resources to obtain renewable resource output simulation data; build a mathematical model according to the energy storage and power distribution network system operating characteristics in the target power distribution network; and construct the simulation environment according to the graph structure data, the load fluctuation simulation data, the renewable resource output simulation data, and the mathematical model. The grid state of the simulation environment reflects the operating state of the target power distribution network at a certain moment, including but not limited to at least one of the following: ordinary load, photovoltaic, etc. basic power, voltage level of the target power distribution network, load change, line capacity, etc. For example, the grid state of the simulation environment can include ordinary load, photovoltaic, etc. basic power information of the target power distribution network.

[0072] Optionally, the target processing system can construct a grid state space based on the sample actions of the user and the grid state of the corresponding simulation environment of the target power distribution network. And based on the resource scheduling scheme of the target power distribution network, a grid action space is constructed.

[0073] Optionally, the grid reward function r g may be as follows:

[0074] r g = w loss r loss + w v r v + w es r es

[0075] wherein w loss , w v , w es are loss penalty weight, voltage offset penalty weight, and energy storage depreciation penalty weight, respectively; r loss , r v , r es are loss penalty, voltage offset penalty, and energy storage depreciation penalty, respectively, and the expression is as follows:

[0076] r loss loss P loss

[0077]

[0078] r es deg P ES

[0079] wherein C loss , C deg are the loss penalty coefficient and the energy storage depreciation coefficient respectively; P loss is the active power loss; V i is the voltage of node i; V ref is the reference voltage amplitude, which is set to 1.0 p.u.

[0080] Optionally, the initial discriminator is trained based on the power grid reward function, the power grid state space and the power grid action space, so that the initial discriminator learns a robust power grid side distributed resource scheduling strategy in the training process, and the target discriminator is obtained.

[0081] For example, the target discriminator is implemented based on a DQN-MLP model, and the input and output data of the trained target discriminator are as follows:

[0082] Input: power grid state (including basic power information such as ordinary load, photovoltaic, etc.), charging and discharging power of user side energy storage device.

[0083] Output: scheduling scheme of power grid side distributed resources, such as:

[0084] A={P ESS,1 ,P ESS,2 ,...,P ESS,n}

[0085] wherein P ESS,i is the charging and discharging power of the i-th power grid side energy storage device, and n is the total number of power grid side energy storage devices.

[0086] Optionally, the DQN-MLP model structure refers to using a three-layer fully connected MLP network as the Q network, and the data processing method is as follows:

[0087] Input layer: receives the power grid state vector s g and the charging and discharging power P u of the user side energy storage device as input, and splices them into an input vector x=[s g ; P u ].

[0088] Hidden layer: three fully connected layers are used as the hidden layer of the Q network:​​​

[0089] First layer: 512 neurons, activation function is ReLU.

[0090] Second layer: 512 neurons, activation function is ReLU.

[0091] Third layer: 512 neurons, activation function is ReLU.

[0092] The mathematical expression is as follows:

[0093] h (1) = ReLU(W (1) x + b (1) )

[0094] h (2) = ReLU(W (2) h (1) + b (2) )

[0095] h (3) = ReLU(W (3) h (2) + b (3) )

[0096] wherein W (i) , b (i) are the weight matrix and bias vector of the i-th layer, respectively.

[0097] Output layer: fully connected layer, output the scheduling scheme of the distributed resource on the power grid side and the corresponding Q value of the action of the power grid, and the activation function is a linear function.

[0098] It should be noted that by the above method, the action space, state space and reward function relied on by the target discriminator are effectively defined, thereby the training effect of the target discriminator can be effectively improved, and the accuracy of the resource scheduling scheme output by the target discriminator in the application process is improved.

[0099] In an optional embodiment, in the process of training the initial discriminator based on the grid reward function, the grid state space and the grid action space to obtain the target discriminator, the target processing system can obtain a current grid state of a simulation environment of the target power distribution network according to the grid state space; generate an experience tuple according to the current grid state, the grid action space and the simulation environment by the initial discriminator, wherein the experience tuple includes the current grid state, a current grid action, a reward obtained by executing the current grid action on the current grid state, and a next grid state obtained by executing the current grid action on the current grid state; store the experience tuple into an experience replay pool, sample the updated experience replay pool to obtain a target experience tuple; determine a loss function value according to the target experience tuple and the grid reward function, and update the parameters in the initial discriminator according to the loss function value; in the case where a preset iteration condition of the initial discriminator is not reached, determine the next grid state in the experience tuple as an updated current grid state of the simulation environment, and repeatedly execute the step of generating the experience tuple according to the updated current grid state until the preset iteration condition of the initial discriminator is reached to obtain the target discriminator.

[0100] For example, the following steps can be performed to train the target discriminator:

[0101] Step S41, action selection. A current grid state s g of a simulation environment of the target power distribution network is obtained according to a grid state space g , and a current grid action a g is selected according to the current grid state s g . The sample action of the user in the grid state is simulated by a lower framework in the inverse reinforcement learning model.

[0102] Step S42, environment interaction. The current grid action a g is applied to the power distribution network simulation environment, and the environment returns a next grid state s g ′ and a grid-side reward r g .

[0103] Step S43, experience storage. An experience tuple (s g , a g , r g , s g ') is stored in a grid-side experience replay pool D G .

[0104] Step S44, experience replay and network update. A batch of experience samples are sampled from the experience replay pool as target experience tuples, and a TD target value is calculated according to the target experience tuples:

[0105]

[0106] wherein y jdenotes a TD target value, γ denotes a discount factor, r g,j denotes a grid reward function, denotes a maximum Q value obtained by performing Q value calculation on the target experience tuple by the target Q network. The gradient error is calculated using the TD target value and the Q network prediction value, and the Q network parameter θ is updated using the gradient descent method to minimize the square of the gradient error, and the loss function is:

[0107]

[0108] wherein, denotes a loss function value, denotes a Q value obtained by performing Q value calculation on the target experience tuple by the Q network, and B is the number of samples obtained from the experience replay buffer. Wherein, after the loss function value is determined, the network parameter value θ of the Q network can be updated according to the loss function value using the Adam optimization algorithm. Wherein, the Q network parameter θ is copied to the target Q network parameter θ' <- θ every certain number of steps to maintain the stability of the target Q network.

[0109] Step S45, in the case where the preset iteration condition of the initial discriminator is not reached, the next grid state in the experience tuple is determined as the updated current grid state of the simulation environment, and steps S41-S44 are repeatedly executed until the preset iteration condition of the initial discriminator is reached, and the target discriminator is obtained. Wherein, the preset iteration condition can be that the iteration reaches a certain number of steps, or the loss function value is less than a preset value.

[0110] It should be noted that, by the above-mentioned manner, the effective training of the target discriminator is realized, so as to improve the accuracy of the resource scheduling scheme output by the target discriminator in the application process.

[0111] In an optional embodiment, the method for determining the resource scheduling scheme of the power distribution network further comprises: determining the behavior characteristics of the user according to the historical load data of the energy storage equipment held by the user and the charge and discharge records of the energy storage equipment; determining the adjustment demand characteristics of the target power distribution network according to the grid voltage deviation, the line overload degree value and the photovoltaic output prediction deviation of the target power distribution network at the target time; processing the behavior characteristics and the adjustment demand characteristics by the target neural network model to obtain a sample resource scheduling scheme of the target power distribution network; determining a sample grid state of the target power distribution network based on the behavior characteristics and the adjustment demand characteristics, and performing expansion processing on the experience replay pool of the initial discriminator based on the sample resource scheduling scheme and the sample grid state.

[0112] Optionally, the target neural network model is designed as a policy fusioner based on a time convolution network (TCN) to fuse long-term user behavior rules with short-term power grid regulation requirements. For example, a TCN-based policy fusioner F is constructed, the input of which is long-term user behavior rule features and short-term power grid regulation requirement features, and the output of which is a sample resource scheduling scheme of the target distribution network. The long-term user behavior rule features (i.e., user behavior features) refer to the user behavior patterns extracted through long-term user behavior data analysis, such as periodic charging and discharging rules, habitual preferences, etc. These features include historical load data, charging and discharging records of energy storage devices, etc., which can be extracted by a long short-term memory network (LSTM) autoencoder. The short-term power grid regulation requirement features (i.e., target distribution network regulation requirement features) include state information of the target distribution network, such as power grid load fluctuation, price change, photovoltaic power output, current state of charge of energy storage devices, etc. These data usually have high time sensitivity and need to quickly respond to the immediate requirements of power grid scheduling. Optionally, the aforementioned target time can be the current time.

[0113] Optionally, the input of the policy fusioner is long-term user behavior rule features h long and short-term power grid regulation requirement features h short . The former is extracted from long-term recorded data such as historical charging and discharging data and user battery SOC, while the latter mainly involves real-time data of the state of the target distribution network, including power grid voltage deviation, line overload degree, photovoltaic output prediction deviation, etc. The output of the policy fusioner is a sample resource scheduling scheme of the target distribution network, which determines the charging and discharging power of the energy storage device.

[0114] Optionally, the advantage of TCN is that it can capture long-term dependencies in time series data and has strong causality, which is suitable for modeling time series data in this scenario. The core structure of the TCN model is an inflated causal convolution, the input of which is long-term user behavior rule features and short-term power grid regulation requirement features, and the output of which is a scheduling parameter on the grid side. Assuming that the input sequence is x TCN =[h long ,h short ], the TCN model can be processed by multiple convolution layers, and each layer has a different dilation factor, which can effectively expand the receptive field and capture long-term temporal dependencies. Assuming that the convolution kernel size is k and the dilation factor is d, the output of the convolution layer is:

[0115] y (l) =Conv d,k (y (l-1) ),l=1,2,...,L C

[0116] where y (l)is the output of the l-th layer, Conv is a convolution operation, and L is the total number of convolution layers. C is the total number of convolution layers.

[0117] The last layer of convolution is passed through an activation function (such as ReLU) to obtain the final sample resource scheduling scheme.

[0118] Optionally, the policy fusioner can be trained based on a contrastive learning method, and the training target is to minimize the error of the power grid scheduling policy, so that the output scheduling policy can effectively balance the stability of the power grid and the coordination of the user-side energy storage behavior. The training sample can be a sample user behavior feature and a sample power distribution network regulation demand feature, and the true label of the training sample can be a sample resource scheduling scheme corresponding to the sample power distribution network. In the training process, the Adam and gradient descent algorithms can be used to optimize the parameters of the TCN network, and the network weights are adjusted by minimizing the loss function, so as to continuously improve the accuracy and stability of the scheduling policy. Once the TCN model is trained, the fusioner can dynamically adjust the power grid scheduling policy according to real-time input data.

[0119] Optionally, after obtaining the sample resource scheduling scheme through the policy fusioner, the target processing system can determine the sample power grid state of the target power distribution network based on the behavior feature and the regulation demand feature, for example, determine the power grid state based on the regulation demand feature, determine the user's action at a certain time (i.e. the charging and discharging power of the user-side energy storage device) based on the behavior feature, and thus determine the sample power grid state according to the power grid state and the user's action. Optionally, the sample resource scheduling scheme is determined as the sample power grid action, and then the sample power grid state and the sample power grid action are input to the initial discriminator to determine the corresponding reward and the next state according to the sample power grid state and the sample power grid action through the initial discriminator, thereby obtaining a supplementary experience tuple, and storing the supplementary experience tuple to the experience replay pool to realize the expansion processing of the experience replay pool.

[0120] It should be noted that by the above-mentioned manner, the policy fusioner can extract the internal law and periodic characteristics of user behavior from long-term user behavior data, and dynamically determine the scheduling policy of the power grid side distributed resource in combination with the short-term power grid operating state and regulation demand, realize the effective fusion of user behavior law and power grid regulation demand, so as to realize the effective expansion of the experience pool and improve the training effect of the target discriminator.

[0121] In an optional embodiment, the target processing system can access various data of the power grid (such as power grid load, photovoltaic output, electricity price information, etc.) in real time to adjust the scheduling strategy, and send the scheduling strategy to the substation implementation module, which is responsible for implementing the distributed resource collaborative scheduling of the power grid side, and through the intelligent scheduling system of the substation, the distributed resources such as energy storage devices, electric vehicle charging facilities and photovoltaic are scheduled in real time to ensure the stable operation of the power grid. After receiving the scheduling instruction of the power grid scheduling master station, the substation implementation module executes the specific resource scheduling task, and through the control of the charge and discharge behavior of the energy storage device, the optimization of photovoltaic output scheduling, etc., the power grid load balance and voltage stability are ensured. The results after scheduling execution can be fed back to the target processing system for further optimization and adjustment of the strategy.

[0122] Therefore, the scheme provided in the application achieves the purpose of predicting user behavior based on the inverse reinforcement learning model and determining the resource scheduling scheme of the target power distribution network based on the predicted user behavior, thereby achieving the technical effect of improving the accuracy of the determined resource scheduling scheme, and further solving the technical problem that in the related art, in a power distribution network with high user-side energy storage penetration rate, user behavior is predicted based on a load control strategy or electricity price guidance, and a resource scheduling scheme of the power distribution network is determined according to the user behavior, resulting in low accuracy of the determined resource scheduling scheme.

[0123] Embodiment 2

[0124] According to the embodiment of the application, an embodiment of a device for determining a resource scheduling scheme of a power distribution network is provided, wherein Figure 3 is a schematic diagram of an optional device for determining a resource scheduling scheme of a power distribution network according to an embodiment of the application, as Figure 3 shown, the device comprises:

[0125] The acquisition module 301 is configured to acquire the state information of the user to obtain target state information, wherein the target state information includes the state of charge of the energy storage device held by the user at the current time, the electricity price of the target power distribution network at the current time, and the target power distribution network for the energy storage device to perform charge and discharge operation.

[0126] The first processing module 302 is configured to process the target state information by an inverse reinforcement learning model to obtain an action corresponding to the target state information, wherein the action refers to the charge and discharge power of the energy storage device held by the user at the current time.

[0127] The second processing module 303 is configured to process the power grid state information of the target power distribution network and the action corresponding to the target state information by a target discriminator to obtain a resource scheduling scheme of the target power distribution network, wherein the target discriminator adopts a deep Q network algorithm.

[0128] It should be noted that the above acquisition module 301, the first processing module 302 and the second processing module 303 correspond to steps S101 to S103 in the above embodiment, and the three modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above embodiment 1.

[0129] Optionally, the resource scheduling scheme determination apparatus of the power distribution network further comprises: a first training module configured to train an upper framework of an initial inverse reinforcement learning model according to historical charging and discharging data of a user to obtain a trained upper framework and a charging and discharging reward function, wherein the charging and discharging reward function is learned by the upper framework during the training process; a second training module configured to train a lower framework of the initial inverse reinforcement learning model according to the charging and discharging reward function, a user action space and a user state space to obtain a trained lower framework; and a first determination module configured to determine the inverse reinforcement learning model according to the trained upper framework and the trained lower framework.

[0130] Optionally, the first training module further comprises: a first determination sub-module configured to determine a user state sequence and a user action sequence according to the historical charging and discharging data of the user, wherein the user state sequence comprises state of charge of a storage device held by the user at a plurality of historical time points and electricity price of a target power distribution network at the plurality of historical time points, and the user action sequence comprises charging and discharging power of the storage device held by the user at the plurality of historical time points; a calculation sub-module configured to take the user state sequence and the user action sequence as a user trajectory, calculate a feature expectation of the user trajectory through the upper framework to obtain a first feature expectation, and determine a sampling trajectory through an agent policy in the upper framework to calculate a feature expectation of the sampling trajectory to obtain a second feature expectation; and a first training sub-module configured to update a weight vector in the charging and discharging reward function based on a difference between the first feature expectation and the second feature expectation, and repeatedly perform the steps of determining the sampling trajectory through the agent policy in the upper framework to update the weight vector until a preset iteration condition of the upper framework is reached to obtain the trained upper framework and the charging and discharging reward function, in a case where the preset iteration condition is not reached.

[0131] Optionally, the second training module further comprises: a first construction sub-module configured to construct a user state space according to state of charge of a storage device held by the user and electricity price of a target power distribution network; a second construction sub-module configured to construct a user action space according to charging and discharging power of the storage device held by the user; and a second training sub-module configured to train the lower framework of the initial inverse reinforcement learning model according to the charging and discharging reward function, the user action space and the user state space to obtain the trained lower framework.

[0132] Optionally, the resource scheduling scheme determination apparatus of the power distribution network further includes: a generation module configured to generate sample actions of the user according to sample state information of the user through a lower framework in the inverse reinforcement learning model; a first construction module configured to construct a power grid state space based on the sample actions of the user and a power grid state of a simulation environment corresponding to the target power distribution network; a second construction module configured to construct a power grid action space based on the resource scheduling scheme of the target power distribution network; and a third training module configured to train an initial discriminator based on a power grid reward function, the power grid state space and the power grid action space to obtain a target discriminator.

[0133] Optionally, the third training module further includes: an acquisition sub-module configured to acquire a current power grid state of the simulation environment of the target power distribution network according to the power grid state space; a generation sub-module configured to generate an experience tuple through the initial discriminator according to the current power grid state, the power grid action space and the simulation environment, wherein the experience tuple includes the current power grid state, a current power grid action, a reward obtained by executing the current power grid action on the current power grid state, and a next power grid state obtained by executing the current power grid action on the current power grid state; a processing sub-module configured to store the experience tuple to an experience replay pool and sample the updated experience replay pool to obtain a target experience tuple; an updating sub-module configured to determine a loss function value according to the target experience tuple and the power grid reward function, and update parameters in the initial discriminator according to the loss function value; and a third training sub-module configured to, in a case where a preset iteration condition of the initial discriminator is not reached, determine the next power grid state in the experience tuple as an updated current power grid state of the simulation environment, and repeatedly execute the step of generating the experience tuple according to the updated current power grid state until the preset iteration condition of the initial discriminator is reached to obtain the target discriminator.

[0134] Optionally, the resource scheduling scheme determination apparatus of the power distribution network further includes: a second determination module configured to determine a behavior feature of the user according to historical load data of an energy storage device held by the user and charge-discharge records of the energy storage device; a third determination module configured to determine an adjustment demand feature of the target power distribution network according to a power grid voltage deviation of the target power distribution network at a target time, a line overload degree value and a photovoltaic output prediction deviation; a third processing module configured to process the behavior feature and the adjustment demand feature through the target neural network model to obtain a sample resource scheduling scheme of the target power distribution network; and a fourth processing module configured to determine a sample power grid state of the target power distribution network based on the behavior feature and the adjustment demand feature, and perform expansion processing on an experience replay pool of the initial discriminator based on the sample resource scheduling scheme and the sample power grid state.

[0135] Embodiment 3

[0136] According to another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, having stored therein a computer program, wherein the computer program is arranged to, when executed, perform the method for determining the resource scheduling scheme of the power distribution network.

[0137] Embodiment 4

[0138] According to another aspect of the embodiments of the present application, there is also provided an electronic device, wherein, Figure 4 is a schematic diagram of an optional electronic device according to an embodiment of the present application, as Figure 4 shown, the electronic device comprises one or more processors; a memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement a program for running, wherein the program is arranged to, when executed, perform the method for determining the resource scheduling scheme of the power distribution network.

[0139] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0140] In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0141] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of units can be a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, and can be electrical or other forms.

[0142] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0143] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0144] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0145] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A method for determining a resource scheduling scheme of a power distribution network, characterized in that, The method comprises the following steps: obtaining state information of a user to obtain target state information, wherein the target state information comprises a state of charge of a storage device held by the user at a current time and a price of a target power distribution network at the current time, and the target power distribution network is used for charging and discharging operation of the storage device; processing the target state information through an inverse reinforcement learning model to obtain an action corresponding to the target state information, wherein the action refers to a charging and discharging power of the storage device held by the user at the current time; processing power grid state information of the target power distribution network and the action corresponding to the target state information through a target discriminator to obtain a resource scheduling scheme of the target power distribution network, wherein the target discriminator adopts a deep Q network algorithm.

2. The method of claim 1, wherein, The inverse reinforcement learning model adopts a double-layer inverse reinforcement learning framework, wherein the inverse reinforcement learning model is obtained through the following steps: training an upper layer framework of an initial inverse reinforcement learning model according to historical charging and discharging data of a user to obtain a trained upper layer framework and a charging and discharging reward function, wherein the charging and discharging reward function is learned by the upper layer framework during the training process; training a lower layer framework of the initial inverse reinforcement learning model according to the charging and discharging reward function, a user action space and a user state space to obtain a trained lower layer framework; determining the inverse reinforcement learning model according to the trained upper layer framework and the trained lower layer framework.

3. The method of claim 2, wherein, The method comprises the following steps: determining a user state sequence and a user action sequence according to the historical charging and discharging data of the user, wherein the user state sequence comprises states of charge of the storage device held by the user at a plurality of historical times and prices of the target power distribution network at the plurality of historical times, and the user action sequence comprises charging and discharging powers of the storage device held by the user at the plurality of historical times; calculating a feature expectation of a user trajectory through the upper layer framework by taking the user state sequence and the user action sequence as the user trajectory to obtain a first feature expectation, and determining a sampling trajectory through an agent policy in the upper layer framework to calculate a feature expectation of the sampling trajectory to obtain a second feature expectation; updating a weight vector in the charging and discharging reward function based on a difference between the first feature expectation and the second feature expectation, and repeatedly performing the step of updating the weight vector by determining a sampling trajectory through an agent policy in the upper layer framework until a preset iteration condition of the upper layer framework is reached to obtain the trained upper layer framework and the charging and discharging reward function.

4. The method of claim 2, wherein, The lower layer framework of the initial inverse reinforcement learning model adopts a deep Q network algorithm, wherein training the lower layer framework of the initial inverse reinforcement learning model according to the charging and discharging reward function, the user action space and the user state space to obtain the trained lower layer framework comprises the following steps: construct a user state space according to a state of charge of the energy storage device held by the user and a price of the target power distribution network; construct a user action space according to a charging and discharging power of the energy storage device held by the user; train a lower framework of the initial inverse reinforcement learning model according to the charging and discharging reward function, the user action space and the user state space to obtain a trained lower framework.

5. The method of claim 1, wherein, The target discriminator is obtained by: generating a sample action of the user according to sample state information of the user through the lower framework in the inverse reinforcement learning model; constructing a power grid state space based on the sample action of the user and a power grid state of a simulation environment corresponding to the target power distribution network; constructing a power grid action space based on a resource scheduling scheme of the target power distribution network; training an initial discriminator based on a power grid reward function, the power grid state space and the power grid action space to obtain the target discriminator.

6. The method of claim 5, wherein, Training an initial discriminator based on a power grid reward function, the power grid state space and the power grid action space to obtain the target discriminator includes: obtaining a current power grid state of the simulation environment of the target power distribution network according to the power grid state space; generating an experience tuple according to the current power grid state, the power grid action space and the simulation environment through the initial discriminator, wherein the experience tuple includes the current power grid state, a current power grid action, a reward obtained by executing the current power grid action on the current power grid state, and a next power grid state obtained by executing the current power grid action on the current power grid state; storing the experience tuple to an experience replay pool and sampling the updated experience replay pool to obtain a target experience tuple; determining a loss function value according to the target experience tuple and the power grid reward function, and updating parameters in the initial discriminator according to the loss function value; in the case where a preset iteration condition of the initial discriminator is not reached, determining the next power grid state in the experience tuple as an updated current power grid state of the simulation environment, and repeatedly executing the step of generating an experience tuple according to the updated current power grid state until the preset iteration condition of the initial discriminator is reached to obtain the target discriminator.

7. The method of claim 6, wherein, The method further includes: determining a behavior feature of the user according to historical load data of the energy storage device held by the user and charging and discharging records of the energy storage device; determining an adjustment demand feature of the target power distribution network according to a power grid voltage deviation, a line overload degree value and a photovoltaic output prediction deviation of the target power distribution network at a target time; processing the behavior feature and the adjustment demand feature through a target neural network model to obtain a sample resource scheduling scheme of the target power distribution network; determining a sample power grid state of the target power distribution network based on the behavior feature and the adjustment demand feature, and performing expansion processing on an experience replay pool of the initial discriminator based on the sample resource scheduling scheme and the sample power grid state.

8. A device for determining a resource scheduling scheme of a power distribution network, characterized in that, including: An acquisition module is configured to acquire state information of a user to obtain target state information, wherein the target state information includes a state of charge of an energy storage device held by the user at a current time and a price of a target power distribution network at the current time, and the target power distribution network is used to supply the energy storage device for charging and discharging operation. A first processing module is configured to process the target state information by using an inverse reinforcement learning model to obtain an action corresponding to the target state information, wherein the action refers to charging and discharging power of the energy storage device held by the user at the current time. A second processing module is configured to process power grid state information of the target power distribution network and the action corresponding to the target state information by using a target discriminator to obtain a resource scheduling scheme of the target power distribution network, wherein the target discriminator adopts a deep Q network algorithm.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein the computer program is configured to execute the method for determining the resource scheduling scheme of the power distribution network according to any one of claims 1 to 7 when running.

10. An electronic device, comprising: The electronic device includes one or more processors. A memory is configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement a program for running, wherein the program is configured to execute the method for determining the resource scheduling scheme of the power distribution network according to any one of claims 1 to 7 when running. The electronic device includes one or more processors. A memory is configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement a program for running, wherein the program is configured to execute the method for determining the resource scheduling scheme of the power distribution network according to any one of claims 1 to 7 when running.

Images