Mobile energy storage scheduling method and device based on knowledge constraint deep Q network

Abstract

The application provides a mobile energy storage scheduling method and device based on knowledge constraint deep expected Q network, studies the charging and discharging decision of mobile energy storage as a main load support component, includes the uncertainty of emergency resource space-time coupling, extreme weather scene, energy storage charging and discharging and space-time transfer constraint, and establishes a mobile energy storage elastic optimization scheduling model for load recovery; based on the mobile energy storage elastic optimization scheduling model, the emergency resource space-time coupling, the uncertainty of extreme weather scene, the energy storage charging and discharging and the space-time transfer constraint are integrated into the mathematical derivation of machine learning preprocessing, so as to establish an energy storage Markov decision process covering optimization constraint knowledge fusion; based on the energy storage Markov decision process, an intelligent agent is trained based on the elastic scheduling process of the deep expected Q network algorithm, so as to optimize the scheduling strategy of the mobile energy storage system. The application enhances the flexibility of the system, guarantees the calculation efficiency, and effectively solves the uncertainty problem.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of power system dispatching, and in particular to a mobile energy storage dispatching method and device based on a knowledge-constrained deep expected Q network. Background Art

[0002] In recent years, extreme weather events have occurred frequently, and the operation of power systems with a high proportion of renewable energy often faces extreme scenarios, resulting in load reduction. Therefore, it is imperative to take anti-underload measures for resilient power systems. With the continuous development of modern power systems, the scale and complexity of power grids have increased significantly. As a key component of the power system, the safe and stable operation of regional power grids is directly related to the reliability and economy of the entire power system. However, with the frequent occurrence of extreme weather events caused by climate change and the large-scale integration of new energy, the operating environment faced by the power grid has become more complex and unpredictable. In such highly uncertain extreme scenarios, traditional dispatching methods are often difficult to respond effectively, and the flexible dispatching capabilities of the power grid urgently need to be improved.

[0003] Traditional grid dispatch methods primarily rely on deterministic and static optimization models. These models typically assume a relatively stable grid operating environment and formulate dispatch strategies based on historical data and experience. However, when faced with sudden extreme events such as typhoons, blizzards, earthquakes, and other natural disasters, as well as the uncertainty of renewable energy output, the limitations of these methods become increasingly apparent. Specifically, traditional methods lack the ability to respond quickly to emergencies and fail to fully account for the dynamic and uncertain nature of grid operation, resulting in ineffective dispatch strategies in practical applications.

[0004] With the rapid development of artificial intelligence (AI) in recent years, deep reinforcement learning (DRL) has demonstrated tremendous potential in solving complex dynamic optimization problems. The Deep Expected Q-Network (DQN), a leading approach to deep reinforcement learning, combines deep learning with reinforcement learning to efficiently learn policies in high-dimensional state spaces. DQN continuously interacts with the environment to learn and optimize scheduling policies, resulting in improved adaptability and robustness in uncertain environments.

[0005] However, applying DQN to regional power grid dispatching still faces numerous challenges. For one thing, the grid's state space is complex and high-dimensional, making effective representation and processing of the grid's operational state a key issue. Furthermore, the grid's operating environment changes rapidly under extreme scenarios, posing a challenge for DQN to respond to emergencies. Summary of the Invention

[0006] In view of this, an embodiment of the present invention provides a mobile energy storage scheduling method and device based on a knowledge-constrained deep expected Q network. This method is based on the deep expected Q network and improves the flexibility and response capability of the power grid by optimizing the charging and discharging decisions of the mobile energy storage system (MES), so as to at least partially solve the above-mentioned problems.

[0007] According to a first aspect of an embodiment of the present invention, a mobile energy storage scheduling method based on a knowledge-constrained deep expectation Q network is provided, comprising: establishing a mobile energy storage elastic optimization scheduling model for load recovery based on the spatiotemporal coupling of emergency resources, the uncertainty of extreme weather scenarios, the energy storage charging and discharging, and the spatiotemporal transfer constraints; based on the mobile energy storage elastic optimization scheduling model, integrating the spatiotemporal coupling of emergency resources, the uncertainty of the extreme weather scenarios, the energy storage charging and discharging, and the spatiotemporal transfer constraints into the mathematical derivation of machine learning preprocessing to establish an energy storage Markov decision process covering the fusion of optimization constraint knowledge; based on the energy storage Markov decision process, establishing an elastic scheduling process training agent based on a deep expectation Q network algorithm to optimize the scheduling strategy of the mobile energy storage system.

[0008] In one implementation, a mobile energy storage elastic optimization scheduling model for load recovery is established based on the spatiotemporal coupling of emergency resources, the uncertainty of extreme weather scenarios, and the constraints of energy storage charging and discharging and spatiotemporal transfer, which can be expressed as:

[0009]

[0010] P t ess =-P t dis +P t char

[0011]

[0012] in, Indicates the power exchanged between the regional power grid and the outside world, represents the charge and discharge power of the MES system at time t, Greater than 0 means charging, Less than 0 means discharge. represents the restored power of the load at time t, Indicates the uncertain power output of new energy, ξ s is a random variable related to the scenario s, which is used to characterize the uncertainty of new energy processing under extreme weather scenarios. represents the discharge power at time t, represents the charging power at time t, represents the total stored energy of the energy storage system at time t, represents the new energy injected after the MES arrives at time t, which is used to characterize the spatiotemporal coupling characteristics of emergency resources. represents the load power restored by the regional power grid at time t, η char and η dis They represent the charging efficiency and discharging efficiency respectively. The above formulas are used together to characterize the energy storage charging and discharging and time-space transfer constraints.

[0013] In another implementation, the mobile energy storage elastic optimization scheduling model incorporates the spatiotemporal coupling of emergency resources, the uncertainty of the extreme weather scenario, the energy storage charging and discharging, and the spatiotemporal transfer constraints into the mathematical derivation of machine learning preprocessing to establish an energy storage Markov decision process that incorporates optimization constraint knowledge fusion. Based on the energy storage Markov decision process, a flexible scheduling process training agent based on a deep expected Q network algorithm is established to optimize the scheduling strategy of the mobile energy storage system, which is expressed as:

[0014] A ESS =[0,ΔP dis ,2ΔP dis ...,(n-1)ΔP dis ,nΔP dis ,...,P max _ dis ]

[0015] P max_dis =N0ΔP dis

[0016]

[0017] R t =αR t-1 +(1-α)r t

[0018]

[0019] q(s,a;θ′)=r(s,a)+γE(q(s′,a;θ′))

[0020]

[0021] Among them, A ESS represents the ESS power matrix vector, P max _ dis Indicates the maximum discharge power, ΔP dis Indicates the adjustable discharge range, N0 indicates the adjustable discharge position, r t Represents a single instant reward return, ω i represents the load weight, Indicates the rated power of the load, Γ trepresents the penalty term, R t represents the cumulative reward of t iterations, α is the discount rate, q(s,a;θ) represents the Q-value reward obtained by executing action a under the certain state s and network parameters θ, E(·) refers to the expected reward obtained under the corresponding uncertain state s, and γ is the training discount rate. The above expression is the knowledge equivalent embedding of the mobile energy storage elastic optimization scheduling model, which is specifically expressed as: Formula A ESS =[0,ΔP dis ,2ΔP dis ...,(n-1)ΔP dis ,nΔP dis ,...,P max_dis ] and P max_dis =N0ΔP dis For the discrete decision approximation of mobile energy storage charging and discharging, and R t =αR t-1 +(1-α)r t The first two equations are penalty function corrections for the soft constraints of overcharging and over-discharging of mobile energy storage, and the remaining equations are expected training representations for multi-scenario stochastic optimization. The above equations realize the embedding of elastic optimization scheduling knowledge constraints in the machine learning process.

[0022] In another implementation, the objective function of establishing the flexible scheduling process training agent based on the deep expected Q network algorithm is:

[0023]

[0024] Where L(·) is the objective function of stochastic gradient descent, s' and a' represent the state and action at the next moment, respectively, r is the single reward return, a is the action variable, s is the state variable, γ is the training discount rate, Q(s,a) represents the Q-value return obtained by executing a in state s, and φ represents the parameters of all network training.

[0025] According to a second aspect of an embodiment of the present invention, a mobile energy storage scheduling device based on a knowledge-constrained deep expectation Q network is provided, comprising: a modeling module for establishing a mobile energy storage elastic optimization scheduling model for load recovery based on the spatiotemporal coupling of emergency resources, the uncertainty of extreme weather scenarios, the energy storage charging and discharging, and the spatiotemporal transfer constraints; a knowledge fusion module for integrating the spatiotemporal coupling of emergency resources, the uncertainty of extreme weather scenarios, the energy storage charging and discharging, and the spatiotemporal transfer constraints into the mathematical derivation of machine learning preprocessing based on the mobile energy storage elastic optimization scheduling model, so as to establish an energy storage Markov decision process covering the fusion of optimization constraint knowledge; a training module for establishing an elastic scheduling process training agent based on the deep expectation Q network algorithm based on the energy storage Markov decision process, so as to optimize the scheduling strategy of the mobile energy storage system.

[0026] In one implementation, a mobile energy storage elastic optimization scheduling model for load recovery is established based on the spatiotemporal coupling of emergency resources, the uncertainty of extreme weather scenarios, and the constraints of energy storage charging and discharging and spatiotemporal transfer, which can be expressed as:

[0027]

[0028] P t ess =-P t dis +P t char

[0029]

[0030] in, Indicates the power exchanged between the regional power grid and the outside world, represents the charge and discharge power of the MES system at time t, Greater than 0 means charging, Less than 0 means discharge. represents the restored power of the load at time t, Indicates the uncertain power output of new energy, ξ s is a random variable related to the scenario s, which is used to characterize the uncertainty of new energy processing under extreme weather scenarios. represents the discharge power at time t, represents the charging power at time t, represents the total stored energy of the energy storage system at time t, represents the new energy injected after the MES arrives at time t, which is used to characterize the spatiotemporal coupling characteristics of emergency resources. represents the load power restored by the regional power grid at time t, η char and η dis They represent the charging efficiency and discharging efficiency respectively. The above formulas are used together to characterize the energy storage charging and discharging and time-space transfer constraints.

[0031] In another implementation, the mobile energy storage elastic optimization scheduling model incorporates the spatiotemporal coupling of emergency resources, the uncertainty of the extreme weather scenario, the energy storage charging and discharging, and the spatiotemporal transfer constraints into the mathematical derivation of machine learning preprocessing to establish an energy storage Markov decision process that incorporates optimization constraint knowledge fusion. Based on the energy storage Markov decision process, a flexible scheduling process training agent based on a deep expected Q network algorithm is established to optimize the scheduling strategy of the mobile energy storage system, which is expressed as:

[0032] A ESS =[0,ΔP dis ,2ΔP dis...,(n-1)ΔP dis ,nΔP dis ,...,P max_dis ]

[0033] P max_dis =N0ΔP dis

[0034]

[0035] R t =αR t-1 +(1-α)r t

[0036]

[0037] q(s,a;θ′)=r(s,a)+γE(q(s′,a;θ′))

[0038]

[0039] Among them, A ESS represents the ESS power matrix vector, P max_dis Indicates the maximum discharge power, ΔP dis Indicates the adjustable discharge range, N0 indicates the adjustable discharge position, r t Represents a single instant reward return, ω i represents the load weight, Indicates the rated power of the load, Γ t represents the penalty term, R t represents the cumulative reward of t iterations, α is the discount rate, q(s,a;θ) represents the Q-value reward obtained by executing action a under the certain state s and network parameters θ, E(·) refers to the expected reward obtained under the corresponding uncertain state s, and γ is the training discount rate. The above expression is the knowledge equivalent embedding of the mobile energy storage elastic optimization scheduling model, which is specifically expressed as: Formula A ESS =[0,ΔP dis ,2ΔP dis ...,(n-1)ΔP dis ,nΔP dis ,...,P max_dis ] and P max_dis =N0ΔP dis For the discrete decision approximation of mobile energy storage charging and discharging, and R t =αR t-1 +(1-α)r t The first two equations are penalty function corrections for the soft constraints of overcharging and over-discharging of mobile energy storage, and the remaining equations are expected training representations for multi-scenario stochastic optimization. The above equations realize the embedding of elastic optimization scheduling knowledge constraints in the machine learning process.

[0040] In another implementation, the objective function of establishing the flexible scheduling process training agent based on the deep expected Q network algorithm is:

[0041]

[0042] Where L(·) is the objective function of stochastic gradient descent, s' and a' represent the state and action at the next moment, respectively, r is the single reward return, a is the action variable, s is the state variable, γ is the training discount rate, Q(s,a) represents the Q-value return obtained by executing a in state s, and φ represents the parameters of all network training.

[0043] According to a third aspect of an embodiment of the present invention, there is provided an electronic device comprising a processor and a memory storing a program, wherein the program comprises instructions that, when executed by the processor, cause the processor to perform the steps of the method according to the first aspect.

[0044] According to a fourth aspect of an embodiment of the present invention, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method of the first aspect is implemented.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] (1) Based on the knowledge constraint fusion method, the present invention introduces random variable factors into the scheduling model. Through the design of intelligent agent rules, it is embedded in the training and learning process to balance the load fluctuations in the power system and effectively represent and process the operating status of the power grid (including extreme scenarios).

[0047] (2) The present invention designs a deep expected Q network (DEQN) algorithm to learn and optimize scheduling strategies from a large amount of uncertain and fluctuating source and load history and forecast data for sudden extreme scenarios, making scheduling decisions more scientific, accurate and robust. Compared with traditional empirical decision-making, this data-driven decision-making method of the present invention is more adaptable to complex and changeable power grid environments and reduces the risk of frequent voltage fluctuations and power outages.

[0048] (3) The solution of the present invention can provide power system operators with more detailed decision support tools, helping operators make more informed decisions and improve operational efficiency and reliability when facing a volatile power market and operating environment. Engineering personnel can then conduct relevant research based on this. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0050] Figure 1 Flowchart of the steps of the mobile energy storage scheduling method based on the knowledge-constrained deep expected Q network of the present invention;

[0051] Figure 2 A rule diagram of the mobile energy storage system of the present invention, including (a) the execution logic of the energy storage system and (b) the charge and discharge curves;

[0052] Figure 3 Schematic diagram of discrete probabilistic processing of uncertain outputs of the present invention;

[0053] Figure 4 This is a schematic diagram of the matching results in the example of the present invention without energy network verification;

[0054] Figure 5 This is a schematic diagram of average power of renewable energy and loads of the present invention;

[0055] Figure 6 Schematic diagram of recovery results in different scenarios of the present invention.

[0056] Figure 7 This is a structural block diagram of a mobile energy storage scheduling device based on a knowledge-constrained deep expected Q network according to the present invention;

[0057] Figure 8 The figure is a schematic structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0058] In order to have a clearer understanding of the technical features, purposes and effects of the embodiments of the present invention, specific implementation methods of the embodiments of the present invention are now described with reference to the accompanying drawings.

[0059] In this document, “exemplary” means “serving as an example, instance or illustration”, and any illustration or implementation described in this document as “exemplary” should not be interpreted as a more preferred or more advantageous technical solution.

[0060] In order to make the personnel in the art better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the embodiments of the present application shall belong to the scope of protection of the embodiments of the present application.

[0061] The specific implementation of the embodiments of the present application will be further described below in combination with the drawings of the embodiments of the present application.

[0062] Referring to Figure 1 The present application provides a mobile energy storage scheduling method based on knowledge constraint deep expected Q network, mainly comprising the following steps:

[0063] Step S1, based on the spatio-temporal coupling of emergency resources, the uncertainty of extreme weather scenarios, the charging and discharging of energy storage and the spatio-temporal transfer constraints, an elastic optimization scheduling model of mobile energy storage for load recovery is established;

[0064] Step S2, based on the mobile energy storage elastic optimization scheduling model, the spatio-temporal coupling of emergency resources, the uncertainty of extreme weather scenarios, the charging and discharging of energy storage and the spatio-temporal transfer constraints are integrated into the mathematical derivation of machine learning preprocessing, to establish an energy storage Markov decision process covering optimization constraint knowledge fusion;

[0065] Step S3, based on the energy storage Markov decision process, an intelligent agent is trained based on the elastic scheduling process of the deep expected Q network algorithm, to optimize the scheduling strategy of the mobile energy storage system.

[0066] Compared with the prior art, the present application has the following advantages:

[0067] (1) The method based on knowledge constraint fusion of the present application introduces random variable factors in the scheduling model, and through the design of agent rules, it is embedded in the training learning process, balances the load fluctuation in the power system, and effectively represents and processes the operation state of the power grid (including extreme scenarios).

[0068] (2) The present application designs a deep expected Q network (DEQN) algorithm, faces sudden extreme scenarios, learns and optimizes the scheduling strategy from a large amount of uncertain and fluctuating source and load historical and predicted data, makes the scheduling decision more scientific, accurate and robust, and the data-driven decision-making method of the present application can better adapt to the complex and variable power grid environment compared with the traditional experience decision-making, and reduces the frequent voltage fluctuation and power failure risk.

[0069] (3) The solution of the present invention can provide power system operators with more detailed decision support tools, helping operators make more informed decisions and improve operational efficiency and reliability when facing a volatile power market and operating environment. Engineering personnel can then conduct relevant research based on this.

[0070] Optionally, based on the spatiotemporal coupling of emergency resources, the uncertainty of extreme weather scenarios, energy storage charging and discharging, and spatiotemporal transfer constraints, a mobile energy storage elastic optimization scheduling model for load recovery is established, which is expressed as:

[0071]

[0072] P t ess =-P t dis +P t char

[0073]

[0074] in, Indicates the power exchanged between the regional power grid and the outside world, represents the charge and discharge power of the MES system at time t, Greater than 0 means charging, Less than 0 means discharge. represents the restored power of the load at time t, Indicates the uncertain power output of new energy, ξ s is a random variable related to the scenario s, which is used to characterize the uncertainty of new energy processing under extreme weather scenarios. represents the discharge power at time t, represents the charging power at time t, represents the total stored energy of the energy storage system at time t, represents the new energy injected after the MES arrives at time t, which is used to characterize the spatiotemporal coupling characteristics of emergency resources. represents the load power restored by the regional power grid at time t, η char and η dis They represent the charging efficiency and discharging efficiency respectively. The above formulas are used together to characterize the energy storage charging and discharging and time-space transfer constraints.

[0075] Optionally, based on the mobile energy storage elastic optimization scheduling model, the spatiotemporal coupling of emergency resources, the uncertainty of the extreme weather scenario, the energy storage charging and discharging, and the spatiotemporal transfer constraints are integrated into the mathematical derivation of machine learning preprocessing to establish an energy storage Markov decision process that incorporates optimization constraint knowledge fusion. Based on the energy storage Markov decision process, a flexible scheduling process training agent based on a deep expected Q network algorithm is established to optimize the scheduling strategy of the mobile energy storage system, which is expressed as:

[0076] A ESS =[0,ΔP dis ,2ΔP dis ...,(n-1)ΔP dis ,nΔP dis ,...,P max_dis ]

[0077] P max_dis =N0ΔP dis

[0078]

[0079] R t =αR t-1 +(1-α)r t

[0080]

[0081] q(s,a;θ′)=r(s,a)+γE(q(s′,a;θ′))

[0082]

[0083] Among them, A ESS represents the ESS power matrix vector, P max _ dis Indicates the maximum discharge power, ΔP dis Indicates the adjustable discharge range, N0 indicates the adjustable discharge position, r t Represents a single instant reward return, ω i represents the load weight, Indicates the rated power of the load, Γ t represents the penalty term, R t represents the cumulative reward of t iterations, α is the discount rate, q(s,a;θ) represents the Q-value reward obtained by executing action a under the certain state s and network parameters θ, E(·) refers to the expected reward obtained under the corresponding uncertain state s, and γ is the training discount rate. The above expression is the knowledge equivalent embedding of the mobile energy storage elastic optimization scheduling model, which is specifically expressed as: Formula A ESS =[0,ΔP dis ,2ΔPdis ...,(n-1)ΔP dis ,nΔP dis ,...,P max_dis ] and P max_dis =N0ΔP dis For the discrete decision approximation of mobile energy storage charging and discharging, and R t =αR t-1 +(1-α)r t The first two equations are penalty function corrections for the soft constraints of overcharging and over-discharging of mobile energy storage, and the remaining equations are expected training representations for multi-scenario stochastic optimization. The above equations realize the embedding of elastic optimization scheduling knowledge constraints in the machine learning process.

[0084] Optionally, the objective function of establishing the flexible scheduling process training agent based on the deep expected Q network algorithm is:

[0085]

[0086] Where L(·) is the objective function of stochastic gradient descent, s' and a' represent the state and action at the next moment, respectively, r is the single reward return, a is the action variable, s is the state variable, γ is the training discount rate, Q(s,a) represents the Q-value return obtained by executing a in state s, and φ represents the parameters of all network training.

[0087] Specifically, the solution of the present invention is further described according to the following examples:

[0088] (1) Mobile energy storage energy management optimization modeling

[0089] In extreme situations, mobile energy storage plays a vital role in supporting loads as a key emergency resource. This paper focuses on using mobile energy storage as the main supporting asset and considers several assumptions in the scheduling process:

[0090] (1) The research focuses on regional power grids without in-depth study of the specific topology of the power grid or the complex power flows.

[0091] (2) Ignoring the spatial state of scheduling, only emphasizing the temporal aspect, where the scheduling time is clearly defined.

[0092] (3) The study assumes that new energy systems such as wind power and photovoltaic power have random characteristics. It does not include the demand-side response process of special loads such as electric vehicles, commercial and industrial loads.

[0093] The response curve of mobile energy storage in dispatch recovery is represented by the energy transfer curve. Mobile energy storage is connected to the regional power grid through traffic behavior and performs discharge actions, such as Figure 2As shown in (a) of the figure, the energy of the entire energy storage system suddenly surges significantly, indicating the arrival of mobile energy storage. Conversely, the discharge of energy storage corresponds to a gradual decrease in the energy of the entire system, as shown in Figure 2 As shown in (b) in .

[0094] The regional power system encompasses energy storage systems, external grid interaction systems, new energy systems, and load systems. In extreme cases, mobile energy storage scheduling also aims to achieve energy balance in the regional power system, with minimum load shedding as the priority objective function. The operating cost corresponds to the minimum load shedding loss cost at each level within the scheduling cycle and can be expressed as:

[0095]

[0096] Where, is the rated power of Class I load at time t, represents the recovered power of the first type load at time t, ω i Indicates the unit loss cost of Class I load.

[0097] At the same time, the following regional power system operation constraints must be met to complete the modeling of the mobile energy storage elastic optimization scheduling model for load recovery:

[0098]

[0099] P t ess =-P t dis +P t char

[0100] 0≤P t char ≤β char P t max_char

[0101] 0≤P t dis ≤β dis P t max_dis

[0102] β char +β dis =1

[0103]

[0104] Where, Indicates the power exchanged between the regional power grid and the outside world, Indicates the charge and discharge power of the MES system at time t. A value greater than 0 indicates charging, and a value less than 0 indicates discharging. represents the restored power of the load at time t, Indicates the uncertain power output of new energy, ξ s is a random variable related to the scenario s, Represents the discharge power at time t. represents the charging power at time t. Represents the total stored energy of the energy storage system at time t. It represents the new energy injected after the MES arrives at time t.

[0105] (2) Reinforcement Learning Preprocessing for Constrained Knowledge Fusion

[0106] 1. Markov Decision Process

[0107] Think of the interaction between the agent and the environment as a discrete time series. The agent begins in an initial state s0 and subsequently chooses an action a0. Consequently, the environment transitions to a new state s1, providing the agent with an immediate reward r1. Subsequently, the agent takes action a1a1 based on state s1, causing the environment to transition to s2, accompanied by a reward r2. This repeated interaction continues indefinitely.

[0108] For MES as an agent, a set of actions a1, a2, ..., a n , and a set of local observations O1, O2, ...O N For the action set, the charging and discharging processes of the energy storage system are reinforcement learning behaviors. For the local observation set, the state of the power system at time t includes the energy storage's charged state, the real-time load state, the real-time renewable energy output, the load output forecast for the next moment, and the renewable energy output forecast for the next moment. The ESS agent selects an action based on its parameterized policy to maximize its expected cumulative reward. The agent is then transitioned to the next state based on the transition probability.

[0109] The objective function of the optimization model is mapped to the reward function of reinforcement learning to establish the reinforcement learning Markov decision process. The reward function can be defined as:

[0110]

[0111] Among them, r t represents the reward at time t, Γ t Indicates punishment reward.

[0112] Therefore, the total reward of the scheduling cycle of the entire elastic scheduling process is:

[0113]

[0114] In the Markov decision process, the discount rate is the focus on future rewards, so the real-time reward function is modified as follows:

[0115] R t =αR t-1 +(1-α)r t

[0116] 2. Rule Modeling of Agents and Environments

[0117] When converting an optimization model to a machine learning model, it is crucial to convert the various constraints of the optimization model into actionable conditions suitable for machine learning. This involves considering the conversion method for each constraint individually.

[0118] In actual extreme application scenarios, the entire mobile energy storage system acts as an intelligent agent. In this case, the charging process is ignored and only the discharge process is considered. Specifically, the discharge power is discretely set to . Therefore, the energy storage discharge action set is:

[0119] A ESS =[0,ΔP dis ,2ΔP dis ...,(n-1)ΔP dis ,nΔP dis ,...,P max _ dis ]

[0120] P max_dis =N0ΔP dis

[0121] (3) Mobile energy storage dispatch solution strategy

[0122] Based on the above knowledge constraint fusion method, a network training process for mobile energy storage scheduling is established. During actual output, the regional power grid outputs the energy storage system's movement or output plan based on the uncertain output of renewable energy at the next moment. The specific training process is as follows:

[0123] Given the probability distribution of the newly added energy output, the probability is processed according to the discretization method, such as Figure 3 shown.

[0124] In order to calculate the value function in the continuous state and action space, a function Q can be designed φ (s,a)≈Q π (s,a) is used to approximate the calculation. The function is a neural network with parameters φ. In order to make the two as equal as possible, the stochastic gradient descent method is used. The objective function of the network training is:

[0125]

[0126] Where L(·) is the objective function of stochastic gradient descent, s' and a' represent the state and action at the next moment, respectively, r is the single reward return, a is the action variable, s is the state variable, γ is the training discount rate, Q(s,a) represents the Q-value return obtained by executing a in state s, and φ represents the parameters of all network training.

[0127] Deep Q Network uses two strategies:

[0128] The first is to freeze the target network, that is, to keep the target parameters fixed for a certain period of time to stabilize the learning target;

[0129] The second is experience replay, which eliminates data correlation by establishing an experience pool (replay buffer). This experience pool contains the agent's recent experiences.

[0130] First, a dispatch strategy for the mobile energy storage system is selected in the estimated Q network. Then, the value function is updated in the target Q network as shown below. The probability density function is discretized.

[0131]

[0132] q(s,a;θ′)=r(s,a)+γE(q(s′,a;θ′))

[0133]

[0134] The specific algorithm flow is shown in Table 1:

[0135] Table 1

[0136]

[0137] (IV) Example analysis

[0138] The scheduling is based on a 24-hour time scale, and the hourly output follows a certain probability distribution. Using the Monte Carlo method, 1000 sets of possible output data are randomly generated every hour. The fluctuation of 10% above and below is used as the 95% confidence interval of the normal distribution and is set as the random value. The average power of renewable energy and load is as follows Figure 4 shown.

[0139] The present invention uses Python language and calls PyTorch package to write the model. The computer CPU model is i7-8550U and the memory is 8GB.

[0140] In order to verify the effectiveness of the model and algorithm proposed in this invention, the following four models are designed for comparative analysis:

[0141] Mode 1: Considering the uncertainty of load output, simulation is performed according to the algorithm proposed by the model.

[0142] Mode 2: Use traditional optimization methods for simulation without considering the uncertainty of load output.

[0143] Mode 3: Similar to Mode 1, this mode also takes into account the uncertainty of the load output. However, it uses traditional optimization methods for simulation.

[0144] Figure 5 The results of energy storage dispatch under different scenarios are presented. The figure depicts an initial gradual decrease in energy storage discharge power followed by a sudden increase. This pattern is consistent with the relatively low renewable energy output at night and the sudden increase in load power during peak demand, which is consistent with expectations.

[0145] Compared with Modes 2 and 3, the mode proposed by the present invention has lower discharge power at the beginning and higher discharge power at the end. This difference is due to the larger power gap observed later, which allows Mode 1 to retain a larger uncertainty.

[0146] Figure 6 The simulation results of recovery load under different scenarios are shown in Figure 2. In the case of average scenario optimization, for example Figure 6 As shown in (a), mode 2 has better fault recovery, but the overall difference is not significant. Under extreme random fields, the overall performance of modes 1 and 3 is better than that of mode 2.

[0147] Specifically, in five randomly selected scenarios, e.g. Figure 6 As shown in (b), (c), (d), and (e), Models 1 and 3 have higher load levels than Model 2. Each model has its own advantages and disadvantages. This is because Model 2 does not consider uncertainty. Both Models 1 and 3 consider uncertainty, but use different solution methods.

[0148] Table 2 shows the computation time and objective function value for each model. Model 3 takes longer to compute than Models 1 and 2, indicating that traditional stochastic optimization models are not suitable for solving stochastic optimization problems with a large number of uncertain scenarios. Due to the consideration of uncertainty, the objective function value of Model 1 is lower than that of Model 2, resulting in a relatively conservative result at the expense of some objective function value.

[0149] Table 2

[0150] model 1 2 3 Solution time / second 36.8s 14.2s >1000s Objective function 131.4 112.1 /

[0151] In summary, the mobile energy storage scheduling method based on the knowledge-constrained deep expectation Q network proposed in the present invention studies the charging and discharging decisions with mobile energy storage as the main load support component. First, a mobile energy storage elastic optimization scheduling model for load recovery is established, highlighting the impact of the spatiotemporal transfer characteristics of mobile energy storage (MES) on the entire system. Secondly, a Markov decision process of the mobile energy storage system as an intelligent agent is established, and the optimization model constraints are integrated into the mathematical derivation of machine learning preprocessing. Finally, a flexible scheduling process based on the deep Q network algorithm is proposed. Simulation results show that the algorithm and model of the present invention enhance the elasticity of the system, ensure computational efficiency, and effectively solve the uncertainty problem.

[0152] As another example, see Figure 7 The present invention provides a mobile energy storage scheduling device 700 based on a knowledge-constrained deep expectation Q network, comprising:

[0153] Modeling module 710 is used to establish a mobile energy storage elastic optimization scheduling model for load recovery based on the spatiotemporal coupling of emergency resources, the uncertainty of extreme weather scenarios, and the constraints of energy storage charging and discharging and spatiotemporal transfer;

[0154] A knowledge fusion module 720 is configured to integrate the spatiotemporal coupling of emergency resources, the uncertainty of extreme weather scenarios, the energy storage charging and discharging, and the spatiotemporal transfer constraints into the mathematical derivation of machine learning preprocessing based on the mobile energy storage elastic optimization scheduling model, thereby establishing an energy storage Markov decision process that incorporates optimization constraint knowledge fusion.

[0155] The training module 730 is used to establish an elastic scheduling process training agent based on the deep expected Q network algorithm based on the energy storage Markov decision process to optimize the scheduling strategy of the mobile energy storage system.

[0156] Optionally, based on the spatiotemporal coupling of emergency resources, the uncertainty of extreme weather scenarios, energy storage charging and discharging, and spatiotemporal transfer constraints, a mobile energy storage elastic optimization scheduling model for load recovery is established, which is expressed as:

[0157]

[0158] P t ess =-P t dis +P t char

[0159]

[0160] in, Indicates the power exchanged between the regional power grid and the outside world, represents the charge and discharge power of the MES system at time t, Greater than 0 means charging, Less than 0 means discharge. represents the restored power of the load at time t, Indicates the uncertain power output of new energy, ξ s is a random variable related to the scenario s, which is used to characterize the uncertainty of new energy processing under extreme weather scenarios. represents the discharge power at time t, represents the charging power at time t, represents the total stored energy of the energy storage system at time t, represents the new energy injected after the MES arrives at time t, which is used to characterize the spatiotemporal coupling characteristics of emergency resources. represents the load power restored by the regional power grid at time t, η char and η dis They represent the charging efficiency and discharging efficiency respectively. The above formulas are used together to characterize the energy storage charging and discharging and time-space transfer constraints.

[0161] Optionally, based on the mobile energy storage elastic optimization scheduling model, the spatiotemporal coupling of emergency resources, the uncertainty of the extreme weather scenario, the energy storage charging and discharging, and the spatiotemporal transfer constraints are integrated into the mathematical derivation of machine learning preprocessing to establish an energy storage Markov decision process that incorporates optimization constraint knowledge fusion. Based on the energy storage Markov decision process, a flexible scheduling process training agent based on a deep expected Q network algorithm is established to optimize the scheduling strategy of the mobile energy storage system, which is expressed as:

[0162] A ESS =[0,ΔP dis ,2ΔP dis ...,(n-1)ΔP dis ,nΔP dis ,...,P max _ dis ]

[0163] P max_dis =N0ΔP dis

[0164]

[0165] R t =αR t-1 +(1-α)r t

[0166]

[0167] q(s,a;θ′)=r(s,a)+γE(q(s′,a;θ′))

[0168]

[0169] Among them, A ESS represents the ESS power matrix vector, P max_dis Indicates the maximum discharge power, ΔP dis Indicates the adjustable discharge range, N0 indicates the adjustable discharge position, r t represents a single immediate reward return, ψ i represents the load weight, Indicates the rated power of the load, Γ t represents the penalty term, R t represents the cumulative reward of t iterations, α is the discount rate, q(s,a;θ) represents the Q-value reward obtained by executing action a under the certain state s and network parameters θ, E(·) refers to the expected reward obtained under the corresponding uncertain state s, and γ is the training discount rate. The above expression is the knowledge equivalent embedding of the mobile energy storage elastic optimization scheduling model, which is specifically expressed as: Formula A ESS =[0,ΔP dis ,2ΔP dis ...,(n-1)ΔP dis ,nΔP dis ,...,P max_dis ] and P max_dis =N0ΔP dis For the discrete decision approximation of mobile energy storage charging and discharging, and R t =αR t-1 +(1-α)r t The first two equations are penalty function corrections for the soft constraints of overcharging and over-discharging of mobile energy storage, and the remaining equations are expected training representations for multi-scenario stochastic optimization. The above equations realize the embedding of elastic optimization scheduling knowledge constraints in the machine learning process.

[0170] Optionally, the objective function of establishing the flexible scheduling process training agent based on the deep expected Q network algorithm is:

[0171]

[0172] Where L(·) is the objective function of stochastic gradient descent, s' and a' represent the state and action at the next moment, respectively, r is the single reward return, a is the action variable, s is the state variable, γ is the training discount rate, Q(s,a) represents the Q-value return obtained by executing a in state s, and φ represents the parameters of all network training.

[0173] It should be understood that the mobile energy storage scheduling device based on the knowledge-constrained deep expected Q network of this embodiment is used to implement the corresponding methods in the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments.

[0174] As another example, see Figure 8 , an electronic device 800 is provided, and a structural block diagram of the electronic device 800 that can serve as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0175] The electronic device 800 may include a processor 802 , a communications interface 804 , a memory 806 , and a communication bus 808 .

[0176] The processor 802, the communication interface 804, and the memory 806 communicate with each other via a communication bus 808. The communication interface 804 is used to communicate with other electronic devices or servers.

[0177] The processor 802 is configured to execute the program 810 , and specifically may execute the relevant steps in the above method embodiment.

[0178] Specifically, the program 810 may include program codes, which include computer operation instructions.

[0179] Processor 802 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0180] The memory 806 is used to store the program 810. The memory 806 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0181] When executed by processor 802, program 810 is used to enable the electronic device to execute a mobile energy storage scheduling method based on a knowledge-constrained deep expectation Q network, including establishing a mobile energy storage elastic optimization scheduling model for load recovery based on the spatiotemporal coupling of emergency resources, the uncertainty of extreme weather scenarios, the energy storage charging and discharging, and the spatiotemporal transfer constraints; based on the mobile energy storage elastic optimization scheduling model, the spatiotemporal coupling of emergency resources, the uncertainty of the extreme weather scenarios, the energy storage charging and discharging, and the spatiotemporal transfer constraints are integrated into the mathematical derivation of machine learning preprocessing to establish an energy storage Markov decision process that covers the fusion of optimization constraint knowledge; based on the energy storage Markov decision process, a flexible scheduling process training agent based on the deep expectation Q network algorithm is established to optimize the scheduling strategy of the mobile energy storage system.

[0182] In addition, the specific implementation of each step in program 810 can refer to the corresponding description of the corresponding steps and units in the above-mentioned method embodiment, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working process of the above-mentioned devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, and will not be repeated here.

[0183] An exemplary embodiment of the present invention further provides a computer storage medium storing a computer program, wherein when the computer program is executed by a processor, the methods of the various embodiments of the present invention are implemented. The corresponding process descriptions in the aforementioned method embodiments can be referred to and will not be repeated here.

[0184] The method according to the embodiment of the present invention described above can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown here.

[0185] Thus far, specific embodiments of the present invention have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

[0186] It should be understood that although this specification is described according to various embodiments, not every embodiment contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0187] Finally, it should be noted that the above implementation methods are only used to illustrate the embodiments of the present invention, and are not limitations on the embodiments of the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the scope of patent protection of the embodiments of the present invention should be defined by the claims.

Claims

1. A mobile energy storage scheduling method based on knowledge-constrained deep expectation Q network, characterized in that: include: Based on the spatiotemporal coupling of emergency resources, the uncertainty of extreme weather scenarios, and the constraints of energy storage charging and discharging and spatiotemporal transfer, a flexible optimization scheduling model for mobile energy storage oriented to load recovery is established. Based on the mobile energy storage elastic optimization scheduling model, the spatiotemporal coupling of emergency resources, the uncertainty of the extreme weather scenario, the energy storage charging and discharging, and the spatiotemporal transfer constraints are integrated into the mathematical derivation of machine learning preprocessing to establish an energy storage Markov decision process that incorporates optimization constraint knowledge fusion; Based on the energy storage Markov decision process, a flexible scheduling process training agent based on the deep expected Q network algorithm is established to optimize the scheduling strategy of the mobile energy storage system; Based on the spatiotemporal coupling of emergency resources, the uncertainty of extreme weather scenarios, energy storage charging and discharging, and spatiotemporal transfer constraints, a mobile energy storage elastic optimization scheduling model for load recovery is established, which is expressed as: in, Indicates the power exchanged between the regional power grid and the outside world, represents the charge and discharge power of the MES system at time t, Greater than 0 means charging, Less than 0 means discharge. represents the restored power of the load at time t, Indicates the uncertain power output of new energy, ξ s is a random variable related to the scenario s, which is used to characterize the uncertainty of new energy processing under extreme weather scenarios. represents the discharge power at time t, represents the charging power at time t, represents the total stored energy of the energy storage system at time t, represents the new energy injected after the MES arrives at time t, which is used to characterize the spatiotemporal coupling characteristics of emergency resources. represents the load power restored by the regional power grid at time t, η char and η dis They represent the efficiency of charging and discharging respectively. The above formulas are used together to characterize the energy storage charging and discharging and time-space transfer constraints; Based on the mobile energy storage elastic optimization scheduling model, the spatiotemporal coupling of emergency resources, the uncertainty of the extreme weather scenario, the energy storage charging and discharging, and the spatiotemporal transfer constraints are integrated into the mathematical derivation of machine learning preprocessing to establish an energy storage Markov decision process that incorporates optimization constraint knowledge fusion. Based on the energy storage Markov decision process, a flexible scheduling process training agent based on the deep expected Q network algorithm is established to optimize the scheduling strategy of the mobile energy storage system, which is expressed as: A ESS =[0,ΔP dis ,2ΔP dis ...,(n-1)ΔP dis ,nΔP dis ,...,P max_dis ] P max_dis =N0ΔP dis R t =αR t-1 +(1-a)r t q(s,a;θ′)=r(s,a)+γE(q(s′,a;θ′)) Among them, A ESS represents the ESS power matrix vector, n represents the scheduling gear; P max_dis Indicates the maximum discharge power, ΔP dis Indicates the adjustable discharge range, N0 indicates the adjustable discharge gear, r t Represents a single instant reward return, ω i represents the load weight, Indicates the rated power of the load, Γ t represents the penalty term, R t represents the cumulative reward of t iterations, α is the discount rate, q(s,a;θ) represents the Q value reward obtained by executing action a under the determined state s and network parameters θ, and a k ∈A, a k represents the action in the kth uncertainty interval, A represents the corresponding set of executable action spaces; E(·) refers to the expected return obtained under the corresponding uncertain state s, γ is the training discount rate, and s′ k Represents the state of the kth uncertainty interval; the above expression is the knowledge equivalent embedding of the mobile energy storage elastic optimization scheduling model, specifically expressed as: Formula A ESS =[0,ΔP dis ,2ΔP dis ...,(n-1)ΔP dis ,nΔP dis ,...,P max _dis ] and P max_dis =N0ΔP dis Approximate the discrete decision of mobile energy storage charging and discharging, and R t =αR t-1 +(1-α)r t The first two equations are penalty function corrections for the overcharge and over-discharge soft constraints of mobile energy storage, and the remaining equations are expected training representations for multi-scenario stochastic optimization.

2. The method according to claim 1, characterized in that The objective function of establishing a flexible scheduling process training agent based on the deep expected Q network algorithm is: Where L(·) is the objective function of stochastic gradient descent, s' and a' represent the state and action at the next moment, respectively, r is the single reward return, a is the action variable, s is the state variable, γ is the training discount rate, q(s,a) represents the Q-value return obtained by executing a in state s, and φ represents the parameters of all network training.

3. A mobile energy storage scheduling device based on knowledge-constrained deep expectation Q network, characterized in that: include: A modeling module is used to establish a flexible optimization scheduling model for mobile energy storage for load recovery based on the spatiotemporal coupling of emergency resources, the uncertainty of extreme weather scenarios, and the constraints of energy storage charging and discharging and spatiotemporal transfer; A knowledge fusion module is used to integrate the spatiotemporal coupling of emergency resources, the uncertainty of extreme weather scenarios, the energy storage charging and discharging, and the spatiotemporal transfer constraints into the mathematical derivation of machine learning preprocessing based on the mobile energy storage elastic optimization scheduling model, so as to establish an energy storage Markov decision process that includes optimization constraint knowledge fusion; A training module is used to establish an elastic scheduling process training agent based on the deep expected Q network algorithm based on the energy storage Markov decision process to optimize the scheduling strategy of the mobile energy storage system; Based on the spatiotemporal coupling of emergency resources, the uncertainty of extreme weather scenarios, energy storage charging and discharging, and spatiotemporal transfer constraints, a mobile energy storage elastic optimization scheduling model for load recovery is established, which is expressed as: P t ess =-P t dis +P t char in, Indicates the power exchanged between the regional power grid and the outside world, represents the charge and discharge power of the MES system at time t, Greater than 0 means charging, Less than 0 means discharge. represents the restored power of the load at time t, Indicates the uncertain power output of new energy, ξ s is a random variable related to the scenario s, which is used to characterize the uncertainty of new energy processing under extreme weather scenarios. represents the discharge power at time t, represents the charging power at time t, represents the total stored energy of the energy storage system at time t, represents the new energy injected after the MES arrives at time t, which is used to characterize the spatiotemporal coupling characteristics of emergency resources. represents the load power restored by the regional power grid at time t, η char and η dis Respectively represent the efficiency of charging and discharging; Based on the mobile energy storage elastic optimization scheduling model, the spatiotemporal coupling of emergency resources, the uncertainty of the extreme weather scenario, the energy storage charging and discharging, and the spatiotemporal transfer constraints are integrated into the mathematical derivation of machine learning preprocessing to establish an energy storage Markov decision process that incorporates optimization constraint knowledge fusion. Based on the energy storage Markov decision process, a flexible scheduling process training agent based on the deep expected Q network algorithm is established to optimize the scheduling strategy of the mobile energy storage system, which is expressed as: A ESS =[0,ΔP dis ,2ΔP dis ...,(n-1)ΔP dis ,nΔP dis ,...,P max_dis ] P max_dis =N0ΔP dis R t =αR t-1 +(1-a)r t q(s,a;θ′)=r(s,a)+γE(q(s′,a;θ′)) Among them, A ESS represents the ESS power matrix vector, n represents the scheduling gear; P max_dis Indicates the maximum discharge power, ΔP dis Indicates the adjustable discharge range, N0 indicates the adjustable discharge gear, r t Represents a single instant reward return, ω i represents the load weight, Indicates the rated power of the load, Γ t represents the penalty term, R t represents the cumulative reward of t iterations, α is the discount rate, q(s,a;θ) represents the Q value reward obtained by executing action a under the determined state s and network parameters θ, and a k ∈A, a k represents the action in the kth uncertainty interval, A represents the corresponding set of executable action spaces; E(·) refers to the expected return obtained under the corresponding uncertain state s, γ is the training discount rate, and s′ k Represents the state of the kth uncertainty interval; the above expression is the knowledge equivalent embedding of the mobile energy storage elastic optimization scheduling model, specifically expressed as: Formula A ESS =[0,ΔP dis ,2ΔP dis ...,(n-1)ΔP dis ,nΔP dis ,...,P max _dis ] and P max_dis =N0ΔP dis Approximate the discrete decision of mobile energy storage charging and discharging, and R t =αR t-1 +(1-α)r t The first two equations are penalty function corrections for the overcharge and over-discharge soft constraints of mobile energy storage, and the remaining equations are expected training representations for multi-scenario stochastic optimization.

4. The device according to claim 3, characterized in that The objective function of establishing a flexible scheduling process training agent based on the deep expected Q network algorithm is: Where L(·) is the objective function of stochastic gradient descent, s' and a' represent the state and action at the next moment, respectively, r is the single reward return, a is the action variable, s is the state variable, γ is the training discount rate, q(s,a) represents the Q-value return obtained by executing a in state s, and φ represents the parameters of all network training.

5. An electronic device, characterized in that: include: processor; Memory for storing programs; The program includes instructions, which, when executed by the processor, cause the processor to perform the steps of the method according to any one of claims 1 to 2.

6. A computer storage medium, characterized in that A computer program is stored thereon, and when the program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images