Construction method of total power consumption cost prediction model, prediction method, equipment and system

Abstract

The invention provides a total power consumption cost prediction model construction method, prediction method, device and system, and the method comprises the steps: constructing a total power consumption cost prediction model in a preset time period based on the use plan information parameters, charge state parameters and performance constraint parameters of a household electric vehicle in the preset time period, the household electric vehicle can be connected to the charging and discharging models of different energy systems; wherein the different energy systems comprise a household energy system and a third-party energy system; establishing a typical household energy system model corresponding to the household energy system in the preset time period; and a near-end strategy optimization algorithm is adopted, and based on the charging and discharging model and the typical household energy system model, a household total electricity consumption cost prediction model in the preset time period is constructed. According to the system and the method, the electricity utilization cost of the household electric vehicle connected to the household energy system and the third-party energy system can be uniformly optimized, so that the total electricity utilization cost of the household is minimum.

Description

Technical Field

[0001] The present invention relates to the field of smart grids, and in particular to a method for constructing a total electricity cost prediction model, a prediction method, a device, and a system. Background Art

[0002] Because electric vehicles (EVs) have energy storage capabilities and can provide bidirectional charging and discharging, their penetration in the passenger car market continues to expand, particularly in households. Simultaneously, the increasing penetration of EVs has led to the implementation of dynamic electricity pricing, enabling flexible price adjustments. These pricing strategies are an excellent demand response resource, not only providing a foundation for EVs to participate in demand response but also contributing to household electricity cost savings, energy conservation and emission reduction, and the safe, stable, and economical operation of the power system.

[0003] In related technologies, when household electric vehicles participate in demand response, the household electric vehicles and the system to which they are connected are usually considered as a whole for demand response optimization. Typical modes generally include: (1) When considering the household energy system to which the household electric vehicle is connected, the household electric vehicle and the household energy system are considered as a whole for demand response optimization. In this case, the impact of the household electric vehicle's continued participation in demand response after moving to another location on the household's total electricity cost is not considered; (2) When considering the commercial building to which the household electric vehicle is connected, the household electric vehicle and the commercial building's original energy system are considered as a whole for demand response optimization. In this case, the impact of the household electric vehicle's continued participation in demand response after moving back home on the household's total electricity cost is not considered. In this way, the demand response optimization method that separates the impact of household electric vehicles on the household's total electricity cost when connected to different energy systems has the problem of difficulty in achieving the lowest total household electricity cost, and is not conducive to users' compliance with scheduling instructions and the widespread promotion of demand response. Summary of the Invention

[0004] In order to solve the problems of the prior art, the present invention proposes a method for constructing a total electricity cost prediction model, a prediction method, an apparatus and a system, which aim to uniformly optimize the electricity costs when household electric vehicles are connected to a household energy system and a third-party energy system respectively, so as to minimize the total household electricity cost expenditure.

[0005] The purpose of the present invention is achieved by adopting the following technical solutions:

[0006] In one aspect, the present invention provides a method for constructing a household total electricity cost prediction model, the method comprising:

[0007] Based on usage plan information parameters, state of charge parameters, and performance constraint parameters of the household electric vehicle within a preset time period, a charging and discharging model is constructed in which the household electric vehicle can be connected to different energy systems within the preset time period; wherein the different energy systems include: a household energy system and a third-party energy system;

[0008] Establishing a typical home energy system model corresponding to the home energy system within the preset time period;

[0009] A proximal strategy optimization algorithm is adopted to construct a household total electricity cost prediction model within the preset time period based on the charging and discharging model and the typical household energy system model.

[0010] Optionally, the preset time period includes: a plurality of sub-time periods of equal length that are continuous in time sequence;

[0011] The usage plan information parameters of the household electric vehicle in each sub-time period include: a first characterization value of whether the household electric vehicle is connected to or not connected to the household energy system, a second characterization value of whether the household electric vehicle is connected to or not connected to the third-party energy system, and a planned mileage in each sub-time period;

[0012] The state of charge parameters of the household electric vehicle in each sub-time period include: the state of charge of the household electric vehicle in each sub-time period, the corresponding interaction power when connected to the different energy systems, the battery rated capacity, the charging efficiency and discharging efficiency of the internal battery pack, and the statistical average value of the decrease in the state of charge of the internal battery pack when the household electric vehicle travels a preset number of kilometers;

[0013] The performance constraint parameters of the household electric vehicle in each sub-time period include: the minimum state of charge value, maximum state of charge value, minimum charge and discharge power value, maximum charge and discharge power value corresponding to the internal battery pack in each sub-time period, and the first minimum starting power corresponding to the household electric vehicle when the mileage of a single trip is less than a preset number of kilometers, and the second minimum starting power corresponding to the household electric vehicle when the mileage of a single trip is greater than or equal to the preset number of kilometers, the second minimum starting power being a positive multiple of the first minimum starting power.

[0014] Optionally, the proximal strategy optimization algorithm is used to construct a household total electricity cost prediction model within the preset time period based on the charging and discharging model and the typical household energy system model, including:

[0015] Adopting the proximal strategy optimization algorithm, based on the charging and discharging model and the typical home energy system model, constructing an initial model of the total household electricity cost within the preset time period;

[0016] Deep reinforcement learning is performed on the initial model of the total household electricity cost to obtain the total household electricity cost prediction model.

[0017] Optionally, the proximal strategy optimization algorithm is used to construct an initial model of the total household electricity cost within the preset time period based on the charge and discharge model and the typical household energy system model, including:

[0018] Based on external environmental information, the current status of the devices in the different energy systems, the control action instructions output by the mobilization systems corresponding to the different energy systems, and user demand information, the charging and discharging model and the typical home energy system model are used to digitally simulate the different energy systems and the home electric vehicle to obtain an energy system simulation model;

[0019] The proximal strategy optimization algorithm is used to build an initial model of the total household electricity cost corresponding to the energy system simulation model within the preset time period.

[0020] Optionally, before performing deep reinforcement learning on the initial model of the household total electricity cost to obtain the household total electricity cost prediction model, the method further includes:

[0021] Based on the power parameters of the charging and discharging model and the device state parameters of the typical home energy system model, constructing the action space, state space and immediate reward of the proximal policy optimization algorithm in the preset time period;

[0022] The performing deep reinforcement learning on the initial model of the total household electricity cost to obtain the total household electricity cost prediction model includes:

[0023] Based on the action space, the state space, and the immediate reward, deep reinforcement learning is performed on the initial model of the total household electricity cost to obtain the total household electricity cost prediction model.

[0024] Optionally, constructing the action space, state space, and immediate reward of the proximal policy optimization algorithm in the preset time period based on the power parameters of the charging and discharging model and the device state parameters of the typical home energy system model includes:

[0025] During the preset time period, an action space is constructed based on the interaction power corresponding to the home electric vehicle being connected to the different energy systems in the charge-discharge model and the action parameters in the typical home energy system model used to characterize turning on or off the load-controllable device; wherein the home energy system includes at least the load-controllable device;

[0026] Based on the state of charge parameters in the charge-discharge model, the power parameters corresponding to the devices in the typical home energy system model, and the preset electricity price information, respectively constructing the endogenous state parameters of the energy system and the exogenous state parameters of the energy system operating environment corresponding to the preset time period, and constructing a state space based on the endogenous state parameters of the energy system and the exogenous state parameters of the energy system operating environment;

[0027] Based on the performance constraint parameters in the charging and discharging model and the working constraints of the load-controllable equipment corresponding to the typical home energy system model, the total household electricity cost within the preset time period and the penalty for violating the preset constraints are determined, and an immediate reward is constructed based on the total household electricity cost and the penalty.

[0028] Optionally, the deep reinforcement learning module of the initial model of the total household electricity cost includes: a new policy network, an old policy network, and an evaluation network;

[0029] The method of performing deep reinforcement learning on the initial model of the total household electricity cost based on the action space, the state space, and the immediate reward to obtain the total household electricity cost prediction model includes:

[0030] Determining, within the preset time period, a probability ratio corresponding to execution of action parameters in the action space between the new policy network and the old policy network based on the state parameters provided by the state space;

[0031] Based on the immediate reward, using the evaluation network, determining an estimated error of a state value corresponding to the state space of the preset time period;

[0032] determining a multi-step advantage of the state space within the preset time period based on the estimation error;

[0033] Based on the probability ratio, the multi-step advantage and the first learning rate, the network parameters of the new policy network and the network parameters of the old policy network are iteratively optimized, and based on the state value and the second learning rate, the network parameters of the evaluation network are iteratively optimized until the deep reinforcement learning module whose output parameters meet the preset convergence conditions is obtained, and the model corresponding to the deep reinforcement learning module whose output parameters meet the preset convergence conditions is determined as the household total electricity cost prediction model.

[0034] Optionally, the devices in the home energy system include at least: a first device with controllable load, a second device with uncontrollable load, and a third device for generating electricity from new energy;

[0035] The typical home energy system model corresponding to the home energy system established within the preset time period includes:

[0036] Modeling the operating parameters of the first device within the preset time period using the mechanism model and the operating constraints corresponding to the first device, and constructing an operating model expression corresponding to the first device;

[0037] Predicting a total load power prediction value corresponding to the second device and a power generation power prediction value corresponding to the third device within the preset time period;

[0038] The typical home energy system model is constructed based on the working model expression, the total load power prediction value and the generated power prediction value.

[0039] Correspondingly, the present invention also provides a system for constructing a household total electricity cost prediction model, the system comprising:

[0040] A first construction module is configured to construct, based on usage plan information parameters, state of charge parameters, and performance constraint parameters of the household electric vehicle within a preset time period, a charging and discharging model in which the household electric vehicle can be connected to different energy systems within the preset time period; wherein the different energy systems include: a household energy system and a third-party energy system;

[0041] A second building module is configured to build a typical home energy system model corresponding to the home energy system within the preset time period;

[0042] The third building module is used to adopt a proximal strategy optimization algorithm to build a household total electricity cost prediction model within the preset time period based on the charging and discharging model and the typical household energy system model.

[0043] Optionally, the preset time period includes: a plurality of sub-time periods of equal length that are continuous in time sequence;

[0044] The usage plan information parameters of the household electric vehicle in each sub-time period include: a first characterization value of whether the household electric vehicle is connected to or not connected to the household energy system, a second characterization value of whether the household electric vehicle is connected to or not connected to the third-party energy system, and a planned mileage in each sub-time period;

[0045] The state of charge parameters of the household electric vehicle in each sub-time period include: the state of charge of the household electric vehicle in each sub-time period, the corresponding interaction power when connected to the different energy systems, the battery rated capacity, the charging efficiency and discharging efficiency of the internal battery pack, and the statistical average value of the decrease in the state of charge of the internal battery pack when the household electric vehicle travels a preset number of kilometers;

[0046] The performance constraint parameters of the household electric vehicle in each sub-time period include: the minimum state of charge value, maximum state of charge value, minimum charge and discharge power value, maximum charge and discharge power value corresponding to the internal battery pack in each sub-time period, and the first minimum starting power corresponding to the household electric vehicle when the mileage of a single trip is less than a preset number of kilometers, and the second minimum starting power corresponding to the household electric vehicle when the mileage of a single trip is greater than or equal to the preset number of kilometers, the second minimum starting power being a positive multiple of the first minimum starting power.

[0047] Optionally, the third building block includes:

[0048] A first construction submodule is configured to construct an initial model of the total household electricity cost within the preset time period based on the charging and discharging model and the typical household energy system model by using the proximal strategy optimization algorithm;

[0049] The deep reinforcement learning submodule is used to perform deep reinforcement learning on the initial model of the total household electricity cost to obtain the total household electricity cost prediction model.

[0050] Optionally, the first building block includes:

[0051] The first unit is configured to digitally simulate the different energy systems and the household electric vehicle using the charge and discharge model and the typical household energy system model based on external environment information, current states of devices in the different energy systems, control action instructions output by the mobilization systems corresponding to the different energy systems, and user demand information, to obtain an energy system simulation model;

[0052] The second unit is used to use the proximal strategy optimization algorithm to build an initial model of the total household electricity cost corresponding to the energy system simulation model within the preset time period.

[0053] Optionally, the deep reinforcement learning submodule includes:

[0054] A third unit is configured to construct an action space, a state space, and an immediate reward for the proximal policy optimization algorithm in the preset time period based on the power parameters of the charging and discharging model and the device state parameters of the typical home energy system model;

[0055] The fourth unit is used to perform deep reinforcement learning on the initial model of the total household electricity cost based on the action space, the state space and the immediate reward to obtain the total household electricity cost prediction model.

[0056] Optionally, the third unit is further configured to construct an action space within the preset time period based on the interaction power corresponding to when the household electric vehicle is connected to the different energy systems in the charge and discharge model, and the action parameters in the typical household energy system model for characterizing turning on or off the load-controllable device; wherein the household energy system includes at least the load-controllable device;

[0057] Based on the state of charge parameters in the charge-discharge model, the power parameters corresponding to the devices in the typical home energy system model, and the preset electricity price information, respectively constructing the endogenous state parameters of the energy system and the exogenous state parameters of the energy system operating environment corresponding to the preset time period, and constructing a state space based on the endogenous state parameters of the energy system and the exogenous state parameters of the energy system operating environment;

[0058] Based on the performance constraint parameters in the charging and discharging model and the working constraints of the load-controllable equipment corresponding to the typical home energy system model, the total household electricity cost within the preset time period and the penalty for violating the preset constraints are determined, and an immediate reward is constructed based on the total household electricity cost and the penalty.

[0059] Optionally, the deep reinforcement learning module of the initial model of the total household electricity cost includes: a new policy network, an old policy network, and an evaluation network;

[0060] The fourth unit is further configured to determine, within the preset time period, based on the state parameters provided by the state space, a probability ratio of executing the action parameters in the action space between the new policy network and the old policy network;

[0061] Based on the immediate reward, using the evaluation network, determining an estimated error of a state value corresponding to the state space of the preset time period;

[0062] determining a multi-step advantage of the state space within the preset time period based on the estimation error;

[0063] Based on the probability ratio, the multi-step advantage and the first learning rate, the network parameters of the new policy network and the network parameters of the old policy network are iteratively optimized, and based on the state value and the second learning rate, the network parameters of the evaluation network are iteratively optimized until the deep reinforcement learning module whose output parameters meet the preset convergence conditions is obtained, and the model corresponding to the deep reinforcement learning module whose output parameters meet the preset convergence conditions is determined as the household total electricity cost prediction model.

[0064] Optionally, the devices in the home energy system include at least: a first device with controllable load, a second device with uncontrollable load, and a third device for renewable energy power generation; the second building module further includes:

[0065] A second construction submodule is configured to model the operating parameters of the first device within the preset time period using the mechanism model and the operating constraints corresponding to the first device, and to construct an operating model expression corresponding to the first device;

[0066] A third construction submodule is configured to predict a total load power prediction value corresponding to the second device and a generated power prediction value corresponding to the third device within the preset time period;

[0067] The fourth construction submodule is used to construct the typical home energy system model based on the working model expression, the total load power prediction value and the generated power prediction value.

[0068] In another aspect, the present invention provides a method for predicting the total electricity cost of a household, the method comprising:

[0069] Based on the real-time operating parameters of devices within the home energy system and the real-time charging and discharging parameters of home electric vehicles connected to different energy systems, training the home total electricity cost prediction model constructed by any of the above methods until convergence, thereby obtaining a home total electricity cost optimization model; wherein the different energy systems include: the home energy system and a third-party energy system;

[0070] The operating power value output by the household total electricity cost optimization model is used as the operating power reference value of the equipment in the household energy system, and the charging and discharging power reference value output by the household total electricity cost optimization model is used as the charging and discharging power reference value when the household electric vehicle is connected to the different energy systems.

[0071] Correspondingly, the present invention also provides a system for predicting the total electricity cost of a household, the method comprising:

[0072] A training module for training a household total electricity cost prediction model constructed by any of the above-described systems until convergence based on real-time operating parameters of devices within the home energy system and real-time charging and discharging parameters corresponding to home electric vehicles connected to different energy systems, thereby obtaining a household total electricity cost optimization model; wherein the different energy systems include: a home energy system and a third-party energy system;

[0073] The prediction module is used to use the operating power value output by the household total electricity cost optimization model as the operating power reference value of the equipment in the household energy system, and to use the charging and discharging power reference value output by the household total electricity cost optimization model as the charging and discharging power reference value when the household electric vehicle is connected to the different energy systems.

[0074] In yet another aspect, the present invention further provides a computing device comprising: one or more processors;

[0075] a processor for executing one or more programs;

[0076] When the one or more programs are executed by the one or more processors, the method for constructing a household total electricity cost prediction model as described in any one of the above items, or the method for predicting the household total electricity cost as described in any one of the above items is implemented.

[0077] On the other hand, the present invention also provides a readable storage medium, characterized in that an execution program is stored thereon, and when the execution program is executed, it implements the method for constructing a household total electricity cost prediction model as described in any of the above items, or the method for predicting the household total electricity cost as described in any of the above items.

[0078] The present invention provides a method and system for constructing a household total electricity cost prediction model. This model uses a multi-parameter model of a household electric vehicle (EV) that can access different energy systems within a preset time period and a typical household energy system model. The model is then constructed using a proximal policy optimization algorithm (deep reinforcement learning). This method enables unified optimization of household EVs accessing the home energy system and other energy systems within different time periods under a dynamic electricity price environment, minimizing the household's total electricity cost. This creates economic value for the EV user and increases their willingness to participate in demand response, contributing to energy conservation and emission reduction and ensuring the safe, stable, and economical operation of the power system. Furthermore, deep reinforcement learning (proximal policy optimization algorithm) optimizes the household's total electricity cost in an environment with uncertain demand factors. This ensures optimization accuracy based on principle (algorithm), facilitating compliance with dispatch instructions by the EV user, improving the quality of electricity demand response, and ultimately facilitating the safe and economical operation of the EV, the home energy system, and third-party energy systems within the power system.

[0079] The present invention provides a method and system for predicting household total electricity costs. This method trains a household total electricity cost prediction model online until convergence, and uses the output of the corresponding household total electricity cost optimization model after convergence as a reference value for the operation of relevant devices within the home energy system, as well as a reference value for the charge and discharge power of household electric vehicles when connected to different energy systems. This method optimizes the total electricity cost of household electric vehicles connected to a home energy system or a third-party energy system as a whole, thereby minimizing the household's total electricity cost expenditure. This method creates economic value for the users of the household electric vehicles and increases their willingness to participate in demand response, facilitating energy conservation and emission reduction, and ensuring the safe, stable, and economical operation of the power system. Furthermore, the method trains the household total electricity cost prediction model online based on real-time operating parameters and real-time charge and discharge parameters to achieve online optimization of the household total electricity cost. This method effectively addresses the uncertainty of relevant performance and operating parameters within different energy systems, helps ensure the accuracy of demand response for household total electricity consumption, and improves the quality of demand response corresponding to the household total electricity cost. This method further facilitates the safe and economical operation of the household electric vehicle, home energy system, and third-party energy system as a whole within the power system.

[0080] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions provided by the embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which:

[0082] Figure 1 A flowchart of a method for constructing a household total electricity cost prediction model provided by an embodiment of the present invention;

[0083] Figure 2 A schematic flow chart of a method for predicting the total household electricity cost provided by an embodiment of the present invention;

[0084] Figure 3 A schematic diagram of a process for optimizing the total household electricity cost of a household electric vehicle based on multi-location demand participation and response according to an embodiment of the present invention;

[0085] Figure 4 A schematic diagram of a framework for a deep reinforcement learning optimization model for total household electricity costs within the next 24 hours, provided by an embodiment of the present invention;

[0086] Figure 5 A schematic diagram of a framework of a household total electricity cost prediction model established in a PPO within a preset time period provided by an embodiment of the present invention;

[0087] Figure 6 A schematic diagram of a system for constructing a household total electricity cost prediction model provided by an embodiment of the present invention;

[0088] Figure 7 A schematic diagram of the composition of a system for predicting total household electricity costs provided by an embodiment of the present invention;

[0089] Figure 8 A schematic diagram of the composition of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0090] The following describes the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art will readily appreciate the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the various details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are intended only to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0091] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0092] In the following description, the terms "first\second\third" are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0093] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art in the art of the embodiments of the present invention. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the embodiments of the present invention.

[0094] In related technologies, when household electric vehicles participate in demand response, they are generally optimized as a whole, including the system they are connected to. Typical modes include the following two:

[0095] (1) Considering the demand response optimization when household electric vehicles are connected to the home energy system, in this scenario, the household electric vehicles and the home energy system are considered as a whole, and the demand response optimization is performed. At this time, the impact of the household electric vehicle's continued participation in demand response on the total electricity cost of the household after it moves elsewhere is not considered.

[0096] (2) Considering the demand response optimization when household electric vehicles are connected to commercial buildings, in this scenario, the household electric vehicles leave the home, and the original energy system of the household electric vehicles and the commercial building is optimized for demand response as a whole. At this time, the impact of the household electric vehicles continuing to participate in demand response after moving home on the total electricity cost of the household is not considered.

[0097] In this way, the demand response optimization method that separates the impact of connecting household electric vehicles to different energy systems on the total household electricity cost has the problem of difficulty in achieving the lowest total household electricity cost, and is not conducive to users following scheduling instructions and the widespread promotion of demand response.

[0098] In view of the above problems, embodiments of the present invention provide a method for constructing a household total electricity cost prediction model and a method for predicting household total electricity cost.

[0099] Example 1

[0100] See Figure 1 The figure is a flow chart of a method for constructing a household total electricity cost prediction model provided by the present invention. Figure 1 The following steps are explained:

[0101] Step 101: Based on usage plan information parameters, state of charge parameters, and performance constraint parameters of the household electric vehicle within a preset time period, a charging and discharging model is constructed in which the household electric vehicle can be connected to different energy systems within the preset time period.

[0102] The different energy systems include: home energy systems and third-party energy systems.

[0103] In some embodiments of the present invention, the home energy system is the original energy system corresponding to the home to which the home electric vehicle belongs. At the same time, the third-party energy system can refer to an energy system other than the home energy system that can provide charged electricity to the home electric vehicle, such as the energy system of a commercial building.

[0104] It should be noted that household electric vehicles can only be connected to one energy system at the same time, that is, either connected to the household energy system, or connected to a third-party energy system, or not connected to either.

[0105] In some embodiments of the present invention, a household electric vehicle may refer to a vehicle that is driven by an electric motor and whose driving power is derived from an onboard rechargeable energy storage system. Household electric vehicles include, but are not limited to, pure electric vehicles, extended-range electric vehicles, hybrid electric vehicles, and pure electric vehicles.

[0106] In some embodiments of the present invention, the preset time period may refer to the next 24 hours, or may refer to the next week, etc. The present invention does not impose any limitation on this. The following embodiments are described as an example in which the preset time period refers to the next 24 hours.

[0107] In some embodiments of the present invention, a charging and discharging model is constructed in which a household electric vehicle can be connected to different energy systems within the next 24 hours. Factors that need to be considered include but are not limited to: the hourly usage plan of the household electric vehicle within the next 24 hours, the corresponding change in state of charge when the household electric vehicle is connected to the household energy system or the third-party energy system, and the performance constraint parameters of the household electric vehicle (including but not limited to: state of charge constraints at the start of the household electric vehicle, charging and discharging power constraints of the household electric vehicle, upper and lower limit constraints on the state of charge of the household electric vehicle), etc.

[0108] In some embodiments of the present invention, the preset time period includes: a plurality of sub-time periods of equal duration that are continuous in time sequence, such as: the next 24 hours can be divided equally into hours, and correspondingly, the usage plan information parameters, charge state parameters and performance constraint parameters of the household electric vehicle within the preset time period may respectively include: the usage plan information parameters of the household electric vehicle in each sub-time period, the charge state parameters of the household electric vehicle in each sub-time period and the performance constraint parameters of the household electric vehicle in each sub-time period.

[0109] In some embodiments of the present invention, the usage plan information parameters of the household electric vehicle in each sub-time period include: a first characterization value of whether the household electric vehicle is connected to or not connected to the household energy system in each sub-time period, a second characterization value of whether the household electric vehicle is connected to or not connected to the third-party energy system, and a planned mileage.

[0110] Here, taking each sub-time period as one hour as an example, the usage plan information parameters of the household electric vehicle per hour include: (n, x h,n , x c,n , d n ), where n represents the nth sub-time period within the preset time period, and the value of n is [0,23], x h,n Indicates whether the household electric vehicle is connected to the home energy system in the nth sub-time period. When its value is 1, it indicates that it is connected, and when its value is 0, it indicates that it is not connected, that is, the first characterization value mentioned above; c,nIndicates whether the household electric vehicle is connected to the third-party energy system in the nth sub-time period. When its value is 1, it indicates that it is connected, and when its value is 0, it indicates that it is not connected, which is the second characterization value mentioned above; d n Indicates the planned mileage of a household electric vehicle in the nth sub-time period.

[0111] In some embodiments of the present invention, the state of charge parameters of the household electric vehicle in each sub-time period include: the state of charge of the household electric vehicle in each sub-time period, the corresponding interaction power when connected to the different energy systems, the battery rated capacity, the charging efficiency and discharging efficiency of the internal battery pack, and the statistical average value of the state of charge decrease corresponding to the internal battery pack when traveling a preset number of kilometers.

[0112] Here, taking each sub-time period as one hour as an example, the state of charge parameters of the household electric vehicle per hour include: (SoC n , p EV,n , C EV , η ch , η dis ,u SoC ); Among them, SoC n represents the state of charge of the household electric vehicle in the nth sub-time period; p EV,n C represents the interaction power between the household electric vehicle and the energy system it is connected to (household energy system or third-party energy system) in the nth sub-time period. A positive value indicates that the household electric vehicle is charging, and a negative value indicates that the household electric vehicle is discharging (e.g., driving). EV Indicates the rated capacity of the battery of a household electric vehicle; η ch Characterizes the charging power of household electric vehicles; η dis Characterizes the discharge power of household electric vehicles; u SoC It represents the statistical average value of the decrease in the state of charge (SOC) of the energy storage battery corresponding to a household electric vehicle traveling 1 kilometer.

[0113] Here, the usage plan information parameters (n, x h,n , x c,n , d n ) and the state of charge parameter (SoC) of household electric vehicles per hour n , p EV,n , C EV , η ch , η dis ,u SoC ) to determine the state of charge SoC of the household electric vehicle in the n+1th sub-time period. n+1 , as shown in formula (1):

[0114]

[0115] In some embodiments of the present invention, the performance constraint parameters of the household electric vehicle in each sub-time period include: the minimum state of charge value, maximum state of charge value, minimum charge and discharge power value, maximum charge and discharge power value corresponding to the internal battery pack in each sub-time period, and the first minimum starting power corresponding to the household electric vehicle when the mileage of a single trip is less than a preset number of kilometers, and the second minimum starting power corresponding to the household electric vehicle when the mileage of a single trip is greater than or equal to the preset number of kilometers, and the second minimum starting power corresponding to the household electric vehicle when the mileage of a single trip is greater than or equal to the preset number of kilometers, wherein the second minimum starting power is a positive multiple of the first minimum starting power.

[0116] Here, continuing to take each sub-time period as one hour as an example, the performance constraint parameters of household electric vehicles per hour include: the power constraint required for a single trip of the household electric vehicle (including: the first minimum starting power corresponding to the household electric vehicle when the mileage of a single trip is less than the preset kilometers, the second minimum starting power corresponding to the household electric vehicle when the mileage of a single trip is greater than or equal to the preset kilometers), the battery charge threshold constraint of the household electric vehicle, the charging and discharging power threshold constraint of the household electric vehicle, etc.

[0117] Among them, the minimum starting power required for a single trip of a household electric vehicle can be agreed upon as follows: If the planned single trip mileage is less than the preset number of kilometers, such as: L low kilometers, then the minimum starting power of a single trip of a household electric vehicle is guaranteed to be no less than the minimum starting power SoC TH ; If the single trip mileage of the travel plan is greater than or equal to L low kilometers, the minimum starting power should be able to support the planned mileage of k sf times, but the minimum starting power does not exceed the maximum power of a household electric vehicle; for example, L low Can take value 50, SoC TH It can be 40% of the total rated capacity of the battery of a household electric vehicle, k sf is a positive number greater than 1; for example, it can be 1.5.

[0118] Based on the above description, in some embodiments of the present invention, the minimum starting power required for a single trip of a household electric vehicle (where the starting power refers to the state of charge of the household electric vehicle at the time of starting travel) may be determined by referring to the following Algorithm 1:

[0119] 1: Initialize the internal counting variable s of this algorithm = 0, initialize the internal counting array b(1) = -1, e(1) = -1

[0120] 2: For n = 1 to 23 do

[0121] 3: s=s+1, b(s)=-1, e(s)=-1

[0122] 4: While e(s) == -1

[0123] 5: If x h,n-1 +x c,n-1 =0 and x h,n +x c,n =1, then determine that n is the starting time of driving, let b(s) = n

[0124] 6: If x h,n-1 +x c,n-1 =1 and x h,n +x c,n = 0, then determine n-1 as the driving end period, e(s) = n

[0125] 7: End

[0126] 8: Calculate the mileage of the household electric car for the sth time:

[0127] 9: Calculate the starting power constraint for the sth trip of a household electric vehicle

[0128]

[0129] 10: End

[0130] It should be noted that, in the above algorithm 1, "n=1 to 23do" is used as an example to divide the next 24 hours into hours and to determine the corresponding n-th sub-time period.

[0131] In some embodiments of the present application, the battery charge threshold constraint of a household electric vehicle can be shown in formula (2):

[0132] SoC min ≤SoC n ≤SoC max Formula (2);

[0133] Among them, SoC min Refers to the minimum state of charge value corresponding to the internal battery pack of household electric vehicles, SoC max Refers to the maximum state of charge value corresponding to the internal battery pack of a household electric vehicle. For example, SoC min and SoC max These are 20% and 80% of the total rated capacity of the battery in a household electric vehicle, respectively.

[0134] In some embodiments of the present application, the charge and discharge power constraints of household electric vehicles can be shown in formula (3):

[0135] p EV,min ≤p EV,n ≤p EV,max Formula (3);

[0136] Among them, p EV,min Refers to the minimum charging and discharging power value corresponding to household electric vehicles, p EV,max Refers to the maximum charging and discharging power value corresponding to household electric vehicles. The specific value can be determined according to actual needs or customized by the manufacturer of the household electric vehicle.

[0137] In this way, based on the various parameters involved in the connection of household electric vehicles to the home energy system or the third-party energy system, such as: usage plan information parameters, charge state parameters and performance constraint parameters, a charging and discharging model for household electric vehicles to be connected to different household energy systems is constructed, which can provide parameter support for the overall optimization of electricity costs for subsequent household electric vehicles to be connected to the home energy system or the third-party energy system in different time periods.

[0138] Step 102: Establish a typical home energy system model corresponding to the home energy system within the preset time period.

[0139] In some embodiments of the present invention, a corresponding typical home energy system model can be established based on the fact that the equipment involved in the home energy system includes at least: electronic equipment with adjustable loads (such as water heaters, air conditioners), new energy power generation equipment, and equipment with uncontrollable loads (such as televisions, computers, washing machines, electric kitchen appliances, electric lights, fans and other household appliances manually controlled by users).

[0140] In some embodiments of the present invention, when the devices in the home energy system include at least: a first device with controllable load, a second device with uncontrollable load, and a third device for renewable energy power generation, step 102 may be implemented by following steps A1 to A3:

[0141] Step A1: Using the mechanism model and the working constraints corresponding to the first device, model the operating parameters of the first device within the preset time period, and construct a working model expression corresponding to the first device.

[0142] In some embodiments of the present invention, the next 24 hours are evenly divided into multiple hourly segments to construct corresponding operating model expressions. Here, taking the example of a first load-controllable device including a water heater and an air conditioner, a mechanism model is used to construct operating model expressions for the water heater and the air conditioner for each hour of the next 24 hours (e.g., the nth sub-time segment mentioned above).

[0143] It should be noted that a mechanistic model, also known as a white-box model, is a precise mathematical model established based on the internal mechanisms of an object, production process, or material flow transfer mechanism. It is based on mass balance equations, energy balance equations, momentum balance equations, phase balance equations, certain physical property equations, chemical reaction laws, and basic circuit laws to obtain a mathematical model of the object or process. The advantage of a mechanistic model is that the parameters have very clear physical meanings.

[0144] In some embodiments of the present invention, a working model expression of a water heater in a home energy system is constructed as shown in the following formula (4):

[0145]

[0146] Among them, θ WH,n represents the hot water temperature at the start of the nth sub-time period; θ WH,n+1 represents the hot water temperature at the start of the n+1th sub-time period; P WH Indicates the rated power of the water heater; A WH,n is the heating control switch for the nth sub-time period. When its value is 1, it indicates that the water heater is heating at rated power. When its value is 0, it indicates that the water heater is not heating. WH 、R WH 、C WH They are the rated power, equivalent thermal resistance and equivalent heat capacity of the water heater when it is working, all of which are positive constants; θ en,n is the temperature of the water injected into the water heater in the nth sub-time period; M is the volume of the water tank of the water heater; d WH,n is the volume of hot water consumed by the user in the home energy system during the nth sub-period.

[0147] It should be noted that the temperature of the hot water in the water heater should also meet the user comfort requirements of the home energy system. Therefore, θ WH,n The following formula (5) must also be satisfied:

[0148] θ WH,LB ≤ θ WH,n ≤θ WH,UB Formula (5);

[0149] Among them, θ WH,LB and θ WH,UB They represent the lower and upper temperature limits acceptable to the user respectively.

[0150] In some embodiments of the present invention, a working model expression of the air conditioner in the home energy system is constructed as shown in the following formula (6):

[0151]

[0152] Among them, θ AC,n is the temperature of the room where the air conditioner is located at the start time of the nth sub-time period; θ AC,n+1 is the temperature of the room where the air conditioner is located at the start time of the n+1th sub-time period; AC is the rated power of the air conditioner; A AC,n is the air conditioner switch in the nth sub-time period. When its value is 1, it means the air conditioner is working at rated power. When its value is 0, it means the air conditioner is not working. AC is the equivalent heat capacity of the air conditioner; R AC is the equivalent thermal resistance of the air conditioner; θ tm,n is the outside air temperature at the start of the nth sub-period.

[0153] It should be noted that, considering the temperature comfort, the air conditioner needs to meet the room temperature constraint during operation. Therefore, θ AC,n The following formula (7) must also be satisfied:

[0154] θ AC,LB ≤θ AC,n ≤θ AC,UB Formula (7);

[0155] Among them, θ AC,LB Refers to the lower limit of the indoor temperature acceptable to the user θ AC,UB Refers to the upper limit of indoor temperature acceptable to users.

[0156] Step A2: predicting the total load power prediction value corresponding to the second device and the generated power prediction value corresponding to the third device within the preset time period.

[0157] In some embodiments of the present invention, the third device for renewable energy generation and the second device with uncontrollable load are not controlled by the home energy system. The power generation power prediction value p of the third device corresponding to each sub-time period within the preset time period, such as the nth sub-time period, can be directly obtained from the relevant prediction system. RN,n , the total load power prediction value p corresponding to the second device FD,n .

[0158] Step A3: construct the typical home energy system model based on the working model expression, the total load power prediction value and the generated power prediction value.

[0159] In some embodiments of the present invention, the obtained working model expression, the total load power prediction value, and the generated power prediction value are directly integrated to obtain a typical home energy system model.

[0160] In this way, based on the relevant working parameters and constraint information corresponding to the various devices in the home energy system, a typical home energy system model corresponding to the home energy system is built, which can provide rich and relatively comprehensive parameter support for the subsequent construction of a household total electricity cost prediction model.

[0161] Step 103: Using a proximal strategy optimization algorithm, based on the charging and discharging model and the typical home energy system model, construct a prediction model for the total household electricity cost within the preset time period.

[0162] In some embodiments of the present invention, a model for predicting the total household electricity cost within the next 24 hours is constructed based on a charge-discharge model and a typical household energy system model using a proximal policy optimization (PPO) algorithm. The following factors are considered:

[0163] The state of charge of the household electric vehicle during the decision period; the temperature of hot water in the water heater; the indoor temperature of the air-conditioned room, and a series of predicted values ​​for the 24 hours starting one hour after the decision period: whether the household electric vehicle is connected to the home energy system, whether the household electric vehicle is connected to a third-party energy system, the planned mileage of the household electric vehicle, the volume of hot water consumed by the user, the temperature of the water injected into the water heater, the outside air temperature, the power generation power of new energy, the total power of electronic equipment with uncontrollable loads, the electricity purchase price of the third-party energy system, the electricity sales price of the third-party energy system, the electricity purchase price of the household energy system, and the electricity sales price of the household energy system.

[0164] The prediction model of household total electricity cost in the next 24 hours has the following characteristics:

[0165] 1) Home electric vehicles can generate interactive power with both the home energy system and a third-party energy system, but not both at the same time;

[0166] 2) The total household electricity cost in the next 24 hours includes: the electricity cost incurred by the household energy system when the household electric vehicle is not connected to the household energy system in the next 24 hours, the electricity cost incurred by the household energy system and the household electric vehicle when the household electric vehicle is connected to the household energy system in the next 24 hours, and the electricity cost incurred by the household electric vehicle when the household electric vehicle is connected to a third-party energy system in the next 24 hours;

[0167] 3) It can simultaneously process one continuous action variable (the charge and discharge power corresponding to a household electric vehicle) and two discrete action variables (the air conditioner switch and the water heater switch; here, the devices in the household energy system including the air conditioner and water heater are used as an example);

[0168] 4) It can output the operating power of electronic devices with adjustable loads in the home energy system for the next 24 hours, output the charging and discharging power of home electric vehicles during the period when they are connected to the home energy system, and output the charging and discharging power of home electric vehicles during the period when they are connected to a third-party energy system.

[0169] In some embodiments of the present invention, the above step 103 may be implemented by the following steps 1031 and 1032 (not shown in the figure):

[0170] Step 1031: Using the proximal strategy optimization algorithm, based on the charging and discharging model and the typical home energy system model, construct an initial model of the total household electricity cost within the preset time period.

[0171] In some embodiments of the present invention, the controlled objects involved in the initial model of the total household electricity cost within a preset time period include: a household energy system, a household electric vehicle, and a third-party energy system.

[0172] In some embodiments of the present invention, step 1031 may be implemented by the following process:

[0173] First, based on the external environmental information, the current status of the equipment in the different energy systems, the control action instructions output by the mobilization system corresponding to the different energy systems, and user demand information, the charging and discharging model and the typical home energy system model are used to digitally simulate the different energy systems and the home electric vehicle to obtain an energy system simulation model.

[0174] In some embodiments of the present invention, user demand information (including: air conditioning temperature, water heater temperature, driving duration, time and distance of household electric vehicles in the home energy system), external environmental information (such as: external temperature, electricity purchase price of third-party energy system, electricity purchase price of home energy system, etc.), current status of equipment in different energy systems (including but not limited to the current status of each device in the home energy system and the current status of power supply equipment in the third-party energy system) and control action instructions (open, close, hold, etc.) corresponding to the mobilization system output of different energy systems can be obtained first, and then the above-mentioned information obtained, the corresponding input value charging and discharging model and the typical home energy system model are digitally simulated to obtain an energy system simulation model.

[0175] Then, the proximal strategy optimization algorithm is used to build an initial model of the total household electricity cost corresponding to the energy system simulation model within the preset time period.

[0176] In some embodiments of the present invention, a network architecture corresponding to the proximal strategy optimization algorithm can be adopted, with the home energy system, home electric vehicle and third-party energy system as the controlled objects, to build an initial model of the total household electricity cost including the energy system simulation model.

[0177] In this way, based on external environmental information, the current status of equipment in different energy systems, the control action instructions output by the mobilization systems corresponding to different energy systems, and user demand information, with the help of charging and discharging models and typical household energy system models, an initial model of the total household electricity cost corresponding to the household energy system, household electric vehicles, and third-party energy systems as a whole is constructed. This can provide an initial model structure for building the final household total electricity cost prediction model based on the participation of household electric vehicles in demand response in many aspects.

[0178] Step 1032: Perform deep reinforcement learning on the initial model of the total household electricity cost to obtain the total household electricity cost prediction model.

[0179] In some embodiments of the present invention, since the initial model of the total household electricity cost is built based on the network architecture corresponding to PPO, the action space, state space, and immediate reward of PPO can be constructed first, and then the initial model of the total household electricity cost can be subjected to deep reinforcement learning (iterative optimization) with the help of the action space, state space, and immediate reward to obtain a prediction model for the total household electricity cost.

[0180] In some embodiments of the present invention, before executing step 1032, step B1 may be executed first:

[0181] Step B1: Based on the power parameters of the charging and discharging model and the device state parameters of the typical home energy system model, construct the action space, state space and immediate reward of the proximal strategy optimization algorithm in the preset time period.

[0182] In some embodiments of the present invention, some parameters, such as power parameters and device state parameters, can be directly extracted from the charging and discharging model and the typical home energy system model to build the action space, state space, and immediate reward of PPO within a preset time period (the next 24 hours).

[0183] In some embodiments of the present invention, the above step B1 can be implemented by following steps B11 to B13:

[0184] Step B11: Within the preset time period, an action space is constructed based on the interaction power corresponding to when the household electric vehicle is connected to the different energy systems in the charging and discharging model, and the action parameters in the typical household energy system model used to characterize the opening or closing of the load controllable device.

[0185] Wherein, the home energy system at least includes the load controllable device.

[0186] Step B12: Based on the state of charge parameters in the charging and discharging model, the power parameters corresponding to the equipment of the typical home energy system model, and the preset electricity price information, the endogenous state parameters of the energy system and the exogenous state parameters of the energy system operating environment corresponding to the preset time period are respectively constructed, and based on the endogenous state parameters of the energy system and the exogenous state parameters of the energy system operating environment, a state space is constructed.

[0187] Step B13: Based on the performance constraint parameters in the charging and discharging model and the working constraints of the load-controllable device corresponding to the typical home energy system model, determine the total household electricity cost within the preset time period and the penalty for violating the preset constraints, and construct an immediate reward based on the total household electricity cost and the penalty.

[0188] It should be noted that since some of the parameters involved in the above are described in terms of the nth sub-time period, the action space, state space, and immediate reward of the preset time period are also divided into the action space, state space, and immediate reward of multiple sub-time periods for description. The action space, state space, and immediate reward corresponding to the nth sub-time period are also described below.

[0189] In some embodiments of the present invention, the action space a of the PPO in the nth sub-time period is constructed based on the interaction power corresponding to the connection of household electric vehicles to different energy systems in the charge and discharge model in the nth sub-time period and the action parameters in the typical household energy system model used to characterize the opening or closing of load controllable devices. n ={p EV,n ,A WH,n ,A AC,n}; where p EV,n (The interaction power of the household electric vehicle and the energy system it is connected to in the nth sub-time period) is a continuous decision variable, and its corresponding model is a normal distribution. Here, the p in the nth sub-time period can be EV,n The mean of the corresponding normal distribution is set to μ EV,n , the variance is set to The corresponding formula is shown in (8):

[0190]

[0191] Correspondingly, A WH,n and A AC,n They are all switch quantities and can be modeled as Bernoulli distribution, as shown in the following formula (9):

[0192]

[0193] Among them, p WH,n is the nth sub-time period, A WH,n =1 probability; p AC,n is the nth sub-time period, A AC,n =1 probability.

[0194] In some embodiments of the present invention, first, the energy system endogenous state parameter s corresponding to the nth sub-time period can be constructed based on the state of charge parameter in the charge and discharge model in the nth sub-time period, the power parameter corresponding to the equipment of the typical home energy system model, and the preset electricity price information. env,n , and the exogenous state parameter s of the energy system operating environment corresponding to the nth sub-time period ex,n (can be obtained from the prediction system); then, based on the endogenous state parameter s of the energy system corresponding to the nth sub-time period env,n The exogenous state parameter s of the energy system operating environment corresponding to the nth sub-time period ex,n , construct the state space s corresponding to the nth sub-time period n ={s env,n ,s ex,n}. Here, among them, s env,n and s ex,n They can be determined by the following formulas (10) and (11):

[0195] s env,n ={n,SoC n ,θ WH,n ,θ AC,n} Formula (10);

[0196]

[0197] Wherein, in formula (11), i=[n+1, n+24], so the relevant parameters in the formula, such as: h,i 、x c,i d i d WH,i ,θ en,i ,θ tm,i 、p RN,i and p FD,i Corresponding to the x described above h,n 、x c,n d n d WH,n ,θ en,n ,θtm,n 、p RN,n and p FD,n ;at the same time, and The same can be described as: and and and They represent the electricity purchase price and electricity sales price of the third-party energy system in the nth sub-time period, and They represent the electricity purchase price and electricity sales price of the household energy system in the nth sub-time period respectively.

[0198] Correspondingly, the immediate reward r in the nth sub-period is n Includes: the total electricity cost of the home energy system in the nth sub-time period n , the penalty for violating the hot water temperature of the water heater and the indoor temperature constraints of the air-conditioned room in the home energy system in the n-th sub-time period, the penalty for violating the upper and lower limit constraints on the charging and discharging power and the state of charge constraints of the household electric vehicle in the n-th sub-time period, and the penalty caused by the non-zero power of the household electric vehicle interacting with the outside world when the household electric vehicle is neither connected to the home energy system nor to a third-party energy system in the n-th sub-time period.

[0199] In some embodiments of the present invention, the total electricity cost of the home energy system in the nth sub-time period is cost n It can include the following three parts: the electricity cost generated by the household electric vehicle when it is connected to the third-party energy system during the nth sub-time period: c,EV,n ; When the household electric vehicle is not connected to the third-party energy system or the household energy system during the nth sub-time period, the electricity cost generated by the household energy system is cost h,n When a household electric vehicle is connected to the home energy system during the nth sub-time period, the sum of the electricity costs incurred by the household electric vehicle and the home energy system is cost hev,n .

[0200] Here, the electricity cost generated by the household electric vehicle when it is connected to the third-party energy system in the nth sub-time period is cost c,EV,n , the corresponding model can be established by referring to the following formula (12):

[0201] cost c,EV,n =λ c,n x c,n p EV,n Formula (12);

[0202] Among them, λ c,n represents the unified expression of the electricity price of the third-party energy system in the nth sub-time period. Here, λ c,nCorresponding to the following formula (13):

[0203]

[0204] Correspondingly, when the household electric vehicle is not connected to the third-party energy system or the household energy system during the nth sub-time period, the electricity cost generated by the household energy system is cost h,n , which can be expressed by the following formula (14):

[0205]

[0206] Among them, λ h,n represents the unified expression of the electricity price of the home energy system in the nth sub-time period, represents the power of interaction between the home energy system and the grid in the nth sub-time period, where λ h,n Corresponding to the following formula (15):

[0207]

[0208] at the same time, When the household electric vehicle is not connected to the home energy system during the nth sub-time period, the following formula (16) corresponds:

[0209]

[0210] Correspondingly, when the household electric vehicle is connected to the home energy system during the nth sub-time period, the sum of the electricity costs generated by the household electric vehicle and the home energy system is cost hev,n , refer to the following formula (17):

[0211]

[0212] at the same time, When a household electric vehicle is connected to the home energy system in the nth sub-time period, the following formula (18) corresponds:

[0213]

[0214] Continuing from the above description, the total electricity cost of the home energy system in the nth sub-time period is cost n It can be described as the following formula (19):

[0215]

[0216] In some embodiments of the present invention, the penalty for violating the constraints of the water heater hot water temperature and the indoor temperature in the air-conditioned room in the home energy system in the nth sub-time period may be: if the water heater hot water temperature in the nth sub-time period violates the constraint corresponding to formula (5) described in the above embodiment, the penalty is generated. If the indoor temperature in the air-conditioned room in the nth sub-time period violates the constraint corresponding to formula (7) described in the above embodiment, a penalty is generated.

[0217] In some embodiments of the present invention, the penalty for violating the upper and lower limit constraints of the charge and discharge power and the state of charge constraint of the household electric vehicle in the nth sub-time period may be: if the charge and discharge power of the household electric vehicle in the nth sub-time period violates the constraint corresponding to formula (3) described in the above embodiment, the penalty is generated. If the state of charge of the household electric vehicle battery in the nth sub-time period violates the constraint corresponding to formula (2) described in the above embodiment, a penalty is generated.

[0218] In some embodiments of the present invention, when the household electric vehicle is neither connected to the household energy system nor to the third-party energy system in the nth sub-time period, the penalty caused by the non-zero power of the household electric vehicle interacting with the outside world is You can refer to the following formula (20):

[0219]

[0220] Based on the above description, the immediate reward r of the nth sub-period is n The following formula (21) can be used:

[0221]

[0222] In this way, based on the power parameters of the charging and discharging model and the equipment state parameters of the typical household energy system model, as well as information such as renewable energy, electricity prices, natural air temperature, natural water temperature, and user energy demand faced by different energy systems connected to household electric vehicles, the action space, state space, and immediate reward of PPO in a preset time period are constructed, which provides parameter support for the subsequent deep reinforcement learning of the initial model of the total household electricity cost, and can further improve the accuracy of the household total electricity cost prediction model.

[0223] Correspondingly, the above step 1032 can be implemented by following the following step B2:

[0224] Step B2: Based on the action space, the state space, and the immediate reward, deep reinforcement learning is performed on the initial model of the total household electricity cost to obtain the total household electricity cost prediction model.

[0225] In some embodiments of the present invention, deep reinforcement learning can be performed on the initial model of the total household electricity cost based on the action space, state space, and immediate reward of the constructed PPO in a preset time period to obtain a prediction model for the total household electricity cost. Here, performing deep reinforcement learning on the initial model of the total household electricity cost involves training the deep reinforcement learning module of the PPO. That is, based on the action space, state space, and immediate reward, deep reinforcement learning is performed on the deep reinforcement learning module within the initial model of the total household electricity cost (because the initial model of the total household electricity cost is built using PPO, it must include a deep reinforcement learning module) until the output converges or a prediction model for the total household electricity cost corresponding to certain indicators is obtained.

[0226] It should be noted that the deep reinforcement learning module of PPO is usually composed of three networks: a new policy network, an old policy network, and an evaluation network. Correspondingly, in some embodiments of the present invention, the deep reinforcement learning module of the initial model of the total household electricity cost also includes: a new policy network, an old policy network, and an evaluation network. The new policy network, the old policy network, and the evaluation network can be designed as follows:

[0227] On the one hand, the new policy network and the old policy network can have exactly the same structure, which is designed as a fully connected structure between adjacent layers, and both networks are 6-layer neural networks; wherein, the 6-layer neural network can have 232, 512, 512, 256, 128, and 4 neurons from the input layer to the output layer. Here, the state space s of the nth sub-time period can be n The 232 elements of the input layer are connected to the 232 neurons of the input layer, and the 4 neurons of the output layer sequentially output the action space a of the nth sub-time period. n The determinants of μ EV,n 、 p WH,n 、p AC,n Here, to avoid confusion, the corresponding output of the new policy network can be recorded as (Further, it can be used to determine ), The corresponding output of the old policy network is recorded as (Further, it can be used to determine ),

[0228] In addition, using θ new Denote the parameter set of the new policy network, with θ new,j Represents the parameter set of the new strategy network updated for the jth time, parameter set θ new The corresponding action strategy of the new strategy network is recorded as Use θ oldRepresents the parameter set of the old policy network, with θ old,j Represents the parameter set of the old strategy network updated for the jth time, parameter set θ old The corresponding action strategy of the old strategy network is recorded as

[0229] On the other hand, the evaluation network is designed as a fully connected structure between adjacent layers, which can be a 5-layer neural network; wherein, the 5-layer neural network has 232, 512, 256, 64, and 1 neurons from the input layer to the output layer. Here, the state space s of the nth sub-time period is n Take the 232 elements of the 5-layer neural network as an example, and the 232 neurons in the input layer can output s i The corresponding state value function v π (s i ); ω represents the parameter set of the evaluation network, and ω j represents the parameter set of the evaluation network updated for the jth time.

[0230] In some embodiments of the present invention, the above step B2 may be implemented by following steps B21 to B24:

[0231] Step B21: Within the preset time period, based on the state parameters provided by the state space, determine a probability ratio corresponding to the execution of action parameters in the action space between the new policy network and the old policy network.

[0232] In some embodiments of the present invention, the probability ratio corresponding to the action parameters in the action space executed between the new policy network and the old policy network in the nth sub-time period can be first determined, and then multiple sub-time periods (i.e., each sub-time period corresponding to n being [0, 23]) can be counted to obtain the corresponding probability ratio within the preset time period.

[0233] Here, the state space s based on the nth sub-time period n The parameter information provided determines the action space a between the new policy network and the old policy network in the deep reinforcement learning of the initial model of the total household electricity cost, taking the nth sub-time period n The probability ratio ψ n (θ), as shown in formula (22):

[0234]

[0235] Step B22: Based on the immediate reward, use the evaluation network to determine the estimated error of the state value corresponding to the state space of the preset time period.

[0236] In some embodiments of the present invention, first, the state space s of the nth sub-time period is determined.n The state value function v π (s n ), then, the state value function v is determined based on the evaluation network π (s n ) corresponds to the estimated error.

[0237] Among them, when i mentioned above is n, the state value function v π (s n ), can refer to the state space s n In the case of , the expected value of the cumulative reward obtained by selecting actions according to the given strategy π is shown in formula (23):

[0238]

[0239] Where γ is the long-term return discount factor, γ∈(0,1); function E π () means averaging the contents in the brackets under strategy π; q is the number of periods counted forward from the nth sub-period; N is the number of the last sub-period in the preset time period (the next 24 hours); γ q is the qth power of γ; r n+q is the immediate return in the n+qth sub-period.

[0240] Correspondingly, in the evaluation network, δ n Represents the state value function v π (s n ) corresponds to the estimated error, and its corresponding calculation method is shown in formula (24):

[0241] δ n =r n +γ·v π (s n+1 )-v π (s n ) formula (24);

[0242] Among them, v π (s n+1 ) represents the state space s of the n+1th sub-time period n+1 The state value function of .

[0243] Step B23: Determine the multi-step advantage of the state space within the preset time period based on the estimation error.

[0244] In some embodiments of the present invention, the determined estimation error is directly used to determine the corresponding multi-step advantage A n , refer to formula (25):

[0245]

[0246] Among them, δ n+q Represents the state value function v of the n+q sub-time period π (s n+q ) corresponding to the estimation error; λ is the balance factor, λ∈[0,1].

[0247] Step B24: Based on the probability ratio, the multi-step advantage and the first learning rate, the network parameters of the new policy network and the network parameters of the old policy network are iteratively optimized, and based on the state value and the second learning rate, the network parameters of the evaluation network are iteratively optimized until the deep reinforcement learning module whose output parameters meet the preset convergence conditions is obtained, and the model corresponding to the deep reinforcement learning module whose output parameters meet the preset convergence conditions is determined as the household total electricity cost prediction model.

[0248] In some embodiments of the present invention, the loss function L of the evaluation network can be defined as c The calculation of (ω) is shown in formula (26):

[0249]

[0250] Among them, the function E n () means finding the average of the contents in the brackets under strategy n.

[0251] Here, l1 is used to represent the learning rate of the evaluation network, and the parameter ω of the evaluation network is j , can refer to formula (27) for update:

[0252]

[0253] In some embodiments of the present invention, the loss function L of the new policy network is a (θ) can be calculated according to the following formula (28):

[0254] L a (θ)=E n {min[ψ n (θ),clip(ψ n (θ),1-η,1+η)]A n} Formula (28);

[0255] In the formula, the function clip(x) converts ψ n (θ) is limited to the interval [1-η,1+η], where η is the shear coefficient, which is used to limit the degree of deviation between the new policy network and the old policy network.

[0256] In some embodiments of the present invention, the learning rate of the new policy network can be expressed as l2, and the parameter θ of the new policy network is new,j It can be updated according to the following formula (29):

[0257]

[0258] It should be noted that during the training process of the deep reinforcement learning module, in order to prevent the difference between the new policy network and the old two policy networks from being too large, the parameters of the new policy network need to be copied to the old policy network every B rounds. Thus, a step function u(x) is defined. When the variable x≥0, u(x)=1; when the variable x<0, u(x)=0; the specific value of B can be determined according to actual needs, and the present invention does not impose any limitation on this.

[0259] Referring to the above description, the PPO model for predicting the total household electricity cost in the next 24 hours can be implemented using the following algorithm 2:

[0260] 1: Initialize γ, λ, l1, l2, η, q, and initialize the parameters θ of the new policy network new,j , the parameters of the evaluation network ω j , initialize the parameters θ of the old policy network old,j ←θ new,j , initialize the empirical trajectory O, initialize the PPO convergence flag G = 0

[0261] 2:Repeat

[0262] 3: Obtain s from the physical system corresponding to the home energy system, third-party energy system, and home electric vehicle env,0

[0263] 4: Get s from the prediction system ex,0

[0264] 5: Initial state s0 = [s env,0 ,s ex,0 ]

[0265] 6: For n = 0 to 23 do

[0266] 7: Old policy network input s n , the distribution function of the action is adjusted by its output parameters, and a is sampled from it n

[0267] 8: Perform a n , and s is determined by energy system model simulation env,n+1

[0268] 9: Calculate the immediate reward r n

[0269] 10: Get s from the prediction system ex,n+1

[0270] 11: s n+1 =[s env,n+1 ,s ex,n+1 ]

[0271] 12: O n =[O n-1 ;s n ,a n ,r n ,s n+1 ]

[0272] 13: End For

[0273] 14: Calculate L according to the above formula (26) c (ω), and update ω according to the above formula (27) j

[0274] 15: Calculate according to the above formula (28) L a (θ), and update θ according to the above formula (29) new,j

[0275] 16: Every B rounds, θ old,j ←θ new,j

[0276] 17: If the standard deviation of the total return for D consecutive rounds is less than the threshold ξ, and all penalty returns are 0, then the convergence flag G = 1

[0277] 18: Until G==1

[0278] 19: Output And sent to home energy systems, home electric vehicles, and third-party energy systems

[0279] Similarly, refer to the description of Algorithm 1 above. In Algorithm 2, “n=1 to 23do” is to divide the next 24 hours into hours, and to determine the corresponding n-th sub-time period as an example. n is the empirical trajectory of the nth sub-time period, O n-1 is the empirical trajectory of the n-1th sub-time period.

[0280] In some embodiments of the present invention, the network parameters of the new policy network and the network parameters of the old policy network can be iteratively optimized based on the obtained probability ratio, multi-step advantage, and first learning rate. At the same time, the network parameters of the evaluation network can be iteratively optimized based on the state value and second learning rate until a model corresponding to the deep reinforcement learning module whose output parameters meet the preset convergence conditions is obtained, which is a household total electricity cost prediction model. Here, in the synchronous iterative optimization phase of the new policy network and the old policy network, after the new policy network is optimized, the original new policy network will be replaced with the new old policy network in a new round of parameter optimization phase, that is, θ old,j ←θ new,j .

[0281] It should be noted that the first learning rate and the second learning rate are important hyperparameters of the new policy network, the old policy network and the evaluation network in the training process (iterative optimization), respectively. They can determine the step size for updating the weights of the models in which the new and old policy networks and the evaluation network are located in each iteration, thereby affecting the learning speed and optimization effect of the model; wherein, the values ​​corresponding to the first learning rate and the second learning rate can be the same or different, and the present invention does not impose any limitation on this.

[0282] In this way, by taking into account the relevant functions for processing continuous actions (charging and discharging of household electric vehicles) and multiple switch operations (turning on and off devices in the home energy system, etc.), deep reinforcement learning (training optimization) is performed on the deep reinforcement learning module (including: new policy network, old policy network and evaluation network) in the initial model of the total household electricity cost. This can obtain a model that is more in line with the system required for the household electric vehicle, home energy system and third-party energy system to be built as a whole, namely: a household total electricity cost prediction model, thereby improving the accuracy of the subsequent household total electricity cost prediction model in predicting the minimum electricity cost of the system.

[0283] In some embodiments of the present invention, the deep reinforcement learning module involved in the initial model of the total household electricity cost is mainly responsible for the extraction and application of optimization knowledge within the initial model of the total household electricity cost. It can generate deep reinforcement learning samples based on the feedback information formed by the simulation results of the energy system simulation model within the initial model of the total household electricity cost and the prediction results output by the prediction system within the initial model of the total household electricity cost, thereby extracting optimization knowledge from the deep reinforcement learning samples and obtaining the evolution of the corresponding optimization strategy; and before the deep reinforcement learning module within the initial model of the total household electricity cost converges, the deep reinforcement learning module can output control actions corresponding to the optimization knowledge it has mastered according to probability, and act on the energy system simulation model corresponding to the initial model of the total household electricity cost, so that the energy system simulation model continues to simulate. This process is continuously cycled, so that after the deep reinforcement learning module of the initial model of the total household electricity cost converges, it outputs control action reference values ​​and operation reference values ​​to the household energy system, household electric vehicle, and third-party energy system to control the operation of the household energy system, household electric vehicle, and third-party energy system as a whole.

[0284] It should be noted that the prediction results output by the prediction system within the initial model of the total household electricity cost include: whether and when the household electric vehicle is connected to the household energy system, whether and when it is connected to the third-party energy system, and mileage data every hour (each sub-time period within the preset time period) in the next 24 hours; the volume of hot water consumed by users of the household energy system every hour in the next 24 hours, the temperature of the water injected into the water heater, the outside air temperature, the power generation power of the new energy equipment in the household energy system, the total power of the uncontrollable load equipment in the household energy system, the electricity purchase price and electricity sales price of the third-party energy system when the household electric vehicle is connected to the third-party energy system, and the electricity purchase price and electricity sales price of the household energy system when the household electric vehicle is connected to the household energy system.

[0285] In this way, with the help of information such as the power parameters of the charging and discharging model and the equipment state parameters of the typical household energy system model, the action space, state space and immediate reward are constructed, and deep reinforcement learning is performed on the initial model of the household total electricity cost to obtain a household total electricity cost prediction model, which can effectively deal with the uncertainty factors involved in the household energy system and the third-party energy system, thereby helping to ensure the optimization accuracy and prediction quality of the obtained household total electricity cost prediction model, and further conducive to the stable operation of the system corresponding to household electric vehicles, household energy systems and third-party energy systems as a whole within the power system.

[0286] At the same time, a proximal strategy optimization algorithm is adopted to construct a household total electricity cost prediction model corresponding to the optimization algorithm of the household total electricity cost within a preset time period based on the charging and discharging model and the typical household energy system model. This can not only minimize the household total electricity cost based on the household electric vehicle, household energy system and third-party energy system as a whole, but also ensure the optimization accuracy corresponding to the household total electricity cost prediction model from the algorithm principle, so as to further improve the prediction accuracy of the household total electricity cost prediction model.

[0287] Because household electric vehicles are connected to different energy systems, the connected energy systems and the household electric vehicles need to be considered as a whole for demand response optimization. Since household electric vehicles can be connected to both the household energy system and third-party energy systems, it is difficult to ensure the lowest total household electricity cost in related technologies. In addition, because the energy system connected to household electric vehicles faces a series of uncertainties such as renewable energy, electricity prices, natural air temperature, natural water temperature, and user energy demand, based on this, the present invention provides a method for constructing a household total electricity cost prediction model. By using a multi-parameter-based charging and discharging model of household electric vehicles that can be connected to different energy systems within a preset time period and a typical household energy system model, a household total electricity cost prediction model for the preset time period is constructed based on a proximal strategy optimization algorithm (deep reinforcement learning). On the one hand, it can achieve unified optimization of household electric vehicles connected to the household energy system and other energy systems in different time periods under a dynamic electricity price environment, minimizing the total household electricity cost expenditure, thereby creating economic value for the users of the household electric vehicles and enhancing the willingness of household electric vehicles to participate in demand response, which is beneficial to energy conservation and emission reduction and the safe, stable and economic operation of the power system. On the one hand, through deep reinforcement learning (proximal strategy optimization algorithm), it is possible to optimize the total household electricity cost in an environment with uncertain demand factors in many aspects, that is, to ensure the optimization accuracy from the principle (algorithm), which is conducive to users of household electric vehicles to follow relevant scheduling instructions to improve the quality of electricity demand response, and thus help household electric vehicles, household energy systems and third-party energy systems as a whole to operate stably in the power system on the basis of safety and economy.

[0288] Example 2

[0289] For corresponding reference, see Figure 2 The figure is a flow chart of a method for predicting the total electricity cost of a household provided by the present invention. Figure 2 The following steps are explained:

[0290] Step 201: Based on the real-time operating parameters of the equipment in the home energy system and the real-time charging and discharging parameters corresponding to the home electric vehicle connected to different energy systems, the home total electricity cost prediction model constructed by the method described in any of the above embodiments is trained until convergence to obtain a home total electricity cost optimization model.

[0291] The different energy systems include: home energy systems and third-party energy systems.

[0292] Step 202: Use the operating power value output by the household total electricity cost optimization model as the operating power reference value of the equipment in the household energy system, and use the charging and discharging power reference value output by the household total electricity cost optimization model as the charging and discharging power reference value when the household electric vehicle is connected to the different energy systems.

[0293] In some embodiments of the present invention, the real-time operating parameters of the equipment in the home energy system and the real-time charging and discharging parameters corresponding to the home electric vehicle connected to different energy systems are collected in real time, and the home total electricity cost prediction model is trained online until convergence to obtain a home total electricity cost optimization model, so that the predicted value (reference value) output by the home total electricity cost optimization model is sent to the relevant equipment in the home energy system model as an operating reference parameter, such as: when to turn on and off the air conditioner, the temperature and on and off information of the water heater, and the relevant reference value is provided as a reference for when the home electric vehicle is connected to the home energy system or a third-party energy system, the corresponding access time, etc.

[0294] Thus, the present invention provides a method for predicting household total electricity costs. This method trains a household total electricity cost prediction model online until convergence, and uses the output of the corresponding household total electricity cost optimization model after convergence as a reference value for the operation of relevant devices within the home energy system, as well as a reference value for the charge and discharge power of household electric vehicles when connected to different energy systems. On the one hand, this method can achieve unified optimization of the total electricity cost of household electric vehicles connected to a home energy system or a third-party energy system as a whole, thereby minimizing the household's total electricity cost expenditure. This can create economic value for the users of the household electric vehicles and increase their willingness to participate in demand response, thereby facilitating energy conservation and emission reduction and the safe, stable and economical operation of the power system. On the other hand, the household total electricity cost prediction model is trained online based on real-time operating parameters and real-time charge and discharge parameters to achieve online optimization of the household total electricity cost. This effectively addresses the uncertainty of relevant performance parameters and operating parameters within different energy systems, helps ensure the accuracy of demand response for household total electricity consumption, and improves the quality of demand response corresponding to the household total electricity cost. This, in turn, facilitates the safe and economical operation of the household electric vehicle, the home energy system, and the third-party energy system as a whole within the power system.

[0295] Based on the above description, combined with the description of Example 1 and Example 2, take the prediction of the total household electricity cost corresponding to the next 24 hours as an example, the corresponding execution steps can be referred to Figure 3 FIG. 1 is a flow chart showing a process for optimizing the total household electricity cost of a household electric vehicle based on multiple demand participation responses according to an embodiment of the present invention; wherein:

[0296] Step 301: Establish a charge and discharge model considering the connection of household electric vehicles to different energy systems. Here, the preset time period may refer to the next 24 hours, so the corresponding description of step 101 in the above embodiment 1 can be referred to and will not be repeated here.

[0297] Step 302: Establish a typical home energy system model. Please refer to the description of step 102 in the above embodiment 1, and do not repeat it here.

[0298] Step 303: Establish a deep reinforcement learning optimization model for the total household electricity cost in the next 24 hours. Here, you can refer to the description of step 103 in the above embodiment 1, that is, use PPO to build a prediction model for the total household electricity cost in the next 24 hours based on the charging and discharging model established in step 301 and the typical household energy system model established in step 302.

[0299] Step 304: Online training of a deep reinforcement learning optimization model for the total household electricity cost within the next 24 hours, and using the output of the optimization model after convergence as a reference value for the operating power of the adjustable load devices in the home energy system, a reference value for the charge and discharge power during the period when the household electric vehicle is connected to the home energy system, or a reference value for the charge and discharge power during the period when the household electric vehicle is connected to a third-party energy system. Here, reference may be made to steps 201 and 202 provided in Example 2 above, i.e., using real-time data (real-time operating parameters of the devices within the home energy system, real-time charge and discharge parameters corresponding to the household electric vehicle being connected to different energy systems), online training of the deep reinforcement learning optimization model for the total household electricity cost within the next 24 hours obtained in step 303 is performed to obtain an online optimized model, and using the output prediction value of the online optimized model as a reference value for the operating power of the adjustable load devices in the home energy system, a reference value for the charge and discharge power during the period when the household electric vehicle is connected to the home energy system, or a reference value for the charge and discharge power during the period when the household electric vehicle is connected to a third-party energy system.

[0300] Corresponding, please refer to Figure 4 As shown in the figure, it is a schematic diagram of the framework of the deep reinforcement learning optimization model of the total household electricity cost in the next 24 hours established in the above step 303, wherein: the deep reinforcement learning optimization model of the total household electricity cost in the next 24 hours is the household total electricity cost prediction model within the preset time period described in the above embodiment 1. At the same time, the deep reinforcement learning optimization model of the total household electricity cost in the next 24 hours, that is, Figure 4 The controlled objects corresponding to 401 shown in the figure are: home energy system, home electric vehicle and third-party energy system, that is, Figure 4 402 shown in .

[0301] Here, because 401 is a related model built based on PPO, it includes: a deep reinforcement learning module 4011, an energy system simulation model (energy system simulation module) 4013 and a prediction system (prediction module) 4012. Among them, the 4011 is mainly responsible for the optimization and application of the deep reinforcement learning optimization model of the total household electricity cost in the next 24 hours. It can generate deep reinforcement learning samples based on the feedback information formed by the simulation results of the energy system simulation model 4013 (here, the simulation results can be the real-time digital simulation of the real-time operating data corresponding to the household energy system, household electric vehicle and third-party energy system by the energy system simulation model 4013) and the prediction results output by the prediction system 4012, thereby extracting optimization knowledge from the samples and obtaining the evolution of the optimization strategy; and before the deep reinforcement learning module 4011 converges, the deep The deep reinforcement learning module 4011 can output control actions corresponding to the optimization knowledge it possesses according to probability, so as to act on the energy system simulation model 4013, so that the energy system simulation model 4013 continues to simulate. This process is continuously cycled, and after the deep reinforcement learning module 4011 converges, it outputs relevant scheduling instructions to act on the home energy system, home electric vehicles and third-party energy systems, thereby controlling the operation of the home energy system, home electric vehicles and third-party energy systems, and further achieving the minimum total household electricity cost expenditure when the home electric vehicle is connected to the home energy system and the third-party energy system respectively within the next 24 hours.

[0302] At the same time, you can refer to Figure 5 As shown in FIG, a schematic diagram of a framework of a household total electricity cost prediction model based on PPO within a preset time period; here, continuing to refer to the above description, the controlled objects corresponding to the household total electricity cost prediction model based on PPO within a preset time period are: household energy system, household electric vehicle and third-party energy system, that is, Figure 5 501 shown in . And Figure 5 Relative to Figure 4 For example, it further illustrates that the deep reinforcement learning module of the household total electricity cost prediction model within a preset time period built based on PPO includes multiple network structures, namely: strategy network 504 (new strategy network 5041 and old strategy network 5042) and evaluation network 505.

[0303] The new policy network 5041 and the old policy network 5042 are both composed of 6-layer neural networks, and the 6-layer neural network may have 232, 512, 512, 256, 128, and 4 neurons from the input layer to the output layer. nThe 232 elements of are connected to the 232 neurons of the input layer of the new strategy network 5041 and the old strategy network 5042 respectively, then the 4 neurons of the output layer of the new strategy network 5041 sequentially output the action space a of the nth sub-time period n Determinants The four neurons in the output layer of the old strategy network 5042 sequentially output the action space a of the nth sub-time period n Determinants

[0304] Similarly, the evaluation network 505 can be composed of a 5-layer neural network, which can have 232, 512, 256, 64, and 1 neurons from the input layer to the output layer. n The 232 elements of the evaluation network 505 are connected to the 232 neurons of the input layer, and one neuron of the output layer can output s i The corresponding state value function v π (s i ).

[0305] It should be noted that the PPO-based household total electricity cost prediction model within a preset time period mainly involves the flow of two types of signals:

[0306] First, the signal flow of the control loop, that is, the energy system simulation model 502 obtains real-time operation data from the home energy system, the home electric vehicle, and the third-party energy system 501, and simulates multiple real-time parameters based on the real-time operation data, such as: state information (corresponding to the state space involved in the above embodiment), action information (corresponding to the action space involved in the above embodiment), and reward information (corresponding to the immediate reward involved in the above embodiment). The corresponding input forms a corresponding experience trajectory, that is, 503, and then the output information corresponding to the experience trajectory is used to adjust (update) the relevant parameters to the evaluation network 505 and the policy network 504, such as: the parameter set θ of the new policy network 5041. new , the parameter set θ of the old policy network 5042 old And the parameter set ω of the evaluation network 505. For example, with θ new,j represents the parameter set of the new policy network 5041 updated for the jth time, θ old,j represents the parameter set of the old policy network 5042 updated for the jth time, ω j The parameter set of the evaluation network 505 updated for the jth time is represented by: The action strategy (update strategy) of the old strategy network 5042 is recorded as wait.

[0307] Here, if the deep reinforcement learning module of the household total electricity cost prediction model built based on PPO within a preset time period is iteratively optimized based on the above logic, and 508 is used to determine whether the deep reinforcement learning module has converged (here, it can be determined by the probability ratio of the new strategy network 5041 and the old strategy network 5042 under the relevant state information when executing relevant action information), if it has converged, the relevant operating reference value is sent to the home energy system, home electric vehicle, and third-party energy system 501, so that the home energy system, home electric vehicle, and third-party energy system 501 can operate in real time based on the operating reference value. If it does not converge, the optimization iteration link is entered again with the help of relevant data (that is, iterative training is performed based on the previous round of iterative data and real-time sample data).

[0308] Here, it can be set that in the synchronous iterative optimization link of the new strategy network 5041 and the old strategy network 5042, the old strategy network 5041 will be replaced by the new strategy network 5042 every B rounds, that is, θ old,j ←θ new,j ; Among them, the specific value of B can be determined according to actual needs.

[0309] The second is the signal flow of the learning cycle. Here, the learning cycle mainly involves the learning process of the new policy network 5041, the old policy network 5042 and the evaluation network 505. The three directly interact with each other. The specific signal flow can be referred to as follows: The new policy network 5041 and the old policy network 5042 are based on the corresponding action strategies. And the information output by the evaluation network 505, such as the multi-step advantage information to determine the relevant loss 506: L a (θ), and with the help of the loss L a (θ), iteratively optimize the network parameters of the new strategy network 5041 and the old strategy network 5042 respectively; the evaluation network directly determines the corresponding loss based on the output state value function information, that is, 507: L c (ω), and with the help of the loss L c (ω), iteratively optimize the network parameters of the evaluation network 505.

[0310] Based on the above, the description and calculation of relevant model parameters of the household total electricity cost prediction model based on PPO within a preset time period can refer to the description of the network structure built by PPO involved in actual applications, which will not be repeated here.

[0311] Example 3

[0312] The present invention based on the same inventive concept also provides a system 600 for constructing a household total electricity cost prediction model. Figure 6FIG. 1 is a schematic diagram of a system for constructing a household total electricity cost prediction model according to an embodiment of the present invention. The system includes:

[0313] A first constructing module 601 is configured to construct a charge and discharge model for the household electric vehicle to be connected to different energy systems within a preset time period based on usage plan information parameters, state of charge parameters, and performance constraint parameters of the household electric vehicle within the preset time period; wherein the different energy systems include: a household energy system and a third-party energy system;

[0314] The second building module 602 is configured to build a typical home energy system model corresponding to the home energy system within the preset time period;

[0315] The third construction module 603 is configured to construct a household total electricity cost prediction model within the preset time period based on the charging and discharging model and the typical household energy system model by adopting a proximal strategy optimization algorithm.

[0316] In some embodiments of the present invention, the preset time period includes: a plurality of sub-time periods of equal duration that are sequentially connected; the usage plan information parameters of the household electric vehicle for each sub-time period include: a first characterizing value of whether the household electric vehicle is connected to or not connected to the household energy system, a second characterizing value of whether the household electric vehicle is connected to or not connected to the third-party energy system, and a planned mileage in each sub-time period; the state of charge parameters of the household electric vehicle for each sub-time period include: the state of charge of the household electric vehicle in each sub-time period, the corresponding interaction power when connected to the different energy systems, the battery rated capacity, the charging efficiency and discharging efficiency of the internal battery pack, and the statistical average of the state of charge decrease corresponding to the internal battery pack when the vehicle has traveled a preset number of kilometers; the performance constraint parameters of the household electric vehicle for each sub-time period include: the minimum state of charge value, maximum state of charge value, minimum charge and discharge power value, and maximum charge and discharge power value corresponding to the internal battery pack in each sub-time period, a first minimum starting power value corresponding to the household electric vehicle when the mileage of a single trip is less than the preset number of kilometers, and a second minimum starting power value corresponding to the household electric vehicle when the mileage of a single trip is greater than or equal to the preset number of kilometers, where the second minimum starting power value is a positive multiple of the first minimum starting power value.

[0317] In some embodiments of the present invention, the third construction module 603 includes: a first construction sub-module, used to adopt the proximal strategy optimization algorithm, based on the charging and discharging model and the typical home energy system model, to construct an initial model of the total household electricity cost within the preset time period; a deep reinforcement learning sub-module, used to perform deep reinforcement learning on the initial model of the total household electricity cost to obtain the total household electricity cost prediction model.

[0318] In some embodiments of the present invention, the first construction submodule includes: a first unit for digitally simulating the different energy systems and the household electric vehicle using the charging and discharging model and the typical household energy system model based on external environmental information, the current status of the equipment in the different energy systems, the control action instructions output by the mobilization system corresponding to the different energy systems, and user demand information to obtain an energy system simulation model; a second unit for using the proximal strategy optimization algorithm to build an initial model of the total household electricity cost corresponding to the energy system simulation model within the preset time period.

[0319] In some embodiments of the present invention, the deep reinforcement learning submodule includes: a third unit, which is used to construct the action space, state space and immediate reward of the proximal policy optimization algorithm in the preset time period based on the power parameters of the charging and discharging model and the equipment state parameters of the typical home energy system model; a fourth unit, which is used to perform deep reinforcement learning on the initial model of the total household electricity cost based on the action space, the state space and the immediate reward to obtain the total household electricity cost prediction model.

[0320] In some embodiments of the present invention, the third unit is further used to construct an action space within the preset time period based on the interaction power corresponding to the home electric vehicle when it is connected to the different energy systems in the charge and discharge model, and the action parameters in the typical home energy system model used to characterize the turning on or off of the load-controllable device; wherein the home energy system includes at least the load-controllable device; based on the state of charge parameters in the charge and discharge model, the power parameters corresponding to the devices in the typical home energy system model, and the preset electricity price information, the endogenous state parameters of the energy system and the exogenous state parameters of the energy system operating environment corresponding to the preset time period are respectively constructed, and a state space is constructed based on the endogenous state parameters of the energy system and the exogenous state parameters of the energy system operating environment; based on the performance constraint parameters in the charge and discharge model and the working constraints corresponding to the load-controllable device in the typical home energy system model, the total household electricity cost within the preset time period and the penalty for violating the preset constraints are determined, and an immediate reward is constructed based on the total household electricity cost and the penalty.

[0321] In some embodiments of the present invention, the deep reinforcement learning module of the initial model of the total household electricity cost includes: a new policy network, an old policy network and an evaluation network; the fourth unit is further used to determine, within the preset time period, based on the state parameters provided by the state space, the probability ratio of the execution of the action parameters in the action space between the new policy network and the old policy network; based on the immediate reward, using the evaluation network, determine the estimated error of the state value corresponding to the state space of the preset time period; based on the estimated error, determine the multi-step advantage of the state space within the preset time period; based on the probability ratio, the multi-step advantage and the first learning rate, iteratively optimize the network parameters of the new policy network and the network parameters of the old policy network, and iteratively optimize the network parameters of the evaluation network based on the state value and the second learning rate, until the deep reinforcement learning module whose output parameters meet the preset convergence conditions is obtained, and the model corresponding to the deep reinforcement learning module whose output parameters meet the preset convergence conditions is determined as the household total electricity cost prediction model.

[0322] In some embodiments of the present invention, the devices in the home energy system include at least: a first device with controllable load, a second device with uncontrollable load, and a third device for renewable energy power generation; the second construction module also includes: a second construction sub-module, which is used to use the mechanism model and the working constraints corresponding to the first device to model the operating parameters of the first device in the preset time period and construct a working model expression corresponding to the first device; a third construction sub-module, which is used to predict the total load power prediction value corresponding to the second device and the power generation power prediction value corresponding to the third device in the preset time period; and a fourth construction sub-module, which is used to construct the typical home energy system model based on the working model expression, the total load power prediction value, and the power generation power prediction value.

[0323] It should be noted that the description of the corresponding embodiment of the system for constructing a household total electricity cost prediction model is similar to the description of the aforementioned method for constructing a household total electricity cost prediction model, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the system embodiment of the present invention, please refer to the description of the method embodiment of the present invention for understanding.

[0324] Example 4:

[0325] The present invention based on the same inventive concept also provides a prediction system 700 for the total electricity cost of a household. Figure 7 FIG. 1 is a schematic diagram of a system for predicting the total electricity cost of a household provided by an embodiment of the present invention, the system comprising:

[0326] A training module 701 is configured to train a household total electricity cost prediction model constructed by the system described in any of the above embodiments until convergence based on real-time operating parameters of devices within the household energy system and real-time charging and discharging parameters corresponding to household electric vehicles connected to different energy systems, thereby obtaining a household total electricity cost optimization model; wherein the different energy systems include: the household energy system and a third-party energy system;

[0327] The prediction module 702 is used to use the operating power value output by the household total electricity cost optimization model as the operating power reference value of the equipment in the household energy system, and to use the charging and discharging power reference value output by the household total electricity cost optimization model as the charging and discharging power reference value when the household electric vehicle is connected to the different energy systems.

[0328] It should be noted that the description of the corresponding embodiment of the system for predicting total household electricity costs is similar to the description of the aforementioned method embodiment for predicting total household electricity costs, and has similar beneficial effects as the method embodiment. For any technical details not disclosed in the system embodiment of the present invention, please refer to the description of the method embodiment of the present invention for understanding.

[0329] Example 5

[0330] Based on the same inventive concept, Figure 8 As shown, the present invention also provides an electronic device, which may be a computer device, a single-chip microcomputer device, a smart mobile device, etc. The electronic device in this embodiment may include at least one processor 810, a memory 820, a transceiver component 830, etc. The processor 810, the memory 820, and the transceiver component 830 are connected via a bus 840. The memory 820 may be used to store an execution program, which may include instructions. The processor 810 is used to execute the instructions stored in the memory. The memory 820 may also be used to store data, which may be accessed and / or modified during the execution of the instructions.

[0331] The processor may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in a storage medium to implement the corresponding method flow or corresponding function, so as to implement the steps of a method for constructing a household total electricity cost prediction model in the above embodiment, or the steps of a method for predicting a household total electricity cost in the above embodiment.

[0332] Example 6

[0333] Based on the same inventive concept, the present invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It can be understood that the storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more execution programs (including program codes). It should be noted that the storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage. The processor loads and executes one or more instructions stored in the storage medium to implement the steps of the method for constructing a household total electricity cost prediction model in the above embodiment, or the steps of the method for predicting the household total electricity cost in the above embodiment.

[0334] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0335] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0336] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0337] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0338] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for constructing a household total electricity cost prediction model, characterized in that: The method comprises: Based on the usage plan information parameters, state of charge parameters and performance constraint parameters of the household electric vehicle within a preset time period, a charging and discharging model is constructed in which the household electric vehicle can be connected to different energy systems within the preset time period; wherein the different energy systems include: a household energy system and a third-party energy system; Establishing a typical home energy system model corresponding to the home energy system within the preset time period; A proximal strategy optimization algorithm is adopted to construct a prediction model for the total household electricity cost within the preset time period based on the charging and discharging model and the typical household energy system model.

2. The method according to claim 1, characterized in that: The preset time period includes: a plurality of sub-time periods of equal duration that are continuous in time sequence; The usage plan information parameters of the household electric vehicle in each sub-time period include: a first characterization value of whether the household electric vehicle is connected to or not connected to the household energy system in each sub-time period, a second characterization value of whether the household electric vehicle is connected to or not connected to the third-party energy system, and a planned mileage; The state of charge parameters of the household electric vehicle in each sub-time period include: the state of charge of the household electric vehicle in each sub-time period, the corresponding interaction power when connected to the different energy systems, the rated capacity of the battery, the charging efficiency and the discharging efficiency of the internal battery pack, and the statistical average value of the decrease in the state of charge of the internal battery pack when the household electric vehicle travels a preset number of kilometers; The performance constraint parameters of the household electric vehicle in each sub-time period include: the minimum state of charge value, the maximum state of charge value, the minimum charge and discharge power value, the maximum charge and discharge power value corresponding to the internal battery pack in each sub-time period, and the first minimum starting power corresponding to the household electric vehicle when the mileage of a single trip is less than a preset number of kilometers, and the second minimum starting power corresponding to the household electric vehicle when the mileage of a single trip is greater than or equal to the preset number of kilometers, the second minimum starting power being a positive multiple of the first minimum starting power.

3. The method according to claim 1 or 2, characterized in that: The proximal strategy optimization algorithm is used to construct a household total electricity cost prediction model within the preset time period based on the charging and discharging model and the typical household energy system model, including: Adopting the proximal strategy optimization algorithm, based on the charging and discharging model and the typical household energy system model, constructing an initial model of the total household electricity cost within the preset time period; Deep reinforcement learning is performed on the initial model of the total household electricity cost to obtain the prediction model of the total household electricity cost.

4. The method according to claim 3, characterized in that: The method of using the proximal strategy optimization algorithm to construct an initial model of the total household electricity cost within the preset time period based on the charging and discharging model and the typical household energy system model includes: Based on the external environment information, the current status of the equipment in the different energy systems, the control action instructions output by the mobilization system corresponding to the different energy systems, and the user demand information, the charging and discharging model and the typical household energy system model are used to digitally simulate the different energy systems and the household electric vehicle to obtain an energy system simulation model; The proximal strategy optimization algorithm is used to build an initial model of the total household electricity cost corresponding to the energy system simulation model within the preset time period.

5. The method according to claim 4, characterized in that Before performing deep reinforcement learning on the initial model of the total household electricity cost to obtain the total household electricity cost prediction model, the method further includes: Based on the power parameters of the charging and discharging model and the device state parameters of the typical home energy system model, constructing the action space, state space and immediate reward of the proximal strategy optimization algorithm in the preset time period; The performing deep reinforcement learning on the initial model of the total household electricity cost to obtain the total household electricity cost prediction model includes: Based on the action space, the state space and the immediate reward, deep reinforcement learning is performed on the initial model of the total household electricity cost to obtain the prediction model of the total household electricity cost.

6. The method according to claim 5, characterized in that The construction of the action space, state space and immediate reward of the proximal strategy optimization algorithm in the preset time period based on the power parameters of the charging and discharging model and the device state parameters of the typical home energy system model includes: Within the preset time period, an action space is constructed based on the interaction power corresponding to when the household electric vehicle is connected to the different energy systems in the charging and discharging model, and the action parameters in the typical household energy system model used to characterize turning on or off the load controllable device; wherein the household energy system at least includes the load controllable device; Based on the state of charge parameters in the charging and discharging model, the power parameters corresponding to the equipment of the typical home energy system model, and the preset electricity price information, the endogenous state parameters of the energy system and the exogenous state parameters of the energy system operating environment corresponding to the preset time period are respectively constructed, and based on the endogenous state parameters of the energy system and the exogenous state parameters of the energy system operating environment, a state space is constructed; Based on the performance constraint parameters in the charging and discharging model and the working constraints of the load controllable devices corresponding to the typical home energy system model, the total household electricity cost within the preset time period and the penalty for violating the preset constraints are determined, and an immediate reward is constructed based on the total household electricity cost and the penalty.

7. The method according to claim 6, characterized in that The deep reinforcement learning module of the initial model of the total household electricity cost includes: a new strategy network, an old strategy network and an evaluation network; The method of performing deep reinforcement learning on the initial model of the total household electricity cost based on the action space, the state space and the immediate reward to obtain the prediction model of the total household electricity cost includes: Determining, within the preset time period, based on the state parameters provided by the state space, a probability ratio corresponding to the execution of the action parameters in the action space between the new policy network and the old policy network; Based on the immediate reward, using the evaluation network, determining an estimated error of a state value corresponding to the state space of the preset time period; determining a multi-step advantage of the state space within the preset time period based on the estimation error; Based on the probability ratio, the multi-step advantage and the first learning rate, the network parameters of the new strategy network and the network parameters of the old strategy network are iteratively optimized, and based on the state value and the second learning rate, the network parameters of the evaluation network are iteratively optimized until the deep reinforcement learning module whose output parameters meet the preset convergence conditions is obtained, and the model corresponding to the deep reinforcement learning module whose output parameters meet the preset convergence conditions is determined as the household total electricity cost prediction model.

8. The method according to claim 1, characterized in that: The devices in the home energy system at least include: a first device with controllable load, a second device with uncontrollable load, and a third device for generating electricity with new energy; The typical home energy system model corresponding to the home energy system established within the preset time period includes: Using the mechanism model and the working constraints corresponding to the first device, modeling the operating parameters of the first device within the preset time period, and constructing a working model expression corresponding to the first device; Predicting a total load power prediction value corresponding to the second device and a power generation power prediction value corresponding to the third device within the preset time period; The typical household energy system model is constructed based on the working model expression, the total load power prediction value and the power generation power prediction value.

9. A system for constructing a household total electricity cost prediction model, characterized in that: The system comprises: A first construction module is used to construct a charging and discharging model in which the household electric vehicle can be connected to different energy systems within a preset time period based on the usage plan information parameters, charge state parameters and performance constraint parameters of the household electric vehicle within the preset time period; wherein the different energy systems include: a household energy system and a third-party energy system; A second building module is used to establish a typical home energy system model corresponding to the home energy system within the preset time period; The third building module is used to adopt a proximal strategy optimization algorithm to build a household total electricity cost prediction model within the preset time period based on the charging and discharging model and the typical household energy system model.

10. The system according to claim 9, characterized in that The preset time period includes: a plurality of sub-time periods of equal duration that are continuous in time sequence; The usage plan information parameters of the household electric vehicle in each sub-time period include: a first characterization value of whether the household electric vehicle is connected to or not connected to the household energy system in each sub-time period, a second characterization value of whether the household electric vehicle is connected to or not connected to the third-party energy system, and a planned mileage; The state of charge parameters of the household electric vehicle in each sub-time period include: the state of charge of the household electric vehicle in each sub-time period, the corresponding interaction power when connected to the different energy systems, the rated capacity of the battery, the charging efficiency and the discharging efficiency of the internal battery pack, and the statistical average value of the decrease in the state of charge of the internal battery pack when the household electric vehicle travels a preset number of kilometers; The performance constraint parameters of the household electric vehicle in each sub-time period include: the minimum state of charge value, the maximum state of charge value, the minimum charge and discharge power value, the maximum charge and discharge power value corresponding to the internal battery pack in each sub-time period, and the first minimum starting power corresponding to the household electric vehicle when the mileage of a single trip is less than a preset number of kilometers, and the second minimum starting power corresponding to the household electric vehicle when the mileage of a single trip is greater than or equal to the preset number of kilometers, the second minimum starting power being a positive multiple of the first minimum starting power.

11. The system according to claim 9 or 10, characterized in that: The third building block comprises: A first construction submodule is used to construct an initial model of the total household electricity cost within the preset time period based on the charging and discharging model and the typical household energy system model by using the proximal strategy optimization algorithm; The deep reinforcement learning submodule is used to perform deep reinforcement learning on the initial model of the total household electricity cost to obtain the total household electricity cost prediction model.

12. The system according to claim 11, characterized in that The first building block comprises: The first unit is used to digitally simulate the different energy systems and the household electric vehicle using the charging and discharging model and the typical household energy system model based on external environment information, current states of the devices in the different energy systems, control action instructions output by the mobilization system corresponding to the different energy systems, and user demand information to obtain an energy system simulation model; The second unit is used to adopt the proximal strategy optimization algorithm to build an initial model of the total household electricity cost corresponding to the energy system simulation model within the preset time period.

13. The system according to claim 12, characterized in that The deep reinforcement learning submodule includes: A third unit is used to construct an action space, a state space and an immediate reward of the proximal strategy optimization algorithm in the preset time period based on the power parameters of the charging and discharging model and the device state parameters of the typical home energy system model; The fourth unit is used to perform deep reinforcement learning on the initial model of the total household electricity cost based on the action space, the state space and the immediate reward to obtain the prediction model of the total household electricity cost.

14. The system according to claim 13, characterized in that The third unit is further used to construct an action space within the preset time period based on the interaction power corresponding to the home electric vehicle being connected to the different energy systems in the charge and discharge model and the action parameters in the typical home energy system model used to characterize turning on or off the load controllable device; wherein the home energy system at least includes the load controllable device; Based on the state of charge parameters in the charging and discharging model, the power parameters corresponding to the equipment of the typical home energy system model, and the preset electricity price information, the endogenous state parameters of the energy system and the exogenous state parameters of the energy system operating environment corresponding to the preset time period are respectively constructed, and based on the endogenous state parameters of the energy system and the exogenous state parameters of the energy system operating environment, a state space is constructed; Based on the performance constraint parameters in the charging and discharging model and the working constraints of the load controllable devices corresponding to the typical home energy system model, the total household electricity cost within the preset time period and the penalty for violating the preset constraints are determined, and an immediate reward is constructed based on the total household electricity cost and the penalty.

15. The system according to claim 14, characterized in that The deep reinforcement learning module of the initial model of the total household electricity cost includes: a new strategy network, an old strategy network and an evaluation network; The fourth unit is further configured to determine, within the preset time period, based on the state parameters provided by the state space, a probability ratio corresponding to the execution of the action parameters in the action space between the new policy network and the old policy network; Based on the immediate reward, using the evaluation network, determining an estimated error of a state value corresponding to the state space of the preset time period; determining a multi-step advantage of the state space within the preset time period based on the estimation error; Based on the probability ratio, the multi-step advantage and the first learning rate, the network parameters of the new strategy network and the network parameters of the old strategy network are iteratively optimized, and based on the state value and the second learning rate, the network parameters of the evaluation network are iteratively optimized until the deep reinforcement learning module whose output parameters meet the preset convergence conditions is obtained, and the model corresponding to the deep reinforcement learning module whose output parameters meet the preset convergence conditions is determined as the household total electricity cost prediction model.

16. The system according to claim 9, characterized in that The devices in the home energy system at least include: a first device with controllable load, a second device with uncontrollable load and a third device for generating new energy; the second building module also includes: A second construction submodule is used to model the operating parameters of the first device within the preset time period by using the mechanism model and the working constraints corresponding to the first device, and to construct a working model expression corresponding to the first device; A third construction submodule is used to predict the total load power prediction value corresponding to the second device and the power generation prediction value corresponding to the third device within the preset time period; The fourth construction submodule is used to construct the typical household energy system model based on the working model expression, the total load power prediction value and the generated power prediction value.

17. A method for predicting the total electricity cost of a household, characterized in that: The method comprises: Based on the real-time operating parameters of the equipment in the home energy system and the real-time charging and discharging parameters corresponding to the home electric vehicle connected to different energy systems, the home total electricity cost prediction model constructed by the method according to any one of claims 1 to 8 is trained until convergence to obtain a home total electricity cost optimization model; wherein the different energy systems include: a home energy system and a third-party energy system; The operating power value output by the household total electricity cost optimization model is used as the operating power reference value of the equipment in the household energy system, and the charging and discharging power reference value output by the household total electricity cost optimization model is used as the charging and discharging power reference value when the household electric car is connected to the different energy systems.

18. A system for predicting the total electricity cost of a household, characterized in that: The system comprises: A training module, for training the household total electricity cost prediction model constructed by the system according to any one of claims 9 to 16 until convergence based on the real-time operating parameters of the equipment in the household energy system and the real-time charging and discharging parameters corresponding to the household electric vehicle connected to different energy systems, so as to obtain a household total electricity cost optimization model; wherein the different energy systems include: a household energy system and a third-party energy system; The prediction module is used to use the operating power value output by the household total electricity cost optimization model as the operating power reference value of the equipment in the household energy system, and to use the charging and discharging power reference value output by the household total electricity cost optimization model as the charging and discharging power reference value when the household electric vehicle is connected to the different energy systems.

19. An electronic device, characterized in that: include: at least one processor and memory; The memory and the processor are connected via a bus; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the method for constructing a household total electricity cost prediction model as described in any one of claims 1 to 8, or the method for predicting the household total electricity cost as described in claim 17 is implemented.

20. A readable storage medium, characterized in that: An execution program is stored thereon, and when the execution program is executed, it implements the method for constructing a household total electricity cost prediction model as described in any one of claims 1 to 8, or the method for predicting the household total electricity cost as described in claim 17.

Images