Method for charging and discharging energy storage system on basis of reinforcement learning in consideration of constraint, and power control device supporting same

Abstract

An embodiment of the present invention provides a reinforcement learning model-based ESS charging and discharging control method to which a constraint is applied, and a power control device supporting same, the method comprising the steps of: collecting state information related to an ESS and ESS-related power information every time unit corresponding to a training cycle of a reinforcement learning model; determining a reward according to whether a change between a previous time unit and a current time unit is positive or negative; processing the collected information and the determined reward as an input of the reinforcement learning model; outputting an action of a next time unit according to the processing of the input of the reinforcement learning model; and generating a charging and discharging command value related to ESS operation in response to the action, wherein the step of outputting the action includes a step of rewarding for a change in the action by using previous action values in a plurality of previous time units.

Description

A charging / discharging method for an energy storage device based on reinforcement learning considering constraints and a power control device supporting the method

[0001] The present invention relates to charge / discharge control of an energy storage device, and more particularly, to improvement of charge / discharge of an energy storage device based on reinforcement learning considering constraints.

[0002] If power generated at a power plant can be supplied to loads (or consumers) in the amount required by the load, supply and demand can be properly balanced without waste, enabling efficient power management. However, in situations where the load size fluctuates irregularly due to uncertain factors such as weather, and when power generation for the load utilizes power from renewable power plants (e.g., solar or wind power) with uncertain factors, it is difficult to generate enough power to meet the load's demand. To address this, energy storage systems (ESS) are being used. These devices can store some of the power produced in excess of the load's demand, and can also supply the remaining power when the load's demand is lower.

[0003] The above energy storage device is a device that can charge energy and then discharge the charged energy depending on the usage purpose, and can be used to store energy and support use at different times while controlling the load power size at a specific location on a power line. Such an energy storage device can only operate in one of two modes: charging, which performs energy input for a specified period of time, and discharging, which performs energy output for a different specified period of time that does not overlap with the energy input. In addition, the energy storage device has a storage capacity limit, and in order to manage the lifespan of the energy storage device, it is necessary to set appropriate upper and lower limits so that it does not become fully charged and can no longer be charged, or fully discharged and can no longer be discharged. To this end, a daily or section-by-section charging and discharging plan is established and the ESS is operated so that the ESS can perform optimal charging and discharging.

[0004] Conventional ESS optimal charge / discharge methods plan the charge / discharge power for each unit of time, such as linear or dynamic optimization, in advance for economic profit. When the unit time is reached, the planned ESS charge / discharge is basically performed. However, if a constraint is exceeded or is expected to be exceeded, the ESS charge / discharge is performed differently from the planned value to meet the constraint. For example, in the case of ESSs installed in commercial buildings, the ESS charge / discharge plan is typically established and operated to charge in the early morning when electricity prices are low and discharge during the day when electricity prices are high. However, if the building's air conditioning system is used during early morning ESS charging, exceeding or expected to exceed the capacity of the indoor substation, the ESS charge power can be reduced or charging stopped to meet the constraint. As described above, the general ESS charge / discharge plan and the constraint-based operation consider different situations. Therefore, if the constraint-based operation is followed, the ESS charge / discharge plan does not proceed normally, and if the constraint-based operation is followed, the constraint-based operation is not easy. In addition, since the ESS charge / discharge plan is planned for, for example, 24 hours, while the constraint operation occurs in the form of a one-time event, it is not easy to link the ESS charge / discharge plan operation and the constraint operation.

[0005] Accordingly, the present invention provides a charging / discharging method of an energy storage device and a power control device supporting the same, which can improve non-optimization over a long period of time and non-reflection of situation changes due to execution of a constraint control algorithm by applying reinforcement learning to ESS charging / discharging operation and control.

[0006] However, the purpose of the present invention is not limited to the above purpose, and other purposes not mentioned can be clearly understood from the description below.

[0007] In order to achieve the above-described purpose, a power control device for controlling charging and discharging of an energy storage device includes a server memory that stores status information related to the ESS and power information related to charging and discharging of the ESS, and a server processor functionally connected to the server memory, wherein the server processor is configured to collect the status information and the power information related to the ESS for each time unit corresponding to a training period of the reinforcement learning model, determine a reward according to whether a change between a previous time unit and a current time unit is positive or negative, and then process the collected information and the determined reward as inputs of the reinforcement learning model to output an action for a next time unit, and generate a charging and discharging command value related to the operation of the ESS in response to the action, and is characterized in that it is configured to compensate for a change in the action by using previous action values ​​in a plurality of previous time units in relation to the action output.

[0008] Specifically, the server processor may be configured to collect power unit price information by time unit in relation to the compensation determination, and multiply the power unit price information by the charge and discharge amounts of the ESS to determine the compensation.

[0009] Specifically, the server processor may be set to determine the charge amount of the ESS as a negative number and the discharge amount of the ESS as a positive number.

[0010] Specifically, the server processor may be configured to set a constraint on at least one of the status information and the power information, and to process the input of the reinforcement learning model within the constraint.

[0011] Specifically, the server processor may be configured to set an upper limit constraint value and a lower limit constraint value for the state information, and to apply a constraint compensation value that increases by a negative amount by an amount by which the state information value exceeds the upper limit constraint value or the lower limit constraint value.

[0012] Specifically, the server processor may be configured to provide a higher reward as the value of the state information gets closer to the upper limit constraint value or the lower limit constraint value within the constraint condition including the upper limit constraint value and the lower limit constraint value for the state information.

[0013] Specifically, the server processor may be configured to provide a lower reward as the value of the state information moves away from the upper limit constraint value or the lower limit constraint value within the constraint condition including the upper limit constraint value and the lower limit constraint value for the state information.

[0014] Specifically, the server processor may be set to perform preprocessing to remove abnormal data exceeding a predefined reference value from among the collected information or to interpolate missing portions from among the collected information using adjacent data.

[0015] Specifically, the server processor may be configured to construct a virtual environment including a virtual ESS corresponding to status information related to the ESS, and to apply the charge / discharge command value to the ESS of the virtual environment within the virtual environment to collect information for the next time unit.

[0016] A method for controlling ESS charging and discharging based on reinforcement learning with constraints according to an embodiment of the present invention comprises the steps of collecting state information related to the ESS and power information related to the ESS for each time unit corresponding to a training period of the reinforcement learning model, determining a reward according to whether a change between a previous time unit and a current time unit is positive or negative, processing the collected information and the determined reward as inputs of the reinforcement learning model, outputting an action for a next time unit according to the input processing of the reinforcement learning model, and generating a charge and discharge command value related to the operation of the ESS in response to the action, wherein the step of outputting the action comprises the step of compensating for a change in the action by using previous action values ​​in a plurality of previous time units.

[0017] Specifically, the step of determining the compensation may include a step of collecting power unit price information for each time unit and multiplying the charge amount and discharge amount of the ESS by the power unit price information to determine the compensation for each time unit.

[0018] Specifically, the step of determining compensation for each time unit may include a step of determining a negative number for the charge amount of the ESS and a positive number for the discharge amount of the ESS.

[0019] Specifically, the step of processing the input may include the step of setting a constraint on at least one of the state information and the power information and the step of processing the input of the reinforcement learning model within the constraint.

[0020] Specifically, the step of setting the constraint may include the step of setting an upper constraint value and a lower constraint value for the state information, and applying a constraint compensation value that increases by a negative amount by an amount by which the state information value exceeds the upper constraint value or the lower constraint value.

[0021] Specifically, the step of setting the constraint may include a step of providing a higher reward as the value of the state information gets closer to the upper constraint value or the lower constraint value within the constraint including an upper constraint value and a lower constraint value for the state information.

[0022] Specifically, the step of setting the constraint may include a step of providing a lower reward as the value of the state information moves away from the upper constraint value or the lower constraint value within the constraint including an upper constraint value and a lower constraint value for the state information.

[0023] Specifically, the method may further include a step of performing preprocessing to remove abnormal data exceeding a predefined reference value among the collected information or to interpolate missing portions among the collected information using adjacent data.

[0024] Specifically, the method may further include a step of constructing a virtual environment including a virtual ESS corresponding to status information related to the ESS, and a step of applying the charge / discharge command value to the ESS of the virtual environment within the virtual environment to collect information of the next time unit.

[0025] According to the present invention, the present invention can support ESS charging / discharging performance to comply with constraints while simultaneously providing economic optimization by using a reinforcement learning model that takes constraints into account.

[0026] In addition, various effects other than the above-described effects may be disclosed directly or implicitly in the detailed description according to the embodiments of the present invention to be described later.

[0027] FIG. 1 is a drawing showing an example of an improved charging / discharging environment of an ESS device according to an embodiment of the present invention.

[0028] FIG. 2 is a drawing showing an example of a power control device configuration according to an embodiment of the present invention.

[0029] FIG. 3 is a diagram showing an example of a server processor configuration among the power control device configurations according to an embodiment of the present invention.

[0030] FIG. 4 is a diagram showing an example of a reinforcement learning reward unit configuration according to an embodiment of the present invention.

[0031] FIG. 5 is a diagram for explaining compensation of upper and lower limit constraints according to constraint conditions according to an embodiment of the present invention.

[0032] FIG. 6 is a drawing for explaining compensation and additional compensation for upper and lower limit constraints according to constraint conditions according to an embodiment of the present invention.

[0033] FIG. 7 is a drawing showing examples of various shapes related to additional compensation of a constraint compensation unit according to an embodiment of the present invention.

[0034] FIG. 8 is a diagram showing the results of power upper limit constraint control after additional compensation according to an embodiment of the present invention.

[0035] FIG. 9 is a diagram showing a comparison of SoC of an ESS battery before and after additional compensation according to an embodiment of the present invention.

[0036] FIG. 10 is a diagram showing an example of a reinforcement learning modeling method related to ESS charging / discharging control applying constraints according to an embodiment of the present invention.

[0037] In order to more clearly explain the features and advantages of the problem-solving means of the present invention, the present invention will be described in more detail with reference to specific embodiments of the present invention illustrated in the attached drawings.

[0038] However, detailed descriptions of known functions or components that may obscure the gist of the present invention in the following description and attached drawings are omitted. It should also be noted that, where possible, identical components are indicated with identical reference numerals throughout the drawings.

[0039] The terms and words used in the following description and drawings should not be interpreted as limited to their conventional or dictionary meanings, but should be interpreted with meanings and concepts that conform to the technical idea of ​​the present invention based on the principle that the inventor can appropriately define the concept of the term to best describe his or her invention. Therefore, the embodiments described in this specification and the configurations illustrated in the drawings are merely the most preferred embodiments of the present invention and do not represent all of the technical idea of ​​the present invention. Therefore, it should be understood that there may be various equivalents and modified examples that can replace them at the time of filing this application.

[0040] Additionally, terms containing ordinal numbers, such as "first," "second," etc., are used to describe various components, and are only used to distinguish one component from another, not to limit said components. For example, without departing from the scope of the present invention, a second component could be referred to as a "first component," and similarly, a first component could also be referred to as a "second component."

[0041] In addition, the terminology used in this specification is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In addition, it should be understood that the terms "comprises" or "has" described in this specification are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0042] Additionally, terms such as "unit," "device," and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware, software, or a combination of hardware and software. Additionally, terms such as "a" or "an," "one," "the," and similar related words may be used in the context of describing the present invention (especially in the context of the claims below) to include both singular and plural meanings, unless otherwise indicated in the specification or clearly contradicted by the context.

[0043] In addition to the terms described above, specific terms used in the following description are provided to help understanding of the present invention, and the use of these specific terms may be changed to other forms without departing from the technical spirit of the present invention.

[0044] Additionally, embodiments within the scope of the present invention include computer-readable media having or carrying computer-executable instructions or data structures stored on the computer-readable media. Such computer-readable media may be any available media that can be accessed by a general-purpose or special-purpose computer system. By way of example, and not limitation, such computer-readable media may include physical storage media such as RAM, ROM, EPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or any other medium that can be used to store or carry predetermined program code means in the form of computer-executable instructions, computer-readable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer system.

[0045] Hereinafter, reinforcement learning, which is applied to the ESS charging / discharging method of the present invention, is a field of machine learning in which an agent performs an action in a given environment and receives a reward from the result. For example, reinforcement learning is a method in which an agent (e.g., a power plant output control system) selects an action in a given environment, and the environment changes and a reward is given according to that action. The agent learns by adjusting and improving its actions over time to maximize the reward. To this end, reinforcement learning applies concepts such as state, action, reward, policy, value, and value function. The ESS charging / discharging environment to which the above-described reinforcement learning is applied can be designed so that the ESS can comply with constraints while also performing optimal charging / discharging.

[0046] Below, the types and roles of each component included in the environment that supports improvement of charging and discharging of an energy storage device based on reinforcement learning applying the constraints of the present invention will be described.

[0047] FIG. 1 is a drawing showing an example of an improved charging / discharging environment of an ESS device according to an embodiment of the present invention.

[0048] Referring to FIG. 1, an ESS device charging / discharging improvement environment (10) according to an embodiment of the present invention may include a renewable energy power plant (101), a power system unit (102), a customer (103) (or load), an ESS (104) (energy storage system), an external information provision server (300), and a power control device (200). In addition, the ESS device charging / discharging improvement environment (10) may include a network for signal transmission / reception in at least one relationship among the renewable energy power plant (101), the power system unit (102), the customer (103) and the ESS (104), and between the ESS (104), the external information provision server (300), and the power control device (200). This network supports at least one of a wired or wireless method, and may be configured to support at least one of various communication methods or communication types even in a wireless method. Meanwhile, in Fig. 1, the ESS (104) and the power control device (200) are shown as integrated, but they can be separated into separate, independent configurations and connected through a network.

[0049] The renewable energy power plant (101) may include a power plant that produces at least one type of renewable energy. For example, the renewable energy power plant (101) may include at least one of a wind power plant, a solar thermal or photovoltaic power plant, a tidal power plant, a hydroelectric power plant, and a geothermal power plant. As described above, the renewable energy power plant (101) may include a power plant whose power generation is affected by the surrounding weather or environment. The renewable energy power plant (101) may provide the generated power to the ESS (104). In this process, the renewable energy power plant (101) may be connected to the ESS (104) via a power meter installed at a PCC (Point of Common Coupling) (common connection point, or power generation management device).

[0050] The power system unit (102) may include a power system of a power plant capable of producing a fixed amount of power. For example, the power system unit (102) may include at least one of a power plant that generates power using fossil fuels and a power plant that generates power using nuclear power. The power system unit (102) may be connected to at least one of a power control device (200) and an ESS (104) via a power meter. The power system unit (102) may supply power to the ESS (104) in response to the control of the power control device (200). Additionally, the consumer (103) may also be connected to the ESS (104) via the power meter and may receive power from the ESS (104). For example, status information regarding the generated power of the renewable energy power plant (101) and the power system unit (102) may be provided to the power control device (200) via the power meter. For example, the power control device (200) can detect information about the generated power supplied from the renewable power plant (101) and the power system unit (102) (e.g., the amount of power, voltage, and current of the renewable power plant (101), and the amount of power, voltage, and current of the power system unit (102) that supplies the generated power of a general power plant (e.g., power plants other than the renewable power plant (101))) through a power meter and perform control on the generated power. The power meter can provide information on the status of the generated power of the renewable power plant (101) and the power system unit (102) to the power control device (200).

[0051] The above-mentioned user (103) (or load) may be an entity that receives and consumes electricity. The user (103) may refer to, for example, a power-consuming facility such as a building or factory, at least one home or office that consumes electricity, or at least one specific section that consumes electricity. Alternatively, the user (103) may include power meters of various buildings or devices that use high power (e.g., air conditioning and heating devices, industrial equipment, commercial electrical equipment, electric vehicle chargers). In this way, the user (103) is a component that receives and consumes electricity, and is not limited to a specific form, size, shape, location, etc. In terms of the ESS (104), the user (103) may correspond to a load power or a power demand source. The above-mentioned user (103) may be connected to the ESS (104) via a power meter.

[0052] The ESS (104) can perform a charging operation or a discharging operation in response to the control of the power control device (200). The ESS (104) may include a battery module capable of storing energy, a charging circuit capable of charging the battery module, a discharging circuit for discharging power stored in the battery module, a power conversion device for controlling conversion of the charging circuit and the discharging circuit, and a control circuit for controlling the switching of the operation of the charging circuit or the discharging circuit according to a control signal of the power control device (200). As an example, the ESS (104) can supply stored power to a user (103) for a certain period of time in response to the control of the power control device (200). In addition, the ESS (104) can receive power from a renewable power plant (101) or a power system unit (102) in response to the control of the power control device (200) and charge the battery module. In this regard, the ESS (104) may include a power supply line capable of supplying power to a user (103) and a power reception line capable of receiving power from a renewable power plant (101) or a power system (102). Meanwhile, the ESS (104) may perform an optimized charge / discharge operation within preset constraints in response to the control of the power control device (200).

[0053] The external information providing server (300) may include a server device that provides external information to the power control device (200). The external information providing server (300) may, for example, provide the power control device (200) with weather and other surrounding environment information of the area where the renewable energy power plant (101) or the customer (103) is located. Alternatively, the external information providing server (300) may be an entity that can provide power price information (e.g., a Korea Electric Power Corporation server device) or a server device that can collect and provide power price information. As another example, in the ESS device charging / discharging improvement environment (10), when the ESS (104) and the power control device (200) are designed to be separated, the external information provision server (300) can collect information such as the operation mode, output size, AC / DC voltage and current, temperature, etc. of the ESS power conversion device from the ESS (104), the operation mode of the battery, the entire battery cell and each battery cell SoC (state of charging), voltage, current, temperature, etc. of the battery cell, and provide the information to the power control device (200). Alternatively, the external information provision server (300) can additionally collect power information of power generation facilities and power reception facilities, and provide the information to the power control device (200).

[0054] The power control device (200) can collect various information related to the ESS (104) for each time unit (e.g., time step) in which reinforcement learning operates. For example, the power control device (200) can collect information such as the operation mode, output size, AC / DC voltage / current, temperature, etc. of the ESS power conversion device and the operation mode of the battery, the SoC of the entire battery cell and each battery cell, battery voltage, battery current, battery temperature, etc. directly from the ESS (104) (or through an external information provision server (300)). Additionally or alternatively, the power control device (200) can receive power information of power generation facilities and power information of power reception facilities through a power meter connected to a renewable energy power plant (101) and a power system unit (102). In addition, since it can be operated for purposes such as minimizing power usage costs or maximizing sales revenue, it can receive power price information (e.g., received from Korea Electric Power Corporation). The power control device (200) can collect constraint condition information of the ESS (104). The power control device (200) can perform reinforcement learning related to the operation of the ESS (104) based on the above-described information.

[0055] The environment for training the above reinforcement learning model may include at least one of a real environment, a virtual environment based on a mathematical physical model, or a virtual environment based on a data learning model. As an example, the power control device (200) may utilize the generation of a certain number of data or more, the generation of various learning data, various situation changes, and time acceleration based on the virtual environment during the process of creating a reinforcement learning model for the ESS (104). The power control device (200) may perform a service of applying the reinforcement learning model to the real environment in relation to the service operation of the reinforcement learning model for which learning has been completed. In this regard, the power control device (200) may calculate the difference between the virtual environment and the real environment, and perform a virtual environment update in a form that minimizes the difference (e.g., the difference between input and output), or record and store the difference and apply the difference value as a correction value during the process of applying the reinforcement learning model service. In order to support the above-described function, the power control device (200) may include a configuration as illustrated in FIG. 2.

[0056] FIG. 2 is a drawing showing an example of a power control device configuration according to an embodiment of the present invention.

[0057] Referring to FIGS. 1 and 2, a power control device (200) according to an embodiment of the present invention may include a server communication circuit (210), a server input unit (220), a server memory (230), a server display (240), and a server processor (250).

[0058] The server communication circuit (210) may form a communication channel of the power control device (200). The server communication circuit (210) may form a communication channel with a power meter and an ESS (104) connected to a renewable energy power plant (101) and a power system unit (102), and may receive status information of each component (e.g., the renewable energy power plant (101), the power system unit (102), and the ESS (104)) according to a predefined time unit (e.g., time step). Additionally or alternatively, the server communication circuit (210) may also receive information regarding a predicted amount (e.g., at least one of a predicted amount of generated power) of each component (e.g., the renewable energy power plant (101), the power system unit (102), and the ESS (104)).

[0059] The server input unit (220) may receive administrator input required for operating the power control device (200). As an example, the server input unit (220) may receive administrator input for setting constraints (e.g., minimum energy storage amount (or minimum value) or maximum energy storage amount (or maximum value)) of the ESS (104). Alternatively, the server input unit (220) may receive administrator input requesting output of information related to the operation of the ESS (104). In this regard, the server input unit (220) may include at least one of a physical key, a touch key, a touch screen, a voice input device, a gesture input device, a jog shuttle, and a jog key for administrator input.

[0060] The server memory (230) may store data or programs related to the operation of the power control device (200). For example, the server memory (230) may store environmental status information (231), power information (232), ESS-related information (233), and a reinforcement learning model (234). The environmental status information (231) may include various states of an environment that includes a target that the reinforcement learning model interacts with and controls. For example, the environmental status information (231) may include fixed data such as a unit price of electricity. Alternatively, the environmental status information (231) may include weather information or a temperature image. The power information (232) may include information on generated power provided by a renewable energy power plant (101) and a power system unit (102) (e.g., measurement information such as voltage and current). The above ESS-related information (233) may include at least some of the operation mode, output size, AC / DC voltage / current, temperature, operation mode of the battery, SoC, voltage, current, and temperature of the entire battery cell and each battery cell. The reinforcement learning model (234) may include a model produced by applying reinforcement learning to various pieces of information described above, such as environmental status information (231), power information, and ESS-related information (233). The reinforcement learning model (234) may be updated at regular intervals.

[0061] The server display (240) may output at least one screen related to the operation of the power control device (200). For example, the server display (240) may output a screen including at least one of a connection status with at least one of the components (e.g., the renewable energy power plant (101), the power system unit (102), the customer (103), and the ESS (104)), an operation status of each component, real-time information provided by each component, and prediction information provided by each component. The server display (240) may output at least one of a screen for generating a command related to the control of the ESS (104) based on real-time information and prediction information received from at least one of the components (e.g., the renewable energy power plant (101), the power system unit (102), the customer (103), and the ESS (104)), and a screen for a situation in which the ESS (104) performs a charging operation or a discharging operation according to a message including the generated command. For example, the server display (240) can output real-time information related to the modeling of the reinforcement learning model (234). For example, the server display (240) can output a screen that allows for checking comparison values ​​between the state before and after applying the reinforcement learning model (234) of the present invention in relation to ESS charging / discharging control considering the constraints of the present invention. Alternatively, the server display (240) can output a screen comparing ESS charging / discharging efficiency in a situation where the reinforcement learning model (234) of the present invention is applied and in a situation where it is not applied.

[0062] The server processor (250) may perform at least one of transmitting, processing, and outputting or storing results of processing related to the operation of the power control device (200). For example, the server processor (250) may collect various information required for reinforcement learning at predefined time units (or time units in which reinforcement learning operates), preprocess the collected information, classify the preprocessed information, and perform reinforcement learning modeling based on the classified information. At this time, the server processor (250) provides a virtual environment for reinforcement learning, and when the reinforcement learning model in the virtual environment is completed, the server processor (250) may support service operation related to actual ESS (104) control using the reinforcement learning model (234) in which learning has been completed. In this regard, the server processor (250) may include a configuration as illustrated in FIG. 3.

[0063] FIG. 3 is a drawing showing an example of a server processor configuration among the power control device configurations according to an embodiment of the present invention.

[0064] Referring to FIG. 3, the server processor (250) may include an environmental status information collection unit (251), a collection data preprocessing unit (252), a reinforcement learning status unit (253), a reinforcement learning reward unit (254), a reinforcement learning model learning unit (255), a reinforcement learning model service providing unit (256), and a reinforcement learning action postprocessing unit (257).

[0065] The above environmental status information collection unit (251) can collect various types of environmental status information including objects that the reinforcement learning model (234) interacts with and controls. For example, the environmental status information collection unit (251) can collect various types of information related to power generation and power usage, such as measurement information such as voltage and current (e.g., voltage and current related to at least one of the generated power of a renewable power plant (101) or a power system unit (102), the received power of a consumer (103), and the stored power of an ESS (104)), fixed data such as power unit prices (e.g., power unit price information announced by Korea Electric Power Corporation), temperature images captured by an infrared camera (e.g., the heating temperature of an ESS (104)), and weather information from the Korea Meteorological Administration. In relation to the above-described information collection, the environmental status information collection unit (251) can form communication channels with various devices related to the collection of the above-described information, such as a power meter, an external information provision server (300), and other information, and process the collection of the above-described information periodically or in real time. The environmental status information collection unit (251) can provide the collected information to the collection data preprocessing unit (252).

[0066] The above-described collected data preprocessing unit (252) can change the received information of the environmental state information collection unit (251) in various ways so that the training of the reinforcement learning model (234) is successfully performed. For example, the collected data preprocessing unit (252) can interpolate missing or outlier data or perform normalization to adjust the range between data. For example, the collected data preprocessing unit (252) can determine data above a predefined first reference value or data below a second reference value among the information provided by the environmental state information collection unit (251) as abnormal data and remove the abnormal data. Alternatively, the collected data preprocessing unit (252) can interpolate by referring to other adjacent data for the missing section when some sections of continuous data are missing. The collected data preprocessing unit (252) can perform additional data preprocessing for the purpose of ESS (104) charging / discharging and PCC power control. For example, in order to improve the predictive controllability of the reinforcement learning model (234), if the power generation of a renewable power plant (e.g., solar power) for the next time unit (e.g., Time Step) is predicted and used as one of the reinforcement learning states, the collected data preprocessing unit (252) can perform regression analysis based on the power generation and weather information of the renewable power plant (101) for several predefined past time units, and calculate the predicted amount of power generation. Alternatively, if it is impossible to collect sufficient types of data in the environment, the collected data preprocessing unit (252) can supplement the target amount of data for reinforcement learning by generating virtual data through inference and estimation of the currently collectable data. For example, if the collected data preprocessing unit (252) cannot receive power information (232), it can collect voltage and current information, and calculate power information (232) based on the collected voltage and current information.

[0067] The above reinforcement learning state unit (253) selects information among the output information of the environmental state information collection unit (251) and the collection data preprocessing unit (252) that allows the reinforcement learning model (234) to sufficiently recognize the situation of the current time unit (Time Step) and that falls within a pre-specified standard value, and transmits the selected information to the reinforcement learning model learning unit (255) and the reinforcement learning model service providing unit (256). The pre-specified standard value may be a standard for determining anomalies in information to be applied to the reinforcement learning model.

[0068] The above reinforcement learning reward unit (254) can convert information of the previous time unit and information of the current time unit into numbers to determine whether the change between the previous time unit and the current time unit, which includes the output result of the reinforcement learning model (234), has a positive or negative effect. The above reinforcement learning reward unit (254) can transmit the numerically converted numeric value (e.g., a single numeric value) to the reinforcement learning model learning unit (255) and the reinforcement learning model service providing unit (256).

[0069] The above reinforcement learning model learning unit (255) can process the State and Reward information received from the reinforcement learning state unit (253) and the reinforcement learning reward unit (254) in conjunction with the virtual environment as inputs to the reinforcement learning model (234), and output an action for the next time unit. In addition, the reinforcement learning model can be trained and updated using previous State, Reward, and Action pairs so that the next action is optimal. The reinforcement learning algorithm can combine at least one of several methods such as Model Based or Model Free, Value Function Based or Policy Based, and Discrete Control or Continuous Control, or can apply a deep neural network to reinforcement learning. For example, the reinforcement learning algorithm of the present invention can include a form of deep reinforcement learning combined with deep learning.

[0070] The above reinforcement learning model service providing unit (256) can input the State and Reward information received from the reinforcement learning state unit (253) and the reinforcement learning reward unit (254) into the model by linking with the environment like the reinforcement learning model learning unit (255) and output the action of the next time unit. Compared to the reinforcement learning model learning unit (255), the reinforcement learning model service providing unit (256) uses the reinforcement learning model (234) that has reached the target performance through repeated learning in the reinforcement learning model learning unit (255) without a learning process for the model, thereby outputting the optimal action. In addition, the environment that the reinforcement learning model service providing unit (256) links with can be an actual environment in which the service is to be provided.

[0071] The above reinforcement learning action post-processing unit (257) can post-process the actions corresponding to the outputs of the reinforcement learning model learning unit (255) and the reinforcement learning model service providing unit (256) in accordance with the characteristics and purpose of the ESS (104) to generate an ESS charge / discharge command, and transmit the generated ESS charge / discharge command to the ESS (104). For example, even if reinforcement learning is performed by reflecting the constraint setting values ​​set to prevent overcharging and overdischarging of the ESS (104), the reinforcement learning model (234) can output an action that deviates from the constraint conditions. Accordingly, in the process of outputting an action that goes beyond the overcharge and overdischarge ranges of the ESS (104) from the reinforcement learning model learning unit (255) and the reinforcement learning model service providing unit (256), the reinforcement learning action post-processing unit (257) can adjust the action so that it does not go beyond the preset constraints and finally transmit the adjusted value as an ESS charge and discharge command to the ESS (104).

[0072] As described above, the server processor (250) according to the embodiment of the present invention may collect various information related to the operation of the ESS (104) in the previous time unit and various information related to the operation of the ESS (104) in the current time unit in relation to the charge / discharge control of the ESS (104), and may perform model training for the reinforcement learning model (234) in a virtual environment based on the collected information. In the model training process, the server processor (250) may perform data preprocessing to build a virtual environment for modeling the reinforcement learning model (234), and if the data does not reach the standard target value in the virtual environment building process, other data may be generated using collectible data according to a preset method and then applied to the virtual environment to generate a virtual environment-based reinforcement learning model (234). In addition, the server processor (250) may generate and provide a final charge / discharge control command for the ESS (104) that takes constraints into consideration while applying the reinforcement learning model (234) for which training has been completed to an actual environment. Through this, the power control device (200) of the present invention can provide an improved ESS (104) charging / discharging control effect in the process of applying it to an actual environment while shortening the time required to generate a command for charging / discharging control of an ESS (104) in a virtual environment.

[0073] FIG. 4 is a diagram showing an example of a reinforcement learning reward unit configuration according to an embodiment of the present invention.

[0074] Referring to FIG. 4, a reinforcement learning reward unit (254) according to one embodiment may include a power compensation unit (254a), a movement distance compensation unit (254b), and a constraint compensation unit (254c). This reinforcement learning reward unit (254) is an example of the reinforcement learning reward unit (254) described above in FIG. 3, and may select a final reward by summing up the results of various reward procedures required for reinforcement learning.

[0075] The above power amount compensation unit (254a) is intended to reflect the profit for pure power amount in the action, and can calculate the power amount settlement amount for each hour as compensation. The power amount compensation size is determined by multiplying the power amount and the power unit price, and can be set to a negative number when determining that power is used when charging the ESS (104), and can be determined to be a positive number when determining that power is generated when discharging the ESS (104). This power amount compensation size Reward can be set using the following mathematical expression 1.

[0076] [Mathematical Formula 1]

[0077]

[0078] Energy t is the amount of power in t time units, and Price t can be the unit price of electricity in units of time t. The amount of electricity and the unit price of electricity can vary by unit of time.

[0079] For example, if the ESS (104) charges 100 kWh in the early morning hours when the electricity price is 90 won / kWh, the power compensation unit (254a) can determine the compensation for the corresponding time unit (Time Step) to be -9,000. In addition, if the ESS (104) discharges 100 kWh in the afternoon hours when the electricity price is 200 won / kWh, the power compensation unit (254a) can set the compensation for the corresponding time unit (Time Step) to be +20,000.

[0080] The above movement distance compensation unit (254b) can calculate compensation as a negative value for action changes, thereby minimizing unnecessary action changes. For example, the movement distance compensation unit (254b) can calculate the action change size for each time unit during two previous time units (Time Steps) as a reward, as in the following mathematical expression 2, in order to minimize the action from oscillating up and down.

[0081] [Equation 2]

[0082]

[0083] In mathematical expression 2, action t-2 is the action value of two previous time units, action t-1 is the action value for one unit of time, and action t represents the action value in the current time unit, and -d can represent a value calculated as a negative value for the action change.

[0084] The constraint compensation unit (254c) can set a reward for achieving various constraints when operating the ESS (104). The constraints may include conditions for preventing some of the state information from exceeding preset upper and lower limits when controlling the ESS (104). For example, the constraint compensation unit (254c) can set constraint compensation for various state information of the ESS (104), such as SoC constraint compensation and limit power constraint compensation of the ESS (104). The method by which the constraint compensation unit (254c) sets a reward according to the achievement of constraint conditions is as shown in the following mathematical expression 3. The constraint compensation unit (254c) can increase the learning performance for constraints by varying the upper limit exceeding start reward and the lower limit exceeding start reward according to conditions and situations, and the Reward change ratio coefficient according to the upper limit process degree and the lower limit process degree.

[0085] [Equation 3]

[0086]

[0087] In mathematical formula 3 The condition that is less than or equal to 0 is a normal condition in which the upper limit of the state information is not exceeded. ) means, The condition that is less than or equal to 0 is a normal condition that does not exceed the lower bound of the state information, and is compensated for ( ) can mean. is the compensation conversion ratio coefficient according to the upper limit process degree, is the compensation conversion ratio coefficient according to the degree of the lower limit process. is the upper limit exceeding start reward value, can be a starting reward value exceeding the lower limit.

[0088] FIG. 5 is a diagram for explaining compensation of upper and lower limit constraints according to constraint conditions according to an embodiment of the present invention.

[0089] Referring to FIG. 5, the constraint compensation unit (254c) sets the state value for which the upper and lower constraints are to be set and the upper and lower constraints ( ) and lower bound constraints ( ) You can set the distance of the value to the x-axis and set the constraint compensation according to the distance to the y-axis. If the state value exceeds the upper and lower constraints and the x-axis becomes negative, it is negative. Starting from , it indicates that the constraint compensation value can also increase to a negative value as the upper and lower constraints are exceeded. The slope of the constraint compensation value is a conversion ratio coefficient, which can vary depending on the type of state information applied.

[0090] Meanwhile, the constraint compensation unit (254c) can integrate various constraint compensations with different conditions and circumstances to provide a final constraint compensation. In this regard, the constraint compensation unit (254c) can calculate an integrated constraint compensation value using the following mathematical equation (4).

[0091] [Equation 4]

[0092]

[0093] In mathematical expression 4, t represents a time unit, S represents state information, and C S can mean constraints on state information.

[0094] As an example of constraint compensation, in order to operate the battery of ESS (104) within the range of the limit charge value or the limit discharge value, the constraint compensation unit (254c) can set the maximum and minimum values ​​of SoC, which means the degree of remaining charge, and provide negative compensation when the SoC exceeds the maximum and minimum values, thereby training not to exceed the limit charge or limit discharge. In this regard, the constraint compensation unit (254c) can set the SoC constraint compensation through the following mathematical expression 5 and perform a reinforcement learning model through it.

[0095] [Equation 5]

[0096]

[0097] In mathematical expression 5, The condition that is less than or equal to 0 is that the current SoC value is the maximum value set as the upper limit ( ) is a condition to check whether it exceeds The condition that is less than or equal to 0 is a condition that checks whether the current SoC value exceeds the minimum value set as the SoC lower limit value. If the upper and lower limits are exceeded, a negative compensation ( ) can have the meaning of providing. is the compensation conversion ratio coefficient according to the upper limit SoC degree, is the compensation conversion ratio coefficient according to the lower limit SoC degree. is the upper limit SoC exceedance start compensation value, can be a starting compensation value exceeding the lower limit SoC.

[0098] Meanwhile, in relation to charging and discharging of the ESS (104), generated power may be supplied to the ESS (104) or received power may be supplied from the ESS (104) to the consumer (103) through the PCC point, and in this process, the power meter power of the PCC point may exceed the preset upper and lower limits for various reasons (peak power, power facility capacity, power system stability, etc.). Accordingly, the power control device (200) may need to control the power of the PCC point through charging and discharging of the ESS (104), and may control the power supply or demand to be stable by providing negative compensation when the PCC power exceeds the upper and lower limit settings. In relation to this, the constraint compensation unit (254c) may set constraint compensation for the limit power of the PCC through the following mathematical expression 6, and perform learning of the reinforcement learning model (234) based on this.

[0099] [Equation 6]

[0100]

[0101] The above mathematical equation 6 is, In a situation where the current PCC power is less than 0 and exceeds the PCC upper limit value, negative compensation ( ) can have the meaning of providing, The condition that is less than 0 is a negative compensation (if the current PCC power exceeds the PCC lower limit value). ) can have the meaning of providing. is the compensation conversion ratio coefficient according to the upper limit PCC degree, is the compensation conversion ratio coefficient according to the lower limit PCC degree. is the starting compensation value exceeding the upper limit PCC, can be a starting reward value exceeding the lower bound PCC.

[0102] As described above, the power control device (200) of the present invention can learn the reinforcement learning model (234) while maximizing the overall reward to optimize economic efficiency while maintaining constraints during the charging and discharging control of the ESS (104). The constraint reward set by the power control device (200) can be set to provide a negative reward only when the current state information value deviates from the constraint, and can be set to provide a reward of 0 when the constraint is not deviate. Additionally or alternatively, the power control device (200) can improve the optimal operating performance of the charging and discharging of the ESS (104) by providing an additional reward when the constraint is not deviate.

[0103] As an example, the power control device (200) can induce model learning so that the state information value gradually moves toward the constraint value by providing a relatively high reward when the difference between the upper and lower constraint values ​​of a specific state information value within the constraint is small rather than when it is large. When the state information value is induced to gradually move toward the constraint value, the ESS (104) charging and discharging reinforcement learning model (234) can secure the effect of compensating for the changed battery SoC of the ESS (104) while charging or discharging to prevent exceeding the constraint, and the reinforcement learning model (234) can be trained to get as close as possible to the upper and lower limits of the battery SoC in order to charge the ESS (104) battery as much as possible in the early morning and discharge it as much as possible during the day. In this regard, the constraint compensation unit (254c) can set the relevant constraint compensation through the following mathematical expression 7.

[0104] [Equation 7]

[0105]

[0106] The above mathematical expression 7 can provide an additional compensation value using the z function in addition to the expression that provides a compensation value under the condition that the current state information value described in the mathematical expression 3 above does not exceed the upper limit value. is the compensation conversion ratio coefficient according to the degree of the process up to the upper limit, is the compensation conversion ratio coefficient according to the degree of the process up to the lower limit. is the starting reward value just before exceeding the upper limit, can be the starting reward value just before exceeding the lower limit.

[0107] FIG. 6 is a diagram for explaining compensation and additional compensation for upper and lower limit constraints according to constraint conditions according to an embodiment of the present invention.

[0108] Referring to FIG. 6, in mathematical expression 7, the constraint compensation unit (254c) sets the state value for which the upper and lower constraints are to be set and the upper and lower constraints ( ) and lower bound constraints ( ) You can set the distance of the value to the x-axis and set the constraint compensation according to the distance to the y-axis. If the state value exceeds the upper and lower constraints and the x-axis becomes negative, the negative b t Starting from the upper limit constraint and the lower limit constraint, the constraint compensation value can also increase to a negative value as it exceeds the upper limit constraint and the lower limit constraint. The line corresponding to the straight line equation starting from the origin increases to a negative value as it gets farther from the upper limit constraint value and the lower limit constraint value within the upper limit constraint and the lower limit constraint, so that the compensation can be applied within the upper and lower limits. Accordingly, the constraint compensation unit (254c) can induce action training so that the state information value becomes closer to the constraint value during the learning process of the reinforcement learning model (234). Meanwhile, as illustrated, the constraint compensation unit (254c) sets the constraint compensation b when the upper limit constraint and the lower limit constraint are exceeded. t Starting value h t You can set additional reward values ​​to make them larger.

[0109] FIG. 7 is a drawing showing examples of various shapes related to additional compensation of a constraint compensation unit according to an embodiment of the present invention.

[0110] Referring to FIG. 7, the power control device (200) is as shown in Equation 7 above. and Through adjustment, points that can improve learning performance at various points, as shown in graph 701, can be applied to the training of a reinforcement learning model. Furthermore, the power control device (200) can apply not only the Z function in the form of a first-order linear function, but also various monotonic function forms, as shown in graphs 702 and 703, to the training of a reinforcement learning model.

[0111] FIG. 8 is a diagram showing the result of power upper limit constraint control after additional compensation according to an embodiment of the present invention, and FIG. 9 is a diagram showing a comparison of SoC of an ESS battery before and after additional compensation according to an embodiment of the present invention.

[0112] Referring to FIG. 8, as can be seen from the results of the power upper limit constraint control after additional compensation, when looking at the 15-minute average power graph (805) after control, it can be seen that the power usage graph (803) after control uses power closer to the upper limit value than the power usage graph (801) before control, without exceeding the power upper limit constraint value (807).

[0113] In addition, referring to FIG. 9, when comparing the ESS (104) battery SoC before and after applying the additional compensation value, before the additional compensation, the ESS (104) discharge occurs when the power upper limit constraint is exceeded, which shows a tendency for the ESS (104) battery SoC to decrease, but after applying the additional compensation value, the ESS (104) charging is performed within the power upper limit constraint, thereby supplementing the ESS (104) battery power used to prevent the power upper limit constraint. Through this, it can be confirmed that the ESS battery SoC graph (903) after the additional compensation shows an improved efficiency of 10 percent compared to the ESS battery SoC graph (901) before the additional compensation, and the supplemented battery power can be used for the power upper limit constraint that will occur in the future, and can secure higher profits by discharging during a time period when the unit price of electricity is relatively high.

[0114] As described above, through additional compensation for the reinforcement learning model (234), it can be seen that the system to which the present invention is applied can more efficiently operate the storage capacity of the ESS (104) by allowing the ESS (104) to perform charging or discharging close to the constraint value.

[0115] FIG. 10 is a diagram showing an example of a reinforcement learning modeling method related to ESS charging / discharging control applying constraints according to an embodiment of the present invention.

[0116] Referring to FIG. 10, in relation to a reinforcement learning modeling method for applying constraints related to ESS charge / discharge control according to an embodiment, in step 1001, the server processor (250) of the power control device (200) may collect environmental state information. For example, in relation to reinforcement learning model training, the server processor (250) may collect information including the operation mode, output size, AC / DC voltage / current, temperature, etc. of the ESS power conversion device, which is a major environmental component, from the ESS (104), and the operation mode of the battery, and at least a part of the SoC, voltage, current, and temperature of the entire cell and each cell, for example, for each time step in which reinforcement learning operates. Additionally or alternatively, the server processor (250) may collect power information (e.g., at least a part of the generated power information and the received power information) of a common connection point (e.g., Point of Common Coupling, PCC) of a renewable energy power plant (101) and a power system unit (102) as environmental state information. Additionally, the server processor (250) can collect power price information by time unit as environmental status information.

[0117] In step 1003, the server processor (250) may perform preprocessing on the collected environmental state information. In relation to the preprocessing process, the server processor (250) may determine that values ​​exceeding a predefined reference value among the environmental state information are abnormal data and perform a preprocessing process to remove them from the environmental state information. Alternatively, if a portion of continuous data is missing, the server processor (250) may supplement the missing data by interpolating adjacent data. Alternatively, the server processor (250) may perform normalization to adjust the range of data of the environmental state information. Additionally or alternatively, if the target amount of data for training the reinforcement learning model is not met within a predefined period, the server processor (250) may generate insufficient data through at least one of calculation, prediction, and estimation processes based on previously acquired data.

[0118] At step 1005, the server processor (250) may perform data selection. For example, the server processor (250) may collect information highly relevant to the current time unit situation. For example, the server processor (250) may select, from among the various acquired data, information related to an ESS (104) having specifications most similar to those of the ESS (104) to which the reinforcement learning model is to be applied. In addition, the server processor (250) may select status information of the previous time unit and the current time unit of the environment most similar to the surrounding environment (e.g., power generation and power reception environment) of the ESS (104) on which the reinforcement learning model is trained.

[0119] In step 1007, the server processor (250) may train the reinforcement learning model by applying a reinforcement learning reward. For example, the server processor (250) may convert the state information of the previous time unit, including the output result of the reinforcement learning model, and the state information of the current time unit into a numerical value, and by comparing the numerical values, may calculate whether the change between the previous time unit and the current time unit is positive or negative as a single numerical value. In this process, the server processor (250) may train the reinforcement learning model using the selected data while applying predefined constraints to the state information selected by user input among the environmental state information or predefined state information that requires constraints in the operation of the ESS (104) (e.g., the SoC of the ESS (104), the power unit price, the PCC power). In addition, the server processor (250) may apply an additional reward to induce the reinforcement learning model to move the data to the set value of the constraint at a predefined constant speed or slope.

[0120] In step 1009, the server processor (250) can post-process the action corresponding to the output of the trained reinforcement learning model in accordance with the characteristics and purpose of the ESS (104), generate a charge / discharge command of the ESS (104) corresponding to the post-processing, and transmit it to the ESS (104) of the virtual environment. In relation to the post-processing, even if reinforcement learning training is performed by applying constraints, there may be cases where the output value of the reinforcement learning model exceeds the constraints. The server processor (250) can correct the output of the reinforcement learning model so that it does not exceed the constraints through post-processing, and generate a charge / discharge command of the ESS (104) based on the corrected value. Meanwhile, the ESS (104) of the virtual environment can have the same or similar specifications as the ESS (104) of the real environment to which the reinforcement learning model of the present invention is actually applied, and can have the same environment as the surrounding environmental state information of the ESS (104) of the real environment. The ESS (104) in the virtual environment operates according to the transmitted charge / discharge command and can provide current status information at step 1001.

[0121] Next, in step 1011, the server processor (250) can determine whether the trained reinforcement learning model has achieved the target performance. If the target performance has not been achieved, the process can branch back to step 1001 and re-perform the subsequent operations. If the target performance has been achieved, the server processor (250) can complete the training in step 1013 and store the trained reinforcement learning model (234) in the server memory (230).

[0122] Meanwhile, at least some of the steps related to the reinforcement learning model method described below may be omitted, modified, or replaced with other steps. For example, at least one of steps 1003 and 1005 may be omitted. In step 1007, the reward processing may include at least some of the reward decisions described in the mathematical formula described above. Step 1009 may be omitted.

[0123] While this specification contains details of a number of specific implementations, as described above, these should not be construed as limitations on the scope of any invention or claim, but rather as descriptions of features that may be unique to particular embodiments of particular inventions.

[0124] Furthermore, while operations are depicted in the drawings in a specific order, this should not be construed as requiring that such operations be performed in the specific or sequential order depicted to achieve desired results, or that all depicted operations be performed. In certain instances, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components of the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged into multiple software products.

[0125] This description presents the best mode of the present invention and provides examples to illustrate the invention and enable those skilled in the art to make and use the invention. This specification, written in this manner, is not intended to limit the invention to the specific terms presented. Therefore, while the invention has been described in detail with reference to the examples described above, those skilled in the art will appreciate that modifications, variations, and variations can be made to these examples without departing from the scope of the invention.

[0126] Therefore, the scope of the present invention should not be determined by the described embodiments but by the claims.

[0127] [Explanation of symbols]

[0128] 10: Improved charging and discharging environment for ESS devices

[0129] 101: Renewable Power Plants

[0130] 102: Power System Department

[0131] 103: Acceptance

[0132] 104: ESS device

[0133] 200: Power control unit

[0134] 210: Server Processor

[0135] 220: Server input

[0136] 230: Server memory

[0137] 240: Server display

[0138] 250: Server Processor

[0139] 300: External information provider server

Claims

1. In a power control device that supports ESS (energy storage system, ESS) charging and discharging control based on a reinforcement learning model with constraint conditions applied, A server memory that stores status information related to the ESS and power information related to charging and discharging of the ESS; a server processor functionally connected to the server memory; The above server processor, Collect status information and power information related to the ESS for each time unit corresponding to the training cycle of the reinforcement learning model, After determining the reward based on whether the change between the previous time unit and the current time unit is positive or negative, the collected information and the determined reward are processed as inputs to the reinforcement learning model. Based on the results of input processing of the above reinforcement learning model, the action for the next time unit is output, In response to the above action, it is set to generate a charge / discharge command value related to the operation of the ESS, A power control device supporting ESS charge / discharge control based on a constraint-applied reinforcement learning model, characterized in that, in relation to the above action output, the change in the action is compensated for by using previous action values ​​from multiple previous time units.

2. In paragraph 1, The above server processor, In relation to the above compensation decision, power unit price information is collected by time unit, and the compensation is determined by multiplying the power unit price information by the charge and discharge amounts of the ESS. A power control device supporting ESS charging and discharging control based on a constraint-applied reinforcement learning model, characterized in that the charging amount of the ESS is determined as a negative number and the discharging amount of the ESS is determined as a positive number.

3. In paragraph 2, The above server processor, A power control device supporting ESS charging / discharging control based on a constraint-applied reinforcement learning model, characterized in that a constraint condition is set for at least one of the status information and the power information, and the input of the reinforcement learning model is processed within the constraint condition.

4. In paragraph 1, The above server processor, A power control device supporting ESS charging / discharging control based on a constraint-applied reinforcement learning model, characterized in that an upper limit constraint value and a lower limit constraint value are set for the above-mentioned state information, and a constraint compensation value that increases by a negative amount by an amount that the state information value exceeds the upper limit constraint value or the lower limit constraint value is applied.

5. In paragraph 1, The above server processor, A power control device supporting ESS charge / discharge control based on a constraint-applied reinforcement learning model, characterized in that, within the constraints including an upper limit constraint value and a lower limit constraint value for the state information, a higher reward is provided as the value of the state information gets closer to the upper limit constraint value or the lower limit constraint value.

6. In paragraph 1, The above server processor, A power control device supporting ESS charge / discharge control based on a constraint-applied reinforcement learning model, characterized in that, within the constraints including an upper limit constraint value and a lower limit constraint value for the state information, a lower reward is provided as the value of the state information moves away from the upper limit constraint value or the lower limit constraint value.

7. In paragraph 1, The above server processor, A power control device supporting ESS charge / discharge control based on a constraint-applied reinforcement learning model, characterized in that it is set to perform preprocessing to remove abnormal data exceeding a predefined reference value among the collected information or to interpolate missing parts among the collected information using adjacent data.

8. In paragraph 1, The above server processor, A power control device supporting ESS charge / discharge control based on a constraint-applied reinforcement learning model, characterized in that a virtual environment including a virtual ESS corresponding to status information related to the ESS is constructed, and the charge / discharge command value is applied to the ESS of the virtual environment within the virtual environment to collect information for the next time unit.

9. A method for controlling the charging and discharging of an energy storage device based on a power control device that supports the charging and discharging control of an energy storage system (ESS) based on a reinforcement learning model with constraint conditions applied, A step of collecting status information and power information related to the ESS for each hourly unit corresponding to the training cycle of the reinforcement learning model; A step of determining a reward depending on whether the change between the previous time unit and the current time unit is positive or negative; A step of processing the collected information and the determined reward as inputs to the reinforcement learning model; A step of outputting an action for the next time unit according to input processing of the above reinforcement learning model; A step of generating a charge / discharge command value related to the operation of the ESS in response to the above action; The step of outputting the above action is: A method for controlling ESS charging and discharging based on a constraint-applied reinforcement learning model, characterized in that it comprises a step of compensating for changes in the action by using previous action values ​​from multiple previous time units.

10. In paragraph 9, The steps for determining the above compensation are: A step of collecting power unit price information by time unit and multiplying the power unit price information by the charge and discharge amounts of the ESS to determine the compensation by time unit; The step of determining compensation for each time unit is A method for controlling ESS charging and discharging based on a reinforcement learning model with constraints applied, characterized in that it includes a step of determining the charging amount of the ESS as a negative number and determining the discharging amount of the ESS as a positive number.

11. In paragraph 9, The steps for processing the input of the above reinforcement learning model are: An ESS charging / discharging control method based on a constraint-applied reinforcement learning model, characterized in that it comprises a step of setting a constraint condition for at least one of the status information and the power information and a step of processing the input of the reinforcement learning model within the constraint condition.

12. In paragraph 9, The step of setting the above constraints is: A method for controlling ESS charging and discharging based on a constraint-applied reinforcement learning model, characterized in that it comprises the steps of setting an upper limit constraint value and a lower limit constraint value for the above-mentioned state information, and applying a constraint compensation value that increases by a negative amount by an amount by which the state information value exceeds the upper limit constraint value or the lower limit constraint value.

13. In paragraph 9, The step of setting the above constraints is: A method for controlling ESS charging and discharging based on a reinforcement learning model with constraints applied, characterized in that it comprises a step of providing a higher reward as the value of the state information gets closer to the upper limit constraint value or the lower limit constraint value within the constraints including the upper limit constraint value and the lower limit constraint value for the state information.

14. In paragraph 9, The step of setting the above constraints is: A method for controlling ESS charging and discharging based on a reinforcement learning model with constraints applied, characterized in that it comprises a step of providing a lower reward as the value of the state information moves away from the upper limit constraint value or the lower limit constraint value within the constraints including the upper limit constraint value and the lower limit constraint value for the state information.

15. In paragraph 9, A method for controlling ESS charging and discharging based on a reinforcement learning model with constraints applied, characterized in that it further includes a step of performing preprocessing to remove abnormal data exceeding a predefined reference value among the collected information or interpolating missing parts among the collected information using adjacent data.

16. In paragraph 9, A step of constructing a virtual environment including a virtual ESS corresponding to status information related to the above ESS; A method for controlling ESS charging and discharging based on a constraint-applied reinforcement learning model, characterized in that it further includes a step of collecting information of the next time unit by applying the charging and discharging command value to the ESS of the virtual environment within the virtual environment.

Images