Model training method and device
By generating a policy network and updating power consumption data, the problem of low power scheduling efficiency in the prior art is solved, and the power supply adequacy and long-term balance are achieved during peak power consumption.
Patent Information
- Application Number
- CN202311246164.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-25
- Publication Date
- 2025-03-25
AI Technical Summary
In the prior art, power scheduling efficiency is low, especially during peak electricity consumption periods, and it is difficult to ensure the adequacy and long-term balance of power supply.
By obtaining the historical data set, including multiple power consumption data of multiple processing units at the target time, a corresponding policy network is generated for each processing unit, real-time power storage data is calculated, and power consumption data is updated, preset value network and advantage function are called, and long-term benefits are calculated and the policy network is updated.
The peak power scheduling efficiency of multiple processing units is improved, ensuring the adequacy and long-term balance of power supply, and reducing computing power demand and computing complexity.
Smart Images
Figure CN119695986A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a model training method and device. Background Art
[0002] At present, the electricity consumption in cities is increasing day by day. However, due to the limitations of existing power equipment, it is difficult to ensure long-term sufficient power supply.
[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:
[0004] When users use different power dispatching methods to ensure sufficient power supply during peak power consumption, the problem of low power dispatching efficiency is common. On the one hand, some existing technologies use storage devices to store power during the known low power consumption period based on the existing power consumption rules, as a backup power source during the peak power consumption period, without considering the balanced power storage in the long term. Therefore, too much electric energy is usually stored, which may even cause the peak power consumption to shift, bringing greater power supply impact. On the other hand, some existing technologies establish a power storage model and optimize the corresponding objective function to obtain a reasonable power storage strategy for each time period. However, if the balanced power allocation of multiple buildings in the whole time period is comprehensively considered, the corresponding joint action space calculation is very large, the computing power requirements are too high, and the practicality is poor. Summary of the invention
[0005] In view of this, an embodiment of the present invention provides a model training method and device, which can solve the technical problem of low efficiency of power dispatching during peak power consumption.
[0006] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, a model training method is provided, comprising obtaining a historical data set, including a plurality of power consumption data of a plurality of processing units at a target time; generating a corresponding policy network for each processing unit, so as to use each policy network to calculate the corresponding real-time power storage data based on the corresponding power consumption data, and update the power consumption data of each processing unit accordingly to obtain a sample data set; calling a preset value network to approximately calculate the long-term benefits corresponding to the sample data set; calling a preset advantage function to determine the ascending gradient corresponding to each policy network according to the long-term benefits, so as to update and train each policy network.
[0007] Optionally, each of the processing units comprises: at least one electrical appliance and at least one electrical storage device;
[0008] The power consumption data includes: power consumption data of electrical appliances, power storage device data and environmental data of each processing unit in a corresponding unit time.
[0009] Optionally, the power consumption data of each processing unit is updated accordingly, including:
[0010] Each real-time electricity storage data is added to the electrical appliance electricity consumption data in the corresponding electricity consumption data, and the electrical appliance electricity consumption data is updated according to the addition result.
[0011] Optionally, the power consumption data of each processing unit is updated accordingly, including:
[0012] Obtain the battery capacity data, environmental data and real-time energy storage data of each processing unit in the previous unit time of the target time,
[0013] The preset CityLearn energy simulator is called to calculate the simulated value of the battery power data of the processing unit at the target time and update it accordingly.
[0014] Optionally, the long-term benefit is proportional to the peak value of the total power consumption of the multiple processing units at the target time.
[0015] Optionally, before calling the preset value network, include:
[0016] Acquire multiple power consumption data of multiple processing units within multiple unit times;
[0017] According to the preset reward function, calculate the immediate benefits of multiple processing units in each unit time, and accumulate them to obtain the corresponding long-term benefits;
[0018] Based on the Bellman iteration formula, a value network is constructed to approximate the long-term benefits;
[0019] The value network is iteratively updated multiple times to optimize the value network.
[0020] Optionally, the ascending gradient corresponding to each strategy network is determined based on the long-term benefits, including:
[0021] For each policy network:
[0022] Select one of the remaining policy networks as the comparison policy network.
[0023] In the sample data set, the real-time power storage data corresponding to the comparison strategy network is replaced with an arbitrary value to obtain a comparison sample data set.
[0024] The value network is called to approximately calculate the comparative long-term benefits corresponding to the comparative sample data set,
[0025] The difference between the long-term benefit and the comparative long-term benefit is calculated to determine the ascending gradient corresponding to the policy network according to the difference and the policy gradient theorem.
[0026] Optionally, generating a corresponding policy network for each processing unit includes: generating a policy network corresponding to each processing unit based on a preset policy model;
[0027] Performing update training on each policy network includes: performing corresponding update training on the policy model, and obtaining a trained policy model in response to determining that the policy model converges, or determining that the number of training times of the policy model reaches a preset threshold.
[0028] In addition, the present invention also provides a model training device, including an acquisition module for acquiring a historical data set, including multiple power consumption data of multiple processing units at a target time; a processing module for generating a corresponding policy network for each processing unit, so as to use each policy network to calculate the corresponding real-time power storage data based on the corresponding power consumption data, and update the power consumption data of each processing unit accordingly to obtain a sample data set; a training module for calling a preset value network to approximately calculate the long-term benefits corresponding to the sample data set; and also for calling a preset advantage function to determine the corresponding ascending gradient of each policy network according to the long-term benefits, so as to update and train each policy network.
[0029] An embodiment of the above invention has the following advantages or beneficial effects: the present invention obtains historical data sets, including multiple power consumption data of multiple processing units at target times, to fully utilize the historical power consumption data of multiple processing units in the target area, thereby providing data support for training a power dispatch model corresponding to the corresponding power consumption habits; at the same time, the present invention generates a corresponding strategy network for each processing unit, uses each strategy network to calculate the corresponding real-time power storage data based on the corresponding power consumption data, and updates the power consumption data of each processing unit accordingly to obtain a sample data set, uses a preset simulator to simulate the corresponding changed power consumption data according to the virtual power storage instruction, thereby realizing a precise simulation experiment of the strategy network generated based on the model; in addition, the present invention obtains the calculation result of the current strategy network by calling a preset value network to approximately calculate the long-term benefits corresponding to the sample data set, as feedback data for training the current strategy model; finally, the present invention determines the corresponding ascending gradient of each strategy network according to the long-term benefits by calling a preset advantage function, thereby updating and training each strategy network, and completing the process of directional training and optimization of the strategy model.
[0030] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.
[0032] Figure 1 is a schematic diagram of the main process of the model training method according to the first embodiment of the present invention;
[0033] Figure 2 is a schematic diagram of the main process of the model training method according to the second embodiment of the present invention;
[0034] Figure 3 is a schematic diagram of the main process of the model training method according to the third embodiment of the present invention;
[0035] Figure 4 is a schematic diagram of main modules of a model training device according to a first embodiment of the present invention;
[0036] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;
[0037] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION
[0038] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0039] Figure 1 is a schematic diagram of the main process of the model training method according to the first embodiment of the present invention, such as Figure 1 As shown, the model training method includes:
[0040] Step S101, obtaining a historical data set, including a plurality of power consumption data of a plurality of processing units at a target time.
[0041] In an embodiment, after processing in this step, the historical electricity consumption data of the target area including multiple processing units can be fully utilized, so that in the process of training the model, the model can perceive the electricity consumption habits of the area in multiple dimensions including environmental factors, peak electricity consumption, power storage capacity, etc., and design corresponding power scheduling plans in a targeted manner, that is, store a corresponding amount of electricity in each period of low electricity consumption, so that the final updated power scheduling plan can flatten the peak electricity consumption curve as much as possible on the basis of meeting the electricity demand of the area, and achieve sufficient power supply in each time period.
[0042] In some embodiments, the multiple processing units may be multiple residential buildings, office buildings, etc. that are relatively close to each other; thereby, through the processing of the present invention, each processing unit can reserve electric energy correspondingly during non-peak power consumption periods, so as to achieve the technical effect of shaving the peak power consumption of the total peak power consumption of the multiple processing units; at the same time, the present invention can also consider the coordinated processing of power storage and power consumption among multiple processing units. For example, a processing unit can use the reserved electric energy of other nearby processing units as power supply, thereby further enhancing the processing effect of the present invention for optimizing the scheduling of electric energy between multiple buildings (including various commercial and residential buildings).
[0043] In some embodiments, in order to improve the accuracy of the power consumption data updated based on the power storage action, the power consumption data may include the power consumption data of the electrical appliances, the power data of the storage appliances and the environmental data in each unit time. After this step of processing, multiple direct influencing factors and multiple indirect influencing factors of the power consumption state of each processing unit can be obtained, so that when the CityLearn simulator is used to update the power consumption data in the future, and when the value network, advantage function and strategy network are iteratively calculated, the calculation results are more accurate and closer to the practical situation, thereby improving the practicality of the present invention.
[0044] For example, electricity consumption data can be collected in the following two forms:
[0045] s at =(IP at , I.H. at , I.L. at , SoC at );
[0046] The above IP at Indicates the average indoor temperature of processing unit a within unit time t, IH a represents the average indoor humidity of processing unit a within unit time t, L at Represents the power consumption of processing unit a per unit time t, SoCa t ∈[0, 1] represents the energy storage state of the energy storage device of processing unit a at the beginning of unit time t, that is, the device stores a percentage of its capacity of electrical energy; ε t ={OP t , OH t , ODS t , OFS t};
[0047] The above OP t Indicates the average outdoor temperature per unit time t, OH t Indicates the average outdoor humidity per unit time t, ODS tIndicates the average outdoor solar radiation per unit time t, OFS t It represents the outdoor average solar scattering degree per unit time t. We assume that in the same unit time t, the external environments of the processing units are approximately the same because they are in the same city or the same area in the city.
[0048] In a further embodiment, ε t For at to update.
[0049] Step S102, generating a corresponding strategy network for each processing unit, using each strategy network to calculate corresponding real-time power storage data based on corresponding power consumption data, and correspondingly updating the power consumption data of each processing unit to obtain a sample data set.
[0050] In the embodiment, after the processing of this step, the real-time power storage strategy (data) corresponding to each processing unit is obtained, and the power consumption data corresponding to the current unit time and the next unit time are updated in real time accordingly. At the same time, the data corresponding to the power consumption habits in each processing unit are completely retained, so as to achieve the technical effect of accurately and flexibly simulating the change of power consumption data according to the virtual power storage strategy, so as to accurately analyze and evaluate the changed power consumption peak, and further train and optimize the corresponding model. For example, the power storage strategy of processing unit a can be composed of p a Characterized by the formula p a :s at →u at Definition, s in the formula at is the current power consumption status of each processing unit, u at is the energy storage strategy p a The calculated real-time power storage data (which can also be understood as real-time power storage action)
[0051] In some embodiments, in order to clearly and unambiguously define each real-time power storage data, the real-time power storage data greater than zero can be defined as storing the corresponding electric energy through the corresponding storage device, and the real-time power storage data less than zero can be defined as releasing the corresponding electric energy through the corresponding storage device. For example, the reservoir electric energy can be charging the battery through the power bus. During this process, the overall power consumption corresponding to the power bus increases, so the power consumption of the storage device is positive; releasing electric energy can be supplying power to the electrical appliance through the battery, so the power that the power bus needs to provide to the electrical appliance decreases, so the power consumption of the storage device is negative.
[0052] In some embodiments, in order to accurately update the corresponding power consumption data according to each real-time power storage data, each real-time power storage data and the power consumption data of the electrical appliances in the corresponding power consumption data can be added when the power consumption data of each processing unit is updated, and the power consumption data of the electrical appliances can be updated according to the addition result. In the example described above, if the storage battery is charged and the electrical appliance is directly connected to the power bus, the total power consumption in the corresponding unit time increases; if the storage battery is discharged, the electrical appliance uses the storage battery and the power bus for power supply, and according to historical data, it can be known that the power demand of the electrical appliance in each unit time remains unchanged, so the total power consumption in the corresponding unit time decreases.
[0053] In some embodiments, in order to accurately update the corresponding power storage capacity according to each real-time power storage data, when updating the power consumption data of each processing unit, the power storage capacity data, environmental data and real-time energy storage data of each processing unit in the previous unit time of the target time can be obtained, and the preset CityLearn energy simulator can be called to calculate the simulated value of the power storage capacity data of the processing unit at the target time, and update it accordingly. After this step, the open source CityLearn energy simulator can be used to accurately simulate the change of power storage capacity in each processing unit under historical conditions according to each power storage action, while also taking into account the impact of ambient temperature and humidity on the power storage, so that the simulated updated power storage capacity of the power storage is more convincing.
[0054] Step S103, calling a preset value network to approximately calculate the long-term benefits corresponding to the sample data set.
[0055] In an embodiment, after processing in this step, the corresponding expected value of long-term benefits can be approximately obtained based on the electricity consumption data of a certain unit time without iterating the data of all unit time. This enhances the simulation and training efficiency of the present technical solution, and enables the present technical solution to obtain a trained strategy model more quickly.
[0056] In some embodiments, in order to ensure the accuracy of the policy model and the policy network training direction, it can be determined that the long-term benefit is proportional to the peak value of the total power consumption of the multiple processing units at the target time.
[0057] In some embodiments, in order to obtain an accurate value network so that the calculated long-term benefit approximation has a smaller deviation, multiple power consumption data of multiple processing units in multiple unit times can be obtained before calling the preset value network; according to the preset reward function, the instant benefits corresponding to the multiple processing units in each unit time are calculated, and the corresponding long-term benefits are accumulated; based on the Bellman iteration formula, a value network is constructed to approximately calculate the long-term benefits; the value network is iteratively updated multiple times to optimize the value network. For example, the value network can be as follows: Q in the recursive equation below a As shown, where S on the left side of the equation t is the electricity consumption data at time t, u 1t ,u 2t …u nt is the real-time power storage data corresponding to processing units 1-n respectively; the right side of the equation is the corresponding data at time t+1, and γ is a parameter set manually in [0,1]; by updating the left side of the equation for a specified number of times, the right side of the equation is equal to the left side of the equation, completing the value network Q a Optimization.
[0058]
[0059] For example, the above reward function can be defined by the following formula, so it can be known that the reward function is proportional to the peak value of the sum of power consumption of multiple processing units at the target time t. at represents the electrical power consumption of processing unit a within time t, r at is the corresponding immediate profit.
[0060]
[0061] Step S104, calling a preset advantage function to determine the ascending gradient corresponding to each policy network according to the long-term benefits, so as to perform update training on each policy network.
[0062] In some embodiments, in order to obtain an updated policy network and policy model more efficiently, when determining the ascending gradient corresponding to each policy network based on the long-term benefits, for each policy network: select one of the remaining policy networks as the comparison policy network, and replace the real-time power storage data corresponding to the comparison policy network with an arbitrary value in the sample data set to obtain a comparison sample data set.
[0063] For example, the sample data set can be calculated as follows:
[0064]
[0065] Where uj* is the energy storage data set executed by the energy storage device of any processing unit j. For example, we set uj* = 0, that is, the energy storage device of j does not take any action, neither storing energy nor releasing energy. The advantage function Aa changes the action of any other processing unit, by comparing the current energy storage data u of processing unit a at The long-term benefits of changing the power storage data of any other processing unit to a specified value are used to measure the current power storage data u at The advantages and disadvantages of the system make the measurement results more reasonable.
[0066] The value network is called to approximately calculate the comparative long-term benefits corresponding to the comparative sample data set, and the difference between the long-term benefits and the comparative long-term benefits is calculated to determine the ascending gradient corresponding to the policy network according to the difference and the policy gradient theorem.
[0067] After this step, a more reasonable iteration step size can be calculated, thereby comprehensively improving the efficiency of model training.
[0068] For example, when determining the ascending gradient, the following formula can be used for calculation (i.e., the strategy ascending gradient formula):
[0069]
[0070]
[0071] Here σ is the step size of each iterative update, for example, we can set σ = 0.001.
[0072] In some embodiments, in order to obtain a model with expected computing effects, a corresponding policy network can be generated for each processing unit, including: based on a preset policy model, generating a policy network corresponding to each processing unit; and updating and training each policy network, including: performing corresponding update training on the policy model, and obtaining a trained policy model in response to determining that the policy model converges, or determining that the number of training times of the policy model reaches a preset threshold.
[0073] Figure 2 is a schematic diagram of the main process of the model training method according to the second embodiment of the present invention, the model training method comprising:
[0074] Step S201, obtaining initial data, including power consumption data of multiple processing units in multiple unit times.
[0075] Step S202, calculating the initial peak power consumption characteristic value.
[0076] Step S203: Monitor and call the preset model to generate a third network corresponding to each processing unit.
[0077] Step S204, select a unit time as the target time, and execute the following steps in a loop until there is no next unit time.
[0078] Step S205, determining the target time.
[0079] Step S206, determining the power storage action of each processing unit at the target time according to the corresponding third network and the power consumption data at the target time.
[0080] Step S207 , correspondingly updating the power consumption data of each processing unit at the target time and the unit time next to the target time.
[0081] Step S208, replacing the target time with the next unit time.
[0082] Step S209, obtaining updated data and calculating updated peak power consumption characteristic values.
[0083] Step S210, inputting the initial peak power consumption characteristic value and the updated peak power consumption characteristic value into the model, and training the model with the goal of making the corresponding difference value approach the expected value.
[0084] Step S211, obtaining an updated model, and updating each third network accordingly.
[0085] Step S212, obtaining a trained model.
[0086] Figure 3 is a schematic diagram of the main process of the model training method according to the third embodiment of the present invention, the model training method comprising:
[0087] Step S301, obtaining a historical data set, including a plurality of power consumption data of a plurality of processing units at a target time.
[0088] Preferably, each of the processing units includes: at least one electrical appliance and at least one electrical storage device; the power consumption data includes: electrical appliance power consumption data, electrical storage device power data and environmental data of each processing unit in a corresponding unit time.
[0089] Step S302: Generate a policy network corresponding to each processing unit based on a preset policy model.
[0090] Step S303, adding each real-time power storage data and the power consumption data of the electrical appliances in the corresponding power consumption data, and updating the power consumption data of the electrical appliances according to the addition result.
[0091] Step S304: updating the corresponding battery capacity data according to the real-time power storage data.
[0092] Preferably, the battery power data, environmental data and real-time energy storage data of each processing unit in the unit time before the target time are obtained, and the preset CityLearn energy simulator is called to calculate the simulated value of the battery power data of the processing unit at the target time, and update it accordingly.
[0093] Step S305, obtaining a sample data set updated based on the historical data set.
[0094] Step S306, calling a preset value network to approximately calculate the long-term benefits corresponding to the sample data set.
[0095] Preferably, the long-term benefit is determined to be proportional to the peak value of the total power consumption of the multiple processing units at the target time.
[0096] Step S307, calling the preset advantage function to determine the ascending gradient corresponding to each policy network according to the long-term benefits, and performing update training on each policy network.
[0097] Preferably, for each strategy network: select one of the remaining strategy networks as the comparison strategy network, replace the real-time power storage data corresponding to the comparison strategy network with an arbitrary value in the sample data set, obtain the comparison sample data set, call the value network to approximately calculate the comparison long-term benefit corresponding to the comparison sample data set, calculate the difference between the long-term benefit and the comparison long-term benefit, and determine the ascending gradient corresponding to the strategy network based on the difference and the policy gradient theorem.
[0098] Step S308, performing corresponding update training on the policy model, in response to determining that the policy model converges, or determining that the number of training times of the policy model reaches a preset threshold, a trained policy model is obtained.
[0099] Figure 4 is a schematic diagram of the main modules of the model training device according to an embodiment of the present invention, such as Figure 4 As shown, the model training device 400 includes an acquisition module 401, a processing module 402 and a training module 403. Among them, the acquisition module 401 is used to acquire a historical data set, including multiple power consumption data of multiple processing units at a target time; the processing module 402 generates a corresponding policy network for each processing unit, so as to use each policy network to calculate the corresponding real-time power storage data based on the corresponding power consumption data, and update the power consumption data of each processing unit accordingly to obtain a sample data set; and the training module 403 calls a preset value network to approximately calculate the long-term benefits corresponding to the sample data set; the training module 403 is also used to call a preset advantage function to determine the ascending gradient corresponding to each policy network according to the long-term benefits, so as to update and train each policy network.
[0100] In some embodiments, the acquisition module 401 is further used to: determine that each processing unit includes: at least one electrical appliance and at least one electrical storage device; determine that the power consumption data includes: electrical appliance power consumption data, electrical storage device power data and environmental data of each processing unit in the corresponding unit time.
[0101] In some embodiments, the processing module 402 is also used to: when the power consumption data of each processing unit is updated correspondingly, add each real-time power storage data and the power consumption data of the electrical appliance in the corresponding power consumption data, and update the power consumption data of the electrical appliance according to the addition result.
[0102] In some embodiments, the processing module 402 is also used to: when updating the power consumption data of each processing unit, obtain the battery power data, environmental data and real-time energy storage data of each processing unit in the unit time before the target time, call the preset CityLearn energy simulator, calculate the simulation value of the battery power data of the processing unit at the target time, and update it accordingly.
[0103] In some embodiments, the training module 403 is further used to: determine that the long-term benefit is proportional to the peak value of the total power consumption of the multiple processing units at the target time.
[0104] In some embodiments, the training module 403 is also used to: obtain multiple electricity consumption data of multiple processing units in multiple unit times before calling a preset value network; calculate the instantaneous benefits corresponding to the multiple processing units in each unit time according to a preset reward function, and accumulate the corresponding long-term benefits; construct a value network based on the Bellman iterative formula to approximately calculate the long-term benefits; and perform multiple iterative updates on the value network to optimize the value network.
[0105] In some embodiments, the training module 403 is also used for: when determining the ascending gradient corresponding to each policy network according to the long-term benefits, for each policy network: selecting one of the remaining policy networks as the comparison policy network, replacing the real-time power storage data corresponding to the comparison policy network with an arbitrary value in the sample data set, obtaining a comparison sample data set, calling the value network to approximately calculate the comparison long-term benefits corresponding to the comparison sample data set, calculating the difference between the long-term benefits and the comparison long-term benefits, and determining the ascending gradient corresponding to the policy network according to the difference and the policy gradient theorem.
[0106] In some embodiments, the training module 403 is also used to: generate a corresponding policy network for each processing unit, including: based on a preset policy model, generating a policy network corresponding to each processing unit; updating and training each policy network, including: performing corresponding update training on the policy model, and obtaining a trained policy model in response to determining that the policy model converges, or determining that the number of training times of the policy model reaches a preset threshold.
[0107] It should be noted that the model training method and the model training device described in the present invention have a corresponding relationship in terms of specific implementation contents, so the repeated contents will not be described again.
[0108] Figure 5 An exemplary system architecture 500 is shown to which the model training method or model training device according to an embodiment of the present invention can be applied.
[0109] like Figure 5 As shown, the system architecture 500 may include terminal devices 501, 502, 503, a network 505 and a server 505. The network 505 is used to provide a medium for communication links between the terminal devices 501, 502, 503 and the server 505. The network 505 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0110] The user can use the terminal devices 501, 502, 503 to interact with the server 505 through the network 505 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 501, 502, 503.
[0111] The terminal devices 501 , 502 , and 503 may be various electronic devices having a page display processing screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.
[0112] The server 505 may be a server that provides various services, such as a background management server (only an example) that provides support for users to use the terminal devices 501, 502, and 503. The background management server may analyze and process the received data such as the product information query request, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.
[0113] It should be noted that the model training method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the computing device is generally arranged in the server 505 .
[0114] It should be understood that Figure 5The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.
[0115] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0116] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0117] The following components are connected to the I / O interface 606: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal page display processor (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 606 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.
[0118] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.
[0119] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0120] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0121] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes an acquisition module, a processing module and a training module. The names of these modules do not constitute limitations on the modules themselves in some cases.
[0122] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes obtaining a historical data set, including multiple power consumption data of multiple processing units at a target time; generating a corresponding strategy network for each processing unit, so as to use each strategy network to calculate the corresponding real-time power storage data based on the corresponding power consumption data, and update the power consumption data of each processing unit accordingly to obtain a sample data set; calling a preset value network to approximately calculate the long-term benefits corresponding to the sample data set; calling a preset advantage function to determine the corresponding ascending gradient of each strategy network according to the long-term benefits, so as to update and train each strategy network.
[0123] The technical solution according to the embodiment of the present invention can solve the technical problem of low efficiency of power dispatching during the existing peak power consumption period.
[0124] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for processing electricity consumption data, characterized in that: include: Acquire a historical data set including a plurality of power consumption data of a plurality of processing units at a target time; Generate a corresponding strategy network for each processing unit, use each strategy network to calculate corresponding real-time power storage data based on corresponding power consumption data, and update the power consumption data of each processing unit accordingly to obtain a sample data set; Call the preset value network to approximate the long-term benefits corresponding to the sample data set; The preset advantage function is called to determine the corresponding ascending gradient of each policy network according to the long-term benefits, so as to update and train each policy network.
2. The method according to claim 1, characterized in that: Each of the processing units includes: at least one electrical appliance and at least one electrical storage device; The power consumption data includes: power consumption data of electrical appliances, power storage device data and environmental data of each processing unit in a corresponding unit time.
3. The method according to claim 2, characterized in that The power consumption data of each processing unit is updated accordingly, including: Each real-time electricity storage data is added to the corresponding electrical appliance electricity consumption data, and the electrical appliance electricity consumption data is updated according to the addition result.
4. The method according to claim 1, characterized in that The power consumption data of each processing unit is updated accordingly, including: Obtain the battery capacity data, environmental data and real-time energy storage data of each processing unit in the previous unit time of the target time, The preset CityLearn energy simulator is called to calculate the simulated value of the battery power data of the processing unit at the target time and update it accordingly.
5. The method according to claim 1, characterized in that The long-term benefit is proportional to the peak value of the total power consumption of the multiple processing units at the target time.
6. The method according to claim 1, characterized in that Before calling the preset value network, include: Acquire multiple power consumption data of multiple processing units within multiple unit times; According to the preset reward function, calculate the immediate benefits of multiple processing units in each unit time, and accumulate them to obtain the corresponding long-term benefits; Based on the Bellman iteration formula, a value network is constructed to approximate the long-term benefits; The value network is iteratively updated multiple times to optimize the value network.
7. The method according to claim 1, characterized in that Determine the corresponding upward gradient of each strategy network based on long-term returns, including: For each policy network: Select one of the remaining policy networks as the comparison policy network. In the sample data set, the real-time power storage data corresponding to the comparison strategy network is replaced with an arbitrary value to obtain a comparison sample data set. Call the value network to approximately calculate the comparative long-term benefits corresponding to the comparative sample data set, The difference between the long-term benefit and the comparative long-term benefit is calculated to determine the ascending gradient corresponding to the policy network according to the difference and the policy gradient theorem.
8. The method according to any one of claims 1 to 7, characterized in that: Generating a corresponding strategy network for each processing unit, including: generating a strategy network corresponding to each processing unit based on a preset strategy model; Performing update training on each policy network includes: performing corresponding update training on the policy model, and obtaining a trained policy model in response to determining that the policy model converges, or determining that the number of training times of the policy model reaches a preset threshold.
9. An electricity consumption data processing device, characterized in that: include: An acquisition module, used to acquire a historical data set, including a plurality of power consumption data of a plurality of processing units at a target time; A processing module, used to generate a corresponding strategy network for each processing unit, to use each strategy network to calculate corresponding real-time power storage data based on corresponding power consumption data, and to update the power consumption data of each processing unit accordingly to obtain a sample data set; The training module is used to call the preset value network to approximately calculate the long-term benefits corresponding to the sample data set; It is also used to call the preset advantage function to determine the ascending gradient corresponding to each policy network based on the long-term benefits, so as to update the training of each policy network.
10. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.
11. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.