Power transaction optimization decision-making method and system based on multi-target parameter weight

Abstract

The invention discloses a power transaction optimization decision-making method and system based on a multi-target parameter weight. The method comprises the steps of obtaining transaction data of a power market; preprocessing the transaction data of the electricity market to obtain transaction sample data; and sending the transaction sample data to a preset electricity market environment model to obtain a transaction decision of the current electricity market. According to the power transaction multi-target weight automatic selection method, data collection and optimal strategy output are carried out at the same time through reinforcement learning of related algorithms and by means of the powerful computing power of a computer, personnel intervention is avoided, and strategy making is more credible.

Description

Technical Field

[0001] The present invention relates to the technical field of power trading, and more specifically, to a power trading optimization decision method and system based on multi-objective parameter weights. Background Art

[0002] With the reform of the national power system, market vitality has been stimulated. More and more entities are participating in power market transactions, and their willingness is getting stronger. How to formulate trading strategies without relying on human will and minimize the electricity costs of power users (compared with traditional power grid companies directly buying electricity) is a problem that needs to be solved; this decision is a complex system. Now, when selecting multiple objectives such as transaction cost (Cost), supply and demand balance (Balance), energy and environmental protection score (Eco-Score), and transaction efficiency (Efficiency), they are all based on experience judgment or subjective conduct, resulting in subjectivity in the final formulation of electricity prices and strategies, and it is impossible to be fair and just.

[0003] Therefore, the prior art has defects and needs to be improved urgently. Summary of the invention

[0004] In view of the above problems, the purpose of the present invention is to provide a method and system for optimizing power trading decisions based on multi-objective parameter weights, which can avoid human intervention and make the formulation of power trading strategies more trustworthy.

[0005] A first aspect of the present invention provides a method for optimizing power trading decisions based on multi-objective parameter weights, comprising:

[0006] Obtain transaction data from the electricity market;

[0007] Preprocess the transaction data of the power market to obtain transaction sample data;

[0008] The transaction sample data is sent to the preset power market environment model to obtain the current power market transaction decision.

[0009] In this solution, the steps of constructing the preset power market environment model specifically include:

[0010] Divide the preset initialization model into a target network module, a main network module and an agent module;

[0011] Obtain historical transaction data of the electricity market;

[0012] Preprocess the historical transaction data of the electricity market to obtain training samples;

[0013] Send the training samples to the target network module and the main network module to obtain the loss function value;

[0014] If the loss function value is less than or equal to the preset first threshold, the training ends and the corresponding parameter weight coefficient is extracted;

[0015] The parameter weight coefficients are sent to the agent module for storage.

[0016] This plan also includes:

[0017] If the loss function is greater than the preset first threshold, the gradient of the main network parameters is determined according to the loss function, and the parameters of the main network are updated based on the gradient descent method, and the parameters of the main network are set to θ Q , whose formula is where θ′ Q represents the adjustment parameters of the main network, k is the learning rate, Represents the gradient of the loss function L with respect to the main network parameters; the parameters of the main network are replaced by the adjustment parameters of the main network.

[0018] This plan also includes:

[0019] When the loss function is greater than a preset first threshold, extracting the number of times the initialization model is iteratively calculated during training;

[0020] Determine whether the number of iterative calculations is greater than or equal to the preset number threshold. If so, the training ends and the corresponding parameter weight coefficient is extracted;

[0021] Sending the parameter weight coefficients to the agent module for storage;

[0022] If not, continue training the corresponding initialized model.

[0023] In this solution, if no, the steps of continuing training the corresponding initialization model specifically include:

[0024] Based on each preset time step, the agent module arbitrarily selects a factor that affects transaction efficiency and benefits and the corresponding normalized parameter value according to the current state, and extracts the experience under the corresponding normalized parameter value;

[0025] sending the experience to a preset experience playback buffer for storage;

[0026] Extract any batch of experience samples in the experience replay buffer to repeat the training, and save the parameters of the main network during each training process until the training is completed.

[0027] In this solution, the step of obtaining the loss function value specifically includes:

[0028] Extracting normalized parameter values ​​that affect transaction efficiency and benefits in training samples and market environment status under corresponding normalized parameter values;

[0029] The normalized parameter values ​​that affect transaction efficiency and benefits in the training samples and the market environment status under the corresponding normalized parameter values ​​are sent to the target network module and the main network module respectively to obtain the target Q value and the predicted Q value;

[0030] Based on the mean square error loss function, the loss function value is obtained according to the target Q value and the predicted Q value.

[0031] In this scheme, the steps of obtaining the current electricity market transaction decision include:

[0032] Extract the factors that affect transaction efficiency and benefits from the transaction sample data, as well as the normalized parameter values ​​of the corresponding factors;

[0033] Multiply the normalized parameter value of the factor by the corresponding parameter weight coefficient to obtain the state value of the corresponding factor;

[0034] Accumulate the status values ​​of all factors to obtain the status value of the corresponding transaction sample data;

[0035] Based on the preset state value interval that the state value falls into, the corresponding power trading optimization decision is matched.

[0036] A second aspect of the present invention provides a power transaction optimization decision system based on multi-objective parameter weights, comprising a memory and a processor, wherein the memory stores a power transaction optimization decision method program based on multi-objective parameter weights, and when the power transaction optimization decision method program based on multi-objective parameter weights is executed by the processor, the following steps are implemented:

[0037] Obtain transaction data from the electricity market;

[0038] Preprocess the transaction data of the power market to obtain transaction sample data;

[0039] The transaction sample data is sent to the preset power market environment model to obtain the current power market transaction decision.

[0040] In this solution, the steps of constructing the preset power market environment model specifically include:

[0041] Divide the preset initialization model into a target network module, a main network module and an agent module;

[0042] Obtain historical transaction data of the electricity market;

[0043] Preprocess the historical transaction data of the electricity market to obtain training samples;

[0044] Send the training samples to the target network module and the main network module to obtain the loss function value;

[0045] If the loss function value is less than or equal to the preset first threshold, the training ends and the corresponding parameter weight coefficient is extracted;

[0046] The parameter weight coefficients are sent to the agent module for storage.

[0047] This plan also includes:

[0048] If the loss function is greater than the preset first threshold, the gradient of the main network parameters is determined according to the loss function, and the parameters of the main network are updated based on the gradient descent method, and the parameters of the main network are set to θ Q , whose formula is where θ′ Q represents the adjustment parameters of the main network, k is the learning rate, Represents the gradient of the loss function L with respect to the main network parameters; the parameters of the main network are replaced by the adjustment parameters of the main network.

[0049] The present invention discloses a method and system for optimizing decision-making in power trading based on multi-objective parameter weights. Through reinforcement learning related algorithms and with the help of powerful computer computing power, data is collected while the optimal strategy is output. A method for automatically selecting multi-objective weights for power trading is provided, which avoids human intervention and makes strategy formulation more trustworthy. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flow chart showing a method for optimizing power trading decisions based on multi-objective parameter weights according to the present invention is shown;

[0051] Figure 2 A block diagram of an electric power transaction optimization decision system based on multi-objective parameter weights of the present invention is shown. DETAILED DESCRIPTION

[0052] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0053] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0054] Figure 1 A flow chart of a power transaction optimization decision method based on multi-objective parameter weights of the present invention is shown.

[0055] like Figure 1As shown, the present invention discloses a power transaction optimization decision method based on multi-objective parameter weights, comprising:

[0056] S101, obtaining transaction data of the power market;

[0057] S102, pre-processing the transaction data of the power market to obtain transaction sample data;

[0058] S103, sending the transaction sample data to the preset power market environment model to obtain the transaction decision of the current power market.

[0059] According to an embodiment of the present invention, transaction data of the electricity market is collected, including but not limited to electricity demand, supply, price, weather conditions, etc. The preprocessing step includes data cleaning, transformation and normalization processing to eliminate magnitude differences and noise interference, obtain transaction sample data, and through a preset electricity market environment module, obtain a transaction decision that saves the most on the total electricity cost of the entire electricity transaction.

[0060] According to an embodiment of the present invention, the step of constructing the preset power market environment model specifically includes:

[0061] Divide the preset initialization model into a target network module, a main network module and an agent module;

[0062] Obtain historical transaction data of the electricity market;

[0063] Preprocess the historical transaction data of the electricity market to obtain training samples;

[0064] Send the training samples to the target network module and the main network module to obtain the loss function value;

[0065] If the loss function value is less than or equal to the preset first threshold, the training ends and the corresponding parameter weight coefficient is extracted;

[0066] The parameter weight coefficients are sent to the agent module for storage.

[0067] It should be noted that there are multiple factors in the historical transaction data that affect transaction efficiency and benefits, such as transaction costs, supply and demand balance, energy and environmental protection restrictions, etc. Through preprocessing, training samples are obtained, and the training samples include normalized parameter values ​​of various factors; when the training is completed, the loss function value before the end of the training is extracted, and based on the loss function value before the end of the training, the normalized parameter value corresponding to the predicted Q value and the normalized parameter value corresponding to the target Q value in the loss function value calculation process are extracted, and the normalized parameter value corresponding to the target Q value is divided by the normalized parameter value corresponding to the predicted Q value to obtain the parameter weight coefficient of the factor corresponding to the current normalized parameter value.

[0068] According to an embodiment of the present invention, it also includes:

[0069] If the loss function is greater than the preset first threshold, the gradient of the main network parameters is determined according to the loss function, and the parameters of the main network are updated based on the gradient descent method, and the parameters of the main network are set to θ Q , whose formula is where θ′ Q represents the adjustment parameters of the main network, k is the learning rate, Represents the gradient of the loss function L with respect to the main network parameters; the parameters of the main network are replaced by the adjustment parameters of the main network.

[0070] It should be noted that the gradient of the loss function with respect to the parameters of the main network is calculated by back propagation, and the parameters of the main network are updated using the gradient descent method; further, the predicted Q value is adjusted by decoupling the selection of the target Q value action and the calculation of the target Q value, where the formula is: Q(s, a)←(1-k)*Q(s, a)+k*[r+γmax a′ Q′(s′, a′)]; where k is the learning rate, which determines the influence of the newly learned information on the original predicted Q value, r is the immediate reward obtained after executing the action, γ is the discount factor, which is used to measure the current value of future rewards, and max means selecting the largest predicted Q value from all possible next actions. The update of the predicted Q value is an iterative process, which approaches the true optimal value through continuous learning. The action is the normalized parameter value of the factors affecting transaction efficiency and benefits.

[0071] According to an embodiment of the present invention, it also includes:

[0072] When the loss function is greater than a preset first threshold, extracting the number of times the initialization model is iteratively calculated during training;

[0073] Determine whether the number of iterative calculations is greater than or equal to the preset number threshold. If so, the training ends and the corresponding parameter weight coefficient is extracted;

[0074] Sending the parameter weight coefficients to the agent module for storage;

[0075] If not, continue training the corresponding initialized model.

[0076] It should be noted that the iterative calculation refers to the process in which the initialization module uses a batch of data to perform a parameter update. It is not practical to use all the data for training at one time. Instead, the training sample data is divided into multiple small batches. Each time a batch is processed, the model is initialized to perform forward propagation, calculate the loss, back propagate, and update the parameters. This is a step. The number of steps is equal to the total number of samples in the training sample data divided by the batch size, that is, the number of iterative calculations.

[0077] According to an embodiment of the present invention, if not, the step of continuing training the corresponding initialization model specifically includes:

[0078] Based on each preset time step, the agent module arbitrarily selects a factor that affects transaction efficiency and benefits and the corresponding normalized parameter value according to the current state, and extracts the experience under the corresponding normalized parameter value;

[0079] sending the experience to a preset experience playback buffer for storage;

[0080] Extract any batch of experience samples in the experience replay buffer to repeat the training, and save the parameters of the main network during each training process until the training is completed.

[0081] It should be noted that the number of states of reinforcement learning is set in advance. For example, if there are 4 factors affecting transaction efficiency and benefits, then 4 goals are set. If the step size is adjusted to 0.01, that is, 100 state information is set, at each time step, the agent module selects an action according to the current state, observes the feedback from the environment after executing the action, and determines the experience under the corresponding normalized parameter value, wherein the experience includes the next state, timely reward, and whether the termination condition is met. In each training step, a batch of experience samples are randomly selected from the experience playback buffer, and the main network is used to select actions. For the current state in each experience sample, the main network is used to select the best action that can be taken in the current state.

[0082] According to an embodiment of the present invention, the step of obtaining the loss function value specifically includes:

[0083] Extracting normalized parameter values ​​that affect transaction efficiency and benefits in training samples and market environment status under corresponding normalized parameter values;

[0084] The normalized parameter values ​​that affect transaction efficiency and benefits in the training samples and the market environment status under the corresponding normalized parameter values ​​are sent to the target network module and the main network module respectively to obtain the target Q value and the predicted Q value;

[0085] Based on the mean square error loss function, the loss function value is obtained according to the target Q value and the predicted Q value.

[0086] According to an embodiment of the present invention, for the next state in each experience training sample, the target network is used to calculate the maximum Q value (i.e., the target Q value) of all possible actions taken in this state, and the target Q value is set to y, and its formula is: y = r + γmax a′ Q′(s′, a′), where r represents the immediate reward, γ is the discount factor, Q′(s′, a′) is the target network’s prediction of the Q value of all possible actions a′ in the next state s′, and max a′Indicates the selection of the maximum Q value; the predicted Q value is given by the main network. For the current state s and the selected action a, the predicted Q value is expressed as Q(s, a). The predicted Q value is the Q value prediction given by the main network based on the current state s and the action a; the loss function value is set to L, and its formula is Where N is the size of the training sample batch, y i is the target Q value of the i-th sample, Q(s i , a i ) represents the predicted Q value of the ith sample; the action a represents the normalized parameter value of factor a that affects transaction efficiency and benefits.

[0087] According to an embodiment of the present invention, the step of obtaining a transaction decision of the current electricity market includes:

[0088] Extract the factors that affect transaction efficiency and benefits from the transaction sample data, as well as the normalized parameter values ​​of the corresponding factors;

[0089] Multiply the normalized parameter value of the factor by the corresponding parameter weight coefficient to obtain the state value of the corresponding factor;

[0090] Accumulate the status values ​​of all factors to obtain the status value of the corresponding transaction sample data;

[0091] Based on the preset state value interval that the state value falls into, the corresponding power trading optimization decision is matched.

[0092] It should be noted that if the factors affecting transaction efficiency and benefits include transaction cost (Cost), supply and demand balance (Balance), energy and environmental protection score (Eco-Score), and transaction efficiency (Efficiency), if the state value of the transaction sample data is set to F(x), the formula is:

[0093] F(x)=ω1·Cost(x)+ω2·Balance(x)+ω3·Eco-Score(x)+ω4

[0094] Efficiency(x)

[0095] Among them, ω1, ω2, ω3 and ω4 represent the parameter weight coefficients of the corresponding factors, Cost(x), Balance(x), Eco-Score(x) and Efficiency(x) represent the normalized parameter values ​​of the corresponding factors, where “·” represents multiplication; each preset state value interval corresponds to a state, and each state corresponds to an electricity trading decision.

[0096] Figure 2 A block diagram of an electric power transaction optimization decision system based on multi-objective parameter weights of the present invention is shown.

[0097] like Figure 2 As shown, the second aspect of the present invention provides a power transaction optimization decision system 2 based on multi-objective parameter weights, including a memory 21 and a processor 22, wherein the memory stores a power transaction optimization decision method program based on multi-objective parameter weights, and the power transaction optimization decision method program based on multi-objective parameter weights implements the following steps when executed by the processor:

[0098] Obtain transaction data from the electricity market;

[0099] Preprocess the transaction data of the power market to obtain transaction sample data;

[0100] The transaction sample data is sent to the preset power market environment model to obtain the current power market transaction decision.

[0101] According to an embodiment of the present invention, the step of constructing the preset power market environment model specifically includes:

[0102] Divide the preset initialization model into a target network module, a main network module and an agent module;

[0103] Obtain historical transaction data of the electricity market;

[0104] Preprocess the historical transaction data of the electricity market to obtain training samples;

[0105] Send the training samples to the target network module and the main network module to obtain the loss function value;

[0106] If the loss function value is less than or equal to the preset first threshold, the training ends and the corresponding parameter weight coefficient is extracted;

[0107] The parameter weight coefficients are sent to the agent module for storage.

[0108] According to an embodiment of the present invention, it also includes:

[0109] If the loss function is greater than the preset first threshold, the gradient of the main network parameters is determined according to the loss function, and the parameters of the main network are updated based on the gradient descent method, and the parameters of the main network are set to θ Q , whose formula is where θ′ Q represents the adjustment parameters of the main network, k is the learning rate, Represents the gradient of the loss function L with respect to the main network parameters; the parameters of the main network are replaced by the adjustment parameters of the main network.

[0110] The present invention discloses a method and system for optimizing decision-making in power trading based on multi-objective parameter weights. Through reinforcement learning related algorithms and with the help of powerful computer computing power, data is collected while the optimal strategy is output. A method for automatically selecting multi-objective weights for power trading is provided, which avoids human intervention and makes strategy formulation more trustworthy.

[0111] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0112] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0113] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0114] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.

[0115] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

Claims

1. A power transaction optimization decision method based on multi-objective parameter weights, characterized in that: include: Obtain transaction data from the electricity market; Preprocess the transaction data of the power market to obtain transaction sample data; The transaction sample data is sent to the preset power market environment model to obtain the current power market transaction decision.

2. The method for optimizing power trading based on multi-objective parameter weights according to claim 1 is characterized in that: The steps of constructing the preset power market environment model specifically include: Divide the preset initialization model into a target network module, a main network module and an agent module; Obtain historical transaction data of the electricity market; Preprocess the historical transaction data of the electricity market to obtain training samples; Send the training samples to the target network module and the main network module to obtain the loss function value; If the loss function value is less than or equal to the preset first threshold, the training ends and the corresponding parameter weight coefficient is extracted; The parameter weight coefficients are sent to the agent module for storage.

3. The method for optimizing power trading based on multi-objective parameter weights according to claim 2 is characterized in that: Also includes: If the loss function is greater than the preset first threshold, the gradient of the main network parameters is determined according to the loss function, and the parameters of the main network are updated based on the gradient descent method, and the parameters of the main network are set to θ Q , whose formula is where θ ′ Q represents the adjustment parameters of the main network, k is the learning rate, Represents the gradient of the loss function L with respect to the main network parameters; the parameters of the main network are replaced by the adjustment parameters of the main network.

4. The method for optimizing power trading based on multi-objective parameter weights according to claim 2 is characterized in that: Also includes: When the loss function is greater than a preset first threshold, extracting the number of times the initialization model is iteratively calculated during training; Determine whether the number of iterative calculations is greater than or equal to the preset number threshold. If so, the training ends and the corresponding parameter weight coefficient is extracted; Sending the parameter weight coefficients to the agent module for storage; If not, continue training the corresponding initialized model.

5. The method for optimizing power trading based on multi-objective parameter weights according to claim 4 is characterized in that: If not, the steps of continuing training the corresponding initialization model specifically include: Based on each preset time step, the agent module arbitrarily selects a factor that affects transaction efficiency and benefits and the corresponding normalized parameter value according to the current state, and extracts the experience under the corresponding normalized parameter value; sending the experience to a preset experience playback buffer for storage; Extract any batch of experience samples in the experience replay buffer to repeat the training, and save the parameters of the main network during each training process until the training is completed.

6. The method for optimizing power trading based on multi-objective parameter weights according to claim 2 is characterized in that: The step of obtaining the loss function value specifically includes: Extracting normalized parameter values ​​that affect transaction efficiency and benefits in training samples and market environment status under corresponding normalized parameter values; The normalized parameter values ​​that affect transaction efficiency and benefits in the training samples and the market environment status under the corresponding normalized parameter values ​​are sent to the target network module and the main network module respectively to obtain the target Q value and the predicted Q value; Based on the mean square error loss function, the loss function value is obtained according to the target Q value and the predicted Q value.

7. The method for optimizing power trading based on multi-objective parameter weights according to claim 1 is characterized in that: The step of obtaining the current electricity market transaction decision includes: Extract the factors that affect transaction efficiency and benefits from the transaction sample data, as well as the normalized parameter values ​​of the corresponding factors; Multiply the normalized parameter value of the factor by the corresponding parameter weight coefficient to obtain the state value of the corresponding factor; Accumulate the status values ​​of all factors to obtain the status value of the corresponding transaction sample data; Based on the preset state value interval that the state value falls into, the corresponding power trading optimization decision is matched.

8. A power trading optimization decision system based on multi-objective parameter weights, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a power transaction optimization decision method program based on multi-objective parameter weights, and the power transaction optimization decision method program based on multi-objective parameter weights implements the following steps when executed by the processor: Obtain transaction data from the electricity market; Preprocess the transaction data of the power market to obtain transaction sample data; The transaction sample data is sent to the preset power market environment model to obtain the current power market transaction decision.

9. The power transaction optimization decision system based on multi-objective parameter weights according to claim 8 is characterized in that: The steps of constructing the preset power market environment model specifically include: Divide the preset initialization model into a target network module, a main network module and an agent module; Obtain historical transaction data of the electricity market; Preprocess the historical transaction data of the electricity market to obtain training samples; Send the training samples to the target network module and the main network module to obtain the loss function value; If the loss function value is less than or equal to the preset first threshold, the training ends and the corresponding parameter weight coefficient is extracted; The parameter weight coefficients are sent to the agent module for storage.

10. The power transaction optimization decision system based on multi-objective parameter weights according to claim 9 is characterized in that: Also includes: If the loss function is greater than the preset first threshold, the gradient of the main network parameters is determined according to the loss function, and the parameters of the main network are updated based on the gradient descent method, and the parameters of the main network are set to θ Q , whose formula is where θ ′ Q represents the adjustment parameters of the main network, k is the learning rate, Represents the gradient of the loss function L with respect to the main network parameters; the parameters of the main network are replaced by the adjustment parameters of the main network.

Images