Power data pricing method and system based on reinforcement learning

Abstract

The invention discloses an electric power data pricing method and system based on reinforcement learning, and the method comprises the steps: carrying out the preprocessing and dividing of a multivariable time series data set, training a random forest model based on a training set, carrying out the prediction through a test set, evaluating the random forest model through an MSE mean square error, and obtaining the pricing result of the random forest model. The optimal random forest model is obtained, and then the importance sequence of all the variables is obtained; a feature weighted pricing strategy based on date grouping is constructed; based on the day number difference between the current date and the reference date, constructing a discount coefficient, and further obtaining a preliminary pricing strategy; and based on the preliminary pricing strategy, optimizing the pricing strategy by using a Q learning algorithm until the Q value table is converged to a set range, and obtaining the optimal power grid data pricing. According to the method, the relevance between the features is analyzed, and the order of feature importance is determined; in combination with calculation of weight distribution and time-sensitive discounts of different features, data pricing can dynamically adapt to market requirements.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning and relates to a power data pricing method and system based on reinforcement learning. Background Art

[0002] In recent years, the power market has gradually transformed into a competitive market structure. Establishing a reasonable power data pricing mechanism can not only accurately reflect the supply and demand relationship and actual value of data, but also play an incentive role in the circulation and legal use of data. However, the existing power data pricing methods are usually based on static models or simple linear models. The value of power data is affected by many factors, such as the timeliness, accuracy, and completeness of data. However, it is difficult for existing pricing methods to fully consider these factors. Usually, there is a lack of dynamic adjustment mechanism for the real-time value of data, which makes it difficult for data pricing to reflect the changes in actual market demand and data quality. At the same time, power data pricing methods often rely on the subjective judgment of data providers or data markets, resulting in large differences in pricing for the same type of data in different scenarios. In addition, different market participants have different evaluation criteria for data value, which makes it difficult to quantify and compare the actual value of data, increasing the uncertainty and complexity in data transactions. Moreover, traditional power data pricing methods often fail to fully consider the balance of market supply and demand and the distribution of benefits between data providers and consumers in their design. For example, existing methods usually lack effective incentives for data providers, resulting in low willingness to share data and failure to fully utilize data resources. At the same time, the market mechanism has failed to fully consider the diverse needs and willingness to pay of data consumers, resulting in insufficient market activity and inefficient data transactions. Summary of the invention

[0003] The purpose of the present invention is to solve the problems in the prior art of lack of dynamic adjustment mechanism for power data pricing, inconsistent data value assessment standards, and imperfect market mechanisms and incentives, and to provide a power data pricing method and system based on reinforcement learning.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] The power data pricing method based on reinforcement learning includes:

[0006] Collect users' electricity consumption data, construct a multivariate time series data set, and preprocess the multivariate time series data set;

[0007] Divide the preprocessed multivariate time series data set into training set and test set;

[0008] The random forest model is trained based on the training set, predicted through the test set, and evaluated with the mean square error (MSE) until the optimal random forest model is obtained, and then the importance ranking of each variable is obtained;

[0009] Based on the importance of each variable, a feature-weighted pricing strategy based on date grouping is constructed;

[0010] Considering the changes in time and market demand, a discount factor is constructed based on the difference in the number of days between the current date and the reference date;

[0011] Obtain a preliminary pricing strategy based on the feature-weighted pricing strategy, discount factor, and the amount of data on the current date;

[0012] Based on the preliminary pricing strategy, the pricing strategy is optimized using the Q learning algorithm until the Q value table converges to the set range and the optimal grid pricing is obtained.

[0013] A further improvement of the present invention is:

[0014] Furthermore, the multivariate time series dataset is preprocessed, specifically: the multivariate time series dataset is saved, and the columns of the format are used as features; the rows of the format are used as the eigenvalues ​​of the features; the missing values ​​and outliers in the multivariate time series dataset are uniformly treated as missing values, and the missing values ​​are filled by the mean; the correlation between the variables is analyzed based on the Pearson correlation coefficient, and based on the size of the Pearson correlation coefficient between the variables, the variables that are highly correlated with other variables are used as target variables, and the other variables are used as feature vectors.

[0015] Furthermore, the random forest model is trained based on the training set, predicted through the test set, and evaluated with the mean square error (MSE) until the optimal random forest model is obtained, and then the importance ranking of each variable is obtained, specifically:

[0016] The training set includes: feature data X_train and target variable y_train; the test set includes: feature data X_test and target variable y_test;

[0017] Train the random forest model based on the feature data X_train and target variable y_train in the training set;

[0018] The trained random forest model predicts the feature data X_test of the test set and obtains the predicted value y_pred;

[0019] Calculate the MSE mean square error between the true value y_test and the predicted value y_pred of the test set; when the MSE mean square error is less than the preset value, the optimized random forest model is obtained;

[0020] During the optimization process of the random forest model, the importance of each variable is obtained based on the built-in function feature_importances, and the importance of each variable is ranked.

[0021] Furthermore, based on the importance of each variable, a feature-weighted pricing strategy based on date grouping is constructed, specifically:

[0022]

[0023] Among them, P i Initial pricing for data; a i is the feature importance calculated based on the random forest model; a0 is a constant; Z i is a characteristic variable;

[0024] Taking into account the changes in time and market demand, a discount factor is constructed based on the difference in days between the current date and the reference date, specifically:

[0025]

[0026] The reference date is a pre-set time.

[0027] Furthermore, based on the feature weighted pricing strategy, discount coefficient, and the amount of data on the current date, a preliminary pricing strategy is obtained, specifically:

[0028]

[0029] The data volume for the current date is obtained by grouping the data by date, obtaining data for several days, and creating a list to record the data price for each day.

[0030] Furthermore, based on the preliminary pricing strategy, the Q learning algorithm is used to optimize the pricing strategy until the Q value table converges to the set range, and the optimal grid pricing is obtained, which is:

[0031] (1) Set the parameters of the Q learning algorithm and initialize the Q value table, adjusted_prices list, and rewards_per_episode list;

[0032] Among them, the adjusted_prices list is used to save the adjusted prices in each round; the rewards_per_episode list is used to save the total rewards of each round; the initial value of the Q value table is a random number between -1 and 1;

[0033] (2) For each training round, the starting state s is initialized to the current first price and converted to the corresponding state through the state conversion function, and the total reward is 0;

[0034] (3) For each time step, i.e., each state, in the current state s, the ε-greedy strategy is used, and the decay mechanism is adopted to calculate ε, and an action is selected based on ε and the action selection function: if a random number generated by the action selection function is less than ε, an action is randomly selected; if the random number is not less than ε, the action with the largest Q value in the Q value table in the current state is selected a = argmaxaQ(s,a);

[0035] (4) If the action value is 0, the price is adjusted to 1.01 times the current price; when the action value is 1, the price is adjusted to 0.99 times the current price; other values ​​are to keep the current price unchanged;

[0036] (5) The agent performs action a, obtains an immediate reward r from the environment and considers the upper limit of the reward to be 0.05, applies the reward decay mechanism and adds it to the total reward; obtains s′ as the next price and converts it into the corresponding state;

[0037] (6) Update Q(s,a): Q(s,a)←Q(s,a)+α·[r+γ·maxa′Q(s′,a′)-Q(s,a)], if it is not the last time step, add the Q value of the next state; otherwise only the reward;

[0038] (7) Update the current price to the adjusted price, update the current state to the next state, save the adjusted price to the adjusted_prices list; determine whether the termination condition is met. If the new state s′ is the termination state, end the current round, add the total reward total_reward of the current round to the rewards_per_episode list, and print the reward information of the current round; otherwise, return to step (3);

[0039] (8) Repeat steps (2) to (7) until the termination condition is met or the maximum number of training rounds is reached.

[0040] Furthermore, the state conversion function is used to convert the price into the corresponding state, specifically: the price is normalized to between 0 and 1, and the normalized price represents the position of the price relative to the entire price range; then it is mapped to the state space, and the discrete state is obtained after rounding;

[0041] The attenuation mechanism is used to calculate ε, specifically:

[0042] epsilon=max(epsilon_min,epsilon_start*pow(decay_rate,episode))

[0043] Among them, the minimum value epsilon_min is set to 0.01, the decay rate decay_rate is set to 0.995; the initial value epsilon_start is set to 1.

[0044] Furthermore, the actions include: price increase, decrease and remain unchanged; the reward decay mechanism is that the influence of the reward gradually decreases over time; the decay factor in the reward decay mechanism is calculated based on the current time step and the maximum time step, and when the two are close, the decay factor tends to 0; the greater the difference, the closer it is to 1; at the same time, the decay factor will not be negative;

[0045] The action selection function is as follows: if a generated random number is less than ε, the agent enters the exploration mode and randomly selects an action to enable the agent to explore unknown high-return areas that may exist in the environment; if the random number is not less than ε, the agent enters the utilization mode and selects the action with the largest Q value in the Q value table in the current state, so that the agent uses the learned knowledge to select the action that is currently considered the best; when the action value is 0, it indicates an increase, and the price is adjusted to 1.01 times the current price; when the action value is 1, it indicates a decrease, and the price is adjusted to 0.99 times the current price; other values ​​are to keep the current price unchanged.

[0046] Furthermore, the immediate reward is that if the price increases compared to the previous state, the reward is equal to the percentage of the price change, but does not exceed the preset maximum value of 0.05; if the price decreases compared to the previous state, the reward is the negative value of half the percentage of the price change.

[0047] The power data pricing system based on reinforcement learning includes:

[0048] A preprocessing module, wherein the preprocessing module collects the user's electricity consumption data, constructs a multivariate time series data set, and preprocesses the multivariate time series data set;

[0049] A partitioning module, wherein the partitioning module partitions the preprocessed multivariate time series data set to obtain a training set and a test set;

[0050] A training module, wherein the training module trains the random forest model based on the training set, makes predictions through the test set, and evaluates the random forest model using the mean square error (MSE) until the optimal random forest model is obtained, thereby obtaining the importance ranking of each variable;

[0051] A first building module, wherein the first building module builds a feature weighted pricing strategy based on date grouping based on the importance of each variable;

[0052] a second building module, wherein the second building module constructs a discount coefficient based on the difference in days between the current date and the reference date, taking into account changes in time and market demand;

[0053] An acquisition module, wherein the acquisition module acquires a preliminary pricing strategy based on a feature-weighted pricing strategy, a discount coefficient, and a data volume on a current date;

[0054] The optimization module optimizes the pricing strategy based on the preliminary pricing strategy by using the Q learning algorithm until the Q value table converges to a set range, thereby obtaining the optimal grid pricing.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] The present invention uses a random forest algorithm to analyze the importance of features in power data, and analyzes the correlation between features and determines the order of feature importance; combined with the weight allocation of different features and the calculation of time sensitivity discounts, data pricing can be more reasonable and dynamically adapted to market demand. At the same time, the Q learning algorithm continuously adjusts the price strategy to maximize the cumulative reward, thereby achieving dynamic price adjustment. The model can optimize the price of power data based on historical data and environmental feedback, solving the problem that traditional power data pricing methods cannot respond to market changes in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 It is a flow chart of the power data pricing method based on reinforcement learning of the present invention;

[0059] Figure 2 It is a structural schematic diagram of the power data pricing system based on reinforcement learning of the present invention;

[0060] Figure 3 (a) is a schematic diagram of missing values ​​in a multivariate time series dataset;

[0061] Figure 3 (b) is a schematic diagram of outliers in a multivariate time series dataset

[0062] Figure 4 Heat map for Pearson correlation coefficient correlation analysis;

[0063] Figure 5 Flowchart for calculating feature importance;

[0064] Figure 6 This is the overall structure diagram of the preliminary pricing model;

[0065] Figure 7 The overall framework diagram for dynamically adjusting the price model;

[0066] Figure 8 Flowchart of dynamically adjusting prices for the Q-learning algorithm. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0068] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0070] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0071] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0072] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0073] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0074] See also Figure 1 The present invention discloses a power data pricing method based on reinforcement learning, comprising:

[0075] S101: Collecting user electricity consumption data, constructing a multivariate time series data set, and preprocessing the multivariate time series data set;

[0076] The multivariate time series data set is saved, and the columns of the format are features; the rows of the format are the eigenvalues ​​of the features; the missing values ​​and outliers in the multivariate time series data set are uniformly treated as missing values, and the missing values ​​are filled by the mean; the correlation between the variables is analyzed based on the Pearson correlation coefficient, and based on the size of the Pearson correlation coefficient between the variables, the variables that are highly correlated with other variables are used as target variables, and the other variables are feature vectors.

[0077] S102: Divide the preprocessed multivariate time series data set into a training set and a test set;

[0078] S103: Train the random forest model based on the training set, make predictions through the test set, and evaluate the random forest model using the mean square error (MSE) until the optimal random forest model is obtained, and then obtain the importance ranking of each variable;

[0079] The training set includes: feature data X_train and target variable y_train; the test set includes: feature data X_test and target variable y_test;

[0080] Train the random forest model based on the feature data X_train and target variable y_train in the training set;

[0081] The trained random forest model predicts the feature data X_test of the test set and obtains the predicted value y_pred;

[0082] Calculate the MSE mean square error between the true value y_test and the predicted value y_pred of the test set; when the MSE mean square error is less than the preset value, the optimized random forest model is obtained;

[0083] During the optimization process of the random forest model, the importance of each variable is obtained based on the built-in function feature_importances, and the importance of each variable is ranked.

[0084] S104: Based on the importance of each variable, construct a feature-weighted pricing strategy based on date grouping;

[0085] P i =a0+∑a i *Z i

[0086] Among them, P i Initial pricing for data; a i is the feature importance calculated based on the random forest model; a0 is a constant; Z i is a characteristic variable;

[0087] S105: Considering the changes in time and market demand, a discount factor is constructed based on the difference in the number of days between the current date and the reference date;

[0088]

[0089] The reference date is a pre-set time.

[0090] S106: Obtain a preliminary pricing strategy based on the feature weighted pricing strategy, the discount coefficient, and the data volume on the current date;

[0091]

[0092] The data volume for the current date is obtained by grouping the data by date, obtaining data for several days, and creating a list to record the data price for each day.

[0093] S107: Based on the preliminary pricing strategy, the pricing strategy is optimized using a Q learning algorithm until the Q value table converges to a set range, thereby obtaining the optimal grid pricing.

[0094] (1) Set the parameters of the Q learning algorithm and initialize the Q value table, adjusted_prices list, and rewards_per_episode list;

[0095] Among them, the adjusted_prices list is used to save the adjusted prices in each round; the rewards_per_episode list is used to save the total rewards of each round; the initial value of the Q value table is a random number between -1 and 1;

[0096] (2) For each training round, the starting state s is initialized to the current first price and converted to the corresponding state through the state conversion function, and the total reward is 0;

[0097] (3) For each time step, i.e., each state, in the current state s, the ε-greedy strategy is used, and the decay mechanism is adopted to calculate ε, and an action is selected based on ε and the action selection function: if a random number generated by the action selection function is less than ε, an action is randomly selected; if the random number is not less than ε, the action with the largest Q value in the Q value table in the current state is selected a = argmaxaQ(s,a);

[0098] (4) If the action value is 0, the price is adjusted to 1.01 times the current price; when the action value is 1, the price is adjusted to 0.99 times the current price; other values ​​are to keep the current price unchanged;

[0099] (5) The agent performs action a, obtains an immediate reward r from the environment and considers the upper limit of the reward to be 0.05, applies the reward decay mechanism and adds it to the total reward; obtains s′ as the next price and converts it into the corresponding state;

[0100] (6) Update Q(s,a): Q(s,a)←Q(s,a)+α·[r+γ·maxa′Q(s′,a′)-Q(s,a)], if it is not the last time step, add the Q value of the next state; otherwise only the reward;

[0101] (7) Update the current price to the adjusted price, update the current state to the next state, save the adjusted price to the adjusted_prices list; determine whether the termination condition is met. If the new state s′ is the termination state, end the current round, add the total reward total_reward of the current round to the rewards_per_episode list, and print the reward information of the current round; otherwise, return to step (3);

[0102] (8) Repeat steps (2) to (7) until the termination condition is met or the maximum number of training rounds is reached.

[0103] The state conversion function is used to convert the price into the corresponding state. Specifically, the price is normalized to between 0 and 1. The normalized price represents the price's position relative to the entire price range. Then it is mapped to the state space and rounded to get the discrete state.

[0104] The attenuation mechanism is used to calculate ε, specifically:

[0105] epsilon=max(epsilon_min,epsilon_start*pow(decay_rate,episode));

[0106] Among them, the minimum value epsilon_min is set to 0.01, the decay rate decay_rate is set to 0.995; the initial value epsilon_start is set to 1.

[0107] Actions include: price increase, decrease and remain unchanged; the reward decay mechanism is that the influence of the reward gradually decreases over time; the decay factor in the reward decay mechanism is calculated based on the current time step and the maximum time step. When the two are close, the decay factor tends to 0; the greater the difference, the closer it is to 1; at the same time, the decay factor will not be negative;

[0108] The action selection function is as follows: if a generated random number is less than ε, the agent enters the exploration mode and randomly selects an action to enable the agent to explore unknown high-return areas that may exist in the environment; if the random number is not less than ε, the agent enters the utilization mode and selects the action with the largest Q value in the Q value table in the current state, so that the agent uses the learned knowledge to select the action that is currently considered the best; when the action value is 0, it indicates an increase, and the price is adjusted to 1.01 times the current price; when the action value is 1, it indicates a decrease, and the price is adjusted to 0.99 times the current price; other values ​​are to keep the current price unchanged.

[0109] The immediate reward is that if the price increases compared to the previous state, the reward is equal to the percentage of the price change, but does not exceed the preset maximum value of 0.05; if the price decreases compared to the previous state, the reward is a negative value of half the percentage of the price change.

[0110] See also Figure 2 The present invention discloses a power data pricing system based on reinforcement learning, comprising:

[0111] A preprocessing module, wherein the preprocessing module collects the user's electricity consumption data, constructs a multivariate time series data set, and preprocesses the multivariate time series data set;

[0112] A partitioning module, wherein the partitioning module partitions the preprocessed multivariate time series data set to obtain a training set and a test set;

[0113] A training module, wherein the training module trains the random forest model based on the training set, makes predictions through the test set, and evaluates the random forest model using the mean square error (MSE) until the optimal random forest model is obtained, thereby obtaining the importance ranking of each variable;

[0114] A first building module, wherein the first building module builds a feature weighted pricing strategy based on date grouping based on the importance of each variable;

[0115] a second building module, wherein the second building module constructs a discount coefficient based on the difference in days between the current date and the reference date, taking into account changes in time and market demand;

[0116] An acquisition module, wherein the acquisition module acquires a preliminary pricing strategy based on a feature-weighted pricing strategy, a discount coefficient, and a data volume on a current date;

[0117] The optimization module optimizes the pricing strategy based on the preliminary pricing strategy by using the Q learning algorithm until the Q value table converges to a set range, thereby obtaining the optimal grid pricing.

[0118] Example:

[0119] The invention discloses a power data pricing method based on reinforcement learning, comprising: using random forest to perform feature importance analysis, a power data pricing model based on a Q learning algorithm, and price adjustment based on the Q learning algorithm.

[0120] 1. Use random forest to perform feature importance analysis, specifically:

[0121] The open source data set used in the present invention comes from the machine learning repository of the University of California, Irvine. The name of the open source data set is Individual household electric power consumption, hereinafter referred to as the data set. The data set is a multivariate time series data set, which collects the electricity consumption of a family in Paris, France for nearly four years (December 2006 to November 2010), with a sampling period of 1 minute. The data set has a total of 2075259 rows and 8 columns. Considering the complexity of the model and the calculation speed, the data from December 16, 2006 to September 29, 2007 are used in the subsequent modeling, with a total of 412911 items.

[0122] The data set has eight variables: Global_active_power is the active power per minute, Global_reactive_power is the reactive power per minute, Voltage is the average voltage per minute, Sub_metering_1, Sub_metering_2, and Sub_metering_3 are the active power of the kitchen, laundry room, air conditioner, and electric water heater respectively, and Global_intensity is the current intensity per minute. The date and time are used as row indexes.

[0123] Among them, the kitchen is considered as a whole, the laundry room is considered as a whole, and the air conditioner and electric water heater are considered as a whole. The three variables Sub_metering_1, Sub_metering_2, and Sub_metering_3 correspond to the actual power consumed by these three parts per minute. The eighth variable is date and time; the eigenvalue is seven.

[0124] The following is the process of calculating feature importance using the feature_importances_ attribute of random forest:

[0125] 1) Data preparation. Save the dataset as a csv file, with one column for one feature and one row for one observation. Use pd.read_csv in the python panda library to read the data and convert it into Datafrane format.

[0126] 2) Data preprocessing. By checking the outliers and missing values ​​in the data, such as Figure 3 (a) and Figure 3 (b); Convert them all into missing values ​​and use the average value of each feature as fill-in data. There are 7 features that can actually be used as features. Since serialized data already contains the order, the date and time should be removed. Therefore, the data that can be used is a multivariate sequence consisting of seven variables.

[0127] The Pearson correlation coefficient is used to analyze the correlation between variables. The coefficient value is between -1 and 1, where 1 indicates a perfect positive correlation and -1 indicates a perfect negative correlation, which is sufficient to describe the linear correlation between two variables. Figure 4 It can be concluded that Global_active_power has a high correlation with multiple variables, is highly correlated with Global_intensity, and also has a high correlation with Sub_metering_1, Sub_metering_2, and Sub_metering_3, such as Figure 4As shown in the figure, Global_active_power is determined as the target variable of the data set, and the rest are feature variables.

[0128] 3) Divide the data set into a training set and a test set. Set the ratio of the training set to the test set to 7:3.

[0129] 4) Initialize the random forest model. Use the RandomForestRegressor class of Sklearn to initialize the random forest regression model; set the model parameters and the number of decision trees n_estimator to 150. The more decision trees there are, the better the performance of the model will be, but this will slow down the training speed. Through continuous search for parameters and practical experience analysis, the best effect is achieved when the parameter value is between 100 and 1000.

[0130] 5) Use the training set and target variable to train the model, use the test set to make predictions, and evaluate the model with MSE. Train the random forest model: Use the training set and target variable to train. The random forest algorithm learns the pattern of the data by building multiple decision trees. In the construction of each tree, the model randomly selects samples and features to improve the generalization ability of the model.

[0131] 6) Feature importance calculation: use the built-in function feature_importances in the random forest model and sort it for subsequent result analysis. The flowchart for calculating feature importance is as follows: Figure 5 As shown in the figure, the features included in the dataset are: Global_active_power, Global_reactive_power, Voltage, Sub_metering_1, Sub_metering_2, Sub_metering_3, and Global_intensity; because date and time are excluded, these seven variables constitute the feature set.

[0132] The importance score of each feature is automatically calculated based on the feature_importances_ attribute in the random forest model and normalized to ensure that the sum of the importance of all features is 1. By sorting these scores, the features that have the greatest impact on the target variable can be identified.

[0133] 2. Power Data Pricing Model Based on Q-Learning Algorithm

[0134] The present invention proposes a reasonable preliminary pricing model. Since the functional relationship between the characteristic variables and the final data pricing cannot be known in advance, the characteristic price model function is selected as the preset preliminary pricing model. The linear function is the main form. The main steps are as follows:

[0135] 1) Data preprocessing, using random forest to perform feature importance analysis. The feature importance calculated by the random forest model is the weight a i Directly affects the initial pricing P i These weights reflect the relative importance of each feature in the pricing decision. For example, if a feature has a high importance score, then when calculating the preliminary pricing, the weight of the feature will also be relatively high, thus affecting the final pricing result.

[0136] 2) Considering the fact that pricing is usually an overall concept rather than a single feature, and pricing for each feature is not a common practice in pricing models, the data is grouped by date to obtain 287 days of data, and a list is created to record the data price for each day.

[0137] 3) Based on the importance of each variable, a feature-weighted pricing strategy based on date grouping is constructed;

[0138] P i =a0+∑a i *Z i

[0139] Among them, P i Initial pricing for data; a i is the feature importance calculated based on the random forest model; a0 is a constant; Z i is a characteristic variable;

[0140] 4) Time-sensitive discount mechanism. Taking into account the changes in time and market demand, the reference date is selected as January 1, 2024. By calculating the difference in days between the current date and the reference date, a discount coefficient is constructed in the following form:

[0141]

[0142] The reference date is a pre-set time. The discount coefficient increases gradually as the number of days from the reference date increases; in order to ensure the rationality of the discount, a discount of at least 10% is set.

[0143] 5) Based on the feature-weighted pricing strategy, discount factor, and the amount of data on the current date, obtain a preliminary pricing strategy:

[0144]

[0145] The data volume under the current date is to group the data by date, get the data of several days, and create a list to record the data price of each day. Figure 6 shown.

[0146] 3. Price adjustment based on Q-learning algorithm

[0147] In the process of dynamically adjusting prices using the Q-learning algorithm, several key factors should be considered to ensure the practicality of the strategy. The prices determined for the preliminary pricing model show an increasing trend, which is contrary to the real market situation, because prices cannot continue to rise without considering the fact that prices should fluctuate. Therefore, the model needs to simulate the complexity of price fluctuations in the real world, including rising, falling or unchanged prices.

[0148] To implement this strategy, the Q learning algorithm is applied to price adjustment, so that the model can learn the optimal pricing behavior based on historical data. Through continuous iteration and optimization, the model makes adaptive adjustments accordingly. To ensure the stability and reliability of the model, a convergence criterion for reward value iteration is set, that is, when the reward value gradually tends to be smooth during the iteration process and the fluctuation range does not exceed the preset interval value, the model can be considered to have converged. This convergence indicates that the model has found a relatively stable pricing strategy. The overall framework of the model is as follows: Figure 7 shown.

[0149] The parameter settings and function functions of the model are as follows:

[0150] 1) Parameter setting of the model. The price of each day is a state, that is, there are 287 states; three actions are defined, namely, price increase, decrease and unchanged; the learning rate α is 0.01, which is the result of grid search for parameters; the discount factor γ is 0.95; the number of training rounds is 2500.

[0151] 2) Initialize the Q value table. The size of the Q value table is a two-dimensional array of the number of states * the number of actions, and the initial value is a random number between -1 and 1.

[0152] 3) ε exponential decay mechanism. The initial value epsilon_start is set to 1, which means that the agent mainly explores in the early stage of training, and makes more use of previous knowledge in the later stage of training. The minimum value epsilon_min is set to 0.01, and the decay rate decay_rate is set to 0.995. Specifically, the update formula of the ε exponential decay mechanism is customized as epsilon = max(epsilon_min, epsilon_start*pow(decay_rate, episode));

[0153] 4) Discretize the state function. In order to more easily apply the table-based Q learning algorithm, the price should be mapped to the state space. The specific approach is: first normalize the price to between 0 and 1. The normalized price represents the position of the price relative to the entire price range; then map it to the state space and get the discrete state after rounding;

[0154] 5) Reward function. The percentage of price change is taken into account, and rewards are given based on this change, which is designed to encourage price increases. If the price increases compared to the previous state, the reward is equal to the percentage of price change, but does not exceed a preset maximum value of 0.05, which can prevent excessive rewards from affecting the learning process; if the price decreases compared to the previous state, the reward is a negative value of half the percentage of price change, indicating that price decreases will be punished, but the degree is smaller than the reward when the price increases. In addition, the maximum value of the cumulative reward is limited to 14.35.

[0155] 6) Reward decay mechanism. The purpose of defining this mechanism is to gradually reduce the impact of rewards over time. This decay method helps to encourage exploration in the early stages of training, and even small rewards will be amplified; in the later stages of training, due to decay, even large rewards will not have much impact on updates, thereby encouraging the agent to make more use of known optimal behaviors and balance current immediate rewards with long-term rewards. The decay factor is calculated based on the current time step and the maximum time step. When the two are close, the decay factor tends to 0; the greater the difference, the closer it is to 1; at the same time, the decay factor will not be negative.

[0156] 7) Action selection function. If the generated random number is less than ε, the agent enters the exploration mode and randomly selects an action to explore the unknown high-return areas that may exist in the environment; if the random number is not less than ε, the agent enters the utilization mode and selects the action with the largest Q value in the Q value table in the current state, so that the agent uses the knowledge it has learned to select the best action currently considered. When the action value is 0, it means an increase, and the price is adjusted to 1.01 times the current price; when the action value is 1, it means a decrease, and the price is adjusted to 0.99 times the current price; other values ​​are to keep the current price unchanged.

[0157] The steps for the Q learning algorithm to dynamically adjust prices are as follows:

[0158] 1) Initialize the Q value table. The initial value is a random number between -1 and 1. The size of the Q value table is the number of states * the number of actions, that is, 287 * 3. Set the learning rate α = 0.01, the discount factor γ = 0.95, and the maximum number of training rounds to 25000; initialize the list adjusted_prices to save the adjusted prices in each round; initialize the list rewards_per_episode to save the total rewards for each round.

[0159] 2) For each training round, the starting state s is initialized to the current first price and converted to the corresponding state through the state conversion function, and the total reward is 0.

[0160] 3) For each time step, that is, each state, in the current state s, use the ε-greedy strategy and adopt the attenuation mechanism to calculate ε, and select an action based on ε and the action selection function: if a random number generated by the action selection function is less than ε, then select an action randomly; if the random number is not less than ε, then select the action with the largest Q value in the Q value table in the current state a = argmaxaQ(s,a).

[0161] 4) If the action value selected is 0, the price is adjusted to 1.01 times the current price; when the action value is 1, the price is adjusted to 0.99 times the current price; other values ​​keep the current price unchanged.

[0162] 5) The agent performs action a, obtains an immediate reward r from the environment and considers the upper limit of the reward to be 0.05, applies the reward decay mechanism and adds it to the total reward; obtains s′ as the next price and converts it into the corresponding state.

[0163] 6) Update Q(s,a): Q(s,a)←Q(s,a)+α·[r+γ·maxa′Q(s′,a′)-Q(s,a)], if it is not the last time step, add the Q value of the next state; otherwise only the reward.

[0164] 7) Update the current price to the adjusted price, update the current state to the next state, save the adjusted price to the adjusted_prices list; determine whether the termination condition is met. If the new state s′ is the termination state, end the current round, add the total reward total_reward of the current round to the rewards_per_episode list, and print the reward information of the current round. Otherwise, return to step (3).

[0165] 8) Repeat steps 2)-7) until the termination condition is met or the maximum number of training rounds is reached. The flowchart of the Q learning algorithm dynamically adjusting the price is as follows: Figure 8 shown.

[0166] A terminal device is provided in one embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0167] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0168] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0169] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0170] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0171] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0172] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification or equivalent substitution made within the spirit and principle of the present invention is acceptable.

Claims

1. A power data pricing method based on reinforcement learning, characterized in that: include: Collect users' electricity consumption data, construct a multivariate time series data set, and preprocess the multivariate time series data set; Divide the preprocessed multivariate time series data set into training set and test set; The random forest model is trained based on the training set, predicted through the test set, and evaluated with the mean square error (MSE) until the optimal random forest model is obtained, and then the importance ranking of each variable is obtained; Based on the importance of each variable, a feature-weighted pricing strategy based on date grouping is constructed; Considering the changes in time and market demand, a discount factor is constructed based on the difference in the number of days between the current date and the reference date; Obtain a preliminary pricing strategy based on the feature-weighted pricing strategy, discount factor, and the amount of data on the current date; Based on the preliminary pricing strategy, the pricing strategy is optimized using the Q learning algorithm until the Q value table converges to the set range and the optimal grid pricing is obtained.

2. The power data pricing method based on reinforcement learning according to claim 1 is characterized in that: The preprocessing of the multivariate time series data set is specifically as follows: saving the multivariate time series data set, with the columns of the format as features; The format line is the characteristic value of the feature; The missing values ​​and outliers in the multivariate time series data set are uniformly treated as missing values, and the missing values ​​are filled by the mean; the correlation between the variables is analyzed based on the Pearson correlation coefficient, and based on the size of the Pearson correlation coefficient between the variables, the variables that are highly correlated with other variables are used as target variables, and the other variables are feature vectors.

3. The power data pricing method based on reinforcement learning according to claim 2 is characterized in that: The random forest model is trained based on the training set, predicted through the test set, and evaluated with the MSE mean square error until the optimal random forest model is obtained, and then the importance ranking of each variable is obtained, specifically: Training set Includes: feature data X_train and target variable y_train; test set includes: feature data X_test and target variable y_test; Train the random forest model based on the feature data X_train and target variable y_train in the training set; The trained random forest model predicts the feature data X_test of the test set and obtains the predicted value y_pred; Calculate the MSE mean square error between the true value y_test and the predicted value y_pred of the test set; when the MSE mean square error is less than the preset value, the optimized random forest model is obtained; During the optimization process of the random forest model, the importance of each variable is obtained based on the built-in function feature_importances, and the importance of each variable is ranked.

4. The power data pricing method based on reinforcement learning according to claim 3 is characterized in that: Based on the importance of each variable, a feature weighted pricing strategy based on date grouping is constructed, specifically: P i =a0+∑a i *WITH i Among them, P i Initial pricing for data; a i is the feature importance calculated based on the random forest model; a0 is a constant; Z i is a characteristic variable; Taking into account the changes in time and market demand, a discount factor is constructed based on the difference in days between the current date and the reference date, specifically: The reference date is a pre-set time.

5. The power data pricing method based on reinforcement learning according to claim 4 is characterized in that: The preliminary pricing strategy is obtained based on the feature weighted pricing strategy, discount coefficient and data volume on the current date, specifically: The data volume for the current date is obtained by grouping the data by date, obtaining data for several days, and creating a list to record the data price for each day.

6. The power data pricing method based on reinforcement learning according to claim 5 is characterized in that: Based on the preliminary pricing strategy, the pricing strategy is optimized using the Q learning algorithm until the Q value table converges to the set range, and the optimal grid pricing is obtained, which is specifically: (1) Set the parameters of the Q learning algorithm and initialize the Q value table, adjusted_prices list, and rewards_per_episode list; Among them, the adjusted_prices list is used to save the adjusted prices in each round; the rewards_per_episode list is used to save the total rewards of each round; the initial value of the Q value table is a random number between -1 and 1; (2) For each training round, the starting state s is initialized to the current first price and converted to the corresponding state through the state conversion function, and the total reward is 0; (3) For each time step, i.e., each state, in the current state s, the ε-greedy strategy is used, and the decay mechanism is adopted to calculate ε, and an action is selected based on ε and the action selection function: if a random number generated by the action selection function is less than ε, an action is randomly selected; if the random number is not less than ε, the action with the largest Q value in the Q value table in the current state is selected a = argmaxaQ(s,a); (4) If the action value is 0, the price is adjusted to 1.01 times the current price; when the action value is 1, the price is adjusted to 0.99 times the current price; other values ​​are to keep the current price unchanged; (5) The agent performs action a, obtains an immediate reward r from the environment and considers the upper limit of the reward to be 0.05, applies the reward decay mechanism and adds it to the total reward; obtains s′ as the next price and converts it into the corresponding state; (6) Update Q(s,a): Q(s,a)←Q(s,a)+α·[r+γ·maxa′Q(s′,a′)-Q(s,a)], if it is not the last time step, add the Q value of the next state; otherwise only the reward; (7) Update the current price to the adjusted price, update the current state to the next state, save the adjusted price to the adjusted_prices list; determine whether the termination condition is met. If the new state s′ is the termination state, end the current round, add the total reward total_reward of the current round to the rewards_per_episode list, and print the reward information of the current round; otherwise, return to step (3); (8) Repeat steps (2) to (7) until the termination condition is met or the maximum number of training rounds is reached.

7. The power data pricing method based on reinforcement learning according to claim 6 is characterized in that: The conversion into the corresponding state through the state conversion function is specifically: normalizing the price to between 0 and 1, the normalized price represents the position of the price relative to the entire price range; then mapping it to the state space, and rounding it to obtain a discrete state; The attenuation mechanism is used to calculate ε, specifically: epsilon = max(epsilon_min,epsilon_start*pow(decay_rate,episode)) where The minimum value epsilon_min is set to 0.01, the decay rate decay_rate is set to 0.995, and the initial value epsilon_start is set to 1.

8. The power data pricing method based on reinforcement learning according to claim 7 is characterized in that: The actions include: price increase, decrease and remain unchanged; the reward decay mechanism is that the influence of the reward gradually decreases over time; the decay factor in the reward decay mechanism is calculated based on the current time step and the maximum time step, and when the two are close, the decay factor tends to 0; the greater the difference, the closer it is to 1; at the same time, the decay factor will not be negative; The action selection function is as follows: if a generated random number is less than ε, the agent enters the exploration mode and randomly selects an action to enable the agent to explore unknown high-return areas that may exist in the environment; if the random number is not less than ε, the agent enters the utilization mode and selects the action with the largest Q value in the Q value table in the current state, so that the agent uses the learned knowledge to select the action that is currently considered the best; when the action value is 0, it indicates an increase, and the price is adjusted to 1.01 times the current price; when the action value is 1, it indicates a decrease, and the price is adjusted to 0.99 times the current price; other values ​​are to keep the current price unchanged.

9. The power data pricing method based on reinforcement learning according to claim 8 is characterized in that: The instant reward is that if the price increases compared to the previous state, the reward is equal to the percentage of the price change, but does not exceed the preset maximum value of 0.05; if the price decreases compared to the previous state, the reward is a negative value of half the price change percentage.

10. The power data pricing system based on reinforcement learning is characterized by: include: A preprocessing module, wherein the preprocessing module collects the user's electricity consumption data, constructs a multivariate time series data set, and preprocesses the multivariate time series data set; A partitioning module, wherein the partitioning module partitions the preprocessed multivariate time series data set to obtain a training set and a test set; A training module, wherein the training module trains the random forest model based on the training set, makes predictions through the test set, and evaluates the random forest model using the mean square error (MSE) until the optimal random forest model is obtained, thereby obtaining the importance ranking of each variable; A first building module, wherein the first building module builds a feature weighted pricing strategy based on date grouping based on the importance of each variable; a second building module, wherein the second building module constructs a discount coefficient based on the difference in days between the current date and the reference date, taking into account changes in time and market demand; An acquisition module, wherein the acquisition module acquires a preliminary pricing strategy based on a feature-weighted pricing strategy, a discount coefficient, and a data volume on a current date; The optimization module optimizes the pricing strategy based on the preliminary pricing strategy by using the Q learning algorithm until the Q value table converges to a set range, thereby obtaining the optimal grid pricing.

Images