Data product dynamic pricing method and system based on deep reinforcement learning

Through the dynamic pricing method of data products based on deep reinforcement learning, the exploration strategy and experience playback mechanism are improved, and the problem of insufficient adaptability of data trading platforms in market changes and personalized needs is solved, and the intelligent pricing of data products is realized, which has improved the maximum The stability of the earnings and pricing strategy.

CN119090551APending Publication Date: 2024-12-06KUNMING UNIV OF SCI & TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411109305.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing data trading platforms lack flexibility and adaptability in dynamic pricing of data products, cannot effectively respond to market changes and customer personalized needs, and fail to make full use of deep reinforcement learning technology.

Method used

Adopting a dynamic pricing method for data products based on deep reinforcement learning, through improving exploration strategies and introducing priority experience playback mechanisms, building a Deep Q-Network algorithm model, optimizing pricing strategies and experience pool management, realizing the intelligence of data products and Automated pricing.

Benefits of technology

It significantly improves the maximum return of data products, improves the adaptability and flexibility of pricing strategies, and increases the maximum return percentage compared with optimal static pricing. The improved exploration strategy and experience playback mechanism accelerate the convergence speed of model and stabilizes the growth of returns.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090551A_ABST
    Figure CN119090551A_ABST
Patent Text Reader

Abstract

The invention relates to a data product dynamic pricing method and system based on deep reinforcement learning, and belongs to the field of data product dynamic pricing, data transaction platforms and computers. The method comprises the following steps: initializing a pricing environment; constructing a data product dynamic pricing model based on deep reinforcement learning; training a data product dynamic pricing model, optimizing parameters and updating the weight of the model; performing dynamic pricing on the data product by using the trained model; in the four steps, a dynamic pricing system is formed through an environment processing module, a model construction module, a training module and a pricing output module. According to the method, the problem of low efficiency of balance between exploration and utilization and experience sample utilization in a deep reinforcement learning Deep Q-Network algorithm is solved, the dynamic pricing performance of the data product is improved, and maximization of accumulated income of the data product is indirectly assisted.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a method and system for dynamic pricing of data products based on deep reinforcement learning, and belongs to the fields of dynamic pricing of data products, data trading platforms and computer technology. Background Art

[0002] An efficient data trading market is an important way to eliminate data silos and promote data trading and circulation. As a form of data factor market, the data trading platform is the core carrier for connecting data supply and demand and improving the efficiency of data factor market allocation, providing trust and support for data buyers and sellers. Compared with traditional commodities, data products have the characteristics of timeliness, two-way value uncertainty, and marginal cost tending to zero. Traditional dynamic pricing is difficult to meet the development of the data trading market.

[0003] Existing data trading platforms mainly use fixed pricing and negotiated pricing for data products. Fixed pricing has lower management costs, but lacks flexibility and poor adaptability to market changes. The negotiated pricing is matched by a third-party data trading platform and negotiated by the buyer and seller. After repeated bargaining and quoting, an agreement is reached. It can be flexibly adjusted to achieve better personalized services, but the complexity and management costs are high. Dynamic pricing can dynamically adjust prices according to the ever-changing external environment to maximize the cumulative benefits of data product sellers. Reinforcement learning aims to maximize cumulative rewards, learns from observations and is used to solve Markov decision processes. It is currently one of the most widely used technologies for solving dynamic pricing. Deep reinforcement learning will integrate the perception ability of deep learning and the decision-making ability of reinforcement learning, so that it can be expanded to problems with high-dimensional states and action spaces, and make better strategies in more complex environments.

[0004] Chinese patent application No. 202311242636.2 discloses an AI-based data asset dynamic pricing and quotation method, which constructs a joint dynamic pricing model through integrated learning, dynamically adjusts the model structure and model parameters of the joint dynamic pricing model, and calculates real-time quotations based on the updated joint dynamic pricing model; sends real-time quotations to customers, and tracks and evaluates quotation information; the present invention considers the impact of customer personalized needs, uses incremental learning to update the model according to the newly added customer data, and achieves the purpose of personalized pricing. This dynamic pricing method does not involve the application of deep reinforcement learning to dynamic pricing of data products.

[0005] Chinese patent application No. 202310725823.X discloses a data commodity dynamic pricing method based on differentiated privacy leakage risk. This dynamic pricing method aims at the dynamic change of privacy leakage risk in the process of data sales, designs a dynamic assessment method for personal data privacy leakage risk, analyzes factors such as privacy associations between data, combines information entropy, conditional entropy, mutual information and other theories, dynamically assesses privacy risk, uses the probability of privacy risk leakage as price compensation, and designs differentiated dynamic pricing, thereby providing a differentiated data commodity dynamic pricing mechanism based on privacy leakage risk. This dynamic pricing method does not involve the application of deep reinforcement learning to the dynamic pricing of data products, nor does it perform dynamic pricing according to changes in the pricing environment. Summary of the invention

[0006] In view of the problems existing in the dynamic pricing of data products in the above-mentioned prior art, the present invention provides a method and system for dynamic pricing of data products based on deep reinforcement learning. The present invention focuses on solving the problems of "exploration" and "utilization" balance and low efficiency of experience sample utilization in the deep reinforcement learning Deep Q-Network algorithm by improving the exploration strategy and introducing an experience replay mechanism with priority. The present invention can indirectly assist in maximizing the cumulative benefits of data products.

[0007] The technical solution of the present invention is: in a first aspect, the present invention provides a data product dynamic pricing method based on deep reinforcement learning, the method comprising:

[0008] Step 1: Initialize the pricing environment, including initializing system parameters, network parameters, experience pool, pricing environment, and market environment;

[0009] Step 2: Construct a dynamic pricing model for data products based on deep reinforcement learning. This model includes building an exploration strategy selection pricing module, an experience pool maintenance module, an evaluation network module, a target network module, and a model update module. The exploration strategy selection pricing module adopts an improved ε-greedy exploration strategy with an annealing mechanism to select pricing actions. The experience pool maintenance module uses an experience replay mechanism with priority to improve the experience replay mechanism in the experience pool.

[0010] Step 3: Use the initialized environment to train the constructed data product dynamic pricing model. Through coordination and data sharing between modules, back propagate the gradient of loss to network weights, optimize parameters and update the model weights.

[0011] Step 4: Use the trained model to dynamically price data products.

[0012] Furthermore, in step 1, initializing the pricing environment includes:

[0013] Initialize system parameters. System parameters are the basic configurations that affect the model behavior, including initializing the total sales period, non-homogeneous Poisson distribution parameters of customer page views, customer retention price distribution, service rates and fixed fees, and price adjustment parameters;

[0014] Initialize network parameters. Network parameters are the parameters of the evaluation network and the target network, including the discount factor γ and the learning rate r. l , the number of rounds C to update the target network parameters, the number of samples at a time batch_size;

[0015] Experience pool initialization: clear the experience in the experience pool;

[0016] Initialize the pricing environment, including setting the pricing action set A, generating the price change list and profit statement;

[0017] Market environment initialization includes simulating the arrival and behavior of customers in the actual market and generating a customer arrival list based on the defined arrival rate.

[0018] Furthermore, in step 2, the specific steps are as follows:

[0019] Step 2.1: Construct an exploration strategy selection pricing module, improve the ε-greedy exploration strategy, and use the improved ε-greedy exploration strategy with annealing mechanism to select pricing actions;

[0020] Step 2.2: Construct an experience pool maintenance module to store, sample, and remove experience samples obtained after the agent interacts with the environment. This includes improving the experience playback mechanism in the experience pool, calculating the priority of each experience sample based on the temporal difference error, and giving different experience samples different probabilities of being sampled.

[0021] Step 2.3: Build an evaluation network module to estimate and update the Q value of each state-action pair, i.e., the Q estimate, expressed as Q(s,a|θ);

[0022] Step 2.4: Construct the target network module. The structure of this module is the same as that of the evaluation network module. It is used to calculate the Q target value, which is expressed as Responsible for ensuring the stability of training;

[0023] Step 2.5: Construct a model updating module, which is used to update and optimize model parameters.

[0024] Furthermore, in step 2.1, exploring the strategy selection pricing module includes:

[0025] Step 2.1.1: Initialize the exploration strategy selection module, including the upper bound ε of the exploration rate ε start , the lower bound of the exploration rate εend , current iteration number S done , exploration rate decay rate r d Initialize;

[0026] Step 2.1.2, generate a random number η∈[0,1] for each iteration;

[0027] Step 2.1.3: Calculate the exploration rate ε value at the current i-th iteration for each iteration. The calculation formula is as follows:

[0028]

[0029] Step 2.1.4: Compare the random number η with ε at each iteration i Value size, when ε i <η, randomly select a pricing action for exploration, otherwise select the optimal pricing action predicted by the current network;

[0030] Step 2.1.5: Interact the pricing action with the environment to obtain a set of experiences.

[0031] Furthermore, in step 2.2, the experience pool maintenance module includes:

[0032] Step 2.2.1, experience pool maintenance module initialization, including buffer capacity N, control priority sampling parameter α, used to calculate the dynamic weight starting value β0 during sampling, control the number of frames of dynamic weight decay β frame , the sampling probability p of each set of experience;

[0033] Step 2.2.2, add experience samples: add a set of experiences gained from interacting with the environment to the buffer;

[0034] Step 2.2.3, sampling experience samples: The time series difference error δ is used as the priority metric. The greater the priority, the greater the probability of being sampled. The calculation formula of the time series difference error is:

[0035]

[0036] Where: s′ and a′ are the next state and next action of the current state s and current action a respectively, A represents the pricing action set, r represents the reward, and γ represents the discount factor;

[0037] Then the sampling probability p(i) of the i-th empirical sample is:

[0038]

[0039] Among them: α is the priority index, which takes values ​​[0,1]. The larger the priority index, the greater the impact on the probability of being sampled; ζ is a very small positive number to ensure that the priority of all experience samples is not zero;

[0040] Step 2.2.4: Correct the sample bias caused by the priority experience playback and introduce the dynamic weight w i To correct the sample deviation, the calculation formula is:

[0041]

[0042] Where: D is the sample set in the experience pool; β is the dynamic weight decay coefficient, which starts from the weight starting value β0 and follows the weight decay frame number β as the iterative training proceeds. frame The control gradually increases to 1, and the calculation formula is:

[0043]

[0044] Furthermore, in step 2.3, the evaluation network module includes:

[0045] Step 2.3.1, network parameter initialization, including: the size of the input state space state_size, the size of the output action space action_size and the hidden layer size hidden_size, the default value of which is 128;

[0046] Step 2.3.2, define the hierarchy of the neural network, use a linear layer to map the state space of the input layer to the hidden layer, use three hidden layers, use the ReLU activation function after each hidden layer, and the last layer maps the hidden layer to the action space of the output layer;

[0047] Step 2.3.3: Define the forward propagation function, which is used to calculate and return the Q estimate after receiving the state through the hierarchy of the neural network.

[0048] Furthermore, in step 2.4, the target network module, which has the same neural network structure as the evaluation network module, is used to calculate the Q target value; for each sampled experience, the maximum Q value of the next state predicted by the target network is selected, multiplied by the discount factor γ and added to the current reward r, as the target Q value, i.e., Q target , the calculation formula is:

[0049]

[0050] Furthermore, in step 2.5, the model updating module includes:

[0051] Step 2.5.1, check the size of the experience pool: if the number of experience samples in the experience pool is less than the defined sampling quantity batch_size, return directly without updating;

[0052] Step 2.5.2, sample and unpack experience samples: sample batch_size experience samples from the experience pool, obtain transitions, indexes, weights, and unpack transitions. Each unpack is a set of experience samples, including state s, pricing action a, next state s′, reward r;

[0053] Step 2.5.3, process non-terminal states: create a mask to identify the next state s′ as a non-terminal state, and extract all non-terminal states from it to form a non-terminal state set non_final_S′;

[0054] Step 2.5.4, Packaging: Pack the decompressed state s, pricing action a, next state s′, and reward r into a state set S, pricing action set A, next state set S′, and reward set R;

[0055] Step 2.5.5. Calculate the current Q estimate: Use the evaluation network to calculate the Q value corresponding to the current state set S, and use the gather function in PyTorch to select the Q value corresponding to the pricing action set A to obtain the Q estimate.

[0056] Step 2.5.6, calculate the expected Q value: use the target network to calculate the maximum Q value in non_final_S′, and assign it to the position corresponding to the Q′ value of the next state-action pair, and finally calculate the expected state-action value The calculation formula is:

[0057]

[0058] Step 2.5.7, calculate the loss: The loss is the square of the difference between the current Q value and the expected Q value, multiplied by the weight weights, and a very small constant is added to avoid the priority of 0 in the next step;

[0059] Step 2.5.8. Update the priority of the experience sample: Use the calculated loss to update the priority in the experience pool;

[0060] Step 2.5.9, backpropagation and optimization: Set the optimizer gradient to zero, and backpropagate the gradient. To prevent the gradient from exploding, limit the gradient to the range of [0,1], and call the optimizer in PyTorch to update the evaluation network.

[0061] Furthermore, the step 3 includes training the constructed data product dynamic pricing model for each iteration number, including a round-by-round training phase, a recording and updating phase, and a current training progress visualization phase. The specific steps are as follows:

[0062] Step 3.1: Train the constructed data product dynamic pricing model for each iteration, and perform operations for each time step under each iteration according to the defined number of iterations;

[0063] Step 3.2, record the accumulated rewards and price paths, and update the target network parameters every certain number of rounds C; C is the number of rounds for updating the target network parameters defined when the environment is initialized;

[0064] Step 3.3: Visualize the current training progress, that is, calculate the percentage of the number of training rounds to the total number of iterations during the training process;

[0065] In step 3.1, under the definition of the total number of iterations of training, the following steps are performed in sequence for each sales period within the total sales period under each number of iterations:

[0066] Step 3.1.1, initialize the environment state;

[0067] Step 3.1.2: Perform training for each sales period;

[0068] Step 3.1.3, determine whether the termination condition is met, that is, whether the predetermined total number of iterations is reached. If the predetermined total number of iterations is not reached, steps 3.1.1 and 3.1.2 will continue to be executed in a loop;

[0069] In step 3.2, the recording and updating phase includes:

[0070] Step 3.2.1. Add the accumulated rewards to the reward list reward_trace;

[0071] Step 3.2.2, add the pricing path to the pricing list p_trace;

[0072] Step 3.2.3: Every C rounds, update the parameters of the target network according to the parameters of the evaluation network.

[0073] In step 4, the trained model is used to dynamically price the data product, and a cumulative reward curve is drawn after the model training is completed to visualize the pricing benefits brought by the pricing strategy made at each iteration, and output the optimal pricing corresponding to each price adjustment stage.

[0074] In a second aspect, the present invention also provides a data product dynamic pricing system based on deep reinforcement learning, which includes a module for executing the method in the first aspect above.

[0075] The beneficial effects of the present invention are:

[0076] 1. In view of the characteristics of data products such as timeliness, two-way value uncertainty, and marginal cost tending to zero, the data trading platform is provided with a dynamic pricing method and system for data products. It can adjust product prices in real time according to changes in market demand and competitive environment, realize intelligent and automated dynamic pricing, and has strong adaptability and flexibility.

[0077] 2. The dynamic pricing model based on the Deep Q-Network algorithm established by the invention can significantly increase the maximum revenue increase percentage of data products. In the simulation experiment, compared with the optimal static pricing, the increase percentage is 6.4% to 15.37%. That is, under different hyperparameter combinations of the dynamic pricing model based on the Deep Q-Network algorithm, the dynamic pricing strategy formulated can increase the maximum revenue by 6.4% to 15.37%.

[0078] 3. The invention improves the ε-greedy exploration strategy on the basis of the dynamic pricing model based on the Deep Q-Network algorithm, and adopts the improved ε-greedy exploration strategy with annealing mechanism. The experimental results show that the improved exploration strategy can set the exploration rate more reasonably, so that it focuses more on "exploration" in the early stage and more on "utilization" in the later stage, and the benefits are more stable with the number of iterations.

[0079] 4. The invention improves the experience replay mechanism in the experience pool on the basis of the dynamic pricing model established based on the Deep Q-Network algorithm, and improves the experience replay mechanism by calculating the sampling probability using the time difference error. Experiments show that the improved experience replay mechanism can accelerate the model convergence speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is an architecture diagram of the data product dynamic pricing system based on deep reinforcement learning in the present invention;

[0081] Figure 2 This is a flowchart of a method for dynamic pricing of data products based on deep reinforcement learning in the present invention;

[0082] Figure 3 This is a graph showing the comparison results of different exploration rates of the ε-DQN algorithm in Experiment 2 of the embodiment of the present invention;

[0083] Figure 4 This is a graph showing the comparison results of different priority index experiments of the pri-DQN algorithm in Experiment 3 of the embodiment of the present invention;

[0084] Figure 5 This is a graph showing the experimental comparison results of different dynamic weight initial values ​​of the pri-DQN algorithm in Experiment 3 of the embodiment of the present invention;

[0085] Figure 6This is a graph showing the comparison results of the pri-DQN algorithm with different weight attenuation frames in Experiment 3 of the embodiment of the present invention;

[0086] Figure 7 This is a graph comparing the performance of different algorithms in Experiment 4 of the embodiment of the present invention. DETAILED DESCRIPTION

[0087] Example 1: Figure 1-Figure 7 As shown, a data product dynamic pricing method and system based on deep reinforcement learning includes:

[0088] Step 1: Initialize the pricing environment, including initializing system parameters, network parameters, experience pool, pricing environment, and market environment;

[0089] In some feasible implementations, initializing the pricing environment includes:

[0090] Initialize system parameters, including total sales period, non-homogeneous Poisson distribution parameters of customer page views, customer retention price distribution, service fee rate and fixed fee, and price adjustment parameters;

[0091] Initialize network parameters. Network parameters are the parameters of the evaluation network and the target network, including the discount factor γ and the learning rate r. l , the number of rounds C to update the target network parameters, the number of samples at a time batch_size;

[0092] Initialize the experience pool and clear the experience in the experience pool;

[0093] Initialize the pricing environment, including setting the pricing action set A, generating the price change list and profit statement;

[0094] Market environment initialization includes simulating the arrival and behavior of customers in the actual market, generating a customer arrival list based on the defined arrival rate, etc.

[0095] Step 2: Construct a dynamic pricing model for data products based on deep reinforcement learning. This model includes building an exploration strategy selection pricing module, an experience pool maintenance module, an evaluation network module, a target network module, and a model update module. The exploration strategy selection pricing module adopts an improved ε-greedy exploration strategy with an annealing mechanism to select pricing actions. The experience pool maintenance module uses an experience replay mechanism with priority to improve the experience replay mechanism in the experience pool.

[0096] In some feasible implementations, the specific steps of step 2 are as follows:

[0097] Step 2.1, construct an exploration strategy to select a pricing module, and use an improved ε-greedy strategy with an annealing mechanism to select pricing actions. The ε-greedy strategy randomly selects actions with a fixed probability of ε, and selects the action currently estimated to be optimal with a probability of 1-ε. The smaller the exploration rate ε value, the more it tends to "utilize" current experience, and the larger the ε value, the more it tends to "explore" unknown environments. The present invention innovatively improves the ε-greedy strategy, using the annealing mechanism to make the ε value change from large to small with the number of iterations, so that it can focus on "exploration" in the early stage to prevent falling into the local optimum, and fully "utilize" in the later stage to accelerate convergence.

[0098] Step 2.2, construct an experience pool maintenance module, which is used to store and sample the experience samples obtained after the intelligent agent interacts with the environment, including storing the state, action, reward, and next state of each interaction in the experience pool, sampling a batch of experience samples from the experience pool for model training, and removing old experience after the experience pool is full. Randomly sampling a batch of experience samples from the experience pool with equal probability will ignore the importance differences between different experience samples. Poor experience samples will affect the convergence speed of the algorithm. Therefore, different experience samples should be given different sampling probabilities according to their importance in model training to improve the convergence speed of the algorithm. The present invention improves the experience playback mechanism in the experience pool, calculates the priority of each experience sample based on the temporal difference error, and gives different experience samples different sampling probabilities.

[0099] Step 2.3: Build an evaluation network module to estimate and update the Q value of each state-action pair, which is the Q estimate, expressed as Q(s,a|θ).

[0100] Step 2.4: Build the target network module, use the fixed target network technology (Fix Target Network), and use the target network with the same structure as the evaluation network module to calculate the Q target value, which can be expressed as The parameters of the target network are copied periodically from the evaluation network to update the parameters of the target network, solving the problem of weak training stability. The structure of this module is the same as that of the evaluation network, and is responsible for ensuring the stability of training.

[0101] Step 2.5: Construct a model updating module, which is used to update and optimize model parameters.

[0102] In some feasible implementations, the step 2.1 includes:

[0103] Step 2.1.1: Initialize the exploration strategy selection module, including the upper bound of the exploration rate ε start , the lower bound of the exploration rate ε end , current iteration number S done , exploration rate decay rate r d Initialize;

[0104] Step 2.1.2, generate a random number η∈[0,1] for each iteration;

[0105] Step 2.1.3: Calculate the exploration rate ε value at the current i-th iteration for each iteration. The calculation formula is as follows:

[0106]

[0107] Step 2.1.4: Compare the random number η with ε at each iteration i Value size, when ε i <η, an action is randomly selected exploratoryly, that is, a~Uniform(A); otherwise, the optimal pricing action predicted by the current Q network is selected, that is, a=argmax a′ Q(s,a′).

[0108] Step 2.1.5: Interact pricing actions with the environment to obtain a set of experiences<s,a,r,s′> , respectively represent <current state, pricing action, current reward, next state>.

[0109] In some feasible implementations, the step 2.2 includes:

[0110] Step 2.2.1, experience pool maintenance module initialization, including buffer capacity N, control priority sampling parameter α, used to calculate the dynamic weight starting value β0 during sampling, control the number of frames of dynamic weight decay β frame , each group of empirical sampling probability p i ;

[0111] Step 2.2.2: Add experience samples to the experience gained from interacting with the environment<s,a,r,s′> Add to buffer;

[0112] Step 2.2.3, sample empirical samples, and use the time series difference error δ as the priority metric. The greater the priority, the greater the probability of being sampled. The calculation formula for the time series difference error is:

[0113]

[0114] Where: s′ and a′ are the next state and next action of the current state s and current action a respectively, A represents the pricing action set, r represents the reward, and γ represents the discount factor;

[0115] Then the sampling probability p(i) of the i-th empirical sample is:

[0116]

[0117] Where: α is the priority index, which takes values ​​of [0,1]. The priority index is used to control priority sampling. The larger the index, the greater the impact on the sampling probability. ζ is a very small positive number to ensure that the priority of all experience samples is not zero.

[0118] Step 2.2.4: Correct the sample bias caused by the priority experience playback and introduce the dynamic weight w i To correct the sample deviation, the calculation formula is:

[0119]

[0120] Where: D is the sample set in the experience pool; β is the dynamic weight decay coefficient, which starts from the weight starting value β0 and follows the weight decay frame number β as the iterative training proceeds. frame The control gradually increases to 1, and the calculation formula is:

[0121]

[0122] In some feasible implementations, the step 2.3 includes:

[0123] Step 2.3.1, network parameter initialization, including: the size of the input state space state_size, the size of the output action space action_size and the hidden layer size hidden_size, the default value of which is 128;

[0124] Step 2.3.2, define the hierarchy of the neural network, use a linear layer to map the state space of the input layer to the hidden layer, use three hidden layers, use the ReLU activation function after each hidden layer, and the last layer maps the hidden layer to the action space of the output layer;

[0125] Step 2.3.3, define the forward propagation function, receive the state, calculate and return the Q estimate through the hierarchy of the neural network. The Q value is calculated through the Bellman expectation equation, which is used to estimate the value under a given state and action. The specific formula is as follows:

[0126] Q π (s,a)=R(s,a)+γ∑ s' P(s'|s,a)∑ a' π(a'|s')Q π (s',a') (9)

[0127] Among them, Q π(s,a) is the action value of taking action a in state s and following policy π; R(s,a) is the immediate reward of taking action a in state s; γ is a discount factor used to weigh the importance of future rewards; P(s'|s,a) is the probability of transitioning from state s to the next state s' by taking action a; π(a|s) is the probability of transitioning from state s to the next state s' by taking action a; Q π (s,a) is the action value of taking action a' in the new state s' and following the policy π.

[0128] The Bellman expectation equation is used to update the Q value. The update formula is as follows:

[0129] Q(s,a)←Q(s,a)+α[r+γmax a' Q(s′,a′)-Q(s,a)]

[0130] Among them, α is the learning rate, which controls the speed of Q value update.

[0131] In some feasible implementations, the target network module in step 2.4 has the same neural network structure as the evaluation network module, and is used to calculate the Q target value. For each sampled experience, the maximum Q value of the next state predicted by the target network is selected, multiplied by the discount factor γ and added to the current reward r as the target Q value, i.e., Q target , the calculation formula is:

[0132]

[0133] In some feasible implementations, the model updating module in step 2.5 includes:

[0134] Step 2.5.1, check the size of the experience pool: if the number of experience samples in the experience pool is less than the defined sampling quantity batch_size, return directly without updating;

[0135] Step 2.5.2, sample and unpack experience samples: sample batch_size experience samples from the experience pool, obtain transitions, indexes, weights, and unpack transitions. Each unpack is a set of experience samples, including state s, pricing action a, next state s′, reward r;

[0136] Step 2.5.3, process non-terminal states: create a mask to identify the next state s′ as a non-terminal state, and extract all non-terminal states from it to form a non-terminal state set non_final_S′;

[0137] Step 2.5.4, Packaging: Pack the decompressed state s, pricing action a, next state s′, and reward r into a state set S, pricing action set A, next state set S′, and reward set R;

[0138] Step 2.5.5. Calculate the current Q estimate: Use the evaluation network to calculate the Q value corresponding to the current state set S, and use the gather function in PyTorch to select the Q value corresponding to the pricing action set A to obtain the Q estimate.

[0139] Step 2.5.6, calculate the expected Q value: use the target network to calculate the maximum Q value in non_final_S′, and assign it to the position corresponding to the Q′ value of the next state-action pair, and finally calculate the expected state-action value The calculation formula is:

[0140]

[0141] Step 2.5.7, calculate the loss: The loss is the square of the difference between the current Q value and the expected Q value, multiplied by the weights. The Q value gap between the evaluation network module and the target network module is calculated by minimizing the mean square error loss function, and the gradient descent method is used to iteratively return the gradient to the evaluation network to minimize the loss function. The loss function L(θ) calculation formula and the gradient descent method update formula are as follows:

[0142]

[0143] Among them, θ is the current parameter vector; η is the step size, which controls the magnitude of each update. is the gradient of the loss function L(θ) with respect to the parameter θ.

[0144] Step 2.5.8. Update the priority of the experience sample: Use the calculated loss to update the priority in the experience pool;

[0145] Step 2.5.9, backpropagation and optimization: Set the optimizer gradient to zero, and backpropagate the gradient. To prevent the gradient from exploding, limit the gradient to the range of [0,1], and call the optimizer in PyTorch to update the evaluation network.

[0146] In some feasible implementations, step 3 includes training the constructed data product dynamic pricing model for each iteration number, including a round-by-round training phase, a recording and updating phase, and a current training progress visualization phase. The specific steps are as follows:

[0147] Step 3.1: Train the model for each iteration, and perform operations for each time step at each iteration according to the defined number of iterations;

[0148] Step 3.2, record the accumulated rewards and price paths, and update the target network parameters every certain number of rounds C (i.e., the number of rounds for updating the target network parameters defined when the environment is initialized);

[0149] Step 3.3: Visualize the current training progress, that is, calculate the percentage of the number of training rounds to the total number of iterations during the training process.

[0150] In some feasible implementations, in step 3.1, under the definition of the total number of iterations of training, the following steps are sequentially performed for each sales period within the total sales period under each number of iterations:

[0151] Step 3.1.1, initialize the environment state, define the total number of iterations, total sales period, price list, etc.;

[0152] Step 3.1.2: Perform the following steps for each sales period:

[0153] Step 3.1.2.1. Input the state s observed in the environment into the evaluation network and use the evaluation network to calculate the Q estimate;

[0154] Step 3.1.2.2, use the exploration strategy module to select pricing action a and execute it;

[0155] Step 3.1.2.3, interact with the environment, then receive feedback from the environment as the current reward, and update the current state s to the next state s′;

[0156] Step 3.1.2.4: Convert the conversion pair (i.e., the<s,a,r,s′> ) is stored in the experience pool, which is a set of experience samples;

[0157] Step 3.1.2.5, update the current state s to the next state s′, that is, advance the sales period by one period;

[0158] Step 3.1.2.6, update the evaluation network parameters;

[0159] Step 3.1.3: Determine whether the termination condition is met, that is, whether the predetermined total number of iterations is reached. If the predetermined total number of iterations is not reached, steps 3.1.1 and 3.1.2 will continue to be executed in a loop.

[0160] In some feasible implementations, the specific steps of step 3.2 are:

[0161] Step 3.2.1. Add the accumulated rewards to the reward list reward_trace;

[0162] Step 3.2.2, add the pricing path to the pricing list p_trace;

[0163] Step 3.2.3: Update the parameters of the target network according to the parameters of the evaluation network every C rounds.

[0164] In some feasible implementations, step 4 uses a trained model to dynamically price data products, mainly to draw a cumulative reward curve after the model training is completed, which is used to visualize the pricing benefits brought by the pricing strategy made at each iteration, and output the optimal pricing corresponding to each price adjustment stage.

[0165] An embodiment of the present invention provides a data product dynamic pricing system based on deep reinforcement learning, the system comprising:

[0166] Environment processing module: used to initialize the pricing environment, including initializing system parameters, network parameters, experience pool, pricing environment and market environment;

[0167] Model building module: used to build a dynamic pricing model for data products based on deep reinforcement learning;

[0168] Training module: used to train the constructed data product dynamic pricing model using the initialized environment. Through coordination and data sharing between modules, back-propagation is used to calculate the gradient of loss to network weights, optimize parameters and update model weights.

[0169] Pricing output module: used to dynamically price data products using the trained data product dynamic pricing model based on deep reinforcement learning, and output the dynamic pricing strategy for data products.

[0170] In a feasible implementation, the environment processing module is used for system parameter setting, model initialization, environment initialization and experience pool initialization. System parameter setting is to set the hyperparameters of the reinforcement learning algorithm such as learning rate, discount factor, etc. Model initialization is to initialize the parameters of the deep neural network model and create a neural network model. Environment initialization is to set the pricing environment and market environment status, including the total sales period, platform service fee rate, etc. Experience pool initialization is to create an experience pool D and set its capacity N for storing the experience of the agent interacting with the environment.

[0171] In a feasible implementation, the construction module is specifically used to construct an exploration strategy selection pricing module, an experience pool maintenance module for experience storage, sampling, and deletion, an evaluation network module for calculating Q estimation values, a target network module for calculating Q target values, and a model update module for updating the evaluation network.

[0172] It is used to build an exploration strategy selection pricing module, using the ε-greedy strategy with annealing mechanism to select pricing actions;

[0173] An experience pool maintenance module is used to build experience storage, sampling, and deletion, and an experience pool with a priority playback mechanism is used for maintenance;

[0174] Used to construct the evaluation network module and the target network module, both of which use the evaluation network and the target network with a three-layer neural network (MLP) structure for calculation;

[0175] The model update module used to update the evaluation network calculates the loss function by minimizing the mean square error and updates the evaluation network.

[0176] In order to illustrate the effect of the present invention, the following four experiments were performed:

[0177] Experiment 1: Verify the effectiveness of the unimproved exploration strategy and the unimproved priority experience replay mechanism (denoted as DQN), specifically by comparing with the optimal static pricing from the perspective of maximum cumulative benefits. The unimproved exploration strategy uses the ε-greedy strategy (i.e., randomly selects actions with a probability of a fixed ε exploration rate, and selects the action currently estimated to be the best with a probability of 1-ε) for pricing selection, and the unimproved priority experience replay mechanism randomly samples a batch of experience samples from the experience pool with equal probability during experience sampling. The performance and training stability of the algorithm used depend on the choice of hyperparameters. For any r l ∈{0.0001,0.0005,0.001}, γ∈{0.90,0.95,0.99}, ε∈{0.1,0.3,0.5,0.7,0.9}, a total of 3×3×5=45 hyperparameter combination experiments, the results are shown in Table 1: the maximum profit improvement percentages are all greater than 0, which proves that the present invention can effectively solve the dynamic pricing problem of data products, and dynamic pricing can improve the seller's maximum cumulative profit more than static pricing, and the improvement percentage can reach 6.4% to 15.37%.

[0178] Table 1 Experimental results of maximum benefit improvement percentage for different hyperparameter combinations

[0179]

[0180] Experiment 2: Verify the effectiveness of the improved exploration strategy (denoted as ε-DQN) in terms of stability performance. The exploration strategy refers to the ε-greedy strategy with an annealing mechanism. Starting from a larger ε value, the exploration rate decays exponentially with the number of iterations. Setting a reasonable exploration rate can better balance "exploration" and "exploitation". The experiment sets different exploration decay rate hyperparameters, namely r d =0.005, r d =0.0025, r d =0.002, comparing the average profit improvement percentage of the ε-DQN algorithm and the DQN algorithm with the number of iterations I. The experimental results are as follows Figure 3As shown, it can be seen that the ε-DQN algorithm has a more stable performance in terms of average revenue improvement percentage.

[0181] Experiment 3: Verify the effectiveness of the improved experience replay mechanism (referred to as pri-DQN) in terms of convergence speed. The experience in the experience pool is assigned different sampling probabilities based on the temporal difference error. The larger the temporal difference error, the more important the sample is for the model learning. The pri-DQN algorithm with improved experience replay takes different hyperparameters α, β0, β frame Conduct experiments;

[0182] When β0=0.3、β frame =500, set α=0.3, α=0.5, α=0.7 respectively, the results are as follows Figure 4 As shown;

[0183] When α=0.5、β frame =500, set β0=0.2, β0=0.3, β0=0.4 respectively, the results are as follows Figure 5 As shown;

[0184] Under the conditions of α=0.5 and β0=0.3, set β frame =200, β frame =500, β frame =800, the result is Figure 6 As shown;

[0185] It can be seen from Experiment 3 that pri-DQN has a faster convergence speed than DQN, so the improvement of the priority experience replay mechanism is effective in accelerating the convergence speed.

[0186] Experiment 4: Verify the effectiveness of the method that simultaneously improves the exploration strategy and experience replay mechanism (referred to as ε-pri-DQN). The above-mentioned DQN method, ε-DQN method, pri-DQN and ε-pri-DQN method are experimentally compared. The results are as follows: Figure 7 As shown;

[0187] It can be seen that the ε-pri-DQN method with improved exploration strategy and experience replay mechanism can optimally solve the dynamic pricing strategy of data products.

[0188] The above experiments have proven the effectiveness and superiority of the present invention.

[0189] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A data product dynamic pricing method based on deep reinforcement learning, characterized by: The method comprises: Step 1: Initialize the pricing environment, including initializing system parameters, network parameters, experience pool, pricing environment, and market environment; Step 2: Construct a dynamic pricing model for data products based on deep reinforcement learning. This model includes building an exploration strategy selection pricing module, an experience pool maintenance module, an evaluation network module, a target network module, and a model update module. The exploration strategy selection pricing module adopts an improved ε-greedy exploration strategy with an annealing mechanism to select pricing actions. The experience pool maintenance module uses an experience replay mechanism with priority to improve the experience replay mechanism in the experience pool. Step 3: Use the initialized environment to train the constructed data product dynamic pricing model. Through coordination and data sharing between modules, back propagate the gradient of loss to network weights, optimize parameters and update the model weights. Step 4: Use the trained model to dynamically price data products.

2. The method and system for dynamic pricing of data products based on deep reinforcement learning according to claim 1, characterized in that: In step 1, initializing the pricing environment includes: Initialize system parameters, including total sales period, non-homogeneous Poisson distribution parameters of customer page views, customer retention price distribution, service fee rate and fixed fee, and price adjustment parameters; Initialize network parameters. Network parameters are the parameters of the evaluation network and the target network, including the discount factor γ and the learning rate r. l , the number of rounds C to update the target network parameters, the number of samples at a time batch_size; Experience pool initialization: clear the experience in the experience pool; Initialize the pricing environment, including setting the pricing action set A, generating the price change list and profit statement; Market environment initialization includes simulating the arrival and behavior of customers in the actual market and generating a customer arrival list based on the defined arrival rate.

3. The method and system for dynamic pricing of data products based on deep reinforcement learning according to claim 1, characterized in that: In step 2, the specific steps are as follows: Step 2.1: Construct an exploration strategy selection pricing module, improve the ε-greedy exploration strategy, and use the improved ε-greedy exploration strategy with annealing mechanism to select pricing actions; Step 2.2: Build an experience pool maintenance module to store, sample, and remove experience samples obtained after the agent interacts with the environment; This includes improving the experience replay mechanism in the experience pool, calculating the priority of each experience sample based on the time series difference error, and giving different experience samples different probabilities of being sampled; Step 2.3: Build an evaluation network module to estimate and update the Q value of each state-action pair, i.e., the Q estimate, expressed as Q(s,a|θ); Step 2.4: Construct the target network module. The structure of this module is the same as that of the evaluation network module. It is used to calculate the Q target value, which is expressed as Responsible for ensuring the stability of training; Step 2.5: Construct a model updating module, which is used to update and optimize model parameters.

4. The method and system for dynamic pricing of data products based on deep reinforcement learning according to claim 3, characterized in that: In step 2.1, the exploration strategy selection pricing module includes: Step 2.1.1: Initialize the exploration strategy selection module, including the upper bound ε of the exploration rate ε start , the lower bound of the exploration rate ε end , current iteration number S done , exploration rate decay rate r d Initialize; Step 2.1.2, generate a random number η∈[0,1] for each iteration; Step 2.1.3: Calculate the exploration rate ε value at the current i-th iteration for each iteration. The calculation formula is as follows: Step 2.1.4: Compare the random number η with ε at each iteration i Value size, when ε i <η, randomly select a pricing action for exploration, otherwise select the optimal pricing action predicted by the current network; Step 2.1.5: Interact the pricing action with the environment to obtain a set of experiences.

5. The method and system for dynamic pricing of data products based on deep reinforcement learning according to claim 3, characterized in that: In step 2.2, the experience pool maintenance module includes: Step 2.2.1, experience pool maintenance module initialization, including buffer capacity N, control priority sampling parameter α, used to calculate the dynamic weight starting value β0 during sampling, control the number of frames of dynamic weight decay β frame , the sampling probability p of each set of experience; Step 2.2.2, add experience samples: add a set of experiences gained from interacting with the environment to the buffer; Step 2.2.3, sampling experience samples: The time series difference error δ is used as the priority metric. The greater the priority, the greater the probability of being sampled. The calculation formula of the time series difference error is: Among them: s′ and a′ are the next state and next action of the current state s and the current action a respectively, A represents the pricing action set, r represents the reward, and γ represents the discount factor; Then the sampling probability p(i) of the i-th empirical sample is: Among them: α is the priority index, which takes values ​​[0,1]. The larger the priority index, the greater the impact on the probability of being sampled; ζ is a very small positive number to ensure that the priority of all experience samples is not zero; Step 2.2.4: Correct the sample bias caused by the priority experience playback and introduce the dynamic weight w i To correct the sample deviation, the calculation formula is: Where: D is the sample set in the experience pool; β is the dynamic weight decay coefficient, which starts from the weight starting value β0 and follows the weight decay frame number β as the iterative training proceeds. frame The control gradually increases to 1, and the calculation formula is:

6. The method and system for dynamic pricing of data products based on deep reinforcement learning according to claim 1, characterized in that: In step 2.3, the evaluation network module includes: Step 2.3.1, network parameter initialization, including: the size of the input state space state_size, the size of the output action space action_size and the hidden layer size hidden_size, the default value of which is 128; Step 2.3.2, define the hierarchy of the neural network, use a linear layer to map the state space of the input layer to the hidden layer, use three hidden layers, use the ReLU activation function after each hidden layer, and the last layer maps the hidden layer to the action space of the output layer; Step 2.3.3: Define the forward propagation function, which is used to calculate and return the Q estimate after receiving the state through the hierarchy of the neural network.

7. The method and system for dynamic pricing of data products based on deep reinforcement learning according to claim 1, characterized in that: In step 2.4, the target network module has the same neural network structure as the evaluation network module and is used to calculate the Q target value. For each sampled experience, the maximum Q value of the next state predicted by the target network is selected, multiplied by the discount factor γ and added to the current reward r as the target Q value, i.e., Q target , the calculation formula is:

8. The method and system for dynamic pricing of data products based on deep reinforcement learning according to claim 1, characterized in that: In step 2.5, the model updating module includes: Step 2.5.1, check the size of the experience pool: if the number of experience samples in the experience pool is less than the defined sampling quantity batch_size, return directly without updating; Step 2.5.2, sample and unpack experience samples: sample batch_size experience samples from the experience pool, obtain transitions, indexes, weights, and unpack transitions. Each unpack is a set of experience samples, including state s, pricing action a, next state s′, reward r; Step 2.5.3, process non-terminal states: create a mask to identify the next state s′ as a non-terminal state, and extract all non-terminal states from it to form a non-terminal state set non_final_S′; Step 2.5.4, Packaging: Pack the decompressed state s, pricing action a, next state s′, and reward r into a state set S, pricing action set A, next state set S′, and reward set R; Step 2.5.5, calculate the current Q estimate: use the evaluation network to calculate the Q value corresponding to the current state set S, and select the Q value corresponding to the pricing action set A through the gather function in PyTorch to obtain the Q estimate; Step 2.5.6, calculate the expected Q value: use the target network to calculate the maximum Q value in non_final_S′, and assign it to the position corresponding to the Q′ value of the next state-action pair, and finally calculate the expected state-action value The calculation formula is: Step 2.5.7, calculate the loss: The loss is the square of the difference between the current Q value and the expected Q value, multiplied by the weight weights, and a very small constant is added to avoid the priority of 0 in the next step; Step 2.5.

8. Update the priority of the experience sample: Use the calculated loss to update the priority in the experience pool; Step 2.5.9, backpropagation and optimization: Set the optimizer gradient to zero, and backpropagate the gradient. To prevent the gradient from exploding, limit the gradient to the range of [0,1], and call the optimizer in PyTorch to update the evaluation network.

9. The method and system for dynamic pricing of data products based on deep reinforcement learning according to claim 1, characterized in that: The step 3 includes training the constructed data product dynamic pricing model for each iteration number, including a round-by-round training phase, a recording and updating phase, and a current training progress visualization phase. The specific steps are as follows: Step 3.1: Train the constructed data product dynamic pricing model for each iteration, and perform operations for each time step under each iteration according to the defined number of iterations; Step 3.2, record the accumulated rewards and price paths, and update the target network parameters every certain number of rounds C; C is the number of rounds for updating the target network parameters defined when the environment is initialized; Step 3.3: Visualize the current training progress, that is, calculate the percentage of the number of training rounds to the total number of iterations during the training process; In step 3.1, under the definition of the total number of iterations of training, the following steps are performed in sequence for each sales period within the total sales period under each number of iterations: Step 3.1.1, initialize the environment state; Step 3.1.2: Perform training for each sales period; Step 3.1.3, determine whether the termination condition is met, that is, whether the predetermined total number of iterations is reached. If the predetermined total number of iterations is not reached, steps 3.1.1 and 3.1.2 will continue to be executed in a loop; In step 3.2, the recording and updating phase includes: Step 3.2.

1. Add the accumulated rewards to the reward list reward_trace; Step 3.2.2, add the pricing path to the pricing list p_trace; Step 3.2.3: Every C rounds, update the parameters of the target network according to the parameters of the evaluation network. In step 4, the trained model is used to dynamically price the data product, and a cumulative reward curve is drawn after the model training is completed to visualize the pricing benefits brought by the pricing strategy made at each iteration, and output the optimal pricing corresponding to each price adjustment stage.

10. A data product dynamic pricing system based on deep reinforcement learning, characterized by: The system comprises a module for executing the method according to any one of claims 1-9.

Citation Information

Patent Citations

  • Data commodity dynamic pricing method based on differentiated privacy disclosure risk

    CN116681480A

  • Data asset dynamic pricing and quoting method based on AI

    CN117291641A