Pricing method and device based on reinforcement learning model and electronic equipment

Abstract

This application relates to a pricing method, apparatus, and electronic device based on a reinforcement learning model. The method includes: upon receiving a request from a target user to access a target product, querying the recommended price of the target product for the target user in a Redis cache; acquiring the user profile features and historical behavior features of the target user, as well as the product features and temporal context features of the target product, and concatenating these features to construct a state vector for the target user; the target reinforcement learning model calculating the state-action value function value corresponding to each candidate discount action based on the state vector, and selecting the candidate discount action with the largest state-action value function value as the target pricing action; calculating a discount on the base price of the target product based on the target pricing action to obtain the corresponding recommended price, and returning the recommended price to be displayed to the target user. This application solves the technical problem of pricing strategies lacking adaptive optimization capabilities.

Description

Technical Field

[0001] This application relates to the field of dynamic pricing data processing technology, and in particular to a pricing method, apparatus, and electronic device based on a reinforcement learning model. Background Technology

[0002] In e-commerce platforms, content subscriptions, and membership services, product or service prices are typically differentiated based on membership level, historical purchase behavior, and promotional activities to improve conversion rates and user stickiness, thereby increasing total sales and overall profits. As business scales up, product variety expands, membership numbers continue to grow, and market competition and user needs change rapidly, the complexity and real-time requirements of pricing strategies increase significantly. Against this backdrop, traditional pricing methods are gradually revealing their limitations.

[0003] Existing membership pricing systems mainly include rule-based pricing, historical statistical indicator-based pricing, and strategy optimization methods relying on A / B testing. Rule-based methods typically involve manually configuring fixed discounts or tiered pricing rules, which have long update cycles and struggle to adapt to real-time market changes. Historical statistical methods often rely on metrics such as historical conversion rates and average order value for overall pricing, lacking detailed characterization of individual member behavior. A / B testing-driven strategy optimization depends on manual analysis and experimental adjustments, resulting in long optimization cycles and high resource consumption. All of these solutions generally lack intelligent decision-making mechanisms capable of continuous learning and adaptive optimization in dynamic environments, making it difficult to achieve long-term profit maximization. Summary of the Invention

[0004] This application provides a pricing method, apparatus, and electronic device based on a reinforcement learning model to address the technical problem of pricing strategies lacking adaptive optimization capabilities.

[0005] Firstly, this application provides a pricing method based on a reinforcement learning model, comprising: upon receiving an access request from a target user for a target product, querying the recommended price of the target product for the target user in a Redis cache; if the recommended price of the target product for the target user is not found in the Redis cache, acquiring the user profile features and historical behavior features of the target user, as well as the product features and temporal context features of the target product, and concatenating these features to construct a state vector of the target user; inputting the state vector into a target reinforcement learning model, so that the target reinforcement learning model calculates the state-action value function value corresponding to each candidate discount action based on the state vector, and selects the candidate discount action with the largest state-action value function value as the target pricing action under the condition of satisfying a preset price safety boundary constraint; calculating a discount on the base price of the target product based on the target pricing action to obtain the corresponding recommended price, and returning the recommended price to be displayed to the target user, while writing the recommended price into the Redis cache and setting a preset validity period.

[0006] Secondly, this application provides a pricing device based on a reinforcement learning model, comprising: a query module, configured to query the recommended price of the target product for the target user in a Redis cache when a target user requests access to a target product; a first construction module, configured to, when the recommended price of the target product for the target user is not found in the Redis cache, acquire the user profile features and historical behavior features of the target user, as well as the product features and time context features of the target product, and concatenate the features to construct the state vector of the target user; a first calculation module, configured to input the state vector into a target reinforcement learning model, so that the target reinforcement learning model calculates the state-action value function value corresponding to each candidate discount action based on the state vector, and selects the candidate discount action with the largest state-action value function value as the target pricing action under the condition of satisfying a preset price safety boundary constraint; and a second calculation module, configured to perform a discount calculation on the base price of the target product based on the target pricing action, obtain the corresponding recommended price, return the recommended price to be displayed to the target user, and simultaneously write the recommended price into the Redis cache and set a preset validity period.

[0007] As an optional example, the first construction module includes: a first acquisition unit, used to acquire the user profile features of the target user, wherein the user profile features include membership level, account activity, registration duration, historical purchase frequency, average order value, and the time interval between the most recent purchase; a second acquisition unit, used to acquire the historical behavior features of the target user for the target product, wherein the historical behavior features include browsing count, add-to-cart count, historical conversion rate, and price sensitivity estimate; a third acquisition unit, used to acquire the product features and time context features of the target product, wherein the product features include real-time inventory quantity, inventory turnover rate, and historical conversion rate, and the time context features include current time period identifier, holiday identifier, and promotional activity identifier; and a construction unit, used to perform numerical encoding and normalization processing on the user profile features, the historical behavior features, the product features, and the time context features, and then perform vector concatenation to generate the state vector.

[0008] As an optional example, the above apparatus further includes: an acquisition module, used to acquire historical transaction data from a MySQL database before inputting the state vector into the target reinforcement learning model; a second construction module, used to construct a Markov decision model based on the historical transaction data, and define the state, action, reward, and state transition relationship of the Markov decision model, wherein the state is composed of user profile features, historical behavior features, product features, and time context features, the action includes multiple candidate discount actions, the reward is the revenue data generated under the corresponding discount action, and the state transition relationship is determined by the user's behavioral feedback under different discounts; a third construction module, used to construct a state-action value function based on the Markov decision model, and use a deep Q-network to approximate the state-action value function, wherein the state-action value function represents the long-term cumulative revenue corresponding to performing a candidate discount action in a given state; and a first training module, used to construct a training sample set containing state, action, reward, and state transition relationship using the historical transaction data, and perform offline training on the deep Q-network based on the training sample set to obtain the target reinforcement learning model.

[0009] As an optional example, the first training module includes a training unit, which is used to apply a preset price safety boundary constraint to the action space during offline training, and to randomly sample and batch update the training sample set to obtain the target reinforcement learning model.

[0010] As an optional example, the first calculation module includes: a calculation unit, used to input the state vector into the target reinforcement learning model, so that the target reinforcement learning model calculates the state-action value function value corresponding to the state vector under each candidate discount action through forward propagation; a judgment unit, used to determine the legality of each candidate discount action according to the preset price safety boundary constraint; and a selection unit, used to select the candidate discount action with the largest corresponding state-action value function value among the candidate discount actions that satisfy the preset price safety boundary constraint as the target pricing action.

[0011] As an optional example, the above-mentioned device further includes: a collection module, used to collect the target user's click behavior, purchase behavior, and churn behavior on the target product after returning the recommended price to the target user, and record the corresponding status, pricing action, and actual revenue, generate feedback data, and save the feedback data to a MySQL database; a writing module, used to write the feedback data into a training sample set to form an incremental training dataset; and a second training module, used to retrain the target reinforcement learning model using the incremental training dataset according to a preset update cycle, based on a preset online exploration ratio control strategy and an A / B testing mechanism.

[0012] As an optional example, the above apparatus further includes: a return module, configured to, after querying the recommended price of the target product for the target user in the Redis cache, return the corresponding recommended price to be displayed to the target user if the recommended price of the target product for the target user is found in the Redis cache.

[0013] Thirdly, this application provides a storage medium storing a computer program, wherein the computer program is executed by a processor to perform the aforementioned pricing method based on a reinforcement learning model.

[0014] Fourthly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described pricing method based on a reinforcement learning model through the computer program.

[0015] The technical solutions provided in this application have the following advantages compared with the prior art: This application employs the following approach: Upon receiving an access request from a target user for a target product, the application queries the Redis cache for the recommended price of the target product for the target user. If the recommended price is not found in the Redis cache, the application acquires the user profile features and historical behavior features of the target user, as well as the product features and temporal context features of the target product, and concatenates these features to construct the target user's state vector. This state vector is then input into a target reinforcement learning model, which calculates the state-action value function value for each candidate discount action based on the state vector. Under the condition of satisfying a preset price safety boundary constraint, the application selects the candidate discount action with the largest state-action value function value as the target pricing action. Finally, the application adjusts the base price of the target product according to the target pricing action. The method calculates discounts, obtains corresponding recommended prices, and returns these recommended prices to the target users. Simultaneously, it writes these recommended prices to the Redis cache with a preset validity period. This method prioritizes querying the cached recommended price upon receiving a target user's access request. If the cache misses, it constructs a state vector containing user profile features, historical behavior features, product features, and time context features. This vector is then input into a reinforcement learning model to calculate the state-action value function of each candidate discount action. Under the condition of satisfying price safety boundary constraints, the optimal discount is selected to generate the recommended price, and the result is written to the cache for subsequent reuse. This achieves personalized, dynamic, and risk-controllable pricing decisions, improves pricing efficiency and system response performance, and enhances conversion rates and long-term profitability. Ultimately, it solves the technical problem of pricing strategies lacking adaptive optimization capabilities. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0019] Figure 1This is a flowchart of an optional pricing method based on a reinforcement learning model according to an embodiment of this application; Figure 2 This is a system architecture diagram of an optional pricing method based on a reinforcement learning model according to an embodiment of this application; Figure 3 This is a flowchart illustrating the implementation of an optional pricing method based on a reinforcement learning model according to an embodiment of this application. Figure 4 This is a schematic diagram of an optional pricing device based on a reinforcement learning model according to an embodiment of this application; Figure 5 This is a schematic diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0022] According to a first aspect of the embodiments of this application, a pricing method based on a reinforcement learning model is provided, optionally, as follows: Figure 1 As shown, the above method includes: S102, Upon receiving a request from a target user to access a target product, query the Redis cache for the recommended price of the target product for the target user; S104. If the recommended price of the target product for the target user is not found in the Redis cache, obtain the user profile features and historical behavior features of the target user, as well as the product features and time context features of the target product, and concatenate the features to construct the state vector of the target user. S106, Input the state vector into the target reinforcement learning model so that the target reinforcement learning model can calculate the state-action value function value corresponding to each candidate discount action based on the state vector, and select the candidate discount action with the largest state-action value function value as the target pricing action under the condition of satisfying the preset price safety boundary constraint. S108 calculates the discount on the base price of the target product based on the target pricing action, obtains the corresponding recommended price, and returns the recommended price to be displayed to the target user. At the same time, the recommended price is written to the Redis cache and a preset validity period is set.

[0023] Optionally, this embodiment provides a pricing method based on a reinforcement learning model, applied to e-commerce platforms or membership service systems, to achieve personalized discount pricing for target users in a dynamic business environment. By introducing a reinforcement learning decision-making mechanism, user access behavior is combined with pricing strategy optimization, enabling real-time generation and continuous optimization of pricing results while ensuring price safety boundaries.

[0024] Specifically, when a request for access to a target product is received from a target user, the system first checks the Redis cache to see if a recommended price has been generated for that target user. If a corresponding recommended price exists in the cache and is still within a preset validity period, the recommended price is directly read and returned to the front end for display, thus reducing the pressure on the model for real-time inference and ensuring the stability of the price within a certain time window. If no recommended price is found in the cache or the cache has expired, the system proceeds to the reinforcement learning pricing process.

[0025] In the event of a cache miss, multi-dimensional feature information of the target user is obtained, including user profile features and historical behavior features. User profile features include membership level, account activity, and registration duration, while historical behavior features include historical purchase frequency, average order value, number of views, number of add-to-cart occurrences, historical conversion rate, and discount usage rate. Simultaneously, product features and temporal context features of the target product are obtained. Product features include real-time inventory quantity, inventory turnover rate, and historical conversion rate, while temporal context features include current time period identifiers, holiday identifiers, and promotional activity identifiers. After numerical and normalization processing of these features, feature concatenation is performed to construct the target user's state vector in the current access scenario.

[0026] Subsequently, the state vector is input into a pre-trained objective reinforcement learning model. This model, based on a pricing Markov decision process, evaluates multiple candidate discount actions under different states and calculates the state-action value function for each action. Under the constraint of a preset price safety boundary, the discount action with the highest state-action value function is selected as the target pricing action. This ensures that the pricing decision aligns with the profit maximization objective without exceeding the preset discount range, preventing excessive price fluctuations or adverse effects on brand value.

[0027] After determining the target pricing action, a discount is calculated on the base price of the target product based on the target pricing action to generate a corresponding recommended price. This recommended price is then returned to the front-end page to be displayed to the target user. Simultaneously, the recommended price is written to a Redis cache with a preset validity period, so that the cached result can be directly read upon subsequent accesses within the validity period, thereby improving system response speed and reducing the overhead of redundant model calculations.

[0028] It implements an online pricing mechanism of "caching first, model supplementation". While ensuring system performance in high-concurrency scenarios, it introduces a reinforcement learning model to make dynamic decisions for different users, different products and different time scenarios, thereby realizing the automated execution of personalized pricing strategies.

[0029] Optionally, this embodiment can realize real-time decision-making and continuous optimization of pricing strategies in complex and dynamic environments, thereby improving the accuracy of personalized pricing; it ensures that pricing risks are controllable through a price safety boundary constraint mechanism; it significantly improves system response efficiency and reduces computing resource consumption through a caching mechanism; and overall, it helps to improve the conversion rate, increase the total transaction amount, and optimize long-term profit levels.

[0030] As an optional example, the user profile features and historical behavior features of the target user, as well as the product features and temporal context features of the target product, are obtained and concatenated to construct the target user's state vector, including: Obtain user profile features of the target users, including membership level, account activity, registration duration, historical purchase frequency, average order value, and time interval between the most recent purchase. Obtain the historical behavioral characteristics of target users for target products, including the number of times they browse, the number of times they add items to their cart, the historical conversion rate, and the estimated value of price sensitivity. Obtain the product characteristics and time context characteristics of the target product. The product characteristics include real-time inventory quantity, inventory turnover rate, and historical conversion rate. The time context characteristics include current time period identifier, holiday identifier, and promotional activity identifier. After numerical encoding and normalization of user profile features, historical behavior features, product features, and temporal context features, vectors are concatenated to generate a state vector.

[0031] Optionally, in this embodiment, when constructing the state vector for the reinforcement learning model input, a systematic modeling of relevant information about the target user and target product can be performed through multi-dimensional feature fusion. Specifically, firstly, user profile features of the target user are obtained. These user profile features are used to characterize the user's basic attributes and long-term spending power, including membership level, account activity, registration duration, historical purchase frequency, average order value, and the time interval between the most recent purchase. These features can reflect the user's value level, activity level, and consumption habits, providing basic data support for subsequently judging their price sensitivity and potential conversion ability.

[0032] Secondly, it is necessary to obtain the historical behavioral characteristics of target users regarding the target product or similar products. These historical behavioral characteristics include metrics such as the number of views, the number of times users added to their cart, historical conversion rates, and price sensitivity estimates. Among these, the number of views and the number of times users added to their cart reflect the intensity of user interest, the historical conversion rate characterizes the probability of a user making a purchase in similar scenarios, and the price sensitivity estimate can be modeled and calculated based on historical discount usage and purchasing behavior under price changes, thereby providing a more refined characterization of the user's responsiveness to price changes.

[0033] Furthermore, the product characteristics and temporal context characteristics of the target product are obtained. Product characteristics include real-time inventory quantity, inventory turnover rate, and historical conversion rate, which are used to reflect the supply and demand relationship and sales performance of the product; temporal context characteristics include current time period identifiers, holiday identifiers, and promotional activity identifiers, which are used to characterize the current business environment and external influencing factors, so as to enhance the model's ability to perceive the dynamic market environment.

[0034] After acquiring the aforementioned multi-source features, user profile features, historical behavior features, product features, and temporal context features are numerically encoded. Categorical features are represented using one-hot encoding or embedding, while numerical features are normalized or standardized to eliminate the influence of differences in feature dimensions. Subsequently, the processed features are concatenated into vectors according to a preset order to generate a unified-dimensional state vector, which serves as the input to the reinforcement learning model, thereby providing a complete and structured environmental representation for subsequent pricing decisions.

[0035] As an alternative example, the above method may also include the following before inputting the state vector into the target reinforcement learning model: Retrieve historical transaction data from a MySQL database; A Markov decision model is constructed based on historical transaction data, and the state, action, reward and state transition relationship of the Markov decision model are defined. The state is composed of user profile features, historical behavior features, product features and time context features. The action includes multiple candidate discount actions. The reward is the revenue data generated under the corresponding discount action. The state transition relationship is determined by the user's behavioral feedback under different discounts. A state-action value function is constructed based on a Markov decision model, and a deep Q-network is used to approximate the state-action value function. The state-action value function represents the long-term cumulative return corresponding to performing a candidate discounted action in a given state. By utilizing historical transaction data, a training sample set containing states, actions, rewards, and state transition relationships is constructed. Based on the training sample set, a deep Q-network is trained offline to obtain a target reinforcement learning model.

[0036] Optionally, in this embodiment, before inputting the constructed state vector into the target reinforcement learning model for online inference, the reinforcement learning model is further trained offline to obtain a target reinforcement learning model with stable decision-making capabilities. Specifically, historical transaction data is first obtained from the MySQL database. This historical transaction data includes user's historical order records, discount information, transaction amount, browsing and adding-to-cart behavior records, etc., to characterize the user's real behavioral feedback under different pricing strategies.

[0037] After acquiring historical transaction data, a pricing Markov decision model is constructed based on this data, defining the state, action, reward, and state transition relationships within this model. The state is composed of user profile features, historical behavior features, product features, and temporal context features, representing the environmental information during pricing decisions. Actions include multiple candidate discount actions, such as different discount percentages or incentive levels. The reward is the revenue data generated under the corresponding discount action, which can be the actual transaction amount or profit data after deducting costs. State transition relationships are determined by user behavior feedback under different discount conditions, including actions such as clicking, adding to cart, purchasing, or not purchasing, thus reflecting the impact of the current pricing action on the future state.

[0038] Based on Markov decision models, a state-action value function is constructed to represent the long-term cumulative benefit obtained by performing a candidate discounted action in a given state. To achieve effective modeling of the high-dimensional state space, a deep Q-network is used to approximate the state-action value function, and the value mapping relationship between different states and discounted actions is learned through the neural network parameters.

[0039] Furthermore, a training sample set containing states, actions, rewards, and state transition relationships is constructed using historical transaction data. A deep Q-network is then trained offline based on this training sample set. By iteratively updating the network parameters to minimize the difference between the predicted value and the target value, a convergent and stable target reinforcement learning model is obtained. Through this training process, the model can learn the optimal pricing strategy in complex and dynamic environments, providing a reliable decision-making basis for subsequent online personalized pricing.

[0040] As an alternative example, offline training of a deep Q-network based on a training sample set yields target reinforcement learning models, including: During offline training, a preset price safety boundary constraint is applied to the action space, and the training sample set is randomly sampled and batch updated to obtain the target reinforcement learning model.

[0041] Optionally, in this embodiment, during the offline training of the deep Q-network based on the training sample set, in order to ensure the stability and commercial controllability of the model, a price safety boundary constraint and a random batch update mechanism can be introduced during the training phase to obtain a target reinforcement learning model that meets the risk control requirements.

[0042] Specifically, during the offline training phase, a pre-defined price safety boundary constraint is first applied to the action space of the reinforcement learning model. This constraint limits the range of values ​​for candidate discount actions; for example, it limits the discount percentage to a pre-defined fluctuation range of the base price, preventing the model from learning extremely low or unreasonably high price strategies during training. Actions exceeding the safety boundary can be addressed by restricting the action set, assigning punitive rewards, or adding constraints to the loss function, thereby ensuring that the trained pricing strategy is risk-controllable and commercially viable in practical applications.

[0043] Simultaneously, during model training, the training sample set undergoes random sampling and batch updates. Specifically, historical samples containing information about state, action, reward, and next state are stored in an experience sample pool, and a predetermined number of samples are randomly selected to form a mini-batch for updating network parameters during each training iteration. This random sampling mechanism breaks down the temporal correlation between samples, improving training stability; the batch update method enhances training efficiency and strengthens the model's generalization ability to the overall data distribution.

[0044] Under the combined effect of the aforementioned price safety constraints and random batch training mechanism, the deep Q-network can learn the optimal pricing strategy that conforms to business rules while ensuring stable convergence during the training process. Ultimately, it outputs a target reinforcement learning model that satisfies the goal of maximizing revenue and meets the requirements of price risk control, providing reliable support for online personalized pricing.

[0045] As an optional example, the state vector is input into the target reinforcement learning model, which calculates the state-action value function value corresponding to each candidate discount action based on the state vector. Under the condition of satisfying a preset price safety boundary constraint, the candidate discount action with the largest state-action value function value is selected as the target pricing action. This includes: The state vector is input into the target reinforcement learning model so that the target reinforcement learning model can calculate the state-action value function value of the state vector under each candidate discounted action through forward propagation. Based on the preset price safety boundary constraints, the legality of each candidate discount action is determined; Among the candidate discount actions that meet the preset price safety boundary constraints, the candidate discount action with the largest corresponding state-action value function value is selected as the target pricing action.

[0046] Optionally, in this embodiment, after the state vector is constructed, it is input into the target reinforcement learning model for online inference to determine the optimal pricing action in the current scenario. Specifically, the state vector is first input into the trained target reinforcement learning model, and the state-action value function value corresponding to each candidate discount action is calculated through the forward propagation process of the neural network. The state-action value function value is used to represent the expected long-term cumulative revenue that can be obtained by executing the corresponding discount action under the current user state and product environment.

[0047] After obtaining the state-action value function value corresponding to each candidate discount action, the legality of each candidate discount action is determined according to the preset price safety boundary constraints. Price safety boundary constraints may include minimum transaction price limits, maximum discount percentage limits, profit margin lower limits, and price control rules for specific promotional periods, etc., to ensure that the generated pricing strategy complies with platform operating rules and risk control requirements. Candidate discount actions that do not meet the price safety boundary constraints can be directly eliminated, or their corresponding state-action value can be set to an invalid value, thereby preventing them from participating in the subsequent optimal action selection process.

[0048] After completing the legality screening, the candidate discount action with the largest corresponding state-action value function value is selected from the set of candidate discount actions that meet the preset price safety boundary constraints as the target pricing action. Through this method, the model comprehensively considers multi-dimensional factors such as user characteristics, product characteristics, and time context to evaluate the returns of different discount strategies and select the optimal pricing scheme under controllable risk, thereby balancing profit maximization with business rule constraints and achieving intelligent, dynamic, and personalized pricing decisions.

[0049] As an optional example, after returning the recommended price to show to the target user, the above method also includes: Collect target users' click behavior, purchase behavior, and churn behavior for target products, and record the corresponding status, pricing actions, and actual revenue to generate feedback data. At the same time, save the feedback data to a MySQL database. Feedback data is written into the training sample set to form an incremental training dataset; According to the preset update cycle, based on the preset online exploration ratio control strategy and A / B testing mechanism, the target reinforcement learning model is retrained using the incremental training dataset.

[0050] Optionally, in this embodiment, after generating the recommended price and displaying it to the target user, a closed-loop feedback mechanism is also included to achieve continuous optimization and adaptive updates of the model. This mechanism collects, stores, and incrementally trains user behavior data. Specifically, after the recommended price is displayed, the system collects real-time interaction information such as the target user's click behavior, purchase behavior, and churn behavior related to the target product. This information is then combined with the corresponding state vector, the pricing action performed, and the actual revenue generated to generate feedback data that includes the state, action, reward, and subsequent behavioral results. Simultaneously, the feedback data is persistently saved to a MySQL database for unified management and offline analysis.

[0051] After the feedback data is stored, it is written into the training sample set to construct an incremental training dataset as new sample data. By continuously accumulating real online interaction data, the training sample set can dynamically reflect the latest changes in user preferences, market environment, and inventory fluctuations, thereby enhancing the model's ability to perceive environmental changes.

[0052] Furthermore, according to a preset update cycle, such as daily, weekly, or when a preset data scale threshold is reached, the target reinforcement learning model is retrained based on a preset online exploration ratio control strategy and A / B testing mechanism. The online exploration ratio control strategy is used to retain a certain proportion of exploration actions while ensuring overall revenue stability, thus preventing the model from getting trapped in local optima. The A / B testing mechanism is used to compare and evaluate the performance of the old and new models or different strategy versions, ensuring the stability of model updates and the effectiveness of revenue improvement. Through the above incremental training and periodic update mechanisms, the target reinforcement learning model can continuously learn and iteratively optimize during actual business operations, achieving dynamic adaptive adjustment of pricing strategies and maximizing long-term revenue.

[0053] As an optional example, after retrieving the recommended price of the target product for the target user from the Redis cache, the above method also includes: If the recommended price of the target product for the target user is found in the Redis cache, the corresponding recommended price is returned to the target user for display.

[0054] Optionally, in this embodiment, when a recommended price for a target product that has been generated for the target user and is still valid is successfully retrieved from the Redis cache, the recommended price can be read directly and returned to the front-end page as the current display price for display to the target user, without having to perform feature construction and reinforcement learning model inference calculations again.

[0055] During this process, the validity of the cached data can be further verified, such as whether the cached data has exceeded the preset validity period, whether the product inventory status has undergone significant changes, or whether the price safety boundary has been adjusted. If the cached data meets the usage conditions, the recommended price is returned directly, thereby avoiding the system resource consumption caused by repeated calculations. By utilizing a memory-level caching mechanism, pricing results in high-frequency access scenarios can be quickly reused, effectively reducing the computational pressure of online inference, shortening interface response time, and improving the overall system throughput and user experience.

[0056] Meanwhile, by setting a preset validity period for the recommended prices in the cache, a balance can be struck between computational costs and price update frequency while ensuring a certain level of real-time performance. When the cache expires or misses, the reinforcement learning model is triggered to recalculate, thereby optimizing the configuration between system performance and pricing accuracy. Through the above-mentioned direct return mechanism for cache hits, the personalized pricing system can maintain stable and low-latency operation in high-concurrency scenarios, improving overall service quality and business processing efficiency.

[0057] To illustrate with an example, this application relates to a pricing system based on a reinforcement learning model. By formalizing the pricing problem as a Markov Decision Process (MDP) and introducing a reinforcement learning agent, the system continuously optimizes personalized pricing strategies based on user behavior feedback while satisfying the price safety boundary, thereby achieving long-term maximization of GMV (Gross Merchandise Volume) and profits.

[0058] like Figure 2The system architecture diagram shown depicts a pricing service as the core hub. Modules collaborate through standardized interfaces and data flows, and the system is divided into six core parts: front-end interaction, core services, caching and message middleware, data storage, model training, and model services. Clients initiate price query requests to the pricing service, developed using Spring Boot. The pricing service, on one hand, sends a pricing prediction request to the model service built with TensorFlow Serving via gRPC. After the model service returns the corresponding discount, the pricing service combines this with rules to generate the final price and sends it back to the client. On the other hand, the pricing service writes recommended prices for high-frequency users to a Redis cache to improve subsequent query efficiency. Simultaneously, clicks, purchases, and other behavioral events generated by the client are reported by the pricing service to Kafka, forming a real-time behavioral data stream. The Spark training module obtains real-time behavioral data from Kafka and extracts historical transaction data from a MySQL database to retrain the model. The trained model is then synchronized to the model service, enabling rolling model updates and supporting the continuous and accurate predictions of the pricing service.

[0059] The entire pricing decision process in user access scenarios, as follows: Figure 3 As shown, the system optimizes by first retrieving the cache, focuses on model prediction, and uses behavioral feedback collection as a closed loop, with interconnected steps that are both efficient and secure. It begins by receiving a user's price request. First, it constructs corresponding state features based on user profiles and product information, then initiates a query request to the Redis cache. If the cache is hit, the price is retrieved directly from the cache and returned to the user, completing the pricing decision. If the cache is not hit, the model service is called to predict the discount. After applying price safety boundary constraints to the prediction result, the result is written to the Redis cache for later reuse, and then the final price is generated and returned to the user. Simultaneously, the system collects user feedback data regarding this pricing, such as clicks and purchases, providing data support for the online learning and updating of the reinforcement learning model.

[0060] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0061] According to another aspect of the embodiments of this application, a pricing device based on a reinforcement learning model is also provided, such as... Figure 4 As shown, it includes: The query module 402 is used to query the recommended price of the target product for the target user in the Redis cache when a target user requests access to the target product. The first construction module 404 is used to obtain the user profile features and historical behavior features of the target user, as well as the product features and time context features of the target product, when the recommended price of the target product for the target user is not found in the Redis cache, and to concatenate the features to construct the state vector of the target user. The first calculation module 406 is used to input the state vector into the target reinforcement learning model so that the target reinforcement learning model can calculate the state-action value function value corresponding to each candidate discount action based on the state vector, and select the candidate discount action with the largest state-action value function value as the target pricing action under the condition of satisfying the preset price safety boundary constraint. The second calculation module 408 is used to calculate the discount on the base price of the target product based on the target pricing action, obtain the corresponding recommended price, and return the recommended price to be displayed to the target user. At the same time, the recommended price is written to the Redis cache and a preset validity period is set.

[0062] It should be noted that the query module 402 in this embodiment can be used to execute step S102 in this application embodiment, the first construction module 404 in this embodiment can be used to execute step S104 in this application embodiment, the first calculation module 406 in this embodiment can be used to execute step S106 in this application embodiment, and the second calculation module 408 in this embodiment can be used to execute step S108 in this application embodiment.

[0063] As an optional example, the first building block includes: The first acquisition unit is used to acquire the user profile features of the target user, wherein the user profile features include membership level, account activity, registration duration, historical purchase frequency, average order value and the time interval between the most recent purchase; The second acquisition unit is used to acquire the historical behavioral characteristics of the target user for the target product, including the number of times the user browsed, the number of times the user added the product to their cart, the historical conversion rate, and the price sensitivity estimate. The third acquisition unit is used to acquire the product characteristics and time context characteristics of the target product. The product characteristics include real-time inventory quantity, inventory turnover rate and historical conversion rate. The time context characteristics include current time period identifier, holiday identifier and promotion activity identifier. The building unit is used to numerically encode and normalize user profile features, historical behavior features, product features, and time context features, and then concatenate the vectors to generate a state vector.

[0064] As an optional example, the above-described apparatus further includes: The acquisition module is used to retrieve historical transaction data from a MySQL database before inputting the state vector into the target reinforcement learning model; The second building module is used to build a Markov decision model based on historical transaction data and define the state, action, reward and state transition relationship of the Markov decision model. The state is composed of user profile features, historical behavior features, product features and time context features. The action includes multiple candidate discount actions. The reward is the revenue data generated under the corresponding discount action. The state transition relationship is determined by the user's behavioral feedback under different discounts. The third construction module is used to construct the state-action value function based on the Markov decision model and to approximate the state-action value function using a deep Q-network. The state-action value function represents the long-term cumulative return corresponding to performing a candidate discounted action in a given state. The first training module is used to construct a training sample set containing states, actions, rewards, and state transition relationships using historical transaction data, and to perform offline training on the deep Q network based on the training sample set to obtain the target reinforcement learning model.

[0065] As an optional example, the first training module includes: The training unit is used to apply a preset price safety boundary constraint to the action space during offline training, and to randomly sample and batch update the training sample set to obtain the target reinforcement learning model.

[0066] As an optional example, the first computation module includes: The computational unit is used to input the state vector into the target reinforcement learning model so that the target reinforcement learning model can calculate the state-action value function value of the state vector under each candidate discounted action through forward propagation. The determination unit is used to determine the legality of each candidate discount action according to the preset price safety boundary constraint; the selection unit is used to select the candidate discount action with the largest corresponding state-action value function value from among the candidate discount actions that meet the preset price safety boundary constraint as the target pricing action.

[0067] As an optional example, the above-described apparatus further includes: The data collection module is used to collect the target user's click behavior, purchase behavior, and churn behavior after returning the recommended price to the target user for display, and to record the corresponding status, pricing actions, and actual revenue, generate feedback data, and save the feedback data to the MySQL database. The writing module is used to write feedback data into the training sample set to form an incremental training dataset; The second training module is used to retrain the target reinforcement learning model using an incremental training dataset according to a preset update cycle, based on a preset online exploration ratio control strategy and A / B testing mechanism.

[0068] As an optional example, the above-described apparatus further includes: The return module is used to query the recommended price of the target product for the target user in the Redis cache, and if the recommended price of the target product for the target user is found in the Redis cache, return the corresponding recommended price to be displayed to the target user.

[0069] For other examples of this embodiment, please refer to the examples above, which will not be repeated here.

[0070] Figure 5 This is a schematic diagram of an optional electronic device according to an embodiment of this application, such as... Figure 5 As shown, it includes a processor 502, a communication interface 504, a memory 506, and a communication bus 508. The processor 502, communication interface 504, and memory 506 communicate with each other via the communication bus 508. Memory 506 is used to store computer programs; When processor 502 executes a computer program stored in memory 506, it performs the following steps: When a target user requests access to a target product, the recommended price of the target product for the target user is retrieved from the Redis cache. If the recommended price of the target product for the target user is not found in the Redis cache, the user profile features and historical behavior features of the target user, as well as the product features and time context features of the target product, are obtained and concatenated to construct the state vector of the target user. The state vector is input into the target reinforcement learning model so that the target reinforcement learning model can calculate the state-action value function value corresponding to each candidate discount action based on the state vector, and select the candidate discount action with the largest state-action value function value as the target pricing action under the condition of satisfying the preset price safety boundary constraint. The base price of the target product is discounted based on the target pricing action to obtain the corresponding recommended price, which is then returned to be displayed to the target user. At the same time, the recommended price is written to the Redis cache and a preset validity period is set.

[0071] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The symbol is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0072] The memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0073] As an example, the memory 506 described above may include, but is not limited to, the query module 402, the first construction module 404, the first calculation module 406, and the second calculation module 408 in the pricing device based on the reinforcement learning model. Furthermore, it may include, but is not limited to, other module units in the pricing device based on the reinforcement learning model described above, which will not be elaborated upon in this example.

[0074] The processor mentioned above can be a general-purpose processor, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0075] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0076] Those skilled in the art will understand that Figure 5The structure shown is for illustrative purposes only. The device implementing the above-mentioned pricing method based on reinforcement learning models can be a terminal device, such as a smartphone (e.g., Android phone, iOS phone), tablet computer, PDA, mobile Internet Devices (MID), PAD, etc. Figure 5 This does not limit the structure of the aforementioned electronic devices. For example, the electronic device may also include components that are more... Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0077] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0078] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, which, when executed by a processor, performs the steps in the above-described pricing method based on a reinforcement learning model.

[0079] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0080] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0081] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0082] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0084] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0086] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A pricing method based on a reinforcement learning model, characterized in that, include: When a target user requests access to a target product, the recommended price of the target product for the target user is retrieved from the Redis cache. If the recommended price of the target product for the target user is not found in the Redis cache, the user profile features and historical behavior features of the target user, as well as the product features and time context features of the target product, are obtained and concatenated to construct the state vector of the target user. The state vector is input into the target reinforcement learning model so that the target reinforcement learning model calculates the state-action value function value corresponding to each candidate discount action based on the state vector, and selects the candidate discount action with the largest state-action value function value as the target pricing action under the condition of satisfying the preset price safety boundary constraint. The base price of the target product is discounted according to the target pricing action to obtain the corresponding recommended price, and the recommended price is returned to be displayed to the target user. At the same time, the recommended price is written to the Redis cache and a preset validity period is set.

2. The method according to claim 1, characterized in that, The process of acquiring the user profile features and historical behavior features of the target user, as well as the product features and temporal context features of the target product, and concatenating these features to construct the state vector of the target user includes: The user profile features of the target user are obtained, wherein the user profile features include membership level, account activity, registration duration, historical purchase frequency, average order value and the time interval between the most recent purchase; Obtain the historical behavioral characteristics of the target user for the target product, wherein the historical behavioral characteristics include the number of views, the number of times added to cart, the historical conversion rate, and the price sensitivity estimate; Obtain the product characteristics and time context characteristics of the target product, wherein the product characteristics include real-time inventory quantity, inventory turnover rate and historical conversion rate, and the time context characteristics include current time period identifier, holiday identifier and promotional activity identifier; The user profile features, historical behavior features, product features, and time context features are numerically encoded and normalized, and then concatenated into vectors to generate the state vector.

3. The method according to claim 1, characterized in that, Before inputting the state vector into the target reinforcement learning model, the method further includes: Retrieve historical transaction data from a MySQL database; A Markov decision model is constructed based on the historical transaction data, and the state, action, reward and state transition relationship of the Markov decision model are defined. The state includes user profile features, historical behavior features, product features and time context features. The action includes multiple candidate discount actions. The reward is the revenue data generated under the corresponding discount action. The state transition relationship is determined by the user's behavioral feedback under different discounts. A state-action value function is constructed based on the Markov decision model, and a deep Q-network is used to approximate the state-action value function, wherein the state-action value function represents the long-term cumulative return corresponding to performing a candidate discount action in a given state; Using the historical transaction data, a training sample set containing states, actions, rewards, and state transition relationships is constructed, and the deep Q-network is trained offline based on the training sample set to obtain the target reinforcement learning model.

4. The method according to claim 3, characterized in that, The deep Q-network is trained offline based on the training sample set to obtain the target reinforcement learning model, which includes: During offline training, a preset price safety boundary constraint is applied to the action space, and the training sample set is randomly sampled and batch updated to obtain the target reinforcement learning model.

5. The method according to claim 1, characterized in that, The state vector is input into the target reinforcement learning model, so that the target reinforcement learning model calculates the state-action value function value corresponding to each candidate discount action based on the state vector, and selects the candidate discount action with the largest state-action value function value as the target pricing action under the condition of satisfying the preset price safety boundary constraint. The state vector is input into the target reinforcement learning model so that the target reinforcement learning model can calculate the state-action value function value of the state vector under each candidate discount action through forward propagation. Based on the preset price safety boundary constraint, the legality of each candidate discount action is determined; Among the candidate discount actions that satisfy the preset price safety boundary constraint, the candidate discount action with the largest corresponding state-action value function value is selected as the target pricing action.

6. The method according to claim 1, characterized in that, After returning the recommended price to display to the target user, the method further includes: Collect the target user's click behavior, purchase behavior, and churn behavior on the target product, and record the corresponding status, pricing actions, and actual revenue to generate feedback data. At the same time, save the feedback data to a MySQL database. The feedback data is written into the training sample set to form an incremental training dataset; According to the preset update cycle, based on the preset online exploration ratio control strategy and A / B testing mechanism, the target reinforcement learning model is retrained using the incremental training dataset.

7. The method according to any one of claims 1 to 6, characterized in that, After retrieving the recommended price of the target product for the target user from the Redis cache, the method further includes: If the recommended price of the target product for the target user is found in the Redis cache, the corresponding recommended price is returned to be displayed to the target user.

8. A pricing device based on a reinforcement learning model, characterized in that, include: The query module is used to query the recommended price of the target product for the target user in the Redis cache when a target user requests access to the target product. The first construction module is used to obtain the user profile features and historical behavior features of the target user, as well as the product features and time context features of the target product, when the recommended price of the target product for the target user is not found in the Redis cache, and to concatenate the features to construct the state vector of the target user. The first calculation module is used to input the state vector into the target reinforcement learning model, so that the target reinforcement learning model calculates the state-action value function value corresponding to each candidate discount action according to the state vector, and selects the candidate discount action with the largest state-action value function value as the target pricing action under the condition of satisfying the preset price safety boundary constraint. The second calculation module is used to calculate the discount on the base price of the target product based on the target pricing action, obtain the corresponding recommended price, and return the recommended price to be displayed to the target user. At the same time, the recommended price is written to the Redis cache and a preset validity period is set.

9. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 7 through the computer program.

Images