Multi-objective dynamic pricing method and device based on reinforcement learning and price sensitivity

Abstract

The application relates to the technical field of dynamic pricing and intelligent marketing, and particularly provides a multi-target dynamic pricing method and device based on reinforcement learning and price sensitivity; a price sensitivity curve is constructed based on internal business data, the price sensitivity curve is processed to generate corresponding derivative quantities, and pricing state features are constructed in combination with multi-dimensional business features; a pricing action space is constructed based on the pricing state features, business constraints are determined, and a multi-target reward function is set; the reinforcement learning strategy is pre-trained offline and updated online based on the pricing state features, the pricing action space, the business constraints and the multi-target reward function; the weights of the multi-target reward function are adaptively adjusted during the updating; and the updated reinforcement learning strategy is used to output a price decision, and abnormality alarm monitoring is simultaneously performed, so that multi-target dynamic pricing is realized; and the accuracy and reliability of dynamic pricing are improved.

Description

Technical Field

[0001] This invention relates to the field of dynamic pricing and intelligent marketing technology, specifically providing a multi-objective dynamic pricing method and apparatus based on reinforcement learning and price sensitivity. Background Technology

[0002] Traditional methods often employ rule-based, enumeration-based comparison, or offline predictive approaches for pricing decisions. These methods rely on manually configured rules and offline regression prediction models, exhibiting poor adaptive adjustment capabilities in non-stationary demand scenarios such as seasonal fluctuations, marketing campaigns, and competitor price changes. Some systems use historical logs for offline pricing strategy evaluation; however, the effectiveness of offline strategy evaluation is highly dependent on data collection, sample coverage, and modeling assumptions. When samples are scarce, existing pricing strategies are rigid, or new pricing actions occur, the estimation of the price-demand relationship shows significant deviations, making it difficult to directly support online real-time pricing decisions.

[0003] Existing dynamic pricing solutions based on reinforcement learning typically fail to systematically address hard constraints such as cost floor, channel price differences, price fluctuation ceilings, and compliant price lists. The training process struggles to guarantee that these constraints are met iteratively, making safe exploration even more challenging in non-stationary business environments. Furthermore, many solutions treat the relationship between price and demand as a black box, performing only black-box predictions and lacking utilization of structured information such as price elasticity. This makes it difficult to make interpretable trade-offs between profit, conversion rates, price stability, and risk. Summary of the Invention

[0004] This application provides a multi-objective dynamic pricing method and apparatus based on reinforcement learning and price sensitivity to solve the problems of inaccurate and unstable dynamic pricing.

[0005] This application provides a multi-objective dynamic pricing method based on reinforcement learning and price sensitivity, the method comprising: Acquire multi-source data, including internal business data and external environment data; A price sensitivity curve is constructed based on the internal business data. The price sensitivity curve is processed to generate corresponding derived quantities. Pricing status features are then constructed by combining multi-dimensional business features. Based on the pricing state characteristics, a pricing action space is constructed, business constraints are determined, and a multi-objective reward function is set. Based on the pricing state features, pricing action space, business constraints, and multi-objective reward function, the reinforcement learning strategy is pre-trained offline and updated online; and the weights of the multi-objective reward function are adaptively adjusted during the update. The updated reinforcement learning strategy is used to output price decisions and monitor for anomalies, thereby achieving multi-objective dynamic pricing.

[0006] In some embodiments, the process of processing the price sensitivity curve to generate corresponding derived quantities and constructing pricing status features by combining multi-dimensional business characteristics includes: The price sensitivity curve is processed to generate a corresponding derived quantity, which includes at least a slope parameter, an inflection point, a purchase probability at the current price, and a slope at the current price. Obtain product features, environmental features, contextual features, and constraint and risk features; embed the derived values ​​into user features, and combine them with the product features, environmental features, contextual features, and constraint and risk features to construct a concatenated vector of pricing status features: .

[0007] In some embodiments, the step of outputting a price decision based on the updated reinforcement learning policy while simultaneously performing anomaly alert monitoring to achieve multi-objective dynamic pricing includes: Based on the updated reinforcement learning strategy, candidate price actions are output; The candidate price action is pruned a priori according to the business constraints to obtain the final execution price; Simultaneously generate explanatory elements for price decisions, including sensitivity parameter pairs. The contribution of inventory and the contribution of competitors; Each price decision record is stored in the database, with the following fields: current business status, candidate price action, final execution price, reward value, decision reason tag, strategy weight, and strategy version number. After executing the final price online, collect actual user interaction feedback to form a complete reinforcement learning sample for incremental policy updates.

[0008] The multi-objective dynamic pricing method and apparatus based on reinforcement learning and price sensitivity provided in this application explicitly introduce price-sensitive parameters and derived quantities during state construction and action generation. Anchor prices guide the pricing strategy to only make small adjustments within the theoretically optimal range. Simultaneously, a latent potential function is used for reward shaping, and a reward function is constructed using anchor deviations. This accelerates model convergence without changing the optimal strategy, achieving interpretable pricing decisions and rapid iterative convergence based on price sensitivity. Constrained reinforcement learning, such as the CPO algorithm, is employed to continuously restrict hard constraints such as cost floor, channel price difference, price fluctuation ceiling, and compliance list during model training. The same constraints are used for strong pruning during inference, ensuring consistent safety between training and execution. Profit targets, conversion targets, inventory targets, price stability, and pricing risk are integrated into a unified multi-objective reward, avoiding price fluctuations and short-sighted behavior caused by using only a single target. Simultaneously, the fluctuation penalty weight is dynamically adjusted to effectively suppress price jumps in sensitive areas, improving the accuracy of dynamic pricing. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart of the multi-objective dynamic pricing method and apparatus based on reinforcement learning and price sensitivity provided by the present invention; Figure 2 This is a schematic diagram of the fitting results of the multi-objective dynamic pricing method based on reinforcement learning and price sensitivity provided by the present invention; Figure 3 This is a schematic diagram of the reinforcement learning strategy process of the multi-objective dynamic pricing method based on reinforcement learning and price sensitivity provided by the present invention; Figure 4 This is a schematic diagram illustrating the principle and flow of the multi-objective dynamic pricing device based on reinforcement learning and price sensitivity provided by the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0012] The following is combined Figures 1 to 4 The illustrated embodiments describe the technical solution of the present invention: This application provides an embodiment of a multi-objective dynamic pricing method and apparatus based on reinforcement learning and price sensitivity, referring to... Figure 1 As shown, the multi-objective dynamic pricing method based on reinforcement learning and price sensitivity provided in this embodiment includes the following steps: S110: Acquire multi-source data, which includes internal business data and external environment data.

[0013] In some embodiments, the implementation of step S110 (acquiring multi-source data, including internal business data and external environment data) may include: It should be noted that user data comes from the company's internal systems and external monitoring, while basic user information comes from the internal system's management system.

[0014] Through internal data synchronization interfaces and external data acquisition units, multi-source data supporting pricing decisions is obtained. The multi-source data is divided into internal business data and external environmental data. Specifically, internal business data is taken from the company's internal management system, including basic user information, user value and behavior data, historical pricing context, product and inventory information, and channel and regional attribute data. External environment data is collected in real time through an external monitoring and acquisition module, including competition and environmental data, risk and compliance constraint data; for cold start scenarios of new products and new users, cold start prior data is acquired simultaneously to complete the construction of the initial pricing basis.

[0015] For example, it mainly includes the following information:

[0016] User Basic Information Registration time 2024 / 5 / 10 YYYY-MM-DD User Basic Information Network access time 2024 / 5 / 11 YYYY-MM-DD User Basic Information Membership Levels GOLD SILVER, GOLD, PLATINU User Basic Information area East China National / Provincial / Municipal Multi-level Coding User Basic Information Recent M-period retention / activity tags Active in the last 3 months Scope: Nearly M months of active users ≥ X days User value and behavior Periodic consumption amount 128 Tax-inclusive / Tax-exclusive must be consistent User value and behavior Package Levels Enjoy 129 Standard Package Dictionary User value and behavior Usage time 180 Minutes; missing values ​​are filled with the cluster mean. User value and behavior Use traffic 5 GB; missing values ​​were filled with the cluster mean. User value and behavior Add-to-cart event timestamp 2025 / 9 / 2013:00 YYYY-MM-DDhh:mm:ss User value and behavior Order event timestamp 2025 / 9 / 2013:05 YYYY-MM-DDhh:mm:ss User value and behavior Unsubscription event timestamp 2025 / 9 / 2014:00 YYYY-MM-DDhh:mm:ss Historical pricing context Historical price series p_{t−k} [129,129,119,...] Attached timestamp sequence Historical pricing context Click-through rate (CTR) 12.30% 0–100% Historical pricing context Add-to-cart rate ATC 8.70% 0–100% Historical pricing context Transaction Rate CR 3.90% 0–100% Goods and Inventory Existing stock 235 non-negative integers Goods and Inventory Lifespan stage Growth period Introduction, growth stage, maturity, delisting Goods and Inventory Unsold labels 0 Threshold example: Sell-through rate in the past 7 days <2% Competition and Environment Minimum competitor price p^{comp}_{min} 119 N∈[10,50] can be matched Competition and Environment Maximum competitor price p^{comp}_{max} 139 Same diameter as minimum value Competition and Environment Event / Holiday Signs National Day Holiday Dictionary Competition and Environment Platform global discount strength middle Tiers {Low / Medium / High} or percentages Channels / Regions Channel identification Online direct sales Channel Dictionary Channels / Regions Regional policies East China Policy A Policy version management Channels / Regions Historically poor channels 2.50% Threshold δ_ch=3% Risk and Compliance Minimum sellable price p_min 99 Hard constraints, out-of-bounds clipping Risk and Compliance The highest possible selling price p_max 199 Hard constraints, out-of-bounds clipping Risk and Compliance Sensitive product category whitelist / red line marking yes Triggering Alternative Rules Pricing Cold start prior Cluster mean price elasticity α 1.8 Use when sample is insufficient Cold start prior Inflection point distribution β 129 Switch user-level parameters after meeting the requirements.

[0017] It should be noted that the acquired data forms the basis of price sensitivity modeling. The data is used for model sample construction, clustering, parameter constraints, and online iteration. With data-driven pricing as its core, it comprehensively utilizes statistical modeling of user behavior and product supply, price sensitivity and elasticity analysis, as well as reinforcement learning and operations research optimization methods.

[0018] S120: Construct a price sensitivity curve based on internal business data, process the price sensitivity curve to generate corresponding derived quantities, and combine it with multi-dimensional business characteristics to construct pricing status characteristics.

[0019] In some embodiments, the implementation of step S120 (constructing a price sensitivity curve based on internal business data, processing the price sensitivity curve to generate corresponding derived quantities, and combining multi-dimensional business characteristics to construct pricing status features) may include:

[0020] It should be noted that by forcibly incorporating the price sensitivity curve parameters and derived values ​​into the state vector and interpretation field, and using this to generate an anchor price as the benchmark for action search, the strategy is only fine-tuned within the neighborhood of the anchor point, which significantly reduces the exploration space, accelerates convergence, and improves interpretability.

[0021] It should be noted that user-level and group-level price sensitivity curves are constructed based on internal business data and historical price-response samples, a derivative of the price sensitivity curve is generated, and pricing status features are constructed by combining multi-dimensional business features; pricing status features are constructed (price sensitivity curve derivative plus multi-dimensional business features, outputting a vector representation of the current environment / state).

[0022] It should be noted that by deeply embedding price-sensitive parameters and derived quantities into the entire process of RL state construction, action space design, and reward calculation, rather than just using them as auxiliary inputs, the exploration space is reduced from the root, the interpretability of decisions is improved, and the problems of black box decision-making and slow convergence in existing RL pricing are solved.

[0023] It should be noted that the main purpose of constructing the price sensitivity curve is to depict the relationship between price and purchase probability. It also enables online updates at the user and group levels. For users with a sufficient sample size, a maximum likelihood estimate is performed using the historical "price - whether a transaction was completed" ratio, with the specific formula as follows: ;

[0024] in, This represents the price sensitivity curve, which is the user's... In the user group Below is a price list The probability of purchasing at that time; For the sigmoid function, The price of the commodity is used as the independent variable in the model. Let the slope parameter of the curve be constrained to... To avoid the curve being too flat (users are not price-sensitive) or too steep (training is unstable). The inflection point (median price) of the price sensitivity curve satisfies... =0.5, that is, when the price equals At that time, the probability of a user making a purchase was exactly 50%. Generally, the price is limited to the actual marketable price range to prevent the model from learning meaningless inflection points beyond the business scope; the price is higher than... When the probability of purchase is less than 0.5; when the price is lower than... At that time, the probability of purchasing is >0.5.

[0025] It should be noted that, A curve with a value less than 0.1 is almost horizontal; the flatter the curve, the less sensitive users are to price. The price sensitivity curve (>10) approximates a step; the steeper the curve, the more sensitive users are to price. The core parameters of the price sensitivity curve are... The estimation primarily relies on historical transaction behavior from internal business data. External environmental data (such as competitive pricing and risk events) can be used as contextual features to dynamically shift or scale the curve, but do not directly parameterize the maximum likelihood estimation mentioned above.

[0026] The final fitting result is as follows Figure 2 As shown, users with insufficient samples are defined as cold-start users, and their respective cluster parameters are used. , ) as a priori; , The initial constraints are [0.1, 10] and [ , Based on the above estimates, the derived parameters are... and price sensitivity curve; among which, This is the inflection point, satisfying... and current price The slope f' at the point The above quantities are then incorporated into the subsequent state vector.

[0027] It should be noted that the criteria for determining a cold start are: the number of price-response pairs within the most recent window. If the available price is insufficient (e.g., 30), it is considered a cold start.

[0028] Cold start users are segmented according to business criteria (e.g., category × region × channel × customer level) to obtain the group to which the user belongs.

[0029] It should be noted that due to the insufficient sample size (<30), the two parameters cannot be reliably estimated. Therefore, users starting from scratch directly use the aggregation parameters of the corresponding subgroup as initial parameters.

[0030] The basic models used are mainly divided into two categories: a) Logical type: ,in, It is sigmoid. >0 represents the slope parameter (sensitivity coefficient). This is the inflection point (median price). b) Piecewise linear / spline type: In [ , Set node set on ] Price sensitivity curve fitted based on node set .

[0031] It should be noted that, in order to improve the real-time performance of the model, the price sensitivity curve is incrementally corrected online each time pricing is executed. And obtain user response After ∈{0,1}, the incremental gradient of the parameters is based on the log-likelihood function. Perform iterative updates, setting the learning rate η∈[ , For single-point extreme samples, Huber loss or gradient truncation is used to suppress abnormal perturbations; for parameter oscillations within a short time window, a momentum term m∈[0.8,0.95] is introduced for smoothing optimization.

[0032] It should be noted that dynamic pricing requires comprehensive consideration of five dimensions: users, products, external environment, historical behavior, and business constraints. Basic features are used to characterize attributes such as user purchasing power, loyalty, and price resistance; the price sensitivity derivative directly connects to the price-purchase probability curve fitted earlier, injecting explicit quantitative features of user price sensitivity into the state vector, rather than pure black-box features.

[0033] State feature construction: construction time State vector: This is used to build data models of pricing response relationships in real-world environments. The sub-features of each dimension are defined as follows: 1) (User characteristics): Recent M-day consumption (M e.g., 30 days), membership level, historical average revenue per user (ARPU), near-term rejection, unsubscribe flag, price sensitivity derived characteristics. ,in, The inflection point (median price) of the price sensitivity curve. For the current price The probability of purchasing; 2) (Product characteristics): cost, inventory level, shelf life, and slow-moving indicators; 3) (Environmental characteristics): Competitor price range [ [Activity and holiday logos, platform discount intensity;] 4) (Contextual Features): Historical Price Series Price fluctuation range Near-window pricing—response trajectory embedding; 5) (Constraints and Risks): Price Upper and Lower Limits [ , Maximum price adjustment range Price-sensitive product category indicator .

[0034] Feature processing methods: numerical features are standardized using z-score; categorical features are embedded; time-related features (such as holidays and days of the week) are binarized or periodically encoded; all features are concatenated and then fed into the policy network.

[0035] Specifically, a derived value for the price sensitivity curve is added: For user characteristics, a derived value for the price sensitivity curve is set. , This is for use in subsequent decision-making steps.

[0036] It should be noted that by constructing a potential function based on the squared deviation from the anchor price and adding a reward, the policy invariance condition of Ng–Harada–Russell is satisfied, which accelerates learning and suppresses price fluctuations without changing the original optimal MDP policy.

[0037] S130: Construct a pricing action space based on pricing state characteristics, determine business constraints, and set a multi-objective reward function.

[0038] In some embodiments, the implementation of step S130 (constructing a pricing action space based on pricing state characteristics, determining business constraints, and setting a multi-objective reward function) may include: It should be noted that anchor prices are derived using price sensitivity curves, and data-driven adaptive optimization is employed. Based on these anchor points, multi-objective reinforcement learning decision-making with constraints is implemented to achieve price stabilization penalties and monitoring and alarm mechanisms, thus suppressing drastic price fluctuations. Multi-objective unification and price stabilization control: Profit, inventory conversion, and price fluctuations / risk penalties constitute a multi-objective reward system. The weight of fluctuation penalties is dynamically adjusted adaptively to suppress price jumps in sensitive areas, reducing short-term price jumps and customer complaints without sacrificing long-term returns.

[0039] Determine the action space and business constraints: The main purpose of this step is to define the reinforcement learning output, i.e., the action space. The action space can be defined using the following two methods: 1) Discrete range: Divide the price range [ , Divide into L tiers (equal intervals or equal tiers based on gross profit margin), actions ∈{1,2,..,L}; where, The minimum price. The maximum price; At time t, the reinforcement learning agent outputs a decision action. 2) Continuous value: Model output price adjustment range Or directly output the price. The final execution price, through truncation constraints, is as follows: ;

[0040] in, The final execution price of the product at time t; The price of the product at time t-1; The real-time price adjustment range at time t; The minimum price. The maximum price; The truncation constraint function forces the calculated price to be limited to [ , Within the specified range, prevent exceeding the boundary.

[0041] Based on the price sensitivity curve output above, combined with cost... A profit approximation function can be constructed: ;

[0042] in, The unit cost of the commodity; This is an approximate function of the unit profit for commodity prices; Price sensitivity curve; For commodity prices; Solving the first-order optimality condition for this profit approximation function yields the profit-maximizing approximate anchor price. Therefore, policy learning can be optimized into residual learning around the anchor price: the model only learns small price adjustment amounts. Instead of directly learning prices across the entire range, this significantly reduces the exploration cost and output variance of reinforcement learning.

[0043] Additional business and security constraints are also required, with the specific constraint rules as follows: 1) Cost protection constraints: ,in, To achieve the minimum gross profit margin, it can be configured as 0.1 to 2 yuan or a relative gross profit margin of 3%; 2) Channel consistency constraint: The same product in different channels , The price meets ;in, The first threshold for the maximum price difference between channels is 1% to 5%; Let be the selling price of the product at time t and through the i-th sales channel; Let be the selling price of the product at time t and through the j-th sales channel; 3) Price fluctuation constraints: Limits the daily and weekly price adjustment range; daily price adjustment range... Weekly price spread ;in, This represents the absolute amount of the actual price adjustment on a single day. This represents the maximum daily price adjustment limit; The historical selling price was one week ago (7 cycles ago); This is the maximum weekly price fluctuation threshold, limiting the overall price difference range over 7 days.

[0044] It should be noted that the price sensitivity curve is used as the core foundation of reinforcement learning pricing, and the price sensitivity parameter becomes part of the state to derive and generate the anchor price; a potential function is constructed based on the anchor to shape the reward and accelerate convergence; a constraint consistency mechanism is proposed to achieve safe pricing; a weighted multi-objective reward system is constructed with profit, conversion, inventory orientation and price fluctuation and risk penalty, and the dynamic balance of the system is maintained under the multi-objective reward and adaptive weight adjustment, combined with business scenarios.

[0045] Multi-objective reward setting: This step mainly completes the construction of the multi-objective reward function, where the basic reward is defined as follows: ;

[0046] in, For multi-objective rewards; For profit incentives, Profit weighting: , The actual transaction volume is determined by the price sensitivity curve. Approximate fitting; This represents the difference in conversion rates. As a conversion rate weight, And the difference between the current conversion rate and the benchmark conversion rate; For inventory-oriented reward weighting, This is an inventory-oriented incentive program that provides additional positive rewards for lower pricing in scenarios with high inventory or near-expiry products. For price stability constraints, As a weight for price stability, , This indicates the price adjustment range for the current period relative to the previous period; As risk constraint weights, This is a risk constraint item, which deducts items for scenarios such as approaching regulatory red lines, suspected arbitrage groups, and malicious cancellation patterns, with a value range of [0,1].

[0047] It should be noted that the potential function reward shaping is introduced into the immediate reward: the anchor price is used to maximize profit. Construct a potential function and superimpose it into the reward function: Potential function: ;

[0048] in, Let be the potential function. To maximize profits, the anchor price To enhance learning of the current environment (including real-time supply and demand, costs, competitor pricing, and other pricing environment information); This is the weight adjustment coefficient; a positive number used to control the constraint strength of the potential function reward. Reward function correction term: ;

[0049] in, This is the shaping reward correction term corresponding to the potential function; The agent's current action (price adjustment action); The current environmental state before the action is performed; To perform the action The environmental state at the next moment after the transition; To reinforce the learning discount factor (satisfying 0 < γ ≤ 1); Let be the state potential function; Current state The corresponding potential function value; State at the next moment The corresponding potential function value.

[0050] The final reward function is: ; in, This is the original instant reward function; The shaping correction reward is for the aforementioned price deviation constraint; This is the final optimized reward function after the superposition potential function is corrected.

[0051] It should be noted that, according to the Ng–Harada–Russell theorem, this shaping does not change the optimal strategy of the Markov decision process, but significantly accelerates the convergence of the strategy to the neighborhood of the anchor price; its strength is normalized and amplitude calibrated to avoid overshadowing the real business returns, and works in conjunction with constraints such as cost protection, channel price difference, and price fluctuation limit.

[0052] S140: Based on pricing state features, pricing action space, business constraints, and multi-objective reward functions, the reinforcement learning strategy is pre-trained offline and updated online; and the weights of the multi-objective reward function are adaptively adjusted during the update.

[0053] In some embodiments, the implementation of step S140 (offline pre-training and online updating of the reinforcement learning strategy based on pricing state features, pricing action space, business constraints, and multi-objective reward function; and adaptively adjusting the weights of the multi-objective reward function during the update) may include: It should be noted that during the training phase, constrained reinforcement learning such as CPO is used to ensure that the iteration is approximately feasible; during the execution phase, candidate actions are strongly clipped (clip / projected) according to the same criteria to eliminate the "training-deployment" bias and ensure safe deployment.

[0054] The process of training and updating reinforcement learning strategies online is as follows: Figure 3 As shown; The specific offline pre-training steps are as follows: Utilizing historical interaction data Pre-train the policy network (or Q-network) to obtain the initial policy π, which provides a foundation for subsequent online exploration; Online exploration and experience playback: Based on the initial strategy π, online scenario exploration is carried out, new interaction experiences are collected and played back and stored simultaneously, and the pricing strategy is continuously optimized through experience reuse, ultimately obtaining a pricing optimization strategy that adapts to the actual scenario; Incremental Updates and Convergence Monitoring: Mini-batch incremental learning is employed, with an update frequency of daily or one batch per 1000 samples. The learning rate is set to... Set a sliding window K (e.g., 7 days) for convergence monitoring. If the profit margin and conversion rate (CR) are both lower than the baseline threshold within the sliding window K, trigger the strategy rollback mechanism (roll back to the previous effective strategy or adopt a hybrid strategy) to ensure stable strategy convergence.

[0055] Network structure design: The policy network and value (or Q) network adopt a 2-4 layer fully connected structure, and the activation function is ReLU; for the Actor-Critic architecture with continuous actions, a parameter sharing structure or a dual-tower separation structure can be adopted; to prevent model overfitting and improve exploration stability, weight decay (L2) is introduced.

[0056] Algorithm selection: Select the appropriate algorithm based on the action space type. If the action space is discrete, the DQN algorithm is preferred for Q-value learning; if the action space is continuous, the PPO algorithm is preferred for policy-value joint optimization.

[0057] Exploration strategy design: Exploration strategies are designed based on action space type: 1) For discrete action spaces, an ϵ-greedy exploration strategy is adopted, with an early exploration rate of... 1) The mean gradually decreases to 0.01 in the later stage; 2) For continuous action space, a zero-mean Gaussian perturbation is added to the output of the Actor network, and the standard deviation of the perturbation σ decreases from 0.05 to 0.01. Experience replay optimization: Priority experience replay (PER) is adopted, and sampling is performed with TD-error as the sample priority. At the same time, importance sampling correction is provided to improve sample utilization efficiency and accelerate policy convergence.

[0058] Adaptive weighting: The weights are dynamically adjusted based on business scenarios to adapt to different needs. During promotional periods, the weights of w2 and w3 are increased to prioritize promotional effectiveness; during off-seasons, the weights of w1 and w4 are increased to prioritize revenue and user stability. After the weights are adjusted, they need to be recalibrated through small-sample gray-scale testing, which solves problems such as price fluctuations and high customer complaints driven by a single objective.

[0059] It should be noted that, in order to address the insufficient exploration of high-uncertainty scenarios and the cold start problem of new products, it is necessary to maintain the posterior distribution and variance of price-sensitive parameters and actively explore high-uncertainty users / products. In the new product stage, a conservative anchor point is generated based on "cost - competitor price range - prior price elasticity", and the price range is gradually broadened in the future to improve early revenue ramp-up and risk controllability.

[0060] S150: Outputs price decisions based on the updated reinforcement learning policy, while simultaneously monitoring for anomalies to achieve multi-objective dynamic pricing.

[0061] In some embodiments, step S150 (outputting price decisions based on the updated reinforcement learning policy while simultaneously monitoring for anomalies to achieve multi-objective dynamic pricing) includes: Price Decision and Output: This step mainly processes the strategy output results. The specific steps are as follows: 1) Output candidate actions: Output candidate price actions according to the policy function determined in the training steps above. (Such as price adjustment volume or discrete price levels); 2) Price-price prior selection: Candidate actions are selected based on the business constraints defined above. Perform prior trimming to obtain the final execution price. Ensure that the output price is safe and explainable; 3) Generate explanatory elements: Simultaneously generate explanatory elements for price decisions, including sensitive parameter pairs. The contribution value of inventory impact and the contribution value of competitors impact; 4) Data Storage Fields: Each decision record stores the following fields for offline analysis and training sample construction: { ,reason_code,weights,policy_version}; in, This represents the current business status. For candidate price action, For the final execution price, The reward value is represented by reason_code, the decision reason label, weights, the current policy weights or priority sampling weights, and policy_version, which is the policy version number. 5) Gather actual feedback: Online pricing. Afterwards, collect actual user feedback (successful / failed transactions, added to cart / abandoned purchases, complaints / cancellations) to form a complete reinforcement learning sample. ), used for subsequent incremental updates of the strategy; among which, This is the updated state.

[0062] Monitoring and Alarms: In order to maintain the stable operation of the system, it is necessary to monitor multiple indicators such as profit margin, price volatility, channel compliance rate, and customer complaint rate. When any indicator exceeds the preset threshold (the preset threshold is 95%), an alarm will be triggered immediately. It should be noted that the following actions are automatically executed after an alarm: pause policy network updates; revert to the previous stable policy version; and send a notification to the operations and algorithm teams.

[0063] Based on the same inventive concept, embodiments of this application also provide a multi-objective dynamic pricing device based on reinforcement learning and price sensitivity, including: 1) Data Acquisition Unit: Captures and cleans multi-source data in a step-by-step manner; 2) Price sensitivity modeling unit: Trains and updates the price sensitivity function online; 3) State Construction Unit: Standardize / encode features such as user, product and environment, and concatenate them with price-sensitive parameters to form a time-state vector for use by the policy network; 4) Strategy Decision Unit: Based on st, candidate pricing actions are generated through parameterized strategies (or near-greedy strategies derived from Q functions), and are connected with anchor prices and exploration / replay mechanisms; 5) Constraint Control Unit: Enforces business constraints such as lower cost limit, channel difference, and upper fluctuation limit before and after action output; 6) Reward Calculation Unit: Based on multi-objective criteria, profit, conversion, inventory, and price stabilization penalties are combined into an optimizable real-time reward (or a combination of main objective and constraints) to provide stable signals for strategy learning; 7) Policy update unit: Iterates online according to the rhythm of small-batch incremental learning and target network / soft update; Interpretation and Output Unit: Generates traceable elements (price-sensitive parameters, elasticity / inflection point, reason code) for the final price and stores them in the database; 8) Monitoring and Alarm Unit: Continuously monitors key indicators such as profit margin, CR, price fluctuations and channel compliance rate, and sets alarms in combination with anomaly detection to ensure online stability.

[0064] The interactions between each unit are as follows: Figure 4 As shown: The data acquisition unit continuously gathers user, product, environmental, and behavioral data for downstream use; the price sensitivity modeling unit fits a price-purchase probability curve based on the data and outputs price sensitivity parameters and anchor prices, which are then fed to the strategy decision-making unit and the state construction unit, respectively; the state construction unit generates a time-state vector after incorporating price sensitivity elements, historical price-response, and contextual information, and submits it to the strategy decision-making unit; under the mandatory constraints of cost, channel difference, and volatility given by the constraint unit, the strategy decision-making unit generates candidate pricing based on the state vector, and together with the anchor price-potential function, forms a potential potential shaping signal, which is then sent to the reward calculation unit. The reward calculation unit calculates real-time rewards according to multiple objectives (profit, conversion, inventory balance, and price stabilization penalties, etc.) and sends them back to the strategy update unit. Based on this, the strategy update unit incrementally learns the strategy network and generates updated strategies, which are then fed back to the strategy decision unit. The interpretation and output unit performs compliance verification and interpretable generation (including price-sensitive parameters and inflection point deviations) on the final execution price and outputs it to the business side. The monitoring and alarm unit continuously monitors online profit margin, conversion rate, price fluctuations, and compliance hit rate, generates monitoring results for alarms and triggering rollback / limiting, and synchronously feeds back to the strategy update unit and the strategy decision unit, thus forming an adaptive strategy-feedback-update closed loop.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-objective dynamic pricing method based on reinforcement learning and price sensitivity, characterized in that, include: Acquire multi-source data, including internal business data and external environment data; A price sensitivity curve is constructed based on the internal business data. The price sensitivity curve is processed to generate corresponding derived quantities. Pricing status features are then constructed by combining multi-dimensional business features. Based on the pricing state characteristics, a pricing action space is constructed, business constraints are determined, and a multi-objective reward function is set. Based on the aforementioned pricing state characteristics, pricing action space, business constraints, and multi-objective reward function, the reinforcement learning strategy is pre-trained offline and updated online. And during the update, the weights of the multi-objective reward function are adaptively adjusted; The updated reinforcement learning strategy is used to output price decisions and monitor for anomalies, thereby achieving multi-objective dynamic pricing.

2. The multi-objective dynamic pricing method based on reinforcement learning and price sensitivity according to claim 1, characterized in that, The construction of the price sensitivity curve based on the internal business data includes: The expression for the price sensitivity curve constructed based on the aforementioned internal business data is as follows: ； in, Indicates user In the user group Below is a price list The probability of purchasing at that time; For the sigmoid function, For commodity prices; This is the slope parameter of the curve; This represents the inflection point of the price sensitivity curve; Parameters are obtained by performing maximum likelihood estimation based on the historical price and transaction data of the aforementioned internal business data. and .

3. The multi-objective dynamic pricing method based on reinforcement learning and price sensitivity according to claim 1, characterized in that, The process of processing the price sensitivity curve to generate corresponding derived quantities, and combining them with multi-dimensional business characteristics to construct pricing status features, includes: The price sensitivity curve is processed to generate a corresponding derived quantity, which includes at least a slope parameter, an inflection point, a purchase probability at the current price, and a slope at the current price. Obtain product features, environmental features, contextual features, and constraint and risk features; embed the derived values ​​into user features, and combine them with the product features, environmental features, contextual features, and constraint and risk features to construct a concatenated vector of pricing status features: .

4. The multi-objective dynamic pricing method based on reinforcement learning and price sensitivity according to claim 1, characterized in that, The process of constructing a pricing action space based on the pricing state characteristics, determining business constraints, and setting a multi-objective reward function includes: Based on the pricing state characteristics, a pricing action space is defined, and an anchor price is generated according to the price sensitivity curve. A constrained multi-objective reinforcement learning decision is then executed based on the anchor price. The business constraints include at least cost protection constraints, channel consistency constraints, and price fluctuation constraints; wherein, the cost protection constraint limits the execution price to no less than the sum of cost and minimum gross profit margin, the channel consistency constraint limits the price difference of the same product in different channels to no more than a first threshold, and the price fluctuation constraint limits the upper limit of daily and weekly price adjustment range.

5. The multi-objective dynamic pricing method based on reinforcement learning and price sensitivity according to claim 1, characterized in that, The process of constructing a pricing action space based on the pricing state characteristics, determining business constraints, and setting a multi-objective reward function further includes: The multi-objective reward function is in the form of a weighted combination: ； in, For multi-objective rewards; For profit incentives, Weighted by profit; This represents the difference in conversion rates. Conversion rate weighting; For inventory-oriented reward weighting, Inventory-oriented incentive items; For price stability constraints, As a weight for price stability; As risk constraint weights, This is a risk constraint.

6. The multi-objective dynamic pricing method based on reinforcement learning and price sensitivity according to claim 1, characterized in that, The offline pre-training and online updating of the reinforcement learning strategy based on the pricing state features, pricing action space, business constraints, and multi-objective reward function includes: Acquire historical interaction data, use the historical interaction data to perform offline pre-training on the policy network to obtain an initial policy; explore online scenarios based on the initial policy, collect new interaction experiences synchronously and store them in the experience replay pool, and continuously optimize the policy through experience reuse. Online updates are performed using incremental learning in small batches, and convergence monitoring is set up using a sliding window. If the profit margin and conversion rate are both lower than the baseline threshold within the sliding window, a strategy rollback mechanism is triggered to roll back to the previous effective strategy or adopt a hybrid strategy.

7. The multi-objective dynamic pricing method based on reinforcement learning and price sensitivity according to claim 1, characterized in that, The offline pre-training and online updating of the reinforcement learning strategy based on the pricing state features, pricing action space, business constraints, and multi-objective reward function also includes: The weights in the multi-objective reward function are dynamically adjusted according to the business scenario, and then recalibrated through small-sample gray-scale testing after the weights are adjusted. The appropriate reinforcement learning algorithm is selected based on the action space type: the DQN algorithm is used in combination with the ϵ-greedy exploration strategy for discrete action spaces, and the PPO algorithm is used in combination with the Gaussian perturbation exploration strategy for continuous action spaces. Priority Experience Replay (PER) is used, with TD error as the sample priority for sampling, supplemented by importance sampling correction.

8. The multi-objective dynamic pricing method based on reinforcement learning and price sensitivity according to claim 1, characterized in that, The process of outputting price decisions based on the updated reinforcement learning strategy, while simultaneously monitoring for anomalies to achieve multi-objective dynamic pricing, includes: Based on the updated reinforcement learning strategy, candidate price actions are output; The candidate price action is pruned a priori according to the business constraints to obtain the final execution price; Simultaneously generate explanatory elements for price decisions, including sensitivity parameter pairs. The contribution of inventory and the contribution of competitors; Each price decision record is stored in the database, with the following fields: current business status, candidate price action, final execution price, reward value, decision reason tag, strategy weight, and strategy version number. After executing the final price online, collect actual user interaction feedback to form a complete reinforcement learning sample for incremental policy updates.

9. The multi-objective dynamic pricing method based on reinforcement learning and price sensitivity according to claim 1, characterized in that, The method of outputting price decisions based on the updated reinforcement learning strategy, while simultaneously monitoring for anomalies to achieve multi-objective dynamic pricing, further includes: The system acquires profit margin, price volatility, channel compliance rate, and customer complaint rate, and monitors at least one of these indicators in real time; when any of these indicators exceeds a preset threshold, an alarm is immediately triggered.

10. A multi-objective dynamic pricing device based on reinforcement learning and price sensitivity, used to implement the multi-objective dynamic pricing method based on reinforcement learning and price sensitivity as described in any one of claims 1-9, characterized in that, include: Data acquisition unit: used to acquire data from multiple sources; Price sensitivity modeling unit: used to train and update the price sensitivity function online; State building unit: used to standardize features such as users, products and environment, and concatenate them with price-sensitive parameters to form a time-state vector; Strategy Decision Unit: Used to generate candidate pricing actions through parameterized strategies and to connect with anchor prices and exploration and replay mechanisms; Constraint and control unit: Used to enforce business constraints such as lower cost limit, channel difference, and upper fluctuation limit before and after action output; Policy update unit: used for incremental learning and updating the target network using mini-batch incremental learning; Monitoring and alarm unit: Used to continuously monitor key indicators and set alarms in conjunction with anomaly detection.

Images