E-commerce marketing management method and device based on reinforcement learning, and electronic equipment

Abstract

The invention provides an e-commerce marketing management method and device based on reinforcement learning and electronic equipment, and relates to the technical field of marketing management, and the method comprises the steps: carrying out the strict time delay processing of historical transaction data and historical interaction data, and obtaining business interpretable features; executing two-stage inverse reinforcement learning processing based on the business interpretable features to obtain reference marketing strategy distribution and corresponding decision reason codes; under a preset business constraint condition, performing iterative updating on the baseline strategy model by adopting a group relative strategy optimization method to obtain an optimized marketing strategy model; performing quantitative estimation of expected income and risk on the optimized marketing strategy model by using an off-line anti-fact evaluation method; and based on the quantitative estimation result and the real-time business constraint state, using the optimized marketing strategy model to generate and output a session-level promotion decision and a corresponding decision reason code on line. According to the invention, the generation and online execution of the e-commerce marketing strategy which can be explained, audited and verified offline are realized.

Description

Technical Field

[0001] This invention relates to the field of marketing management technology, and in particular to an e-commerce marketing management method, apparatus, and electronic device based on reinforcement learning. Background Technology

[0002] Most existing promotional systems employ rule engines, traditional response models, or recommendation systems. Rule engines struggle to adapt to seasonal / inventory fluctuations, recommendation systems prioritize matching over gating of promotions and explicit constraints on promotional costs, and supervised response models lack post-launch strategy stability and controllable safeguards.

[0003] While online reinforcement learning is adaptive, its exploration on real users carries the risks of over-subsidization and fatigue accumulation. Furthermore, it is difficult to provide financial / compliance-oriented explanations and uncertainties, leading to significant approval resistance.

[0004] Offline assessments often rely on A / B or simple replay, without systematically employing offline counterfactual assessment (OPE) techniques such as inverse bias weighting (IPS / SNIPS) to control bias and variance.

[0005] Multi-channel outreach (such as SMS + email + APP pop-ups) often leads to cross-channel fatigue and conflicts between multiple objectives such as inventory, profit and subsidies, lacking a unified and constrained optimization framework.

[0006] Therefore, a reinforcement learning-based e-commerce marketing management method, device, and electronic device are proposed. Summary of the Invention

[0007] This specification provides an e-commerce marketing management method, device, and electronic device based on reinforcement learning, which realizes the generation and online execution of e-commerce marketing strategies that are interpretable, auditable, and verifiable offline.

[0008] This manual provides a reinforcement learning-based e-commerce marketing management method, including: Obtain historical transaction data and historical interaction data; Strict time-delay processing is applied to the historical transaction data and the historical interaction data to obtain business interpretable features; Based on the interpretable features of the business, a two-stage inverse reinforcement learning process is performed to obtain the reference marketing strategy distribution and the corresponding decision reason code. Using the aforementioned reference marketing strategy distribution as a supervision signal, a baseline strategy model is obtained through list distillation training; Under preset business constraints, the baseline strategy model is iteratively updated using a group relative strategy optimization method to obtain an optimized marketing strategy model. The group relative strategy optimization adopts an intra-group relative advantage calculation method centered on the reference marketing strategy distribution. The expected returns and risks of the optimized marketing strategy model are quantitatively estimated using an offline counterfactual assessment method. Based on the quantitative estimation results and real-time business constraint status, the optimized marketing strategy model is used to generate and output session-level promotional decisions and corresponding decision reason codes online.

[0009] Optionally, the step of performing strict time-delay processing on the historical transaction data and the historical interaction data to obtain business-explainable features includes: Identify future leakable signals in the historical transaction data and historical interaction data, and align the occurrence time of the future leakable signals to the start timestamp of the business session to which they belong; Based on the aligned data, using information whose timestamp is no later than the start timestamp of the business session, a business interpretable feature corresponding to the business session is constructed.

[0010] Optionally, the step of performing a two-stage inverse reinforcement learning process based on the business interpretable features to obtain the reference marketing strategy distribution and the corresponding decision reason code includes: Based on the session-level interpretable features of the business, a first model is trained to predict the probability of performing a promotional action in the current session state. For a session deemed suitable for executing a promotional action, a second model is trained based on the features of the session and promotional action combination level to predict the preference probability distribution on the set of feasible promotional actions, as the reference marketing strategy distribution. The training objectives of both the first model and the second model are based on expert decision-making behavior derived from the historical transaction data and the historical interaction data.

[0011] Optionally, training the first model based on the session-level interpretable features to predict the probability of performing a promotional action in the current session state includes: in, σ(·) is the probability of performing a promotional action given the business interpretable feature x; σ(·) is the Sigmoid function; x∈R d It is a business-interpretable feature vector of dimension d; It is a trainable weight vector; y∈{0,1} is the true label, where 1 indicates that a promotion has occurred and 0 indicates that no promotion has occurred; It is the cross-entropy loss; λ>0 is the L2 regularization coefficient; It is the squared L2 norm of the weight vector, used to prevent overfitting.

[0012] Optionally, for a session deemed suitable for executing a promotional action, a second model is trained based on the features of the session and promotional action combination to predict the preference probability distribution on the set of feasible promotional actions, as the reference marketing strategy distribution, including: For each action k in the set of feasible promotional actions, construct an extended feature vector: in, This is the extended feature vector corresponding to action k; It is an indicator vector, identifying the current candidate action as the k-th one; ⊙ is the Hadamard product; The reference distribution is modeled using listwise softmax to obtain the predicted probability distribution of preferences on the set of feasible promotional actions: in, z represents the probability distribution of preferences for choosing action k given x; k Rate action k; The score is given for the j-th feasible offer; τ>0 is a temperature parameter that controls the smoothness of the distribution. For the model objective; Choose the discount when x is given The probability distribution of preferences.

[0013] Optionally, the step of using the reference marketing strategy distribution as a supervision signal to train the baseline strategy model through list distillation includes: Train a neural scorer with parameter θ. For each action k in the set of feasible promotional actions, the scorer outputs a predicted score: in, The predicted score for action k; (·) represents a neural network with parameter θ; The training loss function of the baseline policy model is a list-based cross-entropy loss, expressed as: in, For list-based cross-entropy loss, Distribution of teachers; Predict the distribution of students; This refers to the temperature parameter.

[0014] Optionally, the step of using the reference marketing strategy distribution as a supervision signal to train the baseline strategy model through list distillation further includes: Construct a fused teacher distribution, and use this fused teacher distribution as a supervision signal to train the baseline policy model through list distillation: in, As the baseline strategy model, The teacher strategy is based on business rules; α∈[0,1] is the fusion weight.

[0015] This specification provides an e-commerce marketing management device based on reinforcement learning, comprising: The acquisition module is used to acquire historical transaction data and historical interaction data; The processing module is used to perform strict time-delay processing on the historical transaction data and the historical interaction data to obtain business interpretable features; The learning module is used to perform a two-stage inverse reinforcement learning process based on the interpretable features of the business to obtain the reference marketing strategy distribution and the corresponding decision reason code; The distillation module is used to train a baseline strategy model by using the reference marketing strategy distribution as a supervision signal and distilling a list to obtain the model. The update module is used to iteratively update the baseline strategy model under preset business constraints using a group relative strategy optimization method to obtain an optimized marketing strategy model. The group relative strategy optimization adopts an intra-group relative advantage calculation method centered on the reference marketing strategy distribution. The estimation module is used to quantitatively estimate the expected returns and risks of the optimized marketing strategy model using an offline counterfactual evaluation method. The output module is used to generate and output session-level promotional decisions and corresponding decision reason codes online using the optimized marketing strategy model, based on the quantitative estimation results and real-time business constraint status.

[0016] Optionally, the processing module includes: Identify future leakable signals in the historical transaction data and historical interaction data, and align the occurrence time of the future leakable signals to the start timestamp of the business session to which they belong; Based on the aligned data, using information whose timestamp is no later than the start timestamp of the business session, a business interpretable feature corresponding to the business session is constructed.

[0017] Optionally, the learning module includes: Based on the session-level interpretable features of the business, a first model is trained to predict the probability of performing a promotional action in the current session state. For a session deemed suitable for executing a promotional action, a second model is trained based on the features of the session and promotional action combination level to predict the preference probability distribution on the set of feasible promotional actions, as the reference marketing strategy distribution. The training objectives of both the first model and the second model are based on expert decision-making behavior derived from the historical transaction data and the historical interaction data.

[0018] Optionally, training the first model based on the session-level interpretable features to predict the probability of performing a promotional action in the current session state includes: in, σ(·) is the probability of performing a promotional action given the business interpretable feature x; σ(·) is the Sigmoid function; x∈R d It is a business-interpretable feature vector of dimension d; It is a trainable weight vector; y∈{0,1} is the true label, where 1 indicates that a promotion has occurred and 0 indicates that no promotion has occurred; It is the cross-entropy loss; λ>0 is the L2 regularization coefficient; It is the squared L2 norm of the weight vector, used to prevent overfitting.

[0019] Optionally, for a session deemed suitable for executing a promotional action, a second model is trained based on the features of the session and promotional action combination to predict the preference probability distribution on the set of feasible promotional actions, as the reference marketing strategy distribution, including: For each action k in the set of feasible promotional actions, construct an extended feature vector: in, This is the extended feature vector corresponding to action k; It is an indicator vector, identifying the current candidate action as the k-th one; ⊙ is the Hadamard product; The reference distribution is modeled using listwise softmax to obtain the predicted probability distribution of preferences on the set of feasible promotional actions: in, z represents the probability distribution of preferences for choosing action k given x; k Rate action k; The score is given for the j-th feasible offer; τ>0 is a temperature parameter that controls the smoothness of the distribution. For the model objective; Choose the discount when x is given The probability distribution of preferences.

[0020] Optionally, the distillation module includes: Train a neural scorer with parameter θ. For each action k in the set of feasible promotional actions, the scorer outputs a predicted score: in, The predicted score for action k; (·) represents a neural network with parameter θ; The training loss function of the baseline policy model is a list-based cross-entropy loss, expressed as: in, For list-based cross-entropy loss, Distribution of teachers; Predict the distribution of students; This refers to the temperature parameter.

[0021] Optionally, the distillation module further includes: Construct a fused teacher distribution, and use this fused teacher distribution as a supervision signal to train the baseline policy model through list distillation: in, As the baseline strategy model, The teacher strategy is based on business rules; α∈[0,1] is the fusion weight.

[0022] This specification also provides an electronic device, wherein the electronic device includes: A processor; and a memory storing computer-executable instructions, which, when executed, cause the processor to perform any of the methods described above.

[0023] This specification also provides a computer-readable storage medium that stores one or more programs that, when executed by a processor, implement any of the methods described above.

[0024] This invention first ensures impartiality from the data source by rigorously processing historical data with time delays, eliminating model bias caused by future information leaks and laying a reliable foundation for decision-making. Second, it achieves transparency and explainability in the decision-making process. Through a two-stage inverse reinforcement learning framework, it not only accurately mimics the complex logic of experts in deciding whether to promote a product or choose which discount, but also outputs business-understandable decision reason codes, greatly satisfying the needs of compliance audits and rapid approval. Third, it constructs a secure and controllable strategy iteration path, following the principle of first mimicking experts and then constraining optimization, starting with a secure baseline strategy. Fine-tuning within multiple business safeguards, such as budget and inventory, effectively avoids the waste of subsidies and user experience risks caused by blind online exploration. At the same time, an offline verifiable evaluation system was established. Utilizing advanced offline counterfactual evaluation technology, the system can quantitatively estimate the benefits and confidence intervals of strategies before they go live, transforming trial-and-error deployment into scientific verification and significantly reducing decision-making risks. Ultimately, an efficient and robust engineering closed loop is formed. Through mechanisms such as model distillation and dual-clock execution, it balances the high efficiency and low latency of online services with real-time response to dynamic business constraints, realizing the generation and online execution of e-commerce marketing strategies that are interpretable, auditable, and verifiable offline. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram illustrating the principle of an e-commerce marketing management method based on reinforcement learning, provided for embodiments of this specification; Figure 2 A schematic diagram of the structure of an e-commerce marketing management device based on reinforcement learning, provided for an embodiment of this specification; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification; Figure 4 This is a schematic diagram of a computer-readable medium provided for embodiments of this specification. Detailed Implementation

[0027] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0028] The following is in conjunction with the appendix Figure 1-4 Exemplary embodiments of the invention will be described more fully here. However, exemplary embodiments can be implemented in many forms and should not be construed as limiting the invention to the embodiments set forth herein. Rather, these exemplary embodiments are provided to make the invention more comprehensive and complete, and to facilitate a full communication of the inventive concept to those skilled in the art. The same reference numerals in the figures denote the same or similar elements, components, or parts, and therefore repeated descriptions of them are omitted.

[0029] Subject to the technical concept of this invention, the features, structures, characteristics or other details described in a particular embodiment may be combined in one or more other embodiments in a suitable manner.

[0030] In the description of specific embodiments, the features, structures, characteristics, or other details described in this invention are intended to enable those skilled in the art to fully understand the embodiments. However, it is not excluded that those skilled in the art can practice the technical solutions of this invention without one or more of the specific features, structures, characteristics, or other details.

[0031] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0032] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0033] The terms “and / or” or “and / or” include all combinations of any one or more of the listed items.

[0034] Figure 1 A schematic diagram illustrating the principle of an e-commerce marketing management method based on reinforcement learning, provided in the embodiments of this specification, is shown. This method may include: S110: Obtain historical transaction data and historical interaction data; S120: Perform strict time-delay processing on the historical transaction data and the historical interaction data to obtain business interpretable features; Optionally, S120 includes: Identify future leakable signals in the historical transaction data and historical interaction data, and align the occurrence time of the future leakable signals to the start timestamp of the business session to which they belong; Based on the aligned data, using information whose timestamp is no later than the start timestamp of the business session, a business interpretable feature corresponding to the business session is constructed.

[0035] In the specific implementation of this specification, strict time-delay alignment is performed on possible forward conversion / average order value / category popularity signals to ensure that training samples only use information prior to their timestamps.

[0036] With business-interpretable features as the core: price_sens, churn, engagement, margin_rate, inventory_idx, coupon_elig, seasonality, ret60_lag1, aov_base_lag1, etc., it supports versioning and replay reproduction.

[0037] S130: Perform a two-stage inverse reinforcement learning process based on the business interpretable features to obtain the reference marketing strategy distribution and the corresponding decision reason code; In the specific implementation of this specification, the reason code is generated using a minor syntax (DSL) of “feature contribution → natural language fragment” to form human-readable evidence.

[0038] Optionally, S130 includes: Based on the session-level interpretable features of the business, a first model is trained to predict the probability of performing a promotional action in the current session state. For a session deemed suitable for executing a promotional action, a second model is trained based on the features of the session and promotional action combination level to predict the preference probability distribution on the set of feasible promotional actions, as the reference marketing strategy distribution. The training objectives of both the first model and the second model are based on expert decision-making behavior derived from the historical transaction data and the historical interaction data.

[0039] Optionally, training the first model based on the session-level interpretable features to predict the probability of performing a promotional action in the current session state includes: in, σ(·) is the probability of performing a promotional action given the business interpretable feature x; σ(·) is the Sigmoid function; x∈R d It is a business-interpretable feature vector of dimension d; It is a trainable weight vector; y∈{0,1} is the true label, where 1 indicates that a promotion has occurred and 0 indicates that no promotion has occurred; It is the cross-entropy loss; λ>0 is the L2 regularization coefficient; It is the squared L2 norm of the weight vector, used to prevent overfitting.

[0040] In the specific implementation of this specification, differential privacy noise (DP-SGD) is used for w to protect sensitive logs.

[0041] Optionally, for a session deemed suitable for executing a promotional action, a second model is trained based on the features of the session and promotional action combination to predict the preference probability distribution on the set of feasible promotional actions, as the reference marketing strategy distribution, including: For each action k in the set of feasible promotional actions, construct an extended feature vector: in, This is the extended feature vector corresponding to action k; It is an indicator vector, identifying the current candidate action as the k-th one; ⊙ is the Hadamard product; The reference distribution is modeled using listwise softmax to obtain the predicted probability distribution of preferences on the set of feasible promotional actions: in, z represents the probability distribution of preferences for choosing action k given x; k Rate action k; The score is given for the j-th feasible offer; τ>0 is a temperature parameter that controls the smoothness of the distribution. For the model objective; Choose the discount when x is given The probability distribution of preferences.

[0042] S140: Using the aforementioned reference marketing strategy distribution as a supervision signal, a baseline strategy model is obtained through list distillation training. Optionally, S140 includes: Train a neural scorer with parameter θ. For each action k in the set of feasible promotional actions, the scorer outputs a predicted score: in, The predicted score for action k; (·) represents a neural network with parameter θ; The training loss function of the baseline policy model is a list-based cross-entropy loss, expressed as: in, For list-based cross-entropy loss, Distribution of teachers; Predict the distribution of students; This refers to the temperature parameter.

[0043] Optionally, S140 further includes: Construct a fused teacher distribution, and use this fused teacher distribution as a supervision signal to train the baseline policy model through list distillation: in, As the baseline strategy model, The teacher strategy is based on business rules; α∈[0,1] is the fusion weight.

[0044] S150: Under preset business constraints, the baseline strategy model is iteratively updated using a group relative strategy optimization method to obtain an optimized marketing strategy model, wherein the group relative strategy optimization adopts an intra-group relative advantage calculation method centered on the reference marketing strategy distribution. In the specific implementation of this specification, the basic loss is: in, This is the basic loss; p represents the relative advantage within the group; p is the current strategy, p ref For reference strategy distribution; KL(· ·) represents the KL divergence; c k The cost of performing action k; The entropy of the policy distribution; , All are constant coefficients.

[0045] Advantages of the reference center: Automatically maintain the KL radius and budget cap. Among these, The reward for action k; represents the expected reward under the reference strategy; MAD(r) represents the median absolute deviation of the reward.

[0046] S160: Using an offline counterfactual assessment method, quantitatively estimate the expected returns and risks of the optimized marketing strategy model; S170: Based on the quantitative estimation results and real-time business constraint status, use the optimized marketing strategy model to generate and output session-level promotional decisions and corresponding decision reason codes online.

[0047] In the specific implementation of this instruction manual, the temperature course is as follows: ( Anneal from large to small, first stabilize and then refine.

[0048] Multi-Constraint CMDP Representation: in, For expected reward, Let be the expected cost, c be the cost incurred by the decision, and B be the upper limit threshold of the cost. To obtain the maximum allowable value of the KL divergence, Conditional risk value, To convert the reward r into a loss, This is the upper limit threshold for tail loss.

[0049] Cross-channel fatigue constraints: The fatigue function F is estimated by the Hawkes survival model, and the constraints are... [ ]≤ max Here: E[F]: Expected fatigue across all sessions and channels, representing the average level of "fatigue" experienced by users due to outreach (such as SMS, email, app push notifications, etc.) under a given strategy p. This fatigue value F is typically estimated using Hawkes processes or survival models, for example, measuring the probability of decreased response time or churn risk per unit of time due to excessive outreach. Specifically, it is expressed as: Where u represents the user, F(u) is the user's fatigue score, and P(u) is the probability of the user appearing.

[0050] Fairness regularization: Imposing a distribution difference penalty on specific sensitive slices. Here, γ>0: the fairness regularization coefficient, which controls the weight of this term in the overall loss function. The larger the value, the stronger the constraint on fairness, and the more the strategy will favor the distribution consistency among different user groups.

[0051] D MMD (·,·): Maximum Mean Discrepancy (MMD) is a kernel method for measuring the difference between two probability distributions.

[0052] For any two distributions p and q, it is defined as: in It is a feature map that maps to the reproducing kernel Hilbert space (RKHS). It corresponds to the Hilbert space. Let be the probability distribution of the strategy for the sensitive user subgroup 1 (such as new users, female users, users in a certain region). This represents the policy probability distribution for sensitive user subgroup 2.

[0053] IPS and SNIPS: Output expected boost and confidence interval. In inverse propensity weighted (IPS) estimation: IPS (Inverse Propensity Scoring) is the estimated expected return of a new strategy based on historical log data; n is the total number of samples (number of historical interaction records); i is the sample index (the i-th historical interaction). This is the context feature vector (such as user profile, session features, etc.) corresponding to the iiith interaction. In the context of history The actions taken (such as the actual promotional strategies pushed out); For the target policy (the new policy to be evaluated) in context Select action The probability of; For the behavior strategy (historical log generation strategy) in context Select action The probability of; Actions in historical records The estimator calculates the actual returns (such as clicks, conversions, revenue, etc.). It corrects the historical behavior distribution to the target policy distribution by adding an importance weight π / μ to each historical sample, achieving revenue estimation without the need for online experiments.

[0054] In the self-normalized inverse tendency weighted (SNIPS) estimation formula: SNIPS (Self-Normalized IPS) is an improved version of IPS that reduces variance by normalizing the importance weights. : Importance weight of the i-th sample, measuring the probability ratio between the target policy and the historical behavior policy. The advantage of SNIPS is that it has lower variance and is more stable for extreme weights (when μ is very small, π / μ becomes too large), so it is more widely used in OPE systems in industry.

[0055] Switch / DR extension: When If the variance is too large, switch to dual robust (DR) estimation to reduce variance.

[0056] Cost-benefit joint assessment: simultaneous estimation The Pareto frontier and budget feasible region are also given.

[0057] Dual-clock execution: "Daily planning" generates the baseline strategy for the day based on offline distribution and inventory / budget; "Real-time rescoring" fine-tunes the strategy at the session level based on the latest features and guardrail status.

[0058] Console and Reason Codes: Budget slider, fatigue cap, inventory weight reduction switch; displays reason codes and OPE confidence intervals one by one; supports manual overwrite / rollback / grayscale.

[0059] Failure safety: When any of KL / budget / fatigue exceeds the threshold, automatically fall back to LTDP or teacher strategy.

[0060] Audit and drift monitoring: Record strategy version, threshold, multiplier trajectory and abnormal weight samples; feature distribution drift alarm.

[0061] This invention first ensures impartiality from the data source by rigorously processing historical data with time delays, eliminating model bias caused by future information leaks and laying a reliable foundation for decision-making. Second, it achieves transparency and explainability in the decision-making process. Through a two-stage inverse reinforcement learning framework, it not only accurately mimics the complex logic of experts in deciding whether to promote a product or choose which discount, but also outputs business-understandable decision reason codes, greatly satisfying the needs of compliance audits and rapid approval. Third, it constructs a secure and controllable strategy iteration path, following the principle of first mimicking experts and then constraining optimization, starting with a secure baseline strategy. Fine-tuning within multiple business safeguards, such as budget and inventory, effectively avoids the waste of subsidies and user experience risks caused by blind online exploration. At the same time, an offline verifiable evaluation system was established. Utilizing advanced offline counterfactual evaluation technology, the system can quantitatively estimate the benefits and confidence intervals of strategies before they go live, transforming trial-and-error deployment into scientific verification and significantly reducing decision-making risks. Ultimately, an efficient and robust engineering closed loop is formed. Through mechanisms such as model distillation and dual-clock execution, it balances the high efficiency and low latency of online services with real-time response to dynamic business constraints, realizing the generation and online execution of e-commerce marketing strategies that are interpretable, auditable, and verifiable offline.

[0062] Figure 2 This specification provides a schematic diagram of a reinforcement learning-based e-commerce marketing management device, which may include: Module 10 is used to acquire historical transaction data and historical interaction data; Processing module 20 is used to perform strict time-delay processing on the historical transaction data and the historical interaction data to obtain business interpretable features; Learning module 30 is used to perform two-stage inverse reinforcement learning processing based on the business interpretable features to obtain the reference marketing strategy distribution and the corresponding decision reason code; Distillation module 40 is used to obtain a baseline strategy model by using the reference marketing strategy distribution as a supervision signal and training it through list distillation. The update module 50 is used to iteratively update the baseline strategy model under preset business constraints using a group relative strategy optimization method to obtain an optimized marketing strategy model. The group relative strategy optimization adopts an intra-group relative advantage calculation method centered on the reference marketing strategy distribution. The estimation module 60 is used to quantitatively estimate the expected returns and risks of the optimized marketing strategy model using an offline counterfactual assessment method. The output module 70 is used to generate and output session-level promotional decisions and corresponding decision reason codes online using the optimized marketing strategy model based on the quantitative estimation results and real-time business constraint status.

[0063] Optionally, the processing module 20 includes: Identify future leakable signals in the historical transaction data and historical interaction data, and align the occurrence time of the future leakable signals to the start timestamp of the business session to which they belong; Based on the aligned data, using information whose timestamp is no later than the start timestamp of the business session, a business interpretable feature corresponding to the business session is constructed.

[0064] Optionally, the learning module 30 includes: Based on the session-level interpretable features of the business, a first model is trained to predict the probability of performing a promotional action in the current session state. For a session deemed suitable for executing a promotional action, a second model is trained based on the features of the session and promotional action combination level to predict the preference probability distribution on the set of feasible promotional actions, as the reference marketing strategy distribution. The training objectives of both the first model and the second model are based on expert decision-making behavior derived from the historical transaction data and the historical interaction data.

[0065] Optionally, training the first model based on the session-level interpretable features to predict the probability of performing a promotional action in the current session state includes: in, σ(·) is the probability of performing a promotional action given the business interpretable feature x; σ(·) is the Sigmoid function; x∈R d It is a business-interpretable feature vector of dimension d; It is a trainable weight vector; y∈{0,1} is the true label, where 1 indicates that a promotion has occurred and 0 indicates that no promotion has occurred; It is the cross-entropy loss; λ>0 is the L2 regularization coefficient; It is the squared L2 norm of the weight vector, used to prevent overfitting.

[0066] Optionally, for a session deemed suitable for executing a promotional action, a second model is trained based on the features of the session and promotional action combination to predict the preference probability distribution on the set of feasible promotional actions, as the reference marketing strategy distribution, including: For each action k in the set of feasible promotional actions, construct an extended feature vector: in, This is the extended feature vector corresponding to action k; It is an indicator vector, identifying the current candidate action as the k-th one; ⊙ is the Hadamard product; The reference distribution is modeled using listwise softmax to obtain the predicted probability distribution of preferences on the set of feasible promotional actions: in, z represents the probability distribution of preferences for choosing action k given x; k Rate action k; The score is given for the j-th feasible offer; τ>0 is a temperature parameter that controls the smoothness of the distribution. For the model objective; Choose the discount when x is given The probability distribution of preferences.

[0067] Optionally, the distillation module 40 includes: Train a neural scorer with parameter θ. For each action k in the set of feasible promotional actions, the scorer outputs a predicted score: in, The predicted score for action k; (·) represents a neural network with parameter θ; The training loss function of the baseline policy model is a list-based cross-entropy loss, expressed as: in, For list-based cross-entropy loss, Distribution of teachers; Predict the distribution of students; This refers to the temperature parameter.

[0068] Optionally, the distillation module 40 further includes: Construct a fused teacher distribution, and use this fused teacher distribution as a supervision signal to train the baseline policy model through list distillation: in, As the baseline strategy model, The teacher strategy is based on business rules; α∈[0,1] is the fusion weight.

[0069] The functions of the apparatus in this embodiment have been described in the above method embodiments. Therefore, for any parts not detailed in this embodiment, please refer to the relevant descriptions in the foregoing embodiments, which will not be repeated here.

[0070] Based on the same inventive concept, embodiments of this specification also provide an electronic device.

[0071] The following describes embodiments of the electronic device of the present invention, which can be considered as specific implementations of the methods and apparatus embodiments of the present invention described above. Details described in the embodiments of the electronic device of the present invention should be considered as supplements to the methods or apparatus embodiments described above; details not disclosed in the embodiments of the electronic device of the present invention can be implemented with reference to the methods or apparatus embodiments described above.

[0072] Figure 3 This is a schematic diagram of an electronic device provided as an embodiment of this specification. Refer to the following... Figure 3 The electronic device 300 according to this embodiment of the present invention will be described. Figure 3 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0073] like Figure 3 As shown, the electronic device 300 is presented in the form of a general-purpose computing device. The components of the electronic device 300 may include, but are not limited to: at least one processing unit 310, at least one storage unit 320, a bus 330 connecting different system components (including storage unit 320 and processing unit 310), a display unit 340, etc.

[0074] The storage unit stores program code that can be executed by the processing unit 310, causing the processing unit 310 to perform the steps described in the processing method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 310 can perform, for example... Figure 1 The steps are shown.

[0075] The storage unit 320 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 3201 and / or a cache storage unit 3202, and may further include a read-only memory unit (ROM) 3203.

[0076] The storage unit 320 may also include a program / utility 3204 having a set (at least one) program module 3205, such program module 3205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0077] Bus 330 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0078] Electronic device 300 can also communicate with one or more external devices 400 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable viewers to interact with electronic device 300, and / or with any device that enables electronic device 300 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 350. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 360. Network adapter 360 can communicate with other modules of electronic device 300 via bus 330. It should be understood that, although... Figure 3 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0079] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described in this invention can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the method described above according to this invention. When the computer program is executed by a data processing device, it enables the computer-readable medium to implement the method described above, i.e.: as... Figure 1 The method shown.

[0080] Figure 4 This is a schematic diagram of a computer-readable medium provided for embodiments of this specification.

[0081] accomplish Figure 1The computer program of the method shown can be stored on one or more computer-readable media. A computer-readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0082] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0083] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the audience's computing device, partially on the audience's device, as a standalone software package, partially on the audience's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the audience's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0084] In summary, the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that in practice, general-purpose data processing devices such as microprocessors or digital signal processors (DSPs) can be used to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0085] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0086] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0087] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An e-commerce marketing management method based on reinforcement learning, characterized in that, include: Obtain historical transaction data and historical interaction data; Strict time-delay processing is applied to the historical transaction data and the historical interaction data to obtain business interpretable features; Based on the interpretable features of the business, a two-stage inverse reinforcement learning process is performed to obtain the reference marketing strategy distribution and the corresponding decision reason code. Using the aforementioned reference marketing strategy distribution as a supervision signal, a baseline strategy model is obtained through list distillation training; Under preset business constraints, the baseline strategy model is iteratively updated using a group relative strategy optimization method to obtain an optimized marketing strategy model. The group relative strategy optimization adopts an intra-group relative advantage calculation method centered on the reference marketing strategy distribution. The expected returns and risks of the optimized marketing strategy model are quantitatively estimated using an offline counterfactual assessment method. Based on the quantitative estimation results and real-time business constraint status, the optimized marketing strategy model is used to generate and output session-level promotional decisions and corresponding decision reason codes online.

2. The e-commerce marketing management method based on reinforcement learning as described in claim 1, characterized in that, The process of performing strict time-delay processing on the historical transaction data and the historical interaction data to obtain business-explainable features includes: Identify future leakable signals in the historical transaction data and historical interaction data, and align the occurrence time of the future leakable signals to the start timestamp of the business session to which they belong; Based on the aligned data, using information whose timestamp is no later than the start timestamp of the business session, a business interpretable feature corresponding to the business session is constructed.

3. The e-commerce marketing management method based on reinforcement learning as described in claim 1, characterized in that, The two-stage inverse reinforcement learning process, based on the interpretable features of the business, yields a reference marketing strategy distribution and the corresponding decision reason code, including: Based on the session-level interpretable features of the business, a first model is trained to predict the probability of performing a promotional action in the current session state. For a session deemed suitable for executing a promotional action, a second model is trained based on the features of the session and promotional action combination level to predict the preference probability distribution on the set of feasible promotional actions, as the reference marketing strategy distribution. The training objectives of both the first model and the second model are based on expert decision-making behavior derived from the historical transaction data and the historical interaction data.

4. The e-commerce marketing management method based on reinforcement learning as described in claim 3, characterized in that, The first model, trained based on the session-level interpretable features of the business, predicts the probability of performing a promotional action in the current session state, including: in, σ(·) is the probability of performing a promotional action given the business interpretable feature x; σ(·) is the Sigmoid function; x∈R d It is a business-interpretable feature vector of dimension d; It is a trainable weight vector; y∈{0,1} is the true label, where 1 indicates that a promotion has occurred and 0 indicates that no promotion has occurred; It is the cross-entropy loss; λ>0 is the L2 regularization coefficient; It is the squared L2 norm of the weight vector, used to prevent overfitting.

5. The e-commerce marketing management method based on reinforcement learning as described in claim 4, characterized in that, For sessions deemed suitable for implementing promotional actions, a second model is trained based on the features of the session and promotional action combination to predict the preference probability distribution across the set of feasible promotional actions, serving as the reference marketing strategy distribution, including: For each action k in the set of feasible promotional actions, construct an extended feature vector: in, This is the extended feature vector corresponding to action k; It is an indicator vector, identifying the current candidate action as the k-th one; ⊙ is the Hadamard product; The reference distribution is modeled using listwise softmax to obtain the predicted probability distribution of preferences on the set of feasible promotional actions: in, z represents the probability distribution of preferences for choosing action k given x; k Rate action k; The score is for the j-th feasible offer; τ>0 is a temperature parameter that controls the smoothness of the distribution. For the model objective; Choose the discount when x is given The probability distribution of preferences.

6. The e-commerce marketing management method based on reinforcement learning as described in claim 1, characterized in that, The step of using the reference marketing strategy distribution as a supervision signal to train a baseline strategy model through list distillation includes: Train a neural scorer with parameter θ. For each action k in the set of feasible promotional actions, the scorer outputs a predicted score: in, The predicted score for action k; (·) represents a neural network with parameter θ; The training loss function of the baseline policy model is a list-based cross-entropy loss, expressed as: in, For list-based cross-entropy loss, Distribution of teachers; Predict the distribution of students; This refers to the temperature parameter.

7. The e-commerce marketing management method based on reinforcement learning as described in claim 6, characterized in that, The step of using the reference marketing strategy distribution as a supervision signal to train the baseline strategy model through list distillation also includes: Construct a fused teacher distribution, and use this fused teacher distribution as a supervision signal to train the baseline policy model through list distillation: in, As the baseline strategy model, The teacher strategy is based on business rules; α∈[0,1] is the fusion weight.

8. An e-commerce marketing management device based on reinforcement learning, characterized in that, include: The acquisition module is used to acquire historical transaction data and historical interaction data; The processing module is used to perform strict time-delay processing on the historical transaction data and the historical interaction data to obtain business interpretable features; The learning module is used to perform a two-stage inverse reinforcement learning process based on the interpretable features of the business to obtain the reference marketing strategy distribution and the corresponding decision reason code; The distillation module is used to train a baseline strategy model by using the reference marketing strategy distribution as a supervision signal and distilling a list to obtain the model. The update module is used to iteratively update the baseline strategy model under preset business constraints using a group relative strategy optimization method to obtain an optimized marketing strategy model. The group relative strategy optimization adopts an intra-group relative advantage calculation method centered on the reference marketing strategy distribution. The estimation module is used to quantitatively estimate the expected returns and risks of the optimized marketing strategy model using an offline counterfactual evaluation method. The output module is used to generate and output session-level promotional decisions and corresponding decision reason codes online using the optimized marketing strategy model, based on the quantitative estimation results and real-time business constraint status.

9. An electronic device, wherein, The electronic device includes: A processor; and a memory storing computer-executable instructions, which, when executed, cause the processor to perform the method according to any one of claims 1-7.

10. A computer-readable storage medium, wherein, The computer-readable storage medium stores one or more programs that, when executed by a processor, implement the method of any one of claims 1-7.

Images