An automated financial product pricing strategy optimization method based on reinforcement learning

Abstract

The present application relates to the technical field of financial services, and in particular to an automatic financial product pricing strategy optimization method based on reinforcement learning, comprising obtaining multi-dimensional data of wealth management products, predicting attribute characteristics and distribution probability of a target customer group, classifying actual customers of the products, performing multi-dimensional deviation analysis, and dynamically post-training a reinforcement learning model using a large model post-training key technology; the present application adopts a scheme of deep fusion of a pre-trained large model in the financial field and reinforcement learning, uses the knowledge reasoning ability of the large model to generate an initial pricing strategy conforming to industry common sense and regulatory requirements, simultaneously uses low-rank adaptation technology to realize lightweight fine-tuning of the large model, and integrates product and customer characteristics extracted by the large model into the state space of reinforcement learning, thereby having the advantages of significantly improving model convergence speed, greatly shortening cold start time, and enabling new online products to quickly deploy effective pricing strategies.

Description

Technical Field

[0001] This invention relates to the field of financial services technology, and in particular to an automated financial product pricing strategy optimization method based on reinforcement learning. Background Technology

[0002] With the rapid development of my country's wealth management market and increasingly fierce product competition, pricing strategies directly determine the market attractiveness of products, customer retention rates, and institutional profitability. Traditional pricing methods are no longer sufficient to meet the needs of refined market operations.

[0003] Traditional wealth management product pricing primarily employs cost-plus pricing, market competition pricing, and customer value pricing. Cost-plus pricing, which adds profit to operating costs, is simple and easy to implement but completely ignores market demand and customer differences, easily leading to inappropriate pricing. Market competition pricing uses the prices of similar products as a benchmark; while adaptable to competition, it easily leads to price wars and fails to reflect product differentiation. Customer value pricing is based on perceived customer value, theoretically optimal but difficult to quantify, highly subjective, and lacks a unified standard.

[0004] While supervised learning-based automated pricing methods can fit historical transaction patterns, they can only handle static problems and cannot optimize long-term sequential decisions. Furthermore, newly launched products suffer from a severe cold-start problem due to a lack of historical data. Existing reinforcement learning-based pricing methods, while capable of handling dynamic decision-making problems, still have several core drawbacks: First, the model is trained from scratch, resulting in low sample efficiency and a highly unpredictable initial strategy during the cold start phase. Second, the use of a uniform pricing model fails to consider the heterogeneity of customer groups in terms of risk preferences and price sensitivity, leading to the churn of high-value customers and excessive concessions to low-value customers. Third, focusing solely on short-term returns fails to identify and correct deviations between actual and expected customer distribution, easily triggering liquidity and return risks. Fourth, reliance on human feedback in reinforcement learning results in high labeling costs and inconsistent quality. Fifth, the use of periodic manual updates cannot quickly respond to market changes and is susceptible to catastrophic forgetting.

[0005] In summary, existing technologies cannot achieve accurate, dynamic, and efficient pricing of wealth management products, and there is an urgent need for an automated and intelligent pricing strategy optimization method. Summary of the Invention

[0006] To overcome the problems mentioned in the background art, this invention proposes an automated financial product pricing strategy optimization method based on reinforcement learning.

[0007] The technical solution of this invention is: an automated financial product pricing strategy optimization method based on reinforcement learning, comprising the following steps:

[0008] S1: Obtain multi-dimensional data on wealth management products, including product attribute data, customer base data, transaction behavior data, and market environment data;

[0009] S2: Based on a pre-trained large model in the financial field, product attribute data is deeply encoded to predict the attribute characteristics and distribution probability of the target customer group;

[0010] S3: Classify the actual customers of the product into three core groups: transient customers, medium-term customers and long-term customers, and extract the actual attribute characteristics and distribution data of each group;

[0011] S4: Compare the predicted customer distribution with the actual customer distribution, conduct multi-dimensional deviation analysis, and generate deviation analysis results;

[0012] S5: Using the bias analysis results as a driving signal, the reinforcement learning model is dynamically post-trained using key technologies for large model post-training to update the pricing strategy network.

[0013] S6: Based on the updated reinforcement learning model, output differentiated pricing decisions for different customer groups;

[0014] S7: Collect data on the effectiveness of pricing decisions and feed it back to step S4 for a new round of deviation analysis to form a closed-loop optimization mechanism.

[0015] Preferably, step S2 specifically includes:

[0016] S21: Input the product's structured and unstructured data into a pre-trained large model in the financial field. The structured data includes the expected rate of return range, risk level, investment period, minimum investment amount, fee structure and liquidity terms. The unstructured data includes the product prospectus, investment strategy, risk disclosure statement and fund manager introduction.

[0017] S22: Use LoRA low-rank adaptation technology to perform lightweight fine-tuning on the large model, extract the product's return characteristics, risk characteristics, liquidity characteristics, threshold characteristics and service characteristics, and generate a product feature vector with a dimension of 768.

[0018] S23: Input the product feature vector into the large model inference module to generate a detailed profile of the target customer group, including age range, income level, risk tolerance, investment experience, asset size, liquidity preference and investment term preference;

[0019] S24: Output the predicted customer group distribution probability matrix, with dimensions of [number of customer groups, feature dimension], and calculate the expected conversion rate, expected average order value and expected lifetime value for each customer group.

[0020] Preferably, step S3 specifically includes:

[0021] S31: Based on the RFM model, extract three core indicators for each actual user: the time of their most recent purchase, the frequency of their purchase, and the amount of their purchase.

[0022] S32: The K-means++ clustering algorithm is used to perform cluster analysis on all actual users, dividing them into three major groups: transient customers, medium-term customers, and long-term customers;

[0023] S33: Establish a corresponding attribute feature set for each customer group, extract the same feature dimensions as the prediction target, and generate the actual customer group distribution probability matrix;

[0024] S34: Calculate the actual conversion rate, actual average order value, actual lifetime value, and actual churn rate for each customer group, and analyze the subscription and redemption patterns, reactions to market fluctuations, and responsiveness to marketing activities for different groups.

[0025] Preferably, the multi-dimensional deviation analysis in step S4 includes basic attribute deviation analysis, specifically:

[0026] Calculate the KL divergence and JS divergence between the predicted distribution and the actual distribution for each feature dimension to quantify the degree of deviation for each dimension;

[0027] Identify the top 5 feature dimensions with the largest deviations and analyze the root causes of these deviations.

[0028] Generate a basic attribute deviation report to indicate the potential impact of the deviation on the product pricing strategy.

[0029] Preferably, the multi-dimensional deviation analysis in step S4 includes population distribution deviation analysis, specifically:

[0030] Calculate the deviation rate between the actual and predicted proportions of transient customers, mid-term customers, and long-term customers;

[0031] Analyze the impact of group distribution bias on overall product returns, liquidity, and stability;

[0032] Generate a population distribution deviation report and label the population types whose deviations exceed a preset threshold.

[0033] Preferably, the multi-dimensional deviation analysis in step S4 includes risk analysis, specifically:

[0034] Liquidity risk analysis: Assess the risk of concentrated redemptions that may result from an excessively high proportion of short-term customers in the actual customer distribution, and use the VaR method to quantify potential liquidity losses;

[0035] Credit risk analysis: Assess the potential for complaints and disputes that may result from customers' actual risk tolerance being lower than the product's risk level, and calculate the probability of risk events occurring;

[0036] Profitability and Risk Analysis: Assess the risk of fluctuations in product scale and management fee income caused by changes in the actual customer structure, and predict the profitability volatility for the next 3 months;

[0037] Generate a comprehensive risk assessment report to provide risk constraints for adjusting pricing strategies.

[0038] Preferably, the multi-dimensional deviation analysis in step S4 also includes strengths analysis and weaknesses analysis, specifically:

[0039] Strengths Analysis: Identify high-value groups in the actual customer distribution that exceed expectations, analyze their common characteristics, assess their contribution to the overall product revenue, and explore cross-selling and upselling opportunities;

[0040] Defect analysis: Identify the target groups that are below expectations in the actual customer distribution, analyze the reasons for their churn, evaluate the specific manifestations of the pricing strategy's lack of appeal to these groups, and identify problems in product design and marketing promotion;

[0041] Generate a strengths and weaknesses analysis report to provide guidance for optimizing pricing strategies.

[0042] Preferably, the basic construction steps of the reinforcement learning model in step S5 are as follows:

[0043] Construct a state space S, which includes product state, customer state, market state, and deviation state. Product state includes current yield, size, and remaining maturity; customer state includes customer type, risk tolerance, and historical transaction records; market state includes market index, interest rate level, and yield of similar products; and deviation state includes deviation values ​​for each dimension.

[0044] Construct action space A, which includes adjustable parameters for pricing strategies, including subscription fee rate, redemption fee rate, management fee, sales service fee, performance fee accrual ratio and accrual method;

[0045] Construct a comprehensive reward function The formula is:

[0046] ;

[0047] in, , , , The pre-set weighting coefficients indicate that the bias penalty term is positively correlated with the results of the multi-dimensional bias analysis.

[0048] The PPO (Proximity Policy Optimization) algorithm is used as the core algorithm for reinforcement learning to initialize the policy network and value network.

[0049] Preferably, the specific steps for dynamic post-training using key technologies for large model post-training in step S5 are as follows:

[0050] S51: Utilize the knowledge reasoning ability of the large model to generate an initial pricing strategy. Input product attributes, target customer groups and market environment information into the large model to generate an initial pricing range that conforms to industry common sense and regulatory requirements, which serves as an action space constraint for the reinforcement learning model.

[0051] S52: Automatically generate a preference dataset based on the deviation analysis results. Each sample contains a state, good actions, and bad actions. Good actions are pricing actions that can effectively reduce the deviation between the predicted distribution and the actual distribution, while increasing returns and reducing risks. Bad actions are pricing actions that will amplify the deviation or lead to increased risks.

[0052] S53: Use the DPO direct preference optimization algorithm to fine-tune the policy network of the reinforcement learning model without training the reward model separately;

[0053] S54: Uses LoRA technology to achieve lightweight fine-tuning, updating only 1%-5% of the parameters in the policy network, improving training efficiency and preventing catastrophic forgetting.

[0054] Preferably, the closed-loop optimization mechanism in step S7 specifically includes:

[0055] Elastic weights are used to reinforce the EWC algorithm to prevent the model from forgetting previously learned effective pricing strategies when learning new bias correction knowledge.

[0056] Collect new customer and transaction data regularly, weekly or monthly, and update the deviation analysis results;

[0057] When the deviation value in any dimension exceeds the preset threshold, the post-training process of the model is automatically triggered.

[0058] Establish a model version management system to record the model parameters, corresponding deviation analysis results, and pricing effect data for each version, and support model rollback and comparative analysis.

[0059] The beneficial effects of this invention are:

[0060] 1. Compared to existing technologies that train reinforcement learning models from scratch, which suffer from drawbacks such as low sample efficiency during the cold start phase, strong blindness in initial strategies, and the inability to quickly generate effective pricing strategies for newly launched products, this invention adopts a solution that deeply integrates pre-trained large models in the financial field with reinforcement learning. It utilizes the knowledge reasoning capabilities of the large model to generate initial pricing strategies that conform to industry common sense and regulatory requirements. At the same time, it uses low-rank adaptation technology to achieve lightweight fine-tuning of the large model, integrating the product and customer features extracted by the large model into the state space of reinforcement learning. This has the advantages of significantly improving model convergence speed, greatly shortening cold start time, and enabling newly launched products to quickly deploy effective pricing strategies.

[0061] 2. Compared to existing technologies that use product revenue as the sole feedback signal for model optimization, which has drawbacks such as the inability to identify discrepancies between actual and expected customer distribution, mismatch between pricing strategies and target customer groups, and the potential for high-value customer churn and excessive concessions to low-value customers, this invention employs a multi-dimensional deviation analysis system as the core driving signal for the reinforcement learning model. It comprehensively compares the differences between the predicted and actual customer distribution from five dimensions: basic attributes, group distribution, risk, strengths, and weaknesses. It deeply analyzes the root causes of the deviations and their impact on product revenue and risk, and has the advantages of accurately identifying problems in pricing strategies, providing a clear direction for model optimization, and enabling pricing strategies to proactively correct expected deviations.

[0062] 3. Compared with existing technologies that use human feedback reinforcement learning for model optimization, which has disadvantages such as requiring a large amount of manually labeled data, high labeling costs, unstable labeling quality, and inability to achieve real-time dynamic optimization, this invention adopts an automatic preference learning scheme based on bias analysis results. It automatically generates a preference dataset containing good and bad actions using multi-dimensional bias analysis results, and uses a direct preference optimization algorithm to fine-tune the policy network of the reinforcement learning model. It does not require separate training of the reward model, and has the advantages of completely eliminating the dependence on manually labeled data, significantly reducing model training costs, and significantly improving model update efficiency.

[0063] 4. Compared to existing technologies that use simple customer tags for differentiated pricing, which have drawbacks such as unscientific customer segmentation, low matching degree between pricing strategies and customer needs, and inability to fully tap the value potential of different customer groups, this invention adopts a scientific customer segmentation system based on the RFM model and clustering algorithm. It divides customers into three core groups: transient customers, mid-term customers, and long-term customers. It deeply analyzes the behavioral patterns and price sensitivity of different groups and deeply integrates the customer segmentation results into the state space and reward function design of reinforcement learning. This has the advantages of achieving accurate differentiated pricing, improving customer conversion and retention rates, and maximizing the lifetime value of different customer groups. Attached Figure Description

[0064] Figure 1 The diagram shown is a flowchart of the automated financial product pricing strategy optimization method based on reinforcement learning of the present invention.

[0065] Figure 2 The chart shown is a comparison of verification effect data in a specific embodiment of the present invention. Detailed Implementation

[0066] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0067] Please see Figure 1 This invention provides an embodiment: an automated financial product pricing strategy optimization method based on reinforcement learning, specifically including the following steps:

[0068] Step S1: Multi-dimensional data acquisition and preprocessing

[0069] First, obtain the four core data types required for pricing wealth management products and perform standardized preprocessing:

[0070] Product attribute data includes both structured and unstructured data. Structured data specifically includes: expected rate of return range (e.g., 4%-6%), risk level (R1-R5), investment period (e.g., 1 month-3 years), minimum investment amount (e.g., 1 yuan, 10,000 yuan, 1 million yuan), fee structure (subscription fee, redemption fee, management fee, sales service fee, performance fee), and liquidity terms (opening frequency, redemption settlement time). Unstructured data specifically includes: the full product prospectus, investment strategy report, risk disclosure statement, fund manager / investment manager resume, and historical performance report.

[0071] Customer basic data includes customer age, gender, occupation, income level, asset size, risk tolerance assessment results, years of investment experience, account opening time, and contact information.

[0072] Transaction behavior data includes customer subscription records (subscription time, subscription amount, subscription channel), redemption records (redemption time, redemption amount, redemption reason), holding records (holding product type, holding amount, holding duration), trading frequency, and historical profit and loss.

[0073] Market environment data includes: CSI 300 Index, CSI 500 Index, Treasury yield curve, bank deposit rates, average yield of similar products, market volatility (VIX index), and macroeconomic indicators (GDP growth rate, CPI, PPI).

[0074] Data preprocessing steps:

[0075] Structured data is imputed (numerical data is imputed with the mean, and categorical data is imputed with the mode) and standardized (Z-score standardization).

[0076] Unstructured data is cleaned by removing special characters, stop words, and irrelevant information, and Chinese words are segmented using a word segmentation tool.

[0077] Align the time series data to unify the time granularity to daily.

[0078] Establish a data quality verification mechanism to remove outliers (such as transaction amounts exceeding 3 times the standard deviation).

[0079] Step S2: Target customer group prediction and attribute analysis based on pre-trained large models in the financial field

[0080] We employ large-scale models pre-trained in the financial field as the base model, preferably fine-tuned versions of Qwen-Finance-7B or Llama3-8B. These models perform exceptionally well in financial text understanding and reasoning tasks, specifically including:

[0081] S21: Product Attribute Vectorization Coding

[0082] The preprocessed structured and unstructured product data are concatenated and input into a large financial model. LoRA (Low-Rank Adaptation) technology is then used to perform lightweight fine-tuning of the model, enabling it to accurately extract the core features of wealth management products. Specifically:

[0083] LoRA fine-tuning of key parameters: Optimal values

[0084] Rank r: preferably 4-32, typical value is 8;

[0085] alpha parameter: preferably twice r, i.e., 16;

[0086] Dropout probability: preferably 0.05-0.1, typical value is 0.05;

[0087] Learning rate: 1e-4~5e-4 is preferred, with a typical value of 2e-4;

[0088] Training rounds: 3-10 rounds are preferred, with 5 rounds being the typical value;

[0089] Batch size: 8-32 is preferred, with a typical value of 16.

[0090] After fine-tuning, the output of the last hidden layer of the large model is extracted as the product feature vector, with an optimal dimension of 768, including the following five core dimensions:

[0091] Return characteristics (153 dimensions): expected return, return volatility, historical maximum drawdown, performance benchmark, etc.

[0092] Risk characteristics (153 dimensions): risk level, risk distribution of investment targets, downside risk, extreme risk, etc.;

[0093] Liquidity characteristics (153 dimensions): investment period, opening frequency, redemption restrictions, early redemption penalties, etc.;

[0094] Threshold features (153 dimensions): minimum investment amount, minimum additional investment amount, qualified investor requirements, etc.;

[0095] Service characteristics (156 dimensions): investment advisory services, information disclosure frequency, customer service level, etc.

[0096] S22: Target Customer Group Forecast

[0097] The 768-dimensional product feature vector is input into the inference module of the large model to generate a detailed profile and probability distribution of the target customer group. The large model establishes a mapping relationship between product attributes and customer attributes by learning from customer data of similar products in the past.

[0098] The output includes:

[0099] Attribute distribution of the target customer group: probability distribution of each attribute dimension (age, income, risk tolerance, etc.);

[0100] Customer group distribution probability matrix: The dimensions are [3,7], where 3 represents the three major customer groups (current customers, mid-term customers, and long-term customers), and 7 represents the seven core attribute dimensions;

[0101] Expected metrics for each customer group: expected conversion rate (optimal range 5%-30%), expected average order value (optimal range 10,000-1,000,000 RMB), and expected customer lifetime value (LTV).

[0102] For example, for a mixed wealth management product with a risk level of R3, an expected return of 4%-6%, and an investment period of 1 year, the target customer distribution predicted by the large model is as follows: 20% of customers are mobile customers, 50% are medium-term customers, and 30% are long-term customers.

[0103] Step S3: Actual Customer Segmentation, Classification, and Attribute Analysis

[0104] Based on the RFM model and K-means++ clustering algorithm, the actual users after product launch are scientifically segmented, including:

[0105] S31: RFM Index Calculation

[0106] Calculate the following three core metrics for each actual user:

[0107] Recency (last purchase date): Number of days since the current date;

[0108] Frequency (purchase frequency): The number of transactions in the past 12 months;

[0109] Monetary (Purchase Amount): The cumulative purchase amount over the past 12 months.

[0110] After standardizing the three metrics, we obtain the RFM feature vector for each user, which has a dimension of 3.

[0111] S32: K-means++ clustering analysis

[0112] The K-means++ clustering algorithm is used to cluster the RFM feature vectors of all users, with the number of clusters k fixed at 3, corresponding to three major groups: transient customers, mid-term customers, and long-term customers. The optimal values ​​for the key parameters of the K-means++ algorithm are as follows:

[0113] Distance metric: Euclidean distance;

[0114] Maximum number of iterations: preferably 100-300 times, typical value is 200 times;

[0115] Convergence threshold: preferably 1e-4~1e-6, typical value is 1e-5;

[0116] Number of initializations: preferably 10-20 times, with a typical value of 10 times.

[0117] After clustering, the following customer segmentation criteria were obtained:

[0118] mobile customers Holding period < 3 months, trading frequency > 5 times / month Pursuing short-term price differences, highly sensitive to fees, with an average holding period of 45 days. Mid-term customers Holding period 3-12 months, trading frequency 1-5 times / month Seeking stable returns, moderately sensitive to fees, with an average holding period of 8 months. Long-term customers Holding period > 12 months, trading frequency < 1 time / month Seeking long-term asset appreciation, sensitive to low fees, with an average holding period of 24 months.

[0119] S33: Analysis of Actual Customer Attributes

[0120] For each customer group, a corresponding attribute feature set is established, and the same 7 core attribute dimensions as the prediction target are extracted to generate the actual customer group distribution probability matrix, which also has dimensions [3,7].

[0121] Simultaneously calculate the actual operational metrics for each customer group:

[0122] Actual conversion rate: Number of actual purchases / Number of people reached;

[0123] Actual average order value: Total sales / Number of customers;

[0124] Actual customer lifetime value: The total revenue a customer generates for the product over their entire lifecycle;

[0125] Actual churn rate: The percentage of customers who do not renew their subscriptions after the expiration date.

[0126] Step S4: Multidimensional Deviation Analysis

[0127] By comparing the predicted customer distribution with the actual customer distribution, a deep deviation analysis is conducted across five dimensions to provide precise guidance signals for the post-training of the reinforcement learning model. Specifically:

[0128] S41: Basic Attribute Deviation Analysis

[0129] Calculate the KL divergence and JS divergence between the predicted distribution P and the actual distribution Q for each feature dimension to quantify the degree of bias:

[0130] KL divergence: ;

[0131] JS divergence: ,in, .

[0132] in, Let KL divergence be the KL divergence. Represented by probability distribution Using this as a baseline, calculate the distribution. Relative to distribution The degree of deviation, In the predicted distribution, the th The probability value corresponding to the customer class. In the actual distribution, the th The probability value corresponding to the customer class. Indicates JS divergence, Indicates the intermediate mixed baseline distribution. Indicates the calculation of the predicted distribution With intermediate distribution KL divergence, Indicates the calculation of the actual distribution With intermediate distribution KL divergence;

[0133] The JS divergence ranges from [0,1], with larger values ​​indicating greater bias. Feature dimensions with JS divergence greater than 0.2 are identified as key bias dimensions to analyze the root causes of the bias.

[0134] For example, if the JS divergence of the risk tolerance dimension is 0.35, it indicates that there is a significant deviation between the actual risk tolerance of customers and the prediction, which may be due to insufficient product risk disclosure or inaccurate marketing positioning.

[0135] S42: Population Distribution Deviation Analysis

[0136] Calculate the deviation rate between the actual and predicted proportions of each customer group:

[0137] ;

[0138] When the absolute value of the deviation rate exceeds 20%, the population distribution is considered to have a significant deviation. Analyze the impact of population distribution deviation on the product:

[0139] If the actual proportion of mobile customers is higher than the forecast of 30%, it will increase the product's liquidity pressure and transaction costs.

[0140] If the actual proportion of long-term customers is lower than the predicted 25%, it will reduce the product's financial stability and long-term returns.

[0141] S43: Risk Analysis

[0142] The VaR (Value at Risk) method is used to quantify the potential losses of various risks, with a preferred confidence level of 95% and a preferred holding period of 10 trading days.

[0143] Liquidity risk: Calculate the maximum liquidity gap that may result from concentrated redemptions at a 95% confidence level;

[0144] Credit risk: Calculate the complaint and dispute rates resulting from the mismatch between the actual customer's risk tolerance and the product's risk level;

[0145] Profitability Risk: Predict the volatility of product size and management fee income over the next 3 months due to changes in customer structure.

[0146] Generate a comprehensive risk assessment report and set risk constraints for adjusting pricing strategies, such as: the proportion of transient customers should not exceed 40% and the volatility of product size should not exceed 10%.

[0147] S44: Strengths Analysis

[0148] Identify high-value groups in the actual customer distribution that exceed expectations (usually those whose actual LTV is more than 50% higher than the predicted LTV), and analyze their common characteristics, such as: age between 35 and 50 years old, asset size of more than 5 million yuan, and investment experience of more than 5 years.

[0149] Assess the contribution of these high-value groups to the overall product returns; typically, the top 20% of high-value clients contribute 80% of the returns. Explore cross-selling and upselling opportunities, such as recommending more upscale private equity products or family trust services to high-value clients.

[0150] S45: Defect Analysis

[0151] Identify the target group whose actual conversion rate is more than 50% lower than the predicted conversion rate in the actual customer distribution. Analyze the reasons for their churn through questionnaires and customer interviews, which mainly include: excessively high pricing, insufficient product liquidity, and poor service experience.

[0152] The specific manifestations of pricing strategies that are not attractive to these groups should be assessed. For example, for younger customers, excessively high minimum investment amounts and excessively long lock-up periods are major obstacles.

[0153] Step S5: Post-training of the bias-driven reinforcement learning model

[0154] The key technologies for post-training of large models (LoRA, DPO) are deeply integrated with the PPO reinforcement learning algorithm. The bias analysis results are used to drive the model for dynamic post-training. The specific process is as follows:

[0155] S51: Building a Basic Reinforcement Learning Model

[0156] The PPO (Proximity Policy Optimization) algorithm is adopted as the core algorithm of reinforcement learning, and the following three core components are constructed:

[0157] State space S: 256 dimensions, containing four subspaces:

[0158] Product status (64 dimensions): current yield, product size, remaining term, net asset value volatility, and ranking among similar products;

[0159] Customer Status (64 dimensions): Customer type, risk tolerance, historical transaction records, holding period, LTV;

[0160] Market conditions (64 dimensions): market index, interest rate level, average yield of similar products, market volatility;

[0161] Deviation status (64 dimensions): Deviation values ​​for each dimension, risk assessment values, and strengths and weaknesses scores.

[0162] Action Space A: Contains 6 adjustable pricing parameters, each with the following adjustable range:

[0163] Subscription fee rate: 0%-1.5%;

[0164] Redemption fee: 0%-2%;

[0165] Management fee: 0.1%-2%;

[0166] Sales service fee: 0%-1%;

[0167] Performance-based compensation rate: 0%-20%;

[0168] Performance-based compensation calculation methods: single high-water mark method, annualized rate of return method, and excess return method.

[0169] Comprehensive reward function :

[0170] ;

[0171] Preferred values ​​for weighting coefficients:

[0172] (Short-term return weight): 0.2-0.4, typical value is 0.3;

[0173] (Long-term value weight): 0.3-0.5, typical value is 0.4;

[0174] (Risk cost weight): 0.1-0.3, typical value is 0.2;

[0175] (Bias penalty weight): 0.05-0.15, typical value is 0.1.

[0176] Among them, the bias penalty term is positively correlated with the average JS divergence of the multidimensional bias analysis. When the average JS divergence exceeds 0.3, the bias penalty term increases significantly.

[0177] Optimal values ​​for key parameters of the PPO algorithm:

[0178] Policy network and value network: Both adopt a 3-layer fully connected neural network with a hidden layer dimension of 256;

[0179] Learning rate: 1e-4~5e-4 is preferred, with a typical value of 3e-4;

[0180] clip range: preferably 0.1-0.3, typical value is 0.2;

[0181] Batch size: 32-128 is preferred, with a typical value of 64;

[0182] Training rounds: 3-10 rounds are preferred, with 5 rounds being the typical value;

[0183] (Discount factor): Preferred value is 0.95-0.99, typical value is 0.99.

[0184] S52: Initial Strategy Generation for Large Model Augmentation

[0185] This approach leverages the knowledge reasoning capabilities of large-scale financial models to generate initial pricing strategies, addressing the cold start problem in reinforcement learning. By inputting product attributes, target customer groups, and market environment information into the large model, it generates an initial pricing range that aligns with industry common sense and regulatory requirements.

[0186] For example, for the aforementioned hybrid wealth management product, the initial pricing range generated by the large model is: subscription fee rate 0.8%-1.2%, redemption fee rate 0.5%-1%, and management fee 1%-1.5%. Using this initial pricing range as the action space constraint of the reinforcement learning model can improve the model's convergence speed by 3-5 times.

[0187] S53: Post-DPO training based on deviation analysis results

[0188] A preference dataset is automatically generated using the bias analysis results, eliminating the need for manual annotation and enabling unsupervised model post-training. The specific steps are as follows:

[0189] Preference dataset construction:

[0190] 10,000 state samples were collected from historical transaction data;

[0191] For each state, generate "good actions" and "bad actions" based on the deviation analysis results:

[0192] Good actions: Pricing actions that can reduce the average JS divergence by more than 10% and increase the reward function value by more than 5%;

[0193] Bad actions: Pricing actions that increase the average JS divergence by more than 10% or decrease the reward function value by more than 5%;

[0194] Construct a preference dataset containing 10,000 samples, each sample in the format of (state, good action, bad action).

[0195] DPO algorithm fine-tuning:

[0196] The DPO (Direct Preference Optimization) algorithm is used to fine-tune the policy network of a reinforcement learning model without the need to train a separate reward model.

[0197] Optimal values ​​for key parameters of the DPO algorithm:

[0198] (Temperature coefficient): Preferably 0.05-0.2, typical value is 0.1;

[0199] Learning rate: Optimal 1e-5-1e-4, typical value is 5e-5;

[0200] Training rounds: 3-5 rounds are preferred, with 3 rounds being the typical value;

[0201] Batch size: 32-64 is preferred, with 32 being the typical value.

[0202] LoRA Lightweight Tuning:

[0203] Using LoRA technology to perform lightweight fine-tuning of the policy network involves updating only the parameters of the last two layers, which account for approximately 2%-3% of the total parameters. This not only significantly improves training efficiency but also effectively prevents catastrophic forgetting.

[0204] Step S6: Differentiated Pricing Decision Output

[0205] Based on the updated reinforcement learning model, a personalized pricing plan is generated for each customer:

[0206] For mobile clients: Appropriately reduce short-term subscription fees (e.g., 0.6%-0.8%) and increase redemption fees (e.g., 1.5% for holding period <7 days, 0.75% for 7-30 days) to encourage them to extend their holding period;

[0207] For medium-term clients: We offer a tiered fee structure, with lower fees for longer holding periods. For example: 0.5% redemption fee for holding periods of 3-6 months, 0.25% for 6-12 months, and 0% for holding periods of over 12 months.

[0208] Long-term clients: We offer the lowest basic fees (subscription fee 0.3%-0.5%, management fee 0.8%-1%), along with free investment advisory services and priority subscription rights;

[0209] High-value clients: We offer customized fee plans. For example, clients with assets exceeding RMB 5 million can enjoy a full waiver of subscription fees and a 50% reduction in management fees.

[0210] Step S7: Closed-loop optimization mechanism

[0211] Establish a closed-loop optimization mechanism encompassing "deviation analysis - post-model training - pricing decision - performance feedback" to achieve continuous iteration of pricing strategies.

[0212] Continuous learning mechanism: The Elastic Weight Consolidation (EWC) algorithm is used to calculate the importance weight of each parameter. When learning new bias correction knowledge, important parameters are protected from being significantly modified to prevent catastrophic forgetting.

[0213] Regular update mechanism: New customer and transaction data are collected weekly, and deviation analysis results are updated; a complete model evaluation and fine-tuning is conducted monthly.

[0214] Automatic triggering mechanism: When the bias rate of any dimension exceeds 30%, or the average JS divergence exceeds 0.3, the post-training process of the model is automatically triggered.

[0215] Model version management: Establish a model version management system to record model parameters, corresponding deviation analysis results, pricing performance data, and launch time for each version. When the performance of a new model version declines, it supports one-click rollback to the previous stable version. Specific Implementation

[0216] The test product is an open-ended mixed wealth management product with a risk level of R3, an expected annualized return of 4.5%-5.5%, and an investment period of one year, with an initial product size of 100 million yuan. The test period is from January 1, 2025 to June 30, 2025. All three options are conducted under the same market environment, the same customer outreach channels, and the same marketing and promotion efforts. A total of 10,000 potential customers are selected and randomly divided into three groups of 3,333 customers each.

[0217] The following two comparative schemes are used to compare the technical solution of the present invention with that of the present invention:

[0218] Comparison Option 1 (Unified Pricing Method): Adopting the industry-standard fixed fee model, without distinguishing customer groups, the following pricing is uniformly implemented: subscription fee rate 1.2%, redemption fee rate 1.5% for holding period <7 days, 0.75% for 7-30 days, and 0.5% for holding period over 30 days, management fee 1.5% / year, and sales service fee 0.2% / year.

[0219] Comparison Scheme 2 (Ordinary Reinforcement Learning Pricing Method): This method uses the traditional PPO reinforcement learning algorithm for pricing optimization. It does not include a large model enhancement module or a multi-dimensional bias analysis closed loop, and only aims to maximize product revenue.

[0220] like Figure 2 As shown, the test results for the three schemes are as follows:

[0221] In terms of revenue metrics: Compared to Solution 1, the total revenue was 4.286 million yuan, with an average revenue per customer of 1,286 yuan, a customer lifetime value of 3,215 yuan, and management fee revenue of 752,000 yuan; compared to Solution 2, the total revenue was 5.123 million yuan, with an average revenue per customer of 1,537 yuan, a customer lifetime value of 3,987 yuan, and management fee revenue of 896,000 yuan; the total revenue of the present invention's technical solution was 6.078 million yuan, with an average revenue per customer of 1,824 yuan, a customer lifetime value of 4,926 yuan, and management fee revenue of 1.063 million yuan. The improvements of the present invention's technical solution compared to Solution 1 were 41.8%, 41.8%, 53.2%, and 41.4%, respectively, and the improvements compared to Solution 2 were 18.6%, 18.7%, 23.6%, and 18.6%, respectively.

[0222] In terms of customer metrics: Compared to Solution 1, the customer conversion rate was 12.3%, the 6-month customer retention rate was 58.7%, the customer churn rate was 41.3%, and the customer satisfaction rating was 7.2 / 10; compared to Solution 2, the customer conversion rate was 15.7%, the 6-month customer retention rate was 65.2%, the customer churn rate was 34.8%, and the customer satisfaction rating was 7.6 / 10; the customer conversion rate of the present invention's technical solution was 19.8%, the 6-month customer retention rate was 76.4%, the customer churn rate was 23.6%, and the customer satisfaction rating was 8.5 / 10. The improvements of the present invention's technical solution compared to Solution 1 were 61.0%, 30.2%, -42.9%, and 18.1%, respectively, and the improvements compared to Solution 2 were 26.1%, 17.2%, -32.2%, and 11.8%, respectively.

[0223] Regarding customer structure indicators: In comparison scheme 1, the proportion of transient customers was 48.2%, medium-term customers was 35.6%, and long-term customers was 16.2%; in comparison scheme 2, the proportion of transient customers was 41.5%, medium-term customers was 38.9%, and long-term customers was 19.6%; in the present invention's technical solution, the proportion of transient customers was 29.7%, medium-term customers was 47.2%, and long-term customers was 23.1%. The changes in the present invention's technical solution relative to comparison scheme 1 were -38.4%, +32.6%, and +42.6%, respectively, and the changes relative to comparison scheme 2 were -28.4%, +21.3%, and +17.9%, respectively.

[0224] Regarding risk indicators: Compared to Scheme 1, the product size volatility is 15.8%, the maximum daily redemption rate is 8.7%, and the customer complaint rate is 0.32%; compared to Scheme 2, the product size volatility is 12.3%, the maximum daily redemption rate is 6.2%, and the customer complaint rate is 0.25%; the product size volatility of the present invention's technical solution is 7.6%, the maximum daily redemption rate is 3.5%, and the customer complaint rate is 0.11%. The reduction rates of the present invention's technical solution compared to Scheme 1 are 51.9%, 59.8%, and 65.6%, respectively, and the reduction rates compared to Scheme 2 are 38.2%, 43.5%, and 56.0%, respectively.

[0225] Regarding model efficiency metrics: Compared to Scheme 1, which requires no training and has a cold start time of 0 days; Scheme 2 has a cold start time of 21 days, 128 model convergence iterations, and a monthly update frequency; the present invention's solution has a cold start time of 5 days, 32 model convergence iterations, and a weekly update frequency. The present invention's solution reduces the cold start time by 76.2% and the number of model convergence iterations by 75.0% compared to Scheme 2.

[0226] In summary, based on the test results, the present invention has the following advantages:

[0227] Significantly improved profitability: The total revenue of the product using the technical solution of this invention is 41.8% higher than that of the traditional uniform pricing method and 18.6% higher than that of the ordinary reinforcement learning pricing method. This is mainly due to the fact that the differentiated pricing strategy can better match the price sensitivity of different customer groups, thereby improving the overall revenue level without losing high-value customers.

[0228] The customer structure has been significantly optimized: the technical solution of this invention successfully reduced the proportion of transient customers from 48.2% to 29.7%, while increasing the proportion of long-term customers from 16.2% to 23.1%. This optimization of the customer structure significantly reduces the product's liquidity risk and improves the stability of funds.

[0229] Enhanced risk control capabilities: The product size volatility of the technical solution of this invention is reduced by 51.9% compared with traditional methods, the maximum single-day redemption ratio is reduced by 59.8%, and the customer complaint rate is reduced by 65.6%. This indicates that multi-dimensional risk analysis and deviation correction mechanisms can effectively identify and control potential risks.

[0230] Model efficiency is significantly improved: the cold start time of the technical solution of this invention is shortened from 21 days to 5 days, and the number of model convergence iterations is reduced from 128 to 32. This is mainly due to the large model generation initialization strategy and the DPO automatic preference learning technology, which significantly improves the sample efficiency of the reinforcement learning model.

[0231] The closed-loop optimization effect is outstanding: During the testing period, the technical solution of this invention triggered three automatic post-training processes. After each post-training, the average JS divergence of the model decreased by more than 15%, and the accuracy of the pricing strategy continued to improve. In contrast, ordinary reinforcement learning methods, due to the lack of bias correction mechanisms, began to show a decline in model performance after the fourth month.

[0232] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for optimizing automated financial product pricing strategies based on reinforcement learning, characterized in that, Includes the following steps: S1: Obtain multi-dimensional data on wealth management products, including product attribute data, customer base data, transaction behavior data, and market environment data; S2: Based on a pre-trained large model in the financial field, product attribute data is deeply encoded to predict the attribute characteristics and distribution probability of the target customer group; S3: Classify the actual customers of the product into three core groups: transient customers, medium-term customers and long-term customers, and extract the actual attribute characteristics and distribution data of each group; S4: Compare the predicted customer distribution with the actual customer distribution, conduct multi-dimensional deviation analysis, and generate deviation analysis results; S5: Using the bias analysis results as a driving signal, the reinforcement learning model is dynamically post-trained using key technologies for large model post-training to update the pricing strategy network. S6: Based on the updated reinforcement learning model, output differentiated pricing decisions for different customer groups; S7: Collect data on the effectiveness of pricing decisions and feed it back to step S4 for a new round of deviation analysis to form a closed-loop optimization mechanism.

2. The method for optimizing automated financial product pricing strategies based on reinforcement learning according to claim 1, characterized in that, Step S2 specifically includes: S21: Input the product's structured and unstructured data into a pre-trained large model in the financial field. The structured data includes the expected rate of return range, risk level, investment period, minimum investment amount, fee structure and liquidity terms. The unstructured data includes the product prospectus, investment strategy, risk disclosure statement and fund manager introduction. S22: Use LoRA low-rank adaptation technology to perform lightweight fine-tuning on the large model, extract the product's return characteristics, risk characteristics, liquidity characteristics, threshold characteristics and service characteristics, and generate a product feature vector with a dimension of 768. S23: Input the product feature vector into the large model inference module to generate a detailed profile of the target customer group, including age range, income level, risk tolerance, investment experience, asset size, liquidity preference and investment term preference; S24: Output the predicted customer group distribution probability matrix, with dimensions of [number of customer groups, feature dimension], and calculate the expected conversion rate, expected average order value and expected lifetime value for each customer group.

3. The method for optimizing automated financial product pricing strategies based on reinforcement learning according to claim 1, characterized in that, Step S3 specifically includes: S31: Based on the RFM model, extract three core indicators for each actual user: the time of their most recent purchase, the frequency of their purchase, and the amount of their purchase. S32: The K-means++ clustering algorithm is used to perform cluster analysis on all actual users, dividing them into three major groups: transient customers, medium-term customers, and long-term customers; S33: Establish a corresponding attribute feature set for each customer group, extract the same feature dimensions as the prediction target, and generate the actual customer group distribution probability matrix; S34: Calculate the actual conversion rate, actual average order value, actual lifetime value, and actual churn rate for each customer group, and analyze the subscription and redemption patterns, reactions to market fluctuations, and responsiveness to marketing activities for different groups.

4. The method for optimizing automated financial product pricing strategies based on reinforcement learning according to claim 1, characterized in that, The multi-dimensional deviation analysis in step S4 includes basic attribute deviation analysis, specifically: Calculate the KL divergence and JS divergence between the predicted distribution and the actual distribution for each feature dimension to quantify the degree of deviation for each dimension; Identify the top 5 feature dimensions with the largest deviations and analyze the root causes of these deviations. Generate a basic attribute deviation report to indicate the potential impact of the deviation on the product pricing strategy.

5. The method for optimizing automated financial product pricing strategies based on reinforcement learning according to claim 4, characterized in that, The multi-dimensional deviation analysis in step S4 includes population distribution deviation analysis, specifically: Calculate the deviation rate between the actual and predicted proportions of transient customers, mid-term customers, and long-term customers; Analyze the impact of group distribution bias on overall product returns, liquidity, and stability; Generate a population distribution deviation report and label the population types whose deviations exceed a preset threshold.

6. The method for optimizing automated financial product pricing strategies based on reinforcement learning according to claim 5, characterized in that, The multi-dimensional deviation analysis in step S4 includes risk analysis, specifically: Liquidity risk analysis: Assess the risk of concentrated redemptions that may result from an excessively high proportion of short-term customers in the actual customer distribution, and use the VaR method to quantify potential liquidity losses; Credit risk analysis: Assess the potential for complaints and disputes that may result from customers' actual risk tolerance being lower than the product's risk level, and calculate the probability of risk events occurring; Profitability and Risk Analysis: Assess the risk of fluctuations in product scale and management fee income caused by changes in the actual customer structure, and predict the profitability volatility for the next 3 months; Generate a comprehensive risk assessment report to provide risk constraints for adjusting pricing strategies.

7. The method for optimizing automated financial product pricing strategies based on reinforcement learning according to claim 1, characterized in that, The multi-dimensional deviation analysis in step S4 also includes strengths analysis and weaknesses analysis, specifically: Strengths Analysis: Identify high-value groups in the actual customer distribution that exceed expectations, and analyze their common characteristics. Assess its contribution to overall product revenue and explore cross-selling and upselling opportunities; Defect analysis: Identify the target groups that are below expectations in the actual customer distribution, analyze the reasons for their churn, evaluate the specific manifestations of the pricing strategy's lack of appeal to these groups, and identify problems in product design and marketing promotion; Generate a strengths and weaknesses analysis report to provide guidance for optimizing pricing strategies.

8. The method for optimizing automated financial product pricing strategies based on reinforcement learning according to claim 7, characterized in that, The basic construction steps of the reinforcement learning model in step S5 are as follows: Construct a state space S, which includes product state, customer state, market state, and deviation state. Product state includes current yield, size, and remaining maturity; customer state includes customer type, risk tolerance, and historical transaction records; market state includes market index, interest rate level, and yield of similar products; and deviation state includes deviation values ​​for each dimension. Construct action space A, which includes adjustable parameters for pricing strategies, including subscription fee rate, redemption fee rate, management fee, sales service fee, performance fee accrual ratio and accrual method; Construct a comprehensive reward function The formula is: ； in, , , , The pre-set weighting coefficients indicate that the bias penalty term is positively correlated with the results of the multi-dimensional bias analysis. The PPO (Proximity Policy Optimization) algorithm is used as the core algorithm for reinforcement learning to initialize the policy network and value network.

9. The method for optimizing automated financial product pricing strategies based on reinforcement learning according to claim 8, characterized in that, The specific steps for dynamic post-training using key technologies for large model post-training in step S5 are as follows: S51: Utilize the knowledge reasoning ability of the large model to generate an initial pricing strategy. Input product attributes, target customer groups and market environment information into the large model to generate an initial pricing range that conforms to industry common sense and regulatory requirements, which serves as an action space constraint for the reinforcement learning model. S52: Automatically generate a preference dataset based on the deviation analysis results. Each sample contains a state, good actions, and bad actions. Good actions are pricing actions that can effectively reduce the deviation between the predicted distribution and the actual distribution, while increasing returns and reducing risks. Bad actions are pricing actions that will amplify the deviation or lead to increased risks. S53: Use the DPO direct preference optimization algorithm to fine-tune the policy network of the reinforcement learning model without training the reward model separately; S54: Uses LoRA technology to achieve lightweight fine-tuning, updating only 1%-5% of the parameters in the policy network, improving training efficiency and preventing catastrophic forgetting.

10. The method for optimizing automated financial product pricing strategies based on reinforcement learning according to claim 1, characterized in that, The closed-loop optimization mechanism in step S7 specifically includes: Elastic weights are used to reinforce the EWC algorithm to prevent the model from forgetting previously learned effective pricing strategies when learning new bias correction knowledge. Collect new customer and transaction data regularly, weekly or monthly, and update the deviation analysis results; When the deviation value in any dimension exceeds the preset threshold, the post-training process of the model is automatically triggered. Establish a model version management system to record the model parameters, corresponding deviation analysis results, and pricing effect data for each version, and support model rollback and comparative analysis.

Images