An individualized explainable recommendation system and method based on deep reinforcement learning

Abstract

The application discloses a kind of personalized explainable recommendation system based on deep reinforcement learning, it includes state modeling module by fusing user historical behavior sequence data, commodity feature data, environmental variable data, obtain comprehensive state vector;Reinforcement learning decision module is based on comprehensive state vector Markov decision framework is constructed, and the optimal recommended action and corresponding output probability are obtained by using proximal policy optimization algorithm to train reinforcement learning strategy network model;Explainability analysis module is by SHAP method to the output probability of optimal recommended action carries out feature contribution degree attribution analysis, obtains the feature contribution value corresponding to each feature, obtains the feature contribution weight corresponding to each feature after normalization processing.The application can provide quantifiable, traceable feature contribution explanation for each recommendation decision.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, specifically to a personalized and explainable recommendation system and method based on deep reinforcement learning. Background Technology

[0002] Personalized recommendation systems in internet platforms are core infrastructure for improving user experience and platform operational efficiency. Recommendation systems have evolved from static prediction to sequential decision-making problems, facing the dual challenges of increasing data and scenario complexity, and higher requirements for interpretability and compliance auditing. Existing recommendation technologies can be divided into three categories: static recommendation based on rules or supervised learning, recommendation decision-making based on reinforcement learning, and recommendation interpretability. Static recommendation cannot characterize feedback loops and is difficult to optimize long-term multi-objectives and continuous actions. Although reinforcement learning recommendation can be dynamically optimized, it has strong black box characteristics, high trust and governance costs, and weak interpretability of multi-objective rewards. Interpretable methods are mostly post-processing of supervised learning models, which is separate from the sequential decision-making of reinforcement learning and cannot form a closed loop from decision-making to interpretation to governance. In engineering practice, dynamic decision-making and interpretable governance are separated, and high-risk recommendation scenarios such as coupons lack auditable decision-making basis. Summary of the Invention

[0003] The purpose of this invention is to provide a personalized and explainable recommendation system and method based on deep reinforcement learning. This invention can provide quantifiable and traceable explanations of feature contributions for each recommendation decision while maintaining the dynamic policy optimization capability.

[0004] To achieve this objective, the present invention presents a personalized, explainable recommendation system based on deep reinforcement learning, comprising: The state modeling module is used to fuse and update the user's historical behavior sequence data through a recurrent neural network to obtain the user's current state vector, to process the product feature data through a multilayer perceptron to obtain the product feature vector, to process the environmental variable data through an embedding layer to obtain the environmental feature vector, and to concatenate the user's current state vector, product feature vector, and environmental feature vector to obtain the comprehensive state vector. The reinforcement learning decision module is used to construct a Markov decision framework based on the comprehensive state vector, and to iteratively optimize the reinforcement learning policy network model under the Markov decision framework using the proximal policy optimization algorithm to obtain the reinforcement learning policy network model with optimal parameters. Using the comprehensive state vector, the optimal recommended action and the output probability of the optimal recommended action are obtained through the reinforcement learning policy network model with optimal parameters. The interpretability analysis module is used to perform feature contribution attribution analysis on the output probability of the optimal recommended action using the SHAP method, to obtain the feature contribution value corresponding to each feature in the user's historical behavior sequence data, product feature data and environmental variable data, and to normalize the absolute value of the feature contribution value corresponding to each feature to obtain the feature contribution weight corresponding to each feature.

[0005] Preferably, the decision output and governance module sorts all features from highest to lowest according to their feature contribution weights, selects the top k features as core features, sets the k core features as core feature sets, calculates the sum of the feature contribution weights of all core features in the core feature set, and if the sum of the feature contribution weights of all core features in the core feature set is greater than or equal to the explanatory consistency threshold, then an explainable recommendation result is generated based on the core features; if the sum of the feature contribution weights of all core features in the core feature set is less than the explanatory consistency threshold, then manual auditing is initiated.

[0006] Preferably, the specific process of obtaining the user's current state vector by fusing and updating the user's historical behavior sequence data through a recurrent neural network is as follows: The sequence of user historical behavior can be represented as ,in This represents the user behavior feature vector at time t; User historical behavior sequences are analyzed using a recurrent neural network (GRU). The processing and encoding are performed, and the expression is: in, This represents the user state vector at time t; The GRU unit reads the user's historical behavior sequence step by step. At each time step t, the current input will be combined. and the user state vector at time t-1 After processing the entire user's historical behavior sequence through a gating structure, the user state vector at the last time step is calculated. This serves as the user's current state vector, representing the user's historical behavior sequence.

[0007] Preferably, the specific process of constructing a Markov decision framework based on the comprehensive state vector is as follows: Based on the comprehensive state vector Define the recommended actions that can be executed. And the reward function reflects the execution of the recommended actions. The decision effect is expressed as: in, This indicates the click-through rate. This represents the coefficient of click-through rate in the total reward. Indicates revenue, This represents the coefficient of the income in the total reward. Indicates satisfaction. This represents the coefficient of satisfaction in the total reward. Indicates cost, This represents the coefficient of cost in the total reward. This represents the reward function value.

[0008] Preferably, the reinforcement learning policy network model under the Markov decision framework is iteratively optimized using a proximal policy optimization algorithm to obtain a reinforcement learning policy network model with optimal parameters. The specific process of obtaining the optimal recommended action and its output probability using the integrated state vector through the optimal reinforcement learning policy network model is as follows: The reinforcement learning policy network model undergoes random parameter initialization before training begins; The reinforcement learning policy network model is based on the received integrated state vector Output recommended actions And recommended actions Based on the output probability, execute the recommended action. Then, based on the reward function, the reinforcement learning policy network model is iteratively optimized using the proximal policy optimization algorithm, and the parameters of the reinforcement learning policy network model are continuously updated. When the output value of the reward function is maximized, the reinforcement learning policy network model with optimal parameters is obtained. The optimal reinforcement learning policy network model, using the comprehensive state vector, can obtain the optimal recommended action and the output probability of the optimal recommended action in real time.

[0009] Preferably, the specific process of performing feature contribution attribution analysis on the output probability of the optimal recommended action using the SHAP method to obtain the feature contribution value corresponding to each feature in the user's historical behavior sequence data, product feature data, and environmental variable data is as follows: The output probability of the optimal recommended action is analyzed for feature contribution attribution using the SHAP method, and the expression is as follows: in, f S represents the optimal reinforcement learning policy network model function, where S denotes the comprehensive state vector. This represents the i-th feature in the overall state vector S. Indicates the first i The feature contribution value of each feature; Optimal Reinforcement Learning Policy Network Model Functionf The expression is: f = π(A|S) Where A represents the optimal recommended action, S represents the comprehensive state vector, and π(A|S) represents the output probability of the optimal recommended action; Ultimately, we obtain the feature contribution values ​​corresponding to each feature in the user's historical behavior sequence data, product feature data, and environmental variable data.

[0010] Preferably, the specific process of normalizing the absolute values ​​of the feature contribution values ​​corresponding to each feature to obtain the feature contribution weights corresponding to each feature is as follows: The absolute values ​​of the feature contribution values ​​corresponding to each feature in the user's historical behavior sequence data, product feature data, and environmental variable data are normalized using the following expression: in, j Represents the total amount of characteristics. express j Sum of the absolute values ​​of the feature contribution values ​​of each feature Indicates the first i The feature contribution weights of each feature.

[0011] A personalized, interpretable recommendation method based on deep reinforcement learning includes the following steps: The user's current state vector is obtained by fusing and updating the user's historical behavior sequence data through a recurrent neural network. The product feature vector is obtained by processing the product feature data through a multilayer perceptron. The environmental feature vector is obtained by processing the environmental variable data through an embedding layer. The user's current state vector, product feature vector, and environmental feature vector are concatenated to obtain a comprehensive state vector. Based on the comprehensive state vector, a Markov decision framework is constructed. The reinforcement learning policy network model under the Markov decision framework is iteratively optimized using the proximal policy optimization algorithm to obtain the reinforcement learning policy network model with optimal parameters. Using the comprehensive state vector, the optimal recommended action and the output probability of the optimal recommended action are obtained through the reinforcement learning policy network model with optimal parameters. The SHAP method is used to perform feature contribution attribution analysis on the output probability of the optimal recommended action, and the feature contribution value corresponding to each feature in the user's historical behavior sequence data, product feature data and environmental variable data is obtained. The absolute value of the feature contribution value corresponding to each feature is normalized to obtain the feature contribution weight corresponding to each feature.

[0012] A computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described method.

[0013] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0014] The beneficial effects of this invention are: This invention directly applies the SHAP feature contribution mechanism to the output probability of the policy network, generating quantified feature contribution weights for each recommendation action, thus reducing the operational and compliance risks caused by algorithm opacity. This invention integrates dynamic policy optimization and interpretability analysis into a unified technical chain, ensuring that the interpretation results truly reflect the decision-making logic, forming a complete closed loop from decision-making to interpretation and governance. This invention significantly improves the stability and controllability of recommendation strategies in multi-objective business scenarios. The reward function can integrate multiple objectives such as conversion rate, revenue, and satisfaction, and demonstrates the trade-offs between these objectives, thereby avoiding business deviations caused by optimizing a single objective. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the structure of the present invention; Figure 2 Example graph to explain the prediction results; Figure 3 A heatmap showing the weights contributed to the SHAP features generated in this invention; Figure 4 This is a flowchart of the present invention. Detailed Implementation

[0016] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1 A personalized, interpretable recommendation system based on deep reinforcement learning, such as... Figure 1 As shown, it includes: The state modeling module is used to fuse and update the user's historical behavior sequence data through a recurrent neural network to obtain the user's current state vector, process the product feature data through a multilayer perceptron to obtain the product feature vector, process the environmental variable data through an embedding layer to obtain the environmental feature vector, and concatenate the user's current state vector, product feature vector, and environmental feature vector to obtain the comprehensive state vector. This design, by fusing and updating the user's historical behavior sequence through a recurrent neural network, processing product features through a multilayer perceptron, and processing environmental variables through an embedding layer and concatenating them to obtain the comprehensive state vector, can completely depict the comprehensive state of the user, product, and environment, making the comprehensive state representation fit the real recommendation scenario. The reinforcement learning decision module is used to construct a Markov decision framework based on the comprehensive state vector. It then uses the proximal policy optimization algorithm (PPO policy update) to iteratively optimize the reinforcement learning policy network model under the Markov decision framework, obtaining the optimal reinforcement learning policy network model. Using the comprehensive state vector, the optimal recommended action and its output probability are obtained through the optimal reinforcement learning policy network model. This design, based on the comprehensive state vector, constructs a Markov decision framework and uses the proximal policy optimization algorithm to iteratively optimize the policy network, obtaining the optimal recommended action and output probability. It can achieve dynamic sequential decision optimization for recommendations, adapt to dynamic changes in user interests and the environment, and take into account multi-objective reward optimization, thereby improving the long-term benefits and decision rationality of recommended actions. The interpretability analysis module is used to perform feature contribution attribution analysis on the output probability of the optimal recommendation action using the SHAP method (SHAP stands for SHapley Additive explanations). This yields the feature contribution values ​​corresponding to each feature in the user's historical behavior sequence data, product feature data, and environmental variable data. The absolute values ​​of the feature contribution values ​​corresponding to each feature are normalized to obtain the feature contribution weights corresponding to each feature. This design uses the SHAP method to perform feature contribution attribution analysis on the output probability of the optimal recommendation action, and then normalizes the results to obtain feature contribution weights. This quantifies the contribution of each feature to the recommendation decision, breaking the black box state of reinforcement learning decision-making and providing accurate feature-level explanations for the recommendation results.

[0017] For specific content such as user historical behavior sequence data, product feature data, and environmental variable data, some optimized technical solutions include: User historical behavior sequence data includes user clicks, browsing, adding to cart, and purchase behavior; product characteristic data includes product price, category, brand, and historical conversion rate; environmental variable data includes inventory (product inventory status), time (such as time of day, season), activity status (such as whether there is a promotional activity), and user satisfaction (such as historical ratings or feedback).

[0018] In the above technical solution, the decision output and governance module sorts all features from highest to lowest according to their contribution weights, selects the top k features (k can be 3) as core features, and sets these k core features as the core feature set. It then calculates the sum of the feature contribution weights of all core features in the core feature set. If the sum of the feature contribution weights of all core features in the core feature set is greater than or equal to the explanatory consistency threshold, then an interpretable recommendation result is generated based on the core features. Figure 2As shown; if the sum of the feature contribution weights of all core features in the core feature set is less than the interpretation consistency threshold, then it enters the manual audit; the above design sorts and filters core features according to feature contribution weight, and judges whether to generate an interpretable recommendation result or enter the manual audit based on the interpretation consistency threshold, which can ensure the effectiveness and credibility of the recommendation explanation and realize the connection between automated governance of recommendation decision and manual audit.

[0019] In the above technical solution, the specific process of obtaining the user's current state vector by fusing and updating the user's historical behavior sequence data through a recurrent neural network is as follows: The sequence of user historical behavior can be represented as ,in This represents the user behavior feature vector at time t; User historical behavior sequences are analyzed using a recurrent neural network (GRU). The processing and encoding are performed, and the expression is: in, This represents the user state vector at time t; The GRU unit reads the user's historical behavior sequence step by step. At each time step t, the current input will be combined. and the user state vector at time t-1 After processing the entire user's historical behavior sequence through a gating structure, the user state vector at the last time step is calculated. The above design uses a GRU recurrent neural network to process the user's historical behavior sequence to obtain the user's current state vector. GRU can efficiently capture the temporal dependence and interest changes of user behavior, accurately represent the user's dynamic state, and improve the reliability of the user's current state vector.

[0020] In some preferred embodiments of this invention, the code for obtaining the comprehensive state vector is as follows: h = GRU(user_seq) g = MLP(item_feat) S = torch.cat([h, g, env], dim=1) In the code above, a gated recurrent unit network is used to process the input user_seq (user's historical behavior sequence), and the output result is h, which is the user's current state vector. The user's current state vector encodes the user's dynamic interests and preferences. A multilayer perceptron is used to process the input item_feat (item feature data), and the output result is g, which is the item feature vector. The PyTorch cat function is used to concatenate the user's current state vector h, the item feature vector g, and the environment feature vector env on the feature dimension (dim=1) to generate a comprehensive state vector S, which serves as the input for subsequent reinforcement learning decisions.

[0021] In the above technical solution, the specific process of constructing the Markov decision framework based on the comprehensive state vector is as follows: Based on the comprehensive state vector Define the recommended actions that can be executed. And the reward function reflects the execution of the recommended actions. The decision effect is expressed as: in, This indicates the click-through rate. This represents the coefficient of click-through rate in the total reward. Indicates revenue, This represents the coefficient of the income in the total reward. Indicates satisfaction. This represents the coefficient of satisfaction in the total reward. Indicates cost, This represents the coefficient of cost in the total reward. This represents the reward function value. The above design defines the recommendation action and designs a reward function that integrates click-through rate, revenue, satisfaction, and cost when constructing the Markov decision framework. This enables multi-objective optimization trade-offs, aligns with real business objectives, avoids the drawbacks of single-objective optimization, and makes decisions more in line with business needs.

[0022] In the above technical solution, the reinforcement learning policy network model under the Markov decision framework is iteratively optimized using the proximal policy optimization algorithm to obtain the reinforcement learning policy network model with optimal parameters. The specific process of obtaining the optimal recommended action and its output probability using the integrated state vector and the optimal reinforcement learning policy network model is as follows: The reinforcement learning policy network model undergoes random parameter initialization before training begins; The reinforcement learning policy network model is based on the received integrated state vector Output recommended actions And recommended actions Based on the output probability, execute the recommended action. Then, based on the reward function, the reinforcement learning policy network model is iteratively optimized using the proximal policy optimization algorithm, and the parameters of the reinforcement learning policy network model are continuously updated. When the output value of the reward function is maximized, the reinforcement learning policy network model with optimal parameters is obtained. The optimal reinforcement learning policy network model, using the comprehensive state vector, can obtain the optimal recommended action and its output probability in real time. The above design iteratively optimizes the reinforcement learning policy network through a proximal policy optimization algorithm until the reward function output is maximized. This can stably and efficiently optimize the policy, avoid excessive fluctuations in policy updates, ensure the stability and convergence of the reinforcement learning policy network model training, and output the optimal recommended action and probability in real time, meeting real-time requirements.

[0023] For the near-end policy optimization algorithm, iterative optimization of reinforcement learning policy network models under the Markov decision framework is performed. Some optimization techniques include: The expression for PPO policy update is: in, The optimization objective function for updating the PPO strategy. This represents the expectation of the sampled interaction trajectory data. This represents the ratio of the probability of the new strategy and the probability of the old strategy on the same action. This indicates the value of the dominance function for the corresponding action; This is a truncation function used to limit the magnitude of policy updates; The preset policy change threshold; The implementation code for PPO strategy update is as follows: ratio = new_prob / old_prob loss = torch.min(ratio * adv, torch.clamp(ratio, 1-eps, 1+eps) * adv) In the code above, the probability ratio before and after the policy update is calculated. `new_prob` is the probability that the policy network will choose the current action after the update, and `old_prob` is the probability that the policy network will choose the action before the update. The `old_prob` ratio measures the magnitude of the policy update. The probability ratio is then pruned, limiting it to the interval [1-eps, 1+eps]. The probability ratio is a hyperparameter used to rigidly limit the maximum change in a single policy update. The loss function for PPO is calculated, taking the smaller value between the unpruned target (ratio * adv) and the pruned target (clamp(ratio, 1-eps, 1+eps) * adv). Here, `adv` is the advantage function, representing the degree to which the current action is better than the average level. By minimizing this loss (or maximizing its negative value), the policy network will steadily optimize in the direction of increasing the probability of advantageous actions while ensuring that the update magnitude is not too large.

[0024] In the above technical solution, the specific process of performing feature contribution attribution analysis on the output probability of the optimal recommended action using the SHAP method to obtain the feature contribution values ​​corresponding to each feature in the user's historical behavior sequence data, product feature data, and environmental variable data is as follows: The output probability of the optimal recommended action is analyzed for feature contribution attribution using the SHAP method, and the expression is as follows: in, f S represents the optimal reinforcement learning policy network model function, where S denotes the comprehensive state vector. This represents the i-th feature in the overall state vector S. Indicates the first i The feature contribution value of each feature; Optimal Reinforcement Learning Policy Network Model Function f The expression is: f = π(A|S) Where A represents the optimal recommended action, S represents the comprehensive state vector, and π(A|S) represents the output probability of the optimal recommended action; Finally, the feature contribution values ​​corresponding to each feature in the user's historical behavior sequence data, product feature data, and environmental variable data are obtained. The above design uses the SHAP method to perform feature contribution attribution analysis on the output probability of the reinforcement learning strategy network model with optimal parameters, accurately calculates the contribution value of each feature, and can provide accurate data for subsequent weight calculation.

[0025] In some preferred embodiments of this invention, when performing feature contribution attribution analysis on the output probability of the optimal recommended action using the SHAP method, the generated heatmap of SHAP feature contribution weights is as follows: Figure 3 As shown, the code for SHAP interpretability analysis is as follows: explainer = shap.Explainer(policy_net) phi = explainer(state) weights = abs(phi) / sum(abs(phi)) In the code above, the SHAP library is used to initialize a SHAP interpreter with the trained policy_net (i.e., the reinforcement learning policy network model with optimal parameters) as the interpreter object. The current integrated state vector `state` is input into the SHAP interpreter. The SHAP interpreter runs the SHAP algorithm to calculate the contribution of each feature in the state vector `state` to the final output of the policy network (i.e., the probability of selecting a specific recommended action). The output is `phi`, which is the SHAP value vector of each feature. Positive values ​​indicate that the feature has a positive impact on the decision, while negative values ​​indicate inhibition. First, the absolute value of the SHAP value `phi` is taken to obtain the absolute influence of each feature's contribution. Then, the absolute influence of all features is summed as the denominator. Finally, the absolute influence of each feature is divided by this sum and normalized. The output `weights` is the normalized feature contribution weight. Each weight value is between 0 and 1, and the sum of all weights is 1, which intuitively reflects the percentage of relative importance of each feature to this decision.

[0026] In the above technical solution, the specific process of normalizing the absolute value of the feature contribution value corresponding to each feature to obtain the feature contribution weight corresponding to each feature is as follows: The absolute values ​​of the feature contribution values ​​corresponding to each feature in the user's historical behavior sequence data, product feature data, and environmental variable data are normalized using the following expression: in, j Represents the total amount of characteristics. express j Sum of the absolute values ​​of the feature contribution values ​​of each feature Indicates the first i The feature contribution weights are calculated by normalizing the absolute values ​​of the feature contribution values. This normalizes the feature contribution weights, unifies the feature contribution measurement standard, eliminates the influence of dimensions, and allows for a direct comparison of the contribution levels of different features, facilitating the selection of core features and the generation of interpretations.

[0027] Example 2 A personalized, interpretable recommendation method based on deep reinforcement learning, such as... Figure 4 As shown, a comprehensive state vector is obtained by fusing user historical behavior sequence data, product feature data, and environmental variable data. A Markov decision framework is constructed based on the comprehensive state vector, and the reinforcement learning policy network model is trained using the proximal policy optimization algorithm to obtain the optimal recommended action and its corresponding output probability. The output probability of the optimal recommended action is analyzed by feature contribution attribution using the SHAP method to obtain the feature contribution value corresponding to each feature. After normalization, the feature contribution weight corresponding to each feature is obtained.

[0028] The specific method for personalized and interpretable recommendations based on deep reinforcement learning includes the following steps: The user's current state vector is obtained by fusing and updating the user's historical behavior sequence data through a recurrent neural network. The product feature vector is obtained by processing the product feature data through a multilayer perceptron. The environmental feature vector is obtained by processing the environmental variable data through an embedding layer. The user's current state vector, product feature vector, and environmental feature vector are concatenated to obtain a comprehensive state vector. Based on the comprehensive state vector, a Markov decision framework is constructed. The reinforcement learning policy network model under the Markov decision framework is iteratively optimized using the proximal policy optimization algorithm to obtain the reinforcement learning policy network model with optimal parameters. Using the comprehensive state vector, the optimal recommended action and the output probability of the optimal recommended action are obtained through the reinforcement learning policy network model with optimal parameters. The SHAP method is used to perform feature contribution attribution analysis on the output probability of the optimal recommended action, and the feature contribution value corresponding to each feature in the user's historical behavior sequence data, product feature data and environmental variable data is obtained. The absolute value of the feature contribution value corresponding to each feature is normalized to obtain the feature contribution weight corresponding to each feature.

[0029] Example 3 A computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in Embodiment 2.

[0030] Example 4 A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in Embodiment 2.

[0031] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0032] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0033] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0034] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0035] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the pending claims of the invention.

[0036] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A personalized, explainable recommendation system based on deep reinforcement learning, characterized in that, It includes: The state modeling module is used to fuse and update the user's historical behavior sequence data through a recurrent neural network to obtain the user's current state vector, to process the product feature data through a multilayer perceptron to obtain the product feature vector, to process the environmental variable data through an embedding layer to obtain the environmental feature vector, and to concatenate the user's current state vector, product feature vector, and environmental feature vector to obtain the comprehensive state vector. The reinforcement learning decision module is used to construct a Markov decision framework based on the comprehensive state vector, and to iteratively optimize the reinforcement learning policy network model under the Markov decision framework using the proximal policy optimization algorithm to obtain the reinforcement learning policy network model with optimal parameters. Using the comprehensive state vector, the optimal recommended action and the output probability of the optimal recommended action are obtained through the reinforcement learning policy network model with optimal parameters. The interpretability analysis module is used to perform feature contribution attribution analysis on the output probability of the optimal recommended action using the SHAP method, to obtain the feature contribution value corresponding to each feature in the user's historical behavior sequence data, product feature data and environmental variable data, and to normalize the absolute value of the feature contribution value corresponding to each feature to obtain the feature contribution weight corresponding to each feature.

2. The personalized, explainable recommendation system based on deep reinforcement learning according to claim 1, characterized in that, It also includes: The decision output and governance module sorts all features from highest to lowest according to their contribution weights, selects the top k features as core features, and sets these k core features as the core feature set. It calculates the sum of the feature contribution weights of all core features in the core feature set. If the sum of the feature contribution weights of all core features in the core feature set is greater than or equal to the explanatory consistency threshold, an interpretable recommendation result is generated based on the core features. If the sum of the feature contribution weights of all core features in the core feature set is less than the explanatory consistency threshold, the system proceeds to manual auditing.

3. The personalized, explainable recommendation system based on deep reinforcement learning according to claim 1, characterized in that: The specific process of obtaining the user's current state vector by fusing and updating the user's historical behavior sequence data through a recurrent neural network is as follows: The sequence of user historical behavior can be represented as ,in This represents the user behavior feature vector at time t; User historical behavior sequences are analyzed using a recurrent neural network (GRU). The processing and encoding are performed, and the expression is: in, This represents the user state vector at time t; The GRU unit reads the user's historical behavior sequence step by step. At each time step t, the current input will be combined. and the user state vector at time t-1 After processing the entire user's historical behavior sequence through a gating structure, the user state vector at the last time step is calculated. This serves as the user's current state vector, representing the user's historical behavior sequence.

4. The personalized, explainable recommendation system based on deep reinforcement learning according to claim 1, characterized in that: The specific process of constructing a Markov decision framework based on the aforementioned integrated state vector is as follows: Based on the comprehensive state vector Define the recommended actions that can be executed. And the reward function reflects the execution of the recommended actions. The decision effect is expressed as: in, This indicates the click-through rate. This represents the coefficient of click-through rate in the total reward. Indicates revenue, This represents the coefficient of the income in the total reward. Indicates satisfaction. This represents the coefficient of satisfaction in the total reward. Indicates cost, This represents the coefficient of cost in the total reward. This represents the reward function value.

5. A personalized, explainable recommendation system based on deep reinforcement learning according to claim 4, characterized in that: The reinforcement learning policy network model under the Markov decision framework is iteratively optimized using the proximal policy optimization algorithm to obtain the reinforcement learning policy network model with optimal parameters. The specific process of obtaining the optimal recommended action and its output probability through the optimal reinforcement learning policy network model using the comprehensive state vector is as follows: The reinforcement learning policy network model undergoes random parameter initialization before training begins; The reinforcement learning policy network model is based on the received integrated state vector Output recommended actions And recommended actions Based on the output probability, execute the recommended action. Then, based on the reward function, the reinforcement learning policy network model is iteratively optimized using the proximal policy optimization algorithm, and the parameters of the reinforcement learning policy network model are continuously updated. When the output value of the reward function is maximized, the reinforcement learning policy network model with optimal parameters is obtained. The optimal reinforcement learning policy network model, using the comprehensive state vector, can obtain the optimal recommended action and the output probability of the optimal recommended action in real time.

6. The personalized, explainable recommendation system based on deep reinforcement learning according to claim 1, characterized in that: The specific process of performing feature contribution attribution analysis on the output probability of the optimal recommended action using the SHAP method to obtain the feature contribution value corresponding to each feature in the user historical behavior sequence data, product feature data, and environmental variable data is as follows: The output probability of the optimal recommended action is analyzed for feature contribution attribution using the SHAP method, and the expression is as follows: in, f S represents the optimal reinforcement learning policy network model function, where S denotes the comprehensive state vector. This represents the i-th feature in the overall state vector S. Indicates the first i The feature contribution value of each feature; Optimal Reinforcement Learning Policy Network Model Function f The expression is: f = π(A|S) Where A represents the optimal recommended action, S represents the comprehensive state vector, and π(A|S) represents the output probability of the optimal recommended action; Ultimately, we obtain the feature contribution values ​​corresponding to each feature in the user's historical behavior sequence data, product feature data, and environmental variable data.

7. A personalized, explainable recommendation system based on deep reinforcement learning according to claim 1, characterized in that: The specific process of normalizing the absolute values ​​of the feature contribution values ​​corresponding to each feature to obtain the feature contribution weights corresponding to each feature is as follows: The absolute values ​​of the feature contribution values ​​corresponding to each feature in the user's historical behavior sequence data, product feature data, and environmental variable data are normalized using the following expression: in, j Represents the total amount of characteristics. express j Sum of the absolute values ​​of the feature contribution values ​​of each feature Indicates the first i The feature contribution weights of each feature.

8. A personalized, interpretable recommendation method based on deep reinforcement learning, characterized in that, It includes the following steps: The user's current state vector is obtained by fusing and updating the user's historical behavior sequence data through a recurrent neural network. The product feature vector is obtained by processing the product feature data through a multilayer perceptron. The environmental feature vector is obtained by processing the environmental variable data through an embedding layer. The user's current state vector, product feature vector, and environmental feature vector are concatenated to obtain a comprehensive state vector. Based on the comprehensive state vector, a Markov decision framework is constructed. The reinforcement learning policy network model under the Markov decision framework is iteratively optimized using the proximal policy optimization algorithm to obtain the reinforcement learning policy network model with optimal parameters. Using the comprehensive state vector, the optimal recommended action and the output probability of the optimal recommended action are obtained through the reinforcement learning policy network model with optimal parameters. The SHAP method is used to perform feature contribution attribution analysis on the output probability of the optimal recommended action, and the feature contribution value corresponding to each feature in the user's historical behavior sequence data, product feature data and environmental variable data is obtained. The absolute value of the feature contribution value corresponding to each feature is normalized to obtain the feature contribution weight corresponding to each feature.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in claim 8.

Images