A deep reinforcement learning commodity recommendation method fusing user emotional features

Abstract

The application discloses a kind of deep reinforcement learning commodity recommendation methods of fusing user emotional characteristics, belong to recommended system and artificial intelligence technical field. Including obtaining user comment text and rating data, extract emotional features by pre-training language model and fuse with rating, construct emotional perception user-item interaction sequence;The sequence is encoded based on sequence modeling network, generate the long-term and short-term preference representation of user;Through hierarchical attention mechanism, the long-term and short-term preference of user is fused, and the comprehensive state representation is generated by combining the context information of item;Input reinforcement learning strategy network, calculate exploration rate based on the preference evolution of user, realize on-demand exploration and generate recommendation result.The application fuses emotional correction feedback bias, long-term and short-term preference decoupling modeling accurately depicts user interest evolution, improves the recommendation effect in cold start and interest mutation scene by combining adaptive exploration mechanism, improves the accuracy of recommendation, user experience and system long-term benefit.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and personalized recommendation technology, specifically to a deep reinforcement learning-based product recommendation method that integrates user emotional features. Background Technology

[0002] Recommender systems, as a key technology for addressing information overload and personalized user needs, have evolved from early collaborative filtering and content filtering to today's deep reinforcement learning methods, aiming to filter personalized content for users from massive amounts of information. Traditional methods mainly rely on explicit behavioral data such as clicks, purchases, and ratings for modeling. However, user ratings are often influenced by objective factors and deviate from their true subjective preferences, making it difficult to accurately reflect user attitudes and intentions based solely on ratings.

[0003] Existing research attempts to incorporate sentiment signals from comment texts to enrich user representations, but most methods remain at the level of superficial text processing or coarse-grained sentiment classification, failing to truly integrate user emotional information with user behavioral history. Meanwhile, while reinforcement learning-based dynamic recommendation methods possess sequential decision-making capabilities, they often fail to explicitly distinguish between short-term interest fluctuations and long-term stable preferences, making it difficult for recommendation strategies to accurately adapt to the dynamic evolution of user interests. Furthermore, the fixed exploration strategies used in traditional methods cannot adapt to complex scenarios such as cold start and sudden interest changes, limiting the overall adaptability and long-term benefits of the system. These shortcomings collectively constrain further improvements in the accuracy, personalization level, and scenario robustness of recommendation systems. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a deep reinforcement learning-based product recommendation method that integrates user emotional features. This invention considers designing an emotion-perception feedback fusion module, which extracts emotional polarity and confidence levels from comment text and weights them with user ratings to achieve mutual correction, constructing an interactive representation that more realistically reflects user attitudes and intentions. Furthermore, this invention designs an adaptive exploratory reinforcement learning framework that dynamically adjusts the exploration rate of the policy network by quantifying changes in users' long-term and short-term preferences, achieving on-demand exploration to improve long-term returns in diverse scenarios.

[0005] To achieve the above-mentioned objectives, this invention provides a deep reinforcement learning-based product recommendation method that integrates user emotional features, characterized by comprising the following steps:

[0006] S1: Obtain historical interaction data between users and items;

[0007] S2: Extract emotional information based on evaluation text, integrate the emotional information into user-item interaction pairs, and construct a sequence of user behaviors with emotional perception;

[0008] S3: Encode emotion perception representation sequences based on sequence modeling networks to generate short-term and long-term preference representations of users;

[0009] S4: Through a hierarchical attention mechanism with bidirectional interaction and context adjustment, short-term and long-term preference representations are dynamically fused to generate a comprehensive representation of the user's state;

[0010] S5: Input the comprehensive state representation into the policy network of the reinforcement learning recommendation model, dynamically adjust the exploration strategy based on the long-short-term preference representation, and generate recommendation results.

[0011] Further, in step S1, relevant user and item data is obtained from Amazon Product Data. User rating records include user ID, product ID, username, valid review rate, review text, rating, review summary, review timestamp, and review time. Product information description data includes product ID, product name, price, related products, discount information, brand, and category. The data is then merged using the product ID.

[0012] S11: Perform data preprocessing on the raw data;

[0013] Check if the original dataset contains missing values; if so, delete them.

[0014] S12: Construction of user behavior sequences;

[0015] For each user, all their interaction records are sorted in ascending order by timestamp to form a user behavior sequence. ,in This represents the length of the user's historical interactions. For users with excessively long interaction lengths, a maximum sequence length is set. The value is 200, so a sliding window is used to select the nearest value. Each record is processed.

[0016] S13: User rating normalization;

[0017] The original scores were linearly normalized to... The interval, the formula is:

[0018]

[0019] Further, in step S2, emotional information is extracted based on the evaluation text, and this emotional information is integrated into the user-item interaction pair to construct a user behavior sequence based on emotional perception, including the following steps:

[0020] S21: Extracting sentiment features from comment text;

[0021] The pre-trained language model DistilBERT is used to extract sentiment features from the comment text. Specifically, a global vector representation for representing the overall semantics is selected, denoted as . Add an emotion classification layer at the top of DistilBERT. This layer contains three output units, corresponding to positive, neutral, and negative emotions.

[0022] The classification layer can adopt one of the following two structures:

[0023] Single-layer linear structure: → Linear transformation (768→3) → Softmax, to obtain the emotion probability vector ;

[0024] Two-layer MLP structure: → Linear transformation (768→256) → ReLU → Dropout (p=0.1) → Linear layer (256→3) → Softmax.

[0025] For each comment text, the model outputs a sentiment polarity vector. ,satisfy Predicting sentiment categories by The component corresponding to the highest probability value in the middle determines the sentiment confidence. The range of values ​​is .

[0026] Among them, the emotion vector Normalized score .

[0027] S22: Emotion confidence gating weighting;

[0028] Based on the emotional confidence level Regarding the emotional polarity information Weighted features are generated by performing weighted analysis. The calculation formula is:

[0029]

[0030] in and For learnable parameters, This indicates element-wise multiplication. The sigmoid activation function is defined as follows:

[0031]

[0032] S23: Integration of emotional characteristics and ratings;

[0033] By concatenating the weighted sentiment features with the normalized score, we obtain the user feedback features:

[0034]

[0035] S24: Item embedding representation;

[0036] The item ID is mapped to the embedding layer. Dense vectors. The embedding layer is a learnable lookup table. ,in This embodiment sets the total number of items to [number]. For items Its embedding vector is .

[0037] S25: Construction of Emotion-Aware Interactive Representations;

[0038] By embedding and concatenating user feedback features with item data, a sentiment perception representation of the interaction can be obtained:

[0039]

[0040] All interactions are arranged in chronological order to form a sequence of user behavior:

[0041]

[0042] Further, in step S3, the emotion perception representation sequence is encoded based on a sequence modeling network to generate the user's short-term and long-term preference representations, including the following steps:

[0043] S31: Sequence Modeling Network Architecture;

[0044] The sequence modeling network uses a Transformer encoder, with the following parameter settings: number of encoder layers. Attention count Hidden layer dimensions Feedforward network dimension Dropout rate Add a learnable location encoding matrix, denoted as .

[0045] S32: Short-term user preference modeling;

[0046] Employing a local window attention mechanism, window size Only allow each location to follow a maximum of [number] previous [positions]. Historical position. Short-term preferences are represented as:

[0047]

[0048] S33: Long-term user preference modeling;

[0049] Employing a local window attention mechanism, window size Only allow each location to follow a maximum of [number] previous [positions]. Historical position. Short-term preferences are represented as:

[0050]

[0051] in, , representing short-term and long-term preference representations, respectively.

[0052] Furthermore, in step S4, a hierarchical attention mechanism based on bidirectional interaction and context adjustment is used to dynamically fuse short-term and long-term preference representations to generate a comprehensive state representation of the user, including the following steps:

[0053] S41: Bidirectional cross-attention calculation;

[0054] Characterizing short-term preferences through linear transformation and long-term preference representation Mapping to query, key, and value spaces:

[0055]

[0056]

[0057] in, For a learnable weight matrix, the scaling factor is... .

[0058] Calculate cross-attention in two directions to capture the interaction between long-term and short-term preferences:

[0059] Short-term to long-term attention, using short-term preferences as the query and long-term preferences as the key:

[0060]

[0061] Long-term to short-term attention, using long-term preferences as the query and short-term preferences as the key:

[0062]

[0063] By concatenating the attention outputs from both directions, a bidirectional reinforced preference representation is obtained:

[0064]

[0065] S42: Context information adjustment;

[0066] Introducing contextual feature vectors of candidate items This includes product category, brand, and price range. Adjustment weights are calculated as follows:

[0067]

[0068] in, The adjusted representation is as follows:

[0069]

[0070] in, for The representation after linear transformation, This indicates element-wise multiplication.

[0071] S43: Adaptive Gated Fusion;

[0072] A gating network is constructed to dynamically generate fusion weights. The gating network is a two-layer MLP structure:

[0073]

[0074] in, , It is the Sigmoid activation function. This is the gating weight.

[0075] The final integrated state is represented by a weighted fusion of the original preferences using gating weights:

[0076]

[0077] Further, in step S5, the comprehensive state representation is input into the policy network of the reinforcement learning recommendation model, and the exploration strategy is dynamically adjusted based on the long-short-term preference representation to generate recommendation results, including the following steps:

[0078] S51: Preference Evolution Indicator;

[0079] Calculate the long-term preference evolution degree It represents the magnitude of changes in a user's long-term interests, and is calculated using the following formula:

[0080]

[0081] in, This represents the long-term preference at the current moment. express Long-term preference representation before time step, For time intervals;

[0082] Calculate the short-term - long-term preference deviation To measure the degree of difference between users' short-term and long-term interests:

[0083]

[0084]

[0085] in, This represents short-term preferences. The cosine similarity function;

[0086] S52: Adaptive exploration rate;

[0087] According to the above and Dynamically calculate adaptive exploration rate The formula is:

[0088]

[0089] in, and These are the weighting coefficients. To minimize the exploration rate, To maximize the exploration rate, The function restricts the result to Inside;

[0090] Policy networks use probability Select the optimal recommended action under the current strategy, based on probability. An action is randomly selected from the set of candidate items to achieve adaptive exploration.

[0091] S53: Policy Network Design;

[0092] In practical recommendation systems, the entire item set Typically, a dataset contains millions of items, making it impossible to directly calculate the probability distribution for all items. Therefore, a recall phase is needed to filter out hundreds of candidate items from the full set of items, forming a candidate set. , Size of the candidate item set.

[0093] Policy Network A three-layer perceptron structure is adopted:

[0094] Input layer: State representation ;

[0095] Hidden Layer 1: Linear transformation (256→128), activation function ReLU, Dropout ( );

[0096] Hidden Layer 2: Linear transformation (128 → 64), activation function ReLU, Dropout ( );

[0097] Output layer: Linear transformation ( The activation function is Softmax, which outputs the probability distribution of candidate items.

[0098] S54: Action Selection;

[0099] use -greedy strategy, based on adaptive exploration rate Control, based on probability Select the optimal action under the current strategy:

[0100]

[0101] With probability Explore by randomly selecting an action from a set of candidate items.

[0102] S55: Reward Function Design;

[0103] The Amazon Product Data dataset used in this embodiment contains user reviews and ratings after purchase. User satisfaction is reflected by both ratings and the sentiment of the review text; therefore, the reward function is designed by fusing normalized ratings with sentiment polarity.

[0104]

[0105] in, The normalized score defined in step S13; Set the value to 1 if the emotional polarity extracted in step S22 is positive, otherwise set it to 0. .

[0106] S56: Model training;

[0107] The policy network is trained using the Proximal Policy Optimization (PPO) algorithm.

[0108] Advantage function estimation (GAE):

[0109]

[0110]

[0111] Among them, discount factor GAE parameters , It is the value function.

[0112] PPO objective function:

[0113]

[0114] in, To determine the probability ratio, set the following parameters: pruning threshold. Value coefficient Entropy coefficient , For policy entropy;

[0115] Training hyperparameters, collecting data each time Step experience, batch size Learning rate Using the Adam optimizer, the target network updates at a frequency of every The next step is to copy the policy network parameters to the target network.

[0116] S57: Recommendation results generation;

[0117] Based on the probability distribution of candidate items output by the policy network, the candidate items are sorted, and the top K items with the highest probabilities are selected to form a recommendation list. Users can interact with the recommendations by purchasing or rating them. These interactions will serve as feedback signals for the subsequent reinforcement learning model, used for continuous model updates and optimization.

[0118] To verify the effectiveness of the method of this invention, four metrics—recall@T, hit rate@T, precision@T, and normalized discounted cumulative gain (NDCG@T)—were used to evaluate the recommendation results. The calculation formulas are as follows:

[0119]

[0120]

[0121]

[0122]

[0123] Among them, Precision@K represents the proportion of items of interest to the user in the first K results of the recommendation list; Recall@K represents the proportion of items of interest to the user that were successfully recommended in the recommendation list out of all relevant items; HitRate@K represents the proportion of users who hit at least one item of interest in the first K results of the recommendation list; NDCG@K is used to measure the ranking quality of the recommendation results, and the higher the value, the more the recommendation ranking matches the user's true preferences.

[0124] The above indicators can be used to comprehensively evaluate the performance improvement of the method of the present invention in terms of recommendation accuracy, ranking quality, and long-term benefits.

[0125] Compared with the prior art, the significant advantages of the present invention are reflected in the following aspects:

[0126] 1. To address the problem that traditional recommendation models make shallow and biased use of user feedback, this invention proposes an emotion-perception feedback fusion method. By co-correcting the confidence-weighted fine-grained emotion features with user ratings, an interactive representation that better reflects the user's true intentions is constructed, thereby improving the accuracy and robustness of the model's perception from the data source.

[0127] 2. To address the problem that existing dynamic recommendation models have a coarse modeling of user interest evolution, this invention designs a long-short-term preference separation encoding and hierarchical attention fusion mechanism to explicitly separate short-term fluctuations and long-term trends. Through bidirectional cross-attention and context gating dynamic fusion, it generates user state representations with higher discriminativeness and richer information, providing support for accurate decision-making.

[0128] 3. To address the bottleneck of rigid exploration strategies in reinforcement learning recommendation, this invention proposes an adaptive exploration mechanism driven by preference evolution. By quantifying long-term preference changes and short-term preference deviations, the exploration rate is dynamically adjusted to achieve in-depth utilization when interests are stable and proactive exploration when interests change or during cold starts. This significantly improves the system's adaptability and long-term benefits in scenarios such as cold starts and sudden changes in interests.

[0129] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0130] Figure 1 This is an overall flowchart of a deep reinforcement learning-based product recommendation method that integrates user emotional features, provided as an embodiment of the present invention. Detailed Implementation

[0131] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings of the embodiments of the present invention. The described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0132] like Figure 1 As shown, this invention provides a deep reinforcement learning-based product recommendation method that integrates user emotional features, comprising the following steps:

[0133] S1: Obtain historical interaction data between users and items;

[0134] Retrieves user and item-related data from Amazon Product Data. User rating records include user ID, product ID, username, valid review rate, review text, rating, review summary, review timestamp, and review time. Product information descriptions include product ID, product name, price, related products, discount information, brand, and category. Data is merged by product ID.

[0135] S11: Perform data preprocessing on the raw data;

[0136] Check if the original dataset contains missing values; if so, delete them.

[0137] S12: Construction of user behavior sequences;

[0138] For each user, all their interaction records are sorted in ascending order by timestamp to form a user behavior sequence. ,in This represents the length of the user's historical interactions. For users with excessively long interaction lengths, a maximum sequence length is set. The value is 200, so a sliding window is used to select the nearest value. Each record is processed.

[0139] S13: User rating normalization;

[0140] The original scores were linearly normalized to... The interval, the formula is:

[0141]

[0142] S2: Extract emotional information based on evaluation text, integrate the emotional information into user-item interaction pairs, and construct a sequence of user behaviors with emotional perception;

[0143] S21: Extracting sentiment features from comment text;

[0144] The pre-trained language model DistilBERT is used to extract sentiment features from the comment text. Specifically, a global vector representation for representing the overall semantics is selected, denoted as . Add an emotion classification layer at the top of DistilBERT. This layer contains three output units, corresponding to positive, neutral, and negative emotions.

[0145] The classification layer can adopt one of the following two structures:

[0146] Single-layer linear structure: →Linear transformation (768→3)→Softmax, to obtain the emotion probability vector ;

[0147] Two-layer MLP structure: → Linear transformation (768→256) → ReLU → Dropout (p=0.1) → Linear layer (256→3) → Softmax.

[0148] For each comment text, the model outputs a sentiment polarity vector. ,satisfy Predicting sentiment categories by The component corresponding to the highest probability value in the middle determines the sentiment confidence. The range of values ​​is .

[0149] Among them, the emotion vector Normalized score .

[0150] S22: Emotion confidence gating weighting;

[0151] Based on the emotional confidence level Regarding the emotional polarity information Weighted features are generated by performing weighted analysis. The calculation formula is:

[0152]

[0153] in and For learnable parameters, This indicates element-wise multiplication. The sigmoid activation function is defined as follows:

[0154]

[0155] S23: Integration of emotional characteristics and ratings;

[0156] By concatenating the weighted sentiment features with the normalized score, we obtain the user feedback features:

[0157]

[0158] S24: Item embedding representation;

[0159] The item ID is mapped to the embedding layer. Dense vectors. The embedding layer is a learnable lookup table. ,in This embodiment sets the total number of items to [number]. For items Its embedding vector is .

[0160] S25: Construction of Emotion-Aware Interactive Representations;

[0161] By embedding and concatenating user feedback features with item data, a sentiment perception representation of the interaction can be obtained:

[0162]

[0163] All interactions are arranged in chronological order to form a sequence of user behavior:

[0164]

[0165] S3: Encode emotion perception representation sequences based on sequence modeling networks to generate short-term and long-term preference representations of users;

[0166] S31: Sequence Modeling Network Architecture;

[0167] The sequence modeling network uses a Transformer encoder, with the following parameter settings: number of encoder layers. Attention count Hidden layer dimensions Feedforward network dimension Dropout rate Add a learnable location encoding matrix, denoted as .

[0168] S32: Short-term user preference modeling;

[0169] Employing a local window attention mechanism, window size Only allow each location to follow a maximum of [number] previous [positions]. Historical position. Short-term preferences are represented as:

[0170]

[0171] S33: Long-term user preference modeling;

[0172] Employing a full attention mechanism Allows you to follow all historical locations:

[0173]

[0174] in, , representing short-term and long-term preference representations, respectively.

[0175] S4: Through a hierarchical attention mechanism with bidirectional interaction and context adjustment, short-term and long-term preference representations are dynamically fused to generate a comprehensive representation of the user's state;

[0176] S41: Bidirectional cross-attention calculation;

[0177] Characterizing short-term preferences through linear transformation and long-term preference representation Mapping to query, key, and value spaces:

[0178]

[0179]

[0180] in, For a learnable weight matrix, the scaling factor is... .

[0181] Calculate cross-attention in two directions to capture the interaction between long-term and short-term preferences:

[0182] Short-term to long-term attention, using short-term preferences as the query and long-term preferences as the key:

[0183]

[0184] Long-term to short-term attention, using long-term preferences as the query and short-term preferences as the key:

[0185]

[0186] By concatenating the attention outputs from both directions, a bidirectional reinforced preference representation is obtained:

[0187]

[0188] S42: Context information adjustment;

[0189] Introducing contextual feature vectors of candidate items This includes product category, brand, and price range. Adjustment weights are calculated as follows:

[0190]

[0191] in, The adjusted representation is as follows:

[0192]

[0193] in, for The representation after linear transformation, This indicates element-wise multiplication.

[0194] S43: Adaptive Gated Fusion;

[0195] A gating network is constructed to dynamically generate fusion weights. The gating network is a two-layer MLP structure:

[0196]

[0197] in, , This is the gating weight.

[0198] The final integrated state is represented by a weighted fusion of the original preferences using gating weights:

[0199]

[0200] S5: Input the comprehensive state representation into the policy network of the reinforcement learning recommendation model, dynamically adjust the exploration strategy based on the long-term and short-term preference representation, and generate recommendation results;

[0201] S51: Preference Evolution Indicator;

[0202] Calculate the long-term preference evolution degree It represents the magnitude of changes in a user's long-term interests, and is calculated using the following formula:

[0203]

[0204] in, This represents the current long-term preferences. express Long-term preference representation before time step, For time intervals;

[0205] Calculate the short-term - long-term preference deviation To measure the degree of difference between users' short-term and long-term interests:

[0206]

[0207]

[0208] in, This represents short-term preferences. The cosine similarity function;

[0209] S52: Adaptive exploration rate;

[0210] According to the above and Dynamically calculate adaptive exploration rate The formula is:

[0211]

[0212] in, and These are the weighting coefficients. To minimize the exploration rate, To maximize the exploration rate, The function restricts the result to Inside;

[0213] Policy networks use probability Select the optimal recommended action under the current strategy, based on probability. An action is randomly selected from the set of candidate items to achieve adaptive exploration.

[0214] S53: Policy Network Design;

[0215] In practical recommendation systems, the entire item set Typically, a dataset contains millions of items, making it impossible to directly calculate the probability distribution for all items. Therefore, a recall phase is needed to filter out hundreds of candidate items from the full set of items, forming a candidate set. , Size of the candidate item set.

[0216] Policy Network A three-layer perceptron structure is adopted:

[0217] Input layer: State representation ;

[0218] Hidden Layer 1: Linear transformation (256→128), activation function ReLU, Dropout ( );

[0219] Hidden Layer 2: Linear transformation (128 → 64), activation function ReLU, Dropout ( );

[0220] Output layer: Linear transformation ( The activation function is Softmax, which outputs the probability distribution of candidate items.

[0221] S54: Action Selection;

[0222] use -greedy strategy, based on adaptive exploration rate Control, based on probability Select the optimal action under the current strategy:

[0223]

[0224] With probability Explore by randomly selecting an action from a set of candidate items.

[0225] S55: Reward Function Design;

[0226] The Amazon Product Data dataset used in this embodiment contains user reviews and ratings after purchase. User satisfaction is reflected by both ratings and the sentiment of the review text; therefore, the reward function is designed by fusing normalized ratings with sentiment polarity.

[0227]

[0228] in, The normalized score defined in step S13; Set the value to 1 if the emotional polarity extracted in step S22 is positive, otherwise set it to 0. .

[0229] S56: Model training;

[0230] The policy network is trained using the Proximal Policy Optimization (PPO) algorithm.

[0231] Advantage function estimation (GAE):

[0232]

[0233]

[0234] Among them, discount factor GAE parameters , It is the value function.

[0235] PPO objective function:

[0236]

[0237] in, To determine the probability ratio, set the following parameters: pruning threshold. Value coefficient Entropy coefficient , For policy entropy;

[0238] Training hyperparameters, collecting data each time Step experience, batch size Learning rate Using the Adam optimizer, the target network updates at a frequency of every The next step is to copy the policy network parameters to the target network.

[0239] S57: Generation and display of recommendation results;

[0240] Based on the probability distribution of candidate items output by the policy network, the candidate items are sorted, and the top K items with the highest probabilities are selected to form a recommendation list. Users can interact with the recommendations by purchasing or rating them. These interactions will serve as feedback signals for the subsequent reinforcement learning model, used for continuous model updates and optimization.

[0241] To verify the effectiveness of the method of this invention, four metrics—recall@T, hit rate@T, precision@T, and normalized discounted cumulative gain (NDCG@T)—were used to evaluate the recommendation results. The calculation formulas are as follows:

[0242]

[0243]

[0244]

[0245]

[0246] Among them, Precision@K represents the proportion of items of interest to the user in the first K results of the recommendation list; Recall@K represents the proportion of items of interest to the user that were successfully recommended in the recommendation list out of all relevant items; HitRate@K represents the proportion of users who hit at least one item of interest in the first K results of the recommendation list; NDCG@K is used to measure the ranking quality of the recommendation results, and the higher the value, the more the recommendation ranking matches the user's true preferences.

[0247] The above indicators can be used to comprehensively evaluate the performance improvement of the method of the present invention in terms of recommendation accuracy, ranking quality, and long-term benefits.

[0248] This invention addresses the shortcomings of traditional recommendation models in their superficial and biased use of user feedback signals by proposing an emotion-based feedback fusion method. Through collaborative correction of confidence-weighted emotion features and user ratings, it constructs an interactive representation that more accurately reflects the user's true intent, improving the accuracy and robustness of the model's perception. To address the issue of existing dynamic recommendation models' coarse modeling of user interest evolution, it designs a long-short-term preference separation encoding and hierarchical attention fusion mechanism to separate short-term fluctuations from long-term trends. Dynamic fusion is achieved through bidirectional cross-attention and context gating, generating a more discriminative and information-rich user state representation. Furthermore, to address the bottleneck of rigid exploration strategies in reinforcement learning recommendation, which cannot adapt to diverse scenarios, it proposes a preference evolution-driven adaptive exploration mechanism. By quantifying long-term preference changes and short-term preference deviations to dynamically adjust the exploration rate, it achieves intelligent control strategies, significantly improving the system's adaptability and long-term gains in challenging scenarios such as cold starts and sudden interest mutations.

[0249] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A deep reinforcement learning-based product recommendation method that integrates user emotional features, characterized in that, include: S1: Obtain historical interaction data between users and items; S2: Extract emotional information based on evaluation text, integrate the emotional information into user-item interaction pairs, and construct a sequence of user behaviors with emotional perception; S3: Encode emotion perception representation sequences based on sequence modeling networks to generate short-term and long-term preference representations of users; S4: Through a hierarchical attention mechanism with bidirectional interaction and context adjustment, short-term and long-term preference representations are dynamically fused to generate a comprehensive representation of the user's state; S5: Input the comprehensive state representation into the policy network of the reinforcement learning recommendation model, dynamically adjust the exploration strategy based on the long-short-term preference representation, and generate recommendation results.

2. The deep reinforcement learning-based product recommendation method incorporating user emotional features according to claim 1, characterized in that, In step S1, relevant user and item data is obtained from Amazon Product Data. User rating record data includes user ID, product ID, username, valid review rate, review text, rating, review summary, review timestamp, and review time. Product information description data includes product ID, product name, price, related products, discount information, brand, and category. The data is merged using the product ID. S11: Perform data preprocessing on the raw data; Check if the original dataset contains missing values; if so, delete them. S12: Construction of user behavior sequences; For each user, all their interaction records are sorted in ascending order by timestamp to form a user behavior sequence. ,in This represents the length of the user's historical interactions. For users with excessively long interaction lengths, a maximum sequence length is set. The value is 200, so a sliding window is used to select the nearest value. Each record is processed. S13: User rating normalization; The original scores were linearly normalized to... The interval, the formula is:

3. The deep reinforcement learning-based product recommendation method incorporating user emotional features according to claim 1, characterized in that, Step S2 specifically includes the following: S21: Extracting sentiment features from comment text; The pre-trained language model DistilBERT is used to extract sentiment features from the comment text. Specifically, a global vector representation for representing the overall semantics is selected, denoted as . Add an emotion classification layer at the top of DistilBERT. This layer contains three output units, corresponding to positive, neutral, and negative emotions. The classification layer can adopt one of the following two structures: Single-layer linear structure: → Linear transformation (768→3) → Softmax, to obtain the emotion probability vector ; Two-layer MLP structure: → Linear transformation (768→256) → ReLU → Dropout (p=0.1) → Linear layer (256→3) → Softmax. For each comment text, the model outputs a sentiment polarity vector. ,satisfy Predicting sentiment categories by The component corresponding to the highest probability value in the middle determines the sentiment confidence. The range of values ​​is . Among them, the emotion vector Normalized score . S22: Emotion confidence gating weighting; Based on the emotional confidence level Regarding the emotional polarity information Weighted features are generated by performing weighted analysis. The calculation formula is: in and For learnable parameters, This indicates element-wise multiplication. The sigmoid activation function is defined as follows: S23: Integration of emotional characteristics and ratings; By concatenating the weighted sentiment features with the normalized score, we obtain the user feedback features: S24: Item embedding representation; The item ID is mapped to the embedding layer. Dense vectors. The embedding layer is a learnable lookup table. ,in This embodiment sets the total number of items to [number]. For items Its embedding vector is . S25: Construction of Emotion-Aware Interactive Representations; By embedding and concatenating user feedback features with item data, a sentiment perception representation of the interaction can be obtained: All interactions are arranged in chronological order to form a sequence of user behavior:

4. The deep reinforcement learning-based product recommendation method incorporating user emotional features according to claim 1, characterized in that, Step S3 specifically includes the following: S31: Sequence Modeling Network Architecture; The sequence modeling network uses a Transformer encoder, with the following parameter settings: number of encoder layers. Attention count Hidden layer dimensions Feedforward network dimension Dropout rate Add learnable positional coding . S32: Short-term user preference modeling; Employing a local window attention mechanism, window size Only allow each location to follow a maximum of [number] previous [positions]. Historical position. Short-term preferences are represented as: S33: Long-term user preference modeling; Employing a full attention mechanism Allows you to follow all historical locations: in, , representing short-term and long-term preference representations, respectively.

5. The deep reinforcement learning-based product recommendation method incorporating user emotional features according to claim 1, characterized in that, Step S4 specifically includes the following: S41: Bidirectional cross-attention calculation; Characterizing short-term preferences through linear transformation and long-term preference representation Mapping to query, key, and value spaces: in, For a learnable weight matrix, the scaling factor is... . Calculate cross-attention in two directions to capture the interaction between long-term and short-term preferences: Short-term to long-term attention, using short-term preferences as the query and long-term preferences as the key: Long-term to short-term attention, using long-term preferences as the query and short-term preferences as the key: By concatenating the attention outputs from both directions, a bidirectional reinforced preference representation is obtained: S42: Context information adjustment; Introducing contextual feature vectors of candidate items This includes product category, brand, and price range. Adjustment weights are calculated as follows: in, The adjusted representation is as follows: in, for The representation after linear transformation, This indicates element-wise multiplication. S43: Adaptive Gated Fusion; A gating network is constructed to dynamically generate fusion weights. The gating network is a two-layer MLP structure: in, , It is the Sigmoid activation function. This is the gating weight. The final integrated state is represented by a weighted fusion of the original preferences using gating weights:

6. The deep reinforcement learning-based product recommendation method incorporating user emotional features according to claim 1, characterized in that, Step S5 specifically includes the following: S51: Preference Evolution Indicator; Calculate the long-term preference evolution degree It represents the magnitude of changes in a user's long-term interests, and is calculated using the following formula: ； in, This represents the current long-term preference. express Long-term preference representation before time step, For time intervals; Calculate the short-term-long-term preference deviation To measure the degree of difference between users' short-term and long-term interests: in, This represents short-term preferences. The cosine similarity function; S52: Adaptive exploration rate; According to the above and Dynamically calculate adaptive exploration rate The formula is: in, and These are the weighting coefficients. To minimize the exploration rate, To maximize the exploration rate, The function restricts the result to Inside; Policy networks use probability Select the optimal recommended action under the current strategy, based on probability. An action is randomly selected from the set of candidate items to achieve adaptive exploration. S53: Policy Network Design; In practical recommendation systems, the entire item set Typically, a dataset contains millions of items, making it impossible to directly calculate the probability distribution for all items. Therefore, a recall phase is needed to filter out hundreds of candidate items from the full set of items, forming a candidate set. , Size of the candidate item set. Policy Network A three-layer perceptron structure is adopted: Input layer: State representation ; Hidden Layer 1: Linear transformation (256→128), activation function ReLU, Dropout ( ); Hidden Layer 2: Linear transformation (128 → 64), activation function ReLU, Dropout ( ); Output layer: Linear transformation ( The activation function is Softmax, which outputs the probability distribution of candidate items. S54: Action Selection; use -greedy strategy, based on adaptive exploration rate Control, based on probability Select the optimal action under the current strategy: With probability Explore by randomly selecting an action from a set of candidate items. S55: Reward Function Design; The Amazon Product Data dataset used in this embodiment contains user reviews and ratings after purchase. User satisfaction is reflected by both ratings and the sentiment of the review text; therefore, the reward function is designed by fusing normalized ratings with sentiment polarity. in, The normalized score defined in step S13; Set the value to 1 if the emotional polarity extracted in step S22 is positive, otherwise set it to 0. . S56: Model training; The policy network is trained using the Proximal Policy Optimization (PPO) algorithm. Advantage function estimation (GAE): Among them, discount factor GAE parameters , It is the value function. PPO objective function: in, To determine the probability ratio, set the following parameters: pruning threshold. Value coefficient Entropy coefficient , For policy entropy; Training hyperparameters, collecting data each time Step experience, batch size Learning rate Using the Adam optimizer, the target network updates at a frequency of every The next step is to copy the policy network parameters to the target network. S57: Generation and display of recommendation results; Based on the probability distribution of candidate items output by the policy network, the candidate items are sorted, and the top K items with the highest probabilities are selected to form a recommendation list. Users can interact with the recommendations by purchasing or rating them. These interactions will serve as feedback signals for the subsequent reinforcement learning model, used for continuous model updates and optimization. To verify the effectiveness of the method of this invention, four metrics—recall@T, hit rate@T, precision@T, and normalized discounted cumulative gain (NDCG@T)—were used to evaluate the recommendation results. The calculation formulas are as follows: Among them, Precision@K represents the proportion of user-interested items in the first K results of the recommendation list; Recall@K represents the proportion of user-interested items successfully recommended in the recommendation list out of all relevant items; Hit Rate@K represents the proportion of users who hit at least one user-interested item in the first K results of the recommendation list; NDCG@K is used to measure the ranking quality of the recommendation results, and the higher the value, the more the recommendation ranking matches the user's true preferences. The above indicators can be used to comprehensively evaluate the performance improvement of the method of the present invention in terms of recommendation accuracy, ranking quality, and long-term benefits.

Images