Reinforcement learning reward function improvement method introducing feature words
By introducing user comment feature words in reinforcement learning and adjusting reward functions, the problem of reward functions ignoring comment information in traditional recommendation algorithms is solved, and a more accurate and personalized recommendation effect is achieved.
Patent Information
- Application Number
- CN202510514952.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-23
- Publication Date
- 2025-08-05
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the user recommendation scenario, the reward function design ignores user comment information, resulting in one-sided recommendation strategy, which cannot accurately reflect the user's real needs, and is not effective for new users or cold starts.
By introducing user comment feature words, adjusting the reward function in the reinforcement learning strategy, using probabilistic potential semantic analysis to generate comment feature words, and optimizing the reward function based on the emotional tendency and project attributes of user comments.
Improve the accuracy and personalization of recommendations, help the agent better understand user feedback, speed up learning, and improve recommendation results.
Smart Images
Figure CN120429499A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of recommendation algorithms, and in particular relates to a method for improving a reinforcement learning reward function by introducing feature words. Background Art
[0002] In today's information explosion, users face a severe information overload problem. Every day, the internet is flooded with massive amounts of data, content, and services, making it difficult for users to filter out information that truly meets their needs and interests. To address this challenge, various recommendation algorithms have emerged, aiming to provide personalized content recommendations by analyzing and predicting user preferences.
[0003] Traditional recommendation algorithms, such as collaborative filtering and content-based recommendations, have alleviated the problem of information overload to some extent, but they still have some shortcomings. For example, collaborative filtering relies on historical user behavior data and is not very effective for new users or cold start problems. Content-based recommendations, on the other hand, rely primarily on item attributes and features, which may not accurately capture potential changes in user interests.
[0004] In recent years, reinforcement learning, an advanced machine learning method, has garnered increasing attention in recommender systems. By enabling intelligent agents to learn strategies through interactions with their environment, with the goal of maximizing cumulative rewards, reinforcement learning can dynamically adjust recommendation strategies to adapt to changing user interests. However, general reinforcement learning algorithms also face challenges in recommendation scenarios, particularly in the design of reward functions. Traditional reward functions are often based on simple user feedback, ignoring rich textual information such as user reviews. This can lead to overly biased recommendation strategies that fail to accurately reflect users' true needs. Summary of the Invention
[0005] To address the technical issues inherent in the aforementioned prior art, the present invention provides a method for improving the reward function of reinforcement learning by introducing feature words. This method adjusts the reward function in the policy network by incorporating feature words from user reviews, thereby fully leveraging the useful information contained in these reviews. This allows for a more detailed characterization of user preferences and needs, improving the accuracy and personalization of recommendations. Furthermore, it helps the intelligent agent better understand user feedback, accelerates learning, and enhances recommendation effectiveness. In a reinforcement learning-based user recommendation scenario, this method adjusts the reward function in the policy network by introducing feature words from user reviews, thereby improving recommendation effectiveness.
[0006] To achieve the above-mentioned purpose, the present invention adopts the following technical solution: a method for improving the reinforcement learning reward function by introducing feature words, comprising the following steps:
[0007] S1: Extract users, items and their interactive comment texts, and perform sentiment analysis on their comment texts;
[0008] S2: Design a reinforcement learning model, define the MDP environment, and develop an action strategy network for the agent;
[0009] S3: Perform preprocessing operations on the text by segmenting and removing stop words to obtain a standardized vocabulary set and convert it into a word-document matrix;
[0010] S4: Input the constructed word-document matrix into the PLSA model, learn the potential topic distribution through the EM algorithm, and generate comment feature words;
[0011] S5: Introduce review feature words to adjust the reward function in reinforcement learning strategy.
[0012] As a preferred technical solution of the present invention, in S1, user, project and their interactive comment texts are extracted from the project dataset; the data is cleaned and preprocessed to ensure the accuracy and consistency of the data: duplicate data, data with missing key fields and obviously erroneous data are removed; the processed dataset is then divided into training, verification and test datasets, and the corresponding sentiment labels are marked for the training and verification test sets; the text data is converted into the input format required by the pre-trained language model, special tags [CLS] and [SEP] are added, and an input tensor is generated, which is then input into the pre-trained language model to obtain the sentiment classification of the corresponding comment text.
[0013] As the preferred technical solution of the present invention, in S2, the reinforcement learning goal is determined, generally taking the long-term cumulative reward as the ultimate goal; then define the state space S = {s0, s1, ..., s n-1 ,s n}, the state at step t is s t ; Action space A={a0,a1,…,a n-1 ,a n}, the action at step t is a t Next, we develop a strategy network for the agent's movement. This network maps the agent's states to actions. This strategy consists of a deterministic strategy a = π(s) where states correspond to actions one by one, and a stochastic strategy P(a|s) where each state corresponds to an action probability distribution. Next, we define the reward r(s, a) that the agent receives from the environment after executing an action. The cumulative reward formula is as follows:
[0014]
[0015] Under the strategy π, the expected cumulative reward of state s, i.e., the state value function, is as follows:
[0016]
[0017] Similarly, under the strategy π, the expected cumulative reward of taking action a in state s is the action value function as follows:
[0018]
[0019] After training the policy network, the agent can perform a probabilistic search starting from a given user to eventually find the item recommendation candidate set with the highest cumulative reward and recommend it to the user.
[0020] As the preferred technical solution of the present invention, in S3, the Python Chinese word segmentation third-party library jieba is used for word segmentation, and the two methods of rule-based and statistics-based are combined to segment the comment text into words or vocabulary units to obtain a word segmentation list; in addition to the word segmentation operation, it is also necessary to remove stop words from the input text, select the Harbin Institute of Technology stop word list, traverse the word list after word segmentation, and for each word in the list, check whether it exists in the stop word list; if not, retain the word; if so, delete it from the list; finally, it is necessary to construct a word-document matrix; the rows of this matrix represent words, and the columns represent documents, and the element values are filled with TF-IDF weighted values to evaluate the importance of a word to a document.
[0021] As a preferred technical solution of the present invention, in S4, the word-document matrix is input into the PLSA probabilistic latent semantic analysis model, and the probability of each word belonging to each topic and the probability of each document belonging to each topic are randomly initialized; then the E step and the M step are iterated; wherein the E step is also called the expectation step, and under the current parameter estimation value, the posterior probability of each word-document pair belonging to each topic is calculated, and this is achieved by decomposing the joint probability distribution of the word-document pair into the product of the probability of the word belonging to the topic and the probability of the document belonging to the topic; the M step is also called the maximization step, and the obtained posterior probability is used to recalculate the probability of each word belonging to each topic and the probability of each document belonging to each topic to maximize the likelihood value, which usually involves updating the parameters; the E step and the M step are repeated until the parameters converge or the preset number of iterations is reached, and finally the probability distribution of each word belonging to each topic and the probability distribution of each document belonging to each topic can be obtained; finally, for each topic, the words are sorted according to the probability of belonging to the topic, and the words with higher probability are selected as the feature words of the topic, i.e., the comment feature words.
[0022] As a preferred technical solution of the present invention, in S5, the correlation between the feature words in the user comments and the recommended item attributes is calculated, and the cosine similarity is used to accurately calculate the correlation score between the feature word vector and the item attribute vector, as shown below:
[0023]
[0024] Among them, v k is the vector representation of the comment feature word k, v a The vector representation of the recommended item attribute a;
[0025] In order to quantify the impact of feature words on the item recommendation system, a feature word reward function is defined. This function comprehensively considers the relevance scores of feature words and multiple item attributes, as well as the weights of each attribute and the sentiment tendency of the review text. Its calculation formula is as follows:
[0026]
[0027] Among them, weight(a) is the weight of item attribute a, which can be adjusted according to importance; λ s is the sentiment coefficient of the review text, which is used to reflect the positive or negative impact of the review on the recommendation effect; when the review is positive, λ s A value of 1 brings positive improvement; when the review is negative, λ s A value of -1 brings negative improvement; if the comment sentiment is not obvious, then λ s The value is 0, maintaining the original reward function unchanged, and its calculation formula is as shown in the formula:
[0028]
[0029] Finally, the feature word reward function is combined with the original reward to obtain the adjusted final reward function, as shown in the formula:
[0030] r′(s,a)=r(s,a)+λ·R feature
[0031] Among them, r(s,a) represents the original reward for taking action a in state s, and r'(s,a) represents the final reward after considering the feature word reward; λ is an adjustment coefficient used to balance the original reward and feature word reward.
[0032] The improved method for reinforcement learning reward function by introducing feature words described in the present invention has the following technical effects compared with the prior art by using the above technical solution:
[0033] (1) This invention is based on the strategy network of reinforcement learning, making the recommendation more logical and accurate.
[0034] (2) The present invention generates feature words by probabilistic latent semantic analysis of comments and accurately extracts key information of user comments.
[0035] (3) The present invention introduces comment feature words to optimize the traditional reward function, so that the final recommendation results are more in line with the user's personalized needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 is a flow chart of the method of the present invention;
[0037] Figure 2 This is the working mode diagram of PLSA probabilistic latent semantic analysis. DETAILED DESCRIPTION
[0038] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, some symbols in the embodiments of the present application are first explained to facilitate understanding by those skilled in the art.
[0039] like Figure 1 As shown, based on the user recommendation scenario of reinforcement learning, the present invention generates feature words by probabilistic latent semantic analysis of comments on the article, and then adjusts the reward function based on the feature words.
[0040] A method for improving the reinforcement learning reward function by introducing feature words includes the following steps:
[0041] S1: Extract users, items and their interactive comment texts, and perform sentiment analysis on their comment texts;
[0042] S2: Design a reinforcement learning model, define the MDP environment, and develop an action strategy network for the agent;
[0043] S3: Perform preprocessing operations on the text by segmenting and removing stop words to obtain a standardized vocabulary set and convert it into a word-document matrix;
[0044] S4: Input the constructed word-document matrix into the PLSA model, learn the potential topic distribution through the EM algorithm, and generate comment feature words;
[0045] S5: Introduce review feature words to adjust the reward function in reinforcement learning strategy.
[0046] Specifically, S1 extracts user, project, and interactive comment text from the project dataset. The data is cleaned and preprocessed to ensure accuracy and consistency, removing duplicate data, data missing key fields, and data with obvious errors. The processed dataset is then divided into training, validation, and test datasets, and the training and validation test sets are annotated with corresponding sentiment labels. The text data is converted to the input format required by the pretrained language model, adding special tags [CLS] and [SEP] to generate an input tensor, which is then fed into the pretrained language model to obtain the sentiment classification of the corresponding comment text.
[0047] In S2, the reinforcement learning goal is determined, and long-term cumulative rewards are generally used as the final goal. Then define the state space S = {s0, s1, ..., s n-1 ,s n}, the state at step t is s t Action space A={a0,a1,…,a n-1 ,a n}, the action at step t is a t Next, we develop a policy network for the agent's movement. This network maps the agent's states to actions. This network consists of a deterministic policy a = π(s), where states correspond one to one, and a stochastic policy P(a|s), where each state corresponds to an action with a probability distribution. Next, we define the reward r(s, a) the agent receives from the environment after executing an action. The cumulative reward formula is as follows:
[0048]
[0049] Under the strategy π, the expected cumulative reward of state s, i.e., the state value function, is as follows:
[0050]
[0051] Similarly, under the strategy π, the expected cumulative reward of taking action a in state s is the action value function as follows:
[0052]
[0053] After training the policy network, the agent can perform a probabilistic search starting from a given user to eventually find the item recommendation candidate set with the highest cumulative reward and recommend it to the user.
[0054] Specifically, S3 uses the Python Chinese word segmentation third-party library Jieba to segment the comment text into words or vocabulary units, combining rule-based and statistical-based methods to obtain a word segmentation list. In addition to the word segmentation operation, the input text also needs to be removed from the stop words. The Harbin Institute of Technology stop word list is selected, and the word list after word segmentation is traversed. For each word in the list, check whether it exists in the stop word list. If it does not exist, the word is retained; if it exists, it is deleted from the list. Finally, a word-document matrix needs to be constructed. The rows of this matrix represent words, and the columns represent documents. At the same time, the element values are filled with TF-IDF weighted values to evaluate the importance of a word to a document.
[0055] In S4, the word-document matrix is input to the PLSA probabilistic latent semantic analysis model, which works in Figure 2As shown. Randomly initialize the probability of each word belonging to each topic and the probability of each document belonging to each topic. Next, iterate the E-step and M-step. The E-step is also called the expectation step. Under the current parameter estimation value, the posterior probability of each word-document pair belonging to each topic is calculated. This can be achieved by decomposing the joint probability distribution of the word-document pair into the product of the probability of the word belonging to the topic and the probability of the document belonging to the topic; the M-step is also called the maximization step. Using the obtained posterior probability, the probability of each word belonging to each topic and the probability of each document belonging to each topic are recalculated to maximize the likelihood value, which usually involves updating the parameters. Repeat the E-step and M-step until the parameters converge or the preset number of iterations is reached, and the probability distribution of each word belonging to each topic and the probability distribution of each document belonging to each topic can be obtained. Finally, for each topic, the words are sorted according to the probability of belonging to the topic, and the words with higher probability are selected as the feature words of the topic, that is, the comment feature words.
[0056] Specifically, S5 calculates the correlation between the feature words in the user comments and the recommended item attributes, and uses cosine similarity to accurately calculate the correlation score between the feature word vector and the item attribute vector, as shown below:
[0057]
[0058] Among them, v k is the vector representation of the comment feature word k, v a A vector representation of the recommended item attribute a.
[0059] In order to quantify the impact of feature words on the item recommendation system, a feature word reward function is defined. This function comprehensively considers the relevance score of the feature word with multiple item attributes, as well as the weight of each attribute and the sentiment tendency of the review text. Its calculation formula is as follows:
[0060]
[0061] Among them, weight(a) is the weight of item attribute a, which can be adjusted according to importance. s Is the sentiment coefficient of the review text, which is used to reflect the positive or negative impact of the review on the recommendation effect. When the review is positive, λ s A value of 1 brings positive improvement; when the review is negative, λ s A value of -1 brings negative improvement; if the comment sentiment is not obvious, then λ s The value is 0, and the original reward function remains unchanged. , its calculation formula is as shown in the formula:
[0062]
[0063] Finally, the feature word reward function is combined with the original reward to obtain the adjusted final reward function, as shown in the formula:
[0064] r′(s,a)=r(s,a)+λ·R feature
[0065] Here, r(s,a) represents the raw reward for taking action a in state s, while r'(s,a) represents the final reward after taking into account the feature word reward. λ is an adjustment coefficient used to balance the raw reward and the feature word reward.
[0066] This paper proposes an improved reinforcement learning reward function that incorporates feature words. Based on the user recommendation scenario of reinforcement learning, this method adjusts the reward function in the policy network by incorporating feature words from user reviews, thereby fully utilizing the useful information in user reviews. This method not only more meticulously characterizes user preferences and needs, improving the accuracy and personalization of recommendations, but also helps the intelligent agent better understand user feedback, accelerate learning, and enhance recommendation effectiveness.
[0067] The specific implementation scheme described above further illustrates in detail the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in this field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.
Claims
1. A method for improving the reinforcement learning reward function by introducing feature words, characterized in that: The following steps are involved: S1: Extract users, items and their interactive comment texts, and perform sentiment analysis on their comment texts; S2: Design a reinforcement learning model, define the MDP environment, and develop an action strategy network for the agent; S3: Perform preprocessing operations on the text by segmenting and removing stop words to obtain a standardized vocabulary set and convert it into a word-document matrix; S4: Input the constructed word-document matrix into the PLSA model, learn the potential topic distribution through the EM algorithm, and generate comment feature words; S5: Introduce review feature words to adjust the reward function in reinforcement learning strategy.
2. The method for improving the reinforcement learning reward function by introducing feature words according to claim 1, characterized in that: In S1, user, project, and their interactive comment texts are extracted from the project dataset; the data is cleaned and preprocessed to ensure data accuracy and consistency: duplicate data, data with missing key fields, and data with obvious errors are removed; the processed dataset is then divided into training, validation, and test datasets, and the corresponding sentiment labels are annotated for the training and validation test sets; the text data is converted into the input format required by the pre-trained language model, special tags [CLS] and [SEP] are added, and an input tensor is generated, which is then input into the pre-trained language model to obtain the sentiment classification of the corresponding comment text.
3. The method for improving the reinforcement learning reward function by introducing feature words according to claim 2, characterized in that: In S2, the reinforcement learning goal is determined, and long-term cumulative rewards are generally used as the ultimate goal; then the state space S is defined as {s0, s1, ..., s n-1 ,s n }, the state at step t is s t ; Action space A={a0,a1,…,a n-1 ,a n }, the action at step t is a t Next, we develop a strategy network for the agent's movement. This network maps the agent's states to actions. This strategy consists of a deterministic strategy a = π(s) where states correspond to actions one by one, and a stochastic strategy P(a|s) where each state corresponds to an action probability distribution. Next, we define the reward r(s, a) that the agent receives from the environment after executing an action. The cumulative reward formula is as follows: Under the strategy π, the expected cumulative reward of state s, i.e., the state value function, is as follows: Similarly, under the strategy π, the expected cumulative reward of taking action a in state s is the action value function as follows: After training the policy network, the agent can perform a probabilistic search starting from a given user to eventually find the item recommendation candidate set with the highest cumulative reward and recommend it to the user.
4. The method for improving the reinforcement learning reward function by introducing feature words according to claim 3, characterized in that: In S3, we use the third-party Python Chinese word segmentation library Jieba to segment the review text into words or lexical units, combining rule-based and statistical methods. This results in a word list. In addition to word segmentation, we also need to remove stop words from the input text. We select the Harbin Institute of Technology stop word list and traverse the word list after segmentation. For each word in the list, we check whether it exists in the stop word list. If it does not exist, keep the word; if it does exist, delete it from the list; finally, you need to build a word-document matrix; the rows of this matrix represent words, the columns represent documents, and the element values are filled with TF-IDF weighted values to evaluate the importance of a word to a document.
5. The method for improving the reinforcement learning reward function by introducing feature words according to claim 4, characterized in that: In S4, the word-document matrix is input into the PLSA probabilistic latent semantic analysis model, and the probability of each word belonging to each topic and the probability of each document belonging to each topic are randomly initialized; then the E step and M step are iterated; among them, the E step is also called the expectation step, and under the current parameter estimation value, the posterior probability of each word-document pair belonging to each topic is calculated, which is achieved by decomposing the joint probability distribution of the word-document pair into the product of the probability of the word belonging to the topic and the probability of the document belonging to the topic; the M step is also called the maximization step, and the obtained posterior probability is used to recalculate the probability of each word belonging to each topic and the probability of each document belonging to each topic to maximize the likelihood value, which usually involves updating the parameters; the E step and M step are repeated until the parameters converge or the preset number of iterations is reached, and finally the probability distribution of each word belonging to each topic and the probability distribution of each document belonging to each topic can be obtained; finally, for each topic, the words are sorted according to the probability of belonging to the topic, and the words with higher probability are selected as the feature words of the topic, that is, the comment feature words.
6. The method for improving the reinforcement learning reward function by introducing feature words according to claim 5, characterized in that: In S5, the correlation between the feature words in the user comments and the recommended item attributes is calculated, and the cosine similarity is used to accurately calculate the correlation score between the feature word vector and the item attribute vector, as shown below: Among them, v k is the vector representation of the comment feature word k, v a The vector representation of the recommended item attribute a; In order to quantify the impact of feature words on the item recommendation system, a feature word reward function is defined. This function comprehensively considers the relevance scores of feature words and multiple item attributes, as well as the weights of each attribute and the sentiment tendency of the review text. Its calculation formula is as follows: Among them, weight(a) is the weight of item attribute a, which can be adjusted according to importance; λ s is the sentiment coefficient of the review text, which is used to reflect the positive or negative impact of the review on the recommendation effect; when the review is positive, λ s A value of 1 brings positive improvement; when the review is negative, λ s A value of -1 brings negative improvement; if the comment sentiment is not obvious, then λ s The value is 0, maintaining the original reward function unchanged, and its calculation formula is as shown in the formula: Finally, the feature word reward function is combined with the original reward to obtain the adjusted final reward function, as shown in the formula: r’(s,a)=r(s,a)+λ·R feature Among them, r(s,a) represents the original reward for taking action a in state s, and r'(s,a) represents the final reward after considering the feature word reward; λ is an adjustment coefficient used to balance the original reward and feature word reward.