Commodity recommendation method based on reinforcement learning novelty perception

Abstract

The invention discloses a commodity recommendation method based on reinforcement learning novelty perception. Comprising the following steps: firstly, obtaining user and commodity interaction historical data of a user online shopping website, and constructing an original data set; thirdly, training a feature extraction module by using the original data set to obtain a trained feature extraction module, and extracting by using the trained feature extraction module to obtain a user feature vector and a commodity feature vector; then, combining the time sequence of interaction between the user and the commodity, constructing an interaction sample set according to the user feature vector, the commodity feature vector and the corresponding historical interaction information, and training a reinforcement learning model of novelty perception by using the interaction sample set to obtain a trained reinforcement learning model; and finally, inputting the target interaction sample into the trained reinforcement learning model, and outputting a commodity recommendation result. According to the method, the problem of popularity deviation can be effectively solved, and the recommendation accuracy and novelty are remarkably improved.

Description

Technical Field

[0001] The present invention relates to a commodity recommendation method in the field of commodity recommendation, and specifically relates to a commodity recommendation method based on reinforcement learning novelty perception. Background Art

[0002] Recommendation systems are considered to be one of the most effective ways to solve the problem of information overload. Recommendation systems can not only help people retrieve the information they need, but also increase traffic and revenue for service providers. Traditional product recommendation methods usually use information such as user behavior data and personal attributes to analyze users' historical purchase behaviors and preferences to provide recommended products that are more in line with user interests and needs. However, this type of method is obsessed with the accuracy of recommendations and ignores long-tail products, which causes the problem of popularity bias. The main reason for this is that most observation data are collected from user-centric systems. Ignoring popularity bias may lead to unfair recommendation results. For example, over-recommendation or exposure of popular products will reduce the visibility of other unpopular products, even if they are suitable matches. In addition, this will also cause the recommendation system to push homogeneous products to users, making users feel bored.

[0003] To address the popularity bias problem, existing techniques use debiasing methods to increase the exposure of long-tail items. However, most current methods are built on supervised learning models, such as introducing regularization terms in the training loss, using adversarial learning to bridge the gap between niche and popular items, or adjusting the ranking of items after training. They focus on static recommendations and hardly capture the dynamic shift of user preferences. In addition, the amplification effect of feedback loops in recommendation systems is severely neglected. Therefore, in dynamic environments, these methods face the problems of instability and inefficiency.

[0004] Reinforcement learning has also received a lot of attention in the research of recommendation systems because of its mechanisms such as adaptive learning and dynamic exploration and utilization. Reinforcement learning can capture the dynamic shift of user preferences and reduce the amplification effect of feedback loops, which can effectively solve the instability and inefficiency of product recommendations in dynamic environments. However, these reinforcement learning methods are still designed based on traditional accuracy indicators and do not pay attention to the exposure rate of long-tail items, so they cannot solve the popularity deviation problem caused by the long-tail effect in the recommendation system. Summary of the invention

[0005] In view of the above technical problems existing in the prior art, the present invention provides a product recommendation method based on novelty perception of reinforcement learning. In addition to using users and their interactive products, the present invention introduces information such as the user's age and the brand of the product to improve the accuracy of the recommendation. At the same time, the popularity of the item is introduced into the reward function of reinforcement learning to characterize the novelty of the product and solve the influence of popularity deviation.

[0006] The technical solution of the present invention is as follows:

[0007] 1. A product recommendation method based on novelty perception based on reinforcement learning

[0008] Step 1: Obtain the historical data of user-product interaction on the user's online shopping website, and construct the original data set based on the user information, product information, and historical interaction information in the historical data of user-product interaction;

[0009] Step 2: After training the feature extraction module with the original data set, a trained feature extraction module is obtained, and then the trained feature extraction module is used to extract features of all user information and product information in the original data set to obtain corresponding user feature vectors and product feature vectors;

[0010] Step 3: Combine the time sequence of user-product interaction, build an interaction sample set based on the user feature vector, product feature vector and corresponding historical interaction information, and use the interaction sample set to train the novelty perception reinforcement learning model to obtain a trained reinforcement learning model;

[0011] Step 4: Input the target interaction sample into the trained reinforcement learning model, and the model outputs the product recommendation results.

[0012] The step 2 is specifically as follows:

[0013] The feature extraction module comprises a user feature mapping layer, a commodity feature mapping layer, a user text representation layer and a commodity text representation layer, wherein the user feature mapping layer is connected to the user text representation layer, and the commodity feature mapping layer is connected to the commodity text representation layer; the user information and commodity information in the original data set are combined according to the historical interaction information to obtain positive samples and negative samples, thereby obtaining positive and negative sample sets; during the training process of the feature extraction module, the user information in each sample is used as the input of the user feature mapping layer, and the commodity information in each sample is used as the input of the commodity feature mapping layer; the loss function is calculated according to the output of the user text representation layer and the commodity text representation layer and the interaction information of each sample; the network parameters of the user text representation layer and the commodity text representation layer are adjusted according to the loss function to complete one training; after repeated training with the positive and negative sample sets for multiple times, the trained user text representation layer and the commodity text representation layer are obtained, thereby obtaining a trained feature extraction module; finally, each user information in the original data set is sequentially passed through the user feature mapping layer and the trained user text representation layer to obtain a corresponding user feature vector, and each commodity information in the original data set is sequentially passed through the commodity feature mapping layer and the trained commodity text representation layer to obtain a corresponding commodity feature vector.

[0014] In the user feature mapping layer, various types of data in each user information are mapped to obtain corresponding mapping features; then, the dimensions of each mapping feature of the current user information are reduced to obtain corresponding features after dimension reduction; finally, all the features after dimension reduction corresponding to the current user information are concatenated to obtain a user input vector; in the product feature mapping layer, various types of data in each product information are mapped to obtain corresponding mapping features, then, the dimensions of each mapping feature of the current product information are reduced to obtain corresponding features after dimension reduction, and all the features after dimension reduction corresponding to the current product information are concatenated to obtain a product input vector.

[0015] In step 2, the user text representation layer and the product text representation layer have the same structure, both of which are fully connected networks.

[0016] In step 2, the loss function satisfies the following formula:

[0017]

[0018] Among them, u represents the output of the user text representation layer, i represents the output of the product text representation layer, cos(u, i) represents the cosine similarity between the outputs of the user text representation layer and the product text representation layer, margin represents the dissimilarity threshold, and JL represents whether there is interaction between the user and the product.

[0019] In step 3, each interaction sample in the interaction sample set includes a user feature vector and a product feature vector corresponding to historical interaction products, a product feature vector corresponding to the current interaction product, and a flag "done" indicating that the user has ended the product interaction, wherein the user feature vector and the product feature vector corresponding to the historical interaction products serve as the state of the reinforcement learning model for novelty perception, and the product feature vector corresponding to the current interaction product serves as the action of the reinforcement learning model for novelty perception.

[0020] In step 3, the reward function of the novelty-aware reinforcement learning model satisfies the following formula:

[0021] r(pd, ina)=r acc (pd, ina)+β·r nov (pd)

[0022]

[0023] Among them, r(pd, ina) represents the total reward value, pd represents the current interactive product, ina represents the interactive behavior, and r acc (pd, ina) represents the accuracy reward value, r nov (pd) represents the novelty reward value, β is the reward weight, and pop(pd) represents the popularity factor.

[0024] In step 3, the learning objective y of the novelty-aware reinforcement learning model satisfies the following formula:

[0025]

[0026] Among them, r is the reward obtained by taking action a in state s in the interaction sample set, γ is the discount factor, done is the sign that the user ends the product interaction, s′ is the state obtained after taking action a in state s in the interaction sample set, and a′ is the final action generated by inputting state s′ into the Actor module. is the action value Q estimated by the j-th target Q network, θ′ j represents the parameters of the j-th target Q network, and λ is the scaling coefficient.

[0027] In step 4, when the user in the target interaction sample is an old user of the current user's online shopping website, the target interaction sample includes user information, product information and historical interaction information; when the user in the target interaction sample is a new user of the current user's online shopping website, the target interaction sample includes user information.

[0028] The fully connected network includes two connected fully connected layers.

[0029] 2. A Product Recommendation System Based on Reinforcement Learning Novelty Perception

[0030] The original data set generation unit is used to obtain the historical data of user-product interaction on the user's online shopping website and generate an original data set;

[0031] A feature extraction unit, used for storing a feature extraction module;

[0032] A first training unit is used to train a feature extraction module in the feature extraction unit using the original data set until the training is completed;

[0033] The product recommendation unit is used to store the reinforcement learning model of novelty perception;

[0034] An interaction sample set generation unit is used to extract the user feature vector and the product feature vector corresponding to the original data set by using the feature extraction module trained in the feature extraction unit, and to construct an interaction sample set according to the user feature vector, the product feature vector and the corresponding historical interaction information in combination with the time sequence of the user-product interaction;

[0035] The second training unit is used to train the novelty perception reinforcement learning model using the interaction sample set until the training is completed;

[0036] The output module is used to generate target interaction samples of target users, input the target interaction samples into the trained reinforcement learning model, and output the product recommendation results.

[0037] 3. A computer device

[0038] The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the product recommendation method based on reinforcement learning novelty perception when executing the computer program.

[0039] 4. A computer-readable storage medium

[0040] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the product recommendation method based on reinforcement learning novelty perception are implemented.

[0041] V. A computer program product

[0042] The computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the method for recommending products based on reinforcement learning novelty perception.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] Compared with those supervised learning methods, the reinforcement learning model used in the present invention can not only solve the popularity bias problem, but also better capture the dynamic transfer of user preferences and reduce the amplification effect of feedback loops. In addition, the batch-restricted deep Q learning algorithm introduced in the reinforcement learning model can reduce the extrapolation error that occurs when training the deep Q network.

[0045] Compared with some common reward functions, the present invention proposes a dual reward function that takes into account both accuracy and novelty. The present invention introduces the popularity of the product into the reward function, which guides the reinforcement learning training to minimize the popularity of the recommendation list while maximizing the accuracy of the recommendation, thereby reducing the mutual interference between accuracy and debiasing in a dynamic environment.

[0046] The method proposed in the present invention can effectively solve the popularity deviation problem existing in the recommendation system, and the accuracy and novelty indicators are significantly improved, and is particularly suitable for the recommendation of long-tail products. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The present invention is a flow chart of the method.

[0048] Figure 2 This is the network structure diagram of the feature extraction module.

[0049] Figure 3 Flowchart for batch-restricted deep Q-learning in reinforcement learning models. DETAILED DESCRIPTION

[0050] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0051] like Figure 1 As shown, a product recommendation method based on reinforcement learning novelty perception proposed by the present invention includes the following steps:

[0052] Step 1: Obtain the historical data of user-product interaction on the user's online shopping website, and construct the original data set based on the user information, product information, and historical interaction information in the historical data of user-product interaction;

[0053] User information includes ID, gender, age, and purchasing power; product information includes ID, category ID, store ID, and brand ID; historical interaction information includes timestamp, user ID, product ID, and interaction type, where interaction types include browsing, collecting, adding to cart, and purchasing.

[0054] The user-product interaction data of the user online shopping website to be trained is obtained. In this embodiment, users with more than 20 interactions are randomly selected from the user-product interaction history data according to the number of interactions between the user and the product, and the corresponding product information and historical interaction information are retained, and those users who interact with the product less than 20 times are excluded. 8,000 users and their interaction information are randomly selected and recorded from users with 20 to 50 interactions, and 24,000 users and their interaction information are randomly selected and recorded from users with more than 50 interactions, and the corresponding product information is collected, so as to obtain the original data set.

[0055] Step 2: After training the feature extraction module with the original data set, a trained feature extraction module is obtained, and then the trained feature extraction module is used to extract features of all user information and product information in the original data set to obtain corresponding user feature vectors and product feature vectors;

[0056] Step 2 is as follows:

[0057] like Figure 2 As shown, the feature extraction module includes a user feature mapping layer, a product feature mapping layer, a user text representation layer and a product text representation layer, the user feature mapping layer is connected to the user text representation layer, and the product feature mapping layer is connected to the product text representation layer; the user information and product information in the original data set are combined according to the historical interaction information to obtain positive samples and negative samples, thereby obtaining positive and negative sample sets; during the training process of the feature extraction module, the user information in each sample is used as the input of the user feature mapping layer, and the product information in each sample is used as the input of the product feature mapping layer; the loss function is calculated according to the output of the user text representation layer and the product text representation layer and the interaction information of each sample; the network parameters of the user text representation layer and the product text representation layer are adjusted according to the loss function to complete a training; after repeated training with the positive and negative sample sets for multiple times, the trained user text representation layer and the product text representation layer are obtained, thereby obtaining a trained feature extraction module; finally, each user information in the original data set is sequentially passed through the user feature mapping layer and the trained user text representation layer to obtain the corresponding user feature vector, and each product information in the original data set is sequentially passed through the product feature mapping layer and the trained product text representation layer to obtain the corresponding product feature vector. Among them, positive samples are samples in which users and products interact, while negative samples include users and randomly generated products that do not interact with the users.

[0058] In the user feature mapping layer, each type of data in each user information is mapped according to their respective frequencies of occurrence, specifically mapped to numbers from 0 to M-1, where M is the number of data after removing the same data, to obtain the corresponding mapping features; then the embedding method is used to reduce the dimensions of each mapping feature of the current user information to obtain the corresponding reduced-dimensional features; that is, the mapping features are converted into low-dimensional dense word vectors, and finally, all the reduced-dimensional features corresponding to the current user information are concatenated to obtain the user input vector.

[0059] In the product feature mapping layer, each type of data in each product information is mapped according to its frequency of occurrence, specifically mapped to numbers from 0 to M-1, where M is the number of data after removing the same data, to obtain the corresponding mapping features, and then use the embedding method to reduce the dimensions of each mapping feature of the current product information to obtain the corresponding reduced-dimensional features, that is, convert the mapping features into low-dimensional dense word vectors, and concatenate all the reduced-dimensional features corresponding to the current product information to obtain the product input vector. The length of the product input vector is 4N, where N is the length of each reduced-dimensional feature. The emmbeding method refers to mapping discrete data to continuous variables, counting the number of each feature as M, and randomly generating an M×N-dimensional vector as a mapping table, where N is the length of the mapped vector. According to the serial number of the product or user feature information in the feature, a fixed length N vector is retrieved to represent the feature information.

[0060] The user text representation layer and the product text representation layer have the same structure, both of which are fully connected networks. The fully connected network consists of two connected fully connected layers, each of which contains two hidden layers with sizes of 256 and 128 respectively. Each hidden layer uses the ReLU activation function. The loss function is defined by the cosine similarity between users and products and the interaction information between users and products, and satisfies the following formula:

[0061]

[0062] Among them, u represents the output of the user text representation layer, i represents the output of the product text representation layer, cos(u,i) represents the cosine similarity between the outputs of the user text representation layer and the product text representation layer, and margin represents the dissimilarity threshold to prevent excessive losses. JL indicates whether there is interaction between the user and the product. If yes, JL = 1, if not, JL = -1.

[0063] Step 3: Combine the time sequence of user-product interaction, build an interaction sample set based on the user feature vector, product feature vector and corresponding historical interaction information, and use the interaction sample set to train the novelty-aware reinforcement learning model to obtain a trained reinforcement learning model.

[0064] Each interaction sample in the interaction sample set includes a user feature vector and a product feature vector corresponding to the historical interaction product, a product feature vector corresponding to the current interaction product, and a flag indicating that the user has ended the product interaction, which is done. The user feature vector and the product feature vector corresponding to the historical interaction product serve as the state of the novelty-aware reinforcement learning model, and the product feature vector corresponding to the current interaction product serves as the action of the novelty-aware reinforcement learning model. After determining the current interaction product, take the 10 products closest to the current interaction product as the historical interaction products. If there are none, replace them with a zero vector of the same length. The products represented by the zero vector should also be included in the total number of products. For convenience, the flag indicating that the user has ended the product interaction is done. The product with which the user last interacted is used as the end flag according to the time. Set done = 1, and the rest of the interaction records are set to done = 0.

[0065] The reward function of the novelty-aware reinforcement learning model is a novelty-aware dual reward function that takes into account both accuracy and novelty, eliminating popularity bias while maintaining recommendation accuracy. Specifically, it satisfies the following formula:

[0066] r(pd, ina)=r acc (pd, ina)+β·r nov (pd)

[0067]

[0068] Among them, r(pd, ina) represents the total reward value, pd represents the current interactive product, ina represents the interactive behavior, including browsing, collecting, adding to the shopping cart and purchasing, r acc (pd, ina) represents the accuracy reward value, r nov (pd) represents the novelty reward value, β is the reward weight, which is used to measure the novelty reward r nov and the accuracy reward r acc The coefficient of the ratio. pop(pd) represents the popularity factor, and pop(pd)∈[0,1] represents the normalized number of occurrences of product pd in the interaction sample set.

[0069] When users interact with products in different ways, they can receive different accuracy rewards. The accuracy rewards for browsing, collecting, adding to cart, and purchasing are 1, 2, 2, and 3 respectively.

[0070] The novelty reward function increases the exposure of low-popularity products by increasing the reward value of such products, that is, increasing the chances of such products being recommended.

[0071] The learning objective y of the novelty-aware reinforcement learning model satisfies the following formula:

[0072]

[0073] Among them, r is the reward obtained by taking action a in state s in the interaction sample set, γ is the discount factor, done is the sign that the user ends the product interaction, s′ is the state obtained after taking action a in state s in the interaction sample set, and a′ is the final action generated by inputting state s′ into the Actor module. is the action value Q estimated by the j-th target Q network, θ′ j represents the parameters of the j-th target Q network, and λ is the proportional coefficient, which represents the proportion of smaller Q values ​​added to the calculation.

[0074] Specifically:

[0075] like Figure 3 As shown, the interaction sample set is input into the Actor module training in the batch-restricted deep Q learning module: first, the information of the user and the product they interact with is input into the Actor module to obtain the user state s, the action a under state s, the next state s′ that state s turns to after action a, and the next action a′ predicted by the Actor module based on state s′, which is used to obtain actions and recommend products.

[0076] The Actor module generates the model G by VAE w (s) and the perturbation model ξ φ (s, a), where w and φ are the parameters of the VAE generation model and the perturbation model, s is the state, and a is the action.

[0077] The VAE generative model consists of an encoder and decoder where w1 and w2 are the parameters of the encoder and decoder respectively, and the latent variable z satisfies the normal distribution.

[0078] First, the state s is composed of the encoding of the user and the most recently interacted item, with a dimension of B×11N, where B is the batch size. The user state s and the item currently interacting with the user are input into the encoder, and the parameters μ and σ of the normal distribution required by the decoder are output, with a size of B×2N. The encoder is composed of a fully connected neural network, with two hidden layers of size 64, each using the ReLU activation function, and two layers for outputting parameters μ and σ respectively. In addition, for σ, the value greater than 15 is limited to 15, and the value less than -4 is limited to -4, to ensure that the action deviation generated during decoding is not too large.

[0079] Secondly, the hidden variable z is calculated using the following formula. The hidden variable z and the state s are used as the input of the decoder. The decoder is also composed of a fully connected layer, and also has two hidden layers, both of size 64, and each layer uses the ReLU activation function. The output is the predicted action obtained The dimension is B×N.

[0080]

[0081] in, represents a normal distribution with mean 0 and variance 1.

[0082] The data for VAE training consists of user state s, action a, encoder and decoder parameters w i (i=1,2) is updated by the following formula:

[0083]

[0084] Among them, KL divergence From a normal distribution with mean μ and variance σ Normal distribution with mean 0 and variance 1 The similarity index definition is as follows:

[0085]

[0086] Among them, f(x) and h(x) are the probability density functions corresponding to the two random variables F and H respectively.

[0087] At the same time, the next state s′ also passes through the VAE generation model, but it is not used to update the parameters, but to obtain the predicted action Used for the following training.

[0088] Finally, the next state s′ and the predicted action Input disturbance model ξ φ , and get the final action a′. Similar to the input of the encoder, the user state s′ and the predicted action The model is also composed of a fully connected layer, with two hidden layers, each with a size of 64, and each layer uses a ReLU activation function. The output is the value out used for perturbation, with a dimension of B×N. The formula for the final action a′ is as follows:

[0089]

[0090] Among them, the range of tanh(out) is [-1,1]. This makes the maximum perturbation range of the model is the maximum perturbation value, maximizing the cumulative reward by making the action a′ not too conservative, while not perturbing too much. The parameters of the perturbation model are calculated by the predicted action generated by VAE. And the results of the Q network are updated:

[0091]

[0092] When the Actor module outputs the final action a′, it is then normalized and vector-multiplied with the embedding of each item. The K items with the highest product are selected for recommendation, where K is the number of items recommended to the user once.

[0093] Then, the next state s′, final action a′, reward r and end flag done are input into the Critic module, and the output Q value is used to evaluate the input action and optimize the strategy.

[0094] The Critic module consists of two Q networks and and two target Q networks and θ1, θ2, θ′1, θ′2 are the parameters of the Q network. Their functions and processes are as follows:

[0095] The inputs of the two Q networks are state s and action a, with a total dimension of B×12N. The two Q networks consist of two fully connected layers with two hidden layers, both of size 64, followed by a ReLU activation function. The output is the estimated Q value of each action in each state, with a dimension of B×1. In order to solve the instability and maximum estimation problems caused by the greedy strategy during training, two target Q networks with the same structure as the original Q network are copied to calculate the Q target value, which is more reliable during the update process.

[0096] First, the learning target y is determined by a convex combination of two values ​​produced by the two target Q networks. Its purpose is to penalize high variance estimates in uncertain areas. The formula is as follows:

[0097]

[0098] Among them, r is the reward obtained by taking action a in state s in the interaction sample set, γ is the discount factor, done is the sign that the user ends the product interaction, s′ is the state obtained after taking action a in state s in the interaction sample set, and a′ is the final action generated by inputting state s′ into the Actor module. is the action value Q estimated by the j-th target Q network, θ′ j represents the parameters of the j-th target Q network, and λ is the proportional coefficient, which represents the proportion of smaller Q values ​​added to the calculation.

[0099] Secondly, according to this goal, the parameters θ of the Q network are l (l=1,2) The update formula is as follows:

[0100]

[0101] Among them, a′ is the final action output after inputting state s into the VAE generation model and perturbation model. The parameter θ′ of the target Q network k (m=1,2) soft update:

[0102] θ′ m ←τθ m +(1-τ)θ′ m

[0103] Among them, τ=0.001.

[0104] Step 4: Input the target interaction sample into the trained reinforcement learning model, and the model outputs the product recommendation results.

[0105] In step 4, when the user in the target interaction sample is an old user of the current user's online shopping website, the target interaction sample includes user information, product information and historical interaction information; when the user in the target interaction sample is a new user of the current user's online shopping website, the target interaction sample includes user information.

[0106] The data set used in the simulation experiment is shown in Table 1.

[0107] Table 1 shows the data set used in the simulation experiment.

[0108]

[0109] The UserBehavior dataset comes from a product recommendation competition and contains the user-product interaction data of online shopping websites. The experimental dataset SelectedUB randomly selects 32,000 users from UserBehavior based on the number of interactions between users and products. Among them, the number of interactions of 8,000 users is greater than 50, and the number of interactions of the rest is greater than 20 and less than 50. The corresponding user products and their interaction information are retained. Users have the following four behaviors for their favorite products: browse (pv), favorite (favor), add to cart (cart), and buy (buy). In addition to the above information, this dataset also has information such as user gender, age, purchasing power, and items have categories, stores, and brand names.

[0110] 20% of the data are randomly selected as the test set, and the rest are the training set.

[0111] The accuracy of the recommendation system is usually evaluated using precision@K and NDCG@K. Precision@K represents the accuracy of the recommendation task when K items are recommended to user u each time. The more recommended items are selected by user u, the larger the precision value is, which means the higher the accuracy of the recommendation is. NDCG@K combines the relevance and ranking position of the ranking to represent the quality of the recommended K item sequence. The higher the recommended item selected by the user, the higher the quality of the recommendation.

[0112] The precision@K formula is defined as follows:

[0113]

[0114] Among them, U represents the entire user set, and hits(u) represents the number of intersection elements between the product list recommended by the system to user u and the test item.

[0115] The NDCG@K formula is defined as follows:

[0116]

[0117] Among them, DCG u @K represents the cumulative revenue when there are K recommended products in the product recommendation list for user u, and IDCG u @K represents the cumulative revenue of user u when there are K recommended products in the product recommendation list under ideal conditions, and |·| represents the size of a set. For a single user, the formula for setting the cumulative revenue is:

[0118]

[0119] Here, k represents the position of the product in the recommendation list.

[0120] To evaluate the novelty of each method, the experiment used Coverage. Coverage describes the ability of the recommendation system to mine long-tail items. The method is the percentage of all possible recommended items to all items. The higher the result, the more types of recommended items there are, and the more it reflects the ability of the recommendation system to recommend long-tail items.

[0121] The Coverage formula is defined as follows:

[0122]

[0123] Among them, ∪ u∈U represents the entire set of user set U, L u represents the list of recommended items generated by the recommendation system for user u, so ∪ u∈U L uRepresents the set of products recommended by the system to all users. I represents the set of all recommended items.

[0124] DDPG uses a deep deterministic policy gradient algorithm to make up for the shortcoming of DQN that it cannot handle continuous control problems. BCQ is a modified version of DDPG in offline scenarios, which solves certain errors caused by learning on a fixed data set. The algorithm uses a generative model to generate state-action combinations similar to those in the data set, and then uses a perturbation model to add some noise to the generated actions to increase diversity.

[0125] The model in the control group is used to process the samples in the SelectedUB dataset. The running results are shown in Table 2:

[0126] Table 2 is a comparison table of the performance of the present invention and the existing method in the test set recommendation

[0127]

[0128]

[0129] As can be seen from Table 2, compared with the existing methods, the method proposed in the present invention achieves the best performance. There are three main reasons: (1) The present invention uses the low-dimensional vectors of users and products generated by the feature extraction module. This method is supervised and takes into account the relationship between users and purchased products, thereby improving the accuracy of product recommendations. (2) The double reward model can capture the situation where some products are overly exposed to users and guide reinforcement learning training, thereby avoiding popularity bias. (3) The BCQ algorithm has the advantage of eliminating extrapolation errors compared to the ordinary Q learning algorithm, and can obtain better recommendation results when data is insufficient.

[0130] It can also be seen from Table 2 that the method proposed in the present invention has significant improvements in different dimensions. The control group DDPG performed the worst, and BCQ improved compared with the above control group. The precision@10 of the method proposed in the present invention increased by 8% compared with the best control group, NDCG@10 increased by 10% compared with the best control group, precision@20 increased by 16% compared with the best control group, and NDCG@20 increased by 8% compared with the best control group. Therefore, the method proposed in the present invention can maintain or even surpass the control group in terms of recommendation accuracy. As for novelty, the Coverage of the method proposed in the present invention is 16% and 26.8%, which is 7.1% and 2.3% higher than BCQ, respectively, reflecting that the modified reward module plays a role in recommending long-tail items and alleviates popularity deviation.

[0131] The present invention also proposes a product recommendation system based on reinforcement learning novelty perception, comprising:

[0132] The original data set generation unit is used to obtain the historical data of user-product interaction on the user's online shopping website and generate an original data set;

[0133] A feature extraction unit, used for storing a feature extraction module;

[0134] A first training unit is used to train a feature extraction module in the feature extraction unit using the original data set until the training is completed;

[0135] The product recommendation unit is used to store the reinforcement learning model of novelty perception;

[0136] An interaction sample set generation unit is used to extract the user feature vector and the product feature vector corresponding to the original data set by using the feature extraction module trained in the feature extraction unit, and to construct an interaction sample set according to the user feature vector, the product feature vector and the corresponding historical interaction information in combination with the time sequence of the user-product interaction;

[0137] The second training unit is used to train the novelty perception reinforcement learning model using the interaction sample set until the training is completed;

[0138] The output module is used to generate target interaction samples of target users, input the target interaction samples into the trained reinforcement learning model, and output the product recommendation results.

[0139] The present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a product recommendation method based on reinforcement learning novelty perception are implemented.

[0140] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of a product recommendation method based on reinforcement learning novelty perception are implemented.

[0141] The present invention also proposes a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a product recommendation method based on reinforcement learning novelty perception.

[0142] Finally, it should be noted that the above embodiments and explanations are only used to illustrate the technical solution of the present invention rather than to limit it. Those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope disclosed in the technical solution of the present invention, which should be included in the scope of protection of the claims of the present invention.

Claims

1. A product recommendation method based on reinforcement learning novelty perception, characterized in that: The following steps are involved: Step 1: Obtain the historical data of user-product interaction on the user's online shopping website, and construct the original data set based on the user information, product information, and historical interaction information in the historical data of user-product interaction; Step 2: After training the feature extraction module with the original data set, a trained feature extraction module is obtained, and then the trained feature extraction module is used to extract features of all user information and product information in the original data set to obtain corresponding user feature vectors and product feature vectors; Step 3: Combine the time sequence of user-product interaction, build an interaction sample set based on the user feature vector, product feature vector and corresponding historical interaction information, and use the interaction sample set to train the novelty perception reinforcement learning model to obtain a trained reinforcement learning model; Step 4: Input the target interaction sample into the trained reinforcement learning model, and the model outputs the product recommendation results.

2. A method for recommending products based on novelty perception through reinforcement learning according to claim 1, characterized in that: The step 2 is specifically as follows: The feature extraction module includes a user feature mapping layer, a commodity feature mapping layer, a user text representation layer and a commodity text representation layer, wherein the user feature mapping layer is connected to the user text representation layer, and the commodity feature mapping layer is connected to the commodity text representation layer; the user information and commodity information in the original data set are combined according to the historical interaction information to obtain positive samples and negative samples, thereby obtaining positive and negative sample sets; during the training process of the feature extraction module, the user information in each sample is used as the input of the user feature mapping layer, and the commodity information in each sample is used as the input of the commodity feature mapping layer; the loss function is calculated according to the output of the user text representation layer and the commodity text representation layer and the interaction information of each sample; the network parameters of the user text representation layer and the commodity text representation layer are adjusted according to the loss function to complete one training; after repeated training using the positive and negative sample sets for multiple times, the trained user text representation layer and the commodity text representation layer are obtained, thereby obtaining a trained feature extraction module; Finally, each user information in the original data set is sequentially passed through the user feature mapping layer and the trained user text representation layer to obtain the corresponding user feature vector, and each product information in the original data set is sequentially passed through the product feature mapping layer and the trained product text representation layer to obtain the corresponding product feature vector.

3. A method for recommending products based on novelty perception through reinforcement learning according to claim 2, characterized in that: In the user feature mapping layer, various types of data in each user information are mapped to obtain corresponding mapping features; then, the dimensions of each mapping feature of the current user information are reduced to obtain corresponding features after dimension reduction; finally, all the features after dimension reduction corresponding to the current user information are concatenated to obtain a user input vector; in the product feature mapping layer, various types of data in each product information are mapped to obtain corresponding mapping features, then, the dimensions of each mapping feature of the current product information are reduced to obtain corresponding features after dimension reduction, and all the features after dimension reduction corresponding to the current product information are concatenated to obtain a product input vector.

4. The method for recommending products based on novelty perception through reinforcement learning according to claim 2, characterized in that: In step 2, the user text representation layer and the product text representation layer have the same structure, both of which are fully connected networks.

5. The method for recommending products based on novelty perception through reinforcement learning according to claim 1, characterized in that: In step 2, the loss function satisfies the following formula: Among them, u represents the output of the user text representation layer, i represents the output of the product text representation layer, cos(u,i) represents the cosine similarity between the outputs of the user text representation layer and the product text representation layer, margin represents the dissimilarity threshold, and JL represents whether there is interaction between the user and the product.

6. The method for recommending products based on novelty perception through reinforcement learning according to claim 1, characterized in that: In step 3, each interaction sample in the interaction sample set includes a user feature vector and a product feature vector corresponding to historical interaction products, a product feature vector corresponding to the current interaction product, and a flag "done" indicating that the user has ended the product interaction, wherein the user feature vector and the product feature vector corresponding to the historical interaction products serve as the state of the reinforcement learning model for novelty perception, and the product feature vector corresponding to the current interaction product serves as the action of the reinforcement learning model for novelty perception.

7. The method for recommending products based on novelty perception through reinforcement learning according to claim 1, characterized in that: In step 3, the reward function of the novelty-aware reinforcement learning model satisfies the following formula: r(pd,ina)=r acc (pd,ina)+β·r nov (pd) Among them, r(pd,ina) represents the total reward value, pd represents the current interactive product, ina represents the interactive behavior, and r acc (pd,ina) represents the accuracy reward value, r nov (pd) represents the novelty reward value, β is the reward weight, and pop(pd) represents the popularity factor.

8. The method for recommending products based on novelty perception through reinforcement learning according to claim 1, characterized in that: In step 3, the learning objective y of the novelty-aware reinforcement learning model satisfies the following formula: Among them, r is the reward obtained by taking action a in state s in the interaction sample set, γ is the discount factor, done is the sign that the user ends the product interaction, s′ is the state obtained after taking action a in state s in the interaction sample set, and a′ is the final action generated by inputting state s′ into the Actor module. is the action value Q estimated by the j-th target Q network, θ′ j represents the parameters of the j-th target Q network, and λ is the scaling coefficient.

9. The method for recommending products based on novelty perception through reinforcement learning according to claim 1, characterized in that: In step 4, when the user in the target interaction sample is an old user of the current user's online shopping website, the target interaction sample includes user information, product information and historical interaction information; when the user in the target interaction sample is a new user of the current user's online shopping website, the target interaction sample includes user information.

10. A product recommendation system based on reinforcement learning novelty perception, characterized in that: include: The original data set generation unit is used to obtain the historical data of user-product interaction on the user's online shopping website and generate an original data set; A feature extraction unit, used for storing a feature extraction module; A first training unit is used to train a feature extraction module in the feature extraction unit using the original data set until the training is completed; The product recommendation unit is used to store the reinforcement learning model of novelty perception; An interaction sample set generation unit is used to extract the user feature vector and the product feature vector corresponding to the original data set by using the feature extraction module trained in the feature extraction unit, and to construct an interaction sample set according to the user feature vector, the product feature vector and the corresponding historical interaction information in combination with the time sequence of the user-product interaction; The second training unit is used to train the novelty perception reinforcement learning model using the interaction sample set until the training is completed; The output module is used to generate target interaction samples of target users, input the target interaction samples into the trained reinforcement learning model, and output the product recommendation results.

Images