A Deep Interest Evolution Recommendation Method, Device, Equipment and Storage Medium

Abstract

The present invention discloses a deep interest evolution recommendation method, device, equipment and storage medium, extracts user features and item features required by the model from the training dataset, groups the extracted user features and item features by user, sorts the grouped items by timestamp, extracts user embedding features and item embedding features from the sorted items, generates position embedding features according to the extracted user features and item features, adds the item embedding features and the position embedding features and inputs them into the established Transformer network to obtain an output result, connects the output result obtained by the Transformer network with the user embedding features and inputs them into a two-layer feedforward neural network, uses GELU as the activation function to obtain the final probability distribution, obtains the final predicted item through the probability distribution, and based on the deep evolution recommendation method of high-order feature fusion, fully utilizes user information and item information to improve the prediction ability of the model and improve the accuracy of prediction recommendation.

Description

Technical Field

[0001] The present invention belongs to the field of recommendation algorithms, and relates to a deep interest evolution recommendation method, device, equipment and storage medium. Background Art

[0002] Recommendation algorithms have become an effective strategy for solving information overload. Its essence is an information filtering system that helps users quickly select information and improves the reach efficiency between users and items. Accurately describing user interests is a core metric of a recommendation system. Traditional recommendation algorithms are mostly generated based on basic patterns such as content and social relationships, and all regard the behaviors generated by users-items as independent information. However, in real life, users' interests change dynamically over time, and there is a strong correlation between their previous and subsequent behaviors. In a typical e-commerce recommendation scenario, when a user buys a mobile phone, it is more reasonable to recommend related peripheral products for the user next. However, traditional recommendations mostly model the relationship between users and items based on the positive feedback information of users on items and do not consider the influence of time factors. Using a sequential modeling method based on users' historical behaviors can well solve such problems. At present, modeling users' dynamic preferences from their historical behaviors is challenging and crucial for recommendation algorithms.

[0003] Hidasi B, Karatzoglou A, Baltrunas L, et al. first used RNN for modeling the user sequence behavior in the recommendation system. By introducing a ranking loss function, they learned the sequential decision-making data of users, and then used the RNN model to model the sparse sequential decision-making data. When the input sequence is too long, RNN will have the problem of gradient vanishing or gradient explosion when learning parameters using the backpropagation algorithm. The LSTM model proposed by Hochreiter S, Schmidhuber J. Long Short-Term Memory solves the problem of gradient vanishing and gradient explosion of RNN to a certain extent through its unique gated unit structure. However, the model structure requires the operation of the next moment to use the operation result of the previous moment as input, resulting in the model being unable to operate in parallel and the training of the model taking too long. In addition, the left-to-right unidirectional model structure of RNN determines that each item can only encode the information of the previous items. However, in actual applications, the user's historical sequence is not necessarily strictly ordered. Therefore, the single-item model limits the ability to learn the hidden representation between items from the user's historical sequence. Fei Sun, Jun Liu, et al. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer proposed a bidirectional serialization model to model user behavior from the contexts in both directions of the user behavior sequence. In addition, this model abandoned the use of RNN or CNN and instead used the Transformer structure based on the attention mechanism, solving the problem of serial computing of RNN. However, this model lacks user information and scenario information and cannot model the relationship between users and items, resulting in relatively low recommendation accuracy. Summary of the Invention

[0004] The purpose of the present invention is to provide a deep interest evolution recommendation method, device, device and storage medium, which solves the problem of the lack of user information and scenario information in the above recommendation algorithm and improves the accuracy of its recommendation.

[0005] A deep interest evolution recommendation method includes the following steps:

[0006] S1, extract the user features and item features required by the model from the training dataset, group the extracted user features and item features by user, sort the grouped items by timestamp, and extract the user embedding features and item embedding features from the sorted items;

[0007] S2. Generate location embedding features based on the extracted user features and item features, add the item embedding features and the location embedding features, and input the result into the established Transformer network to obtain an output result;

[0008] S3: Connect the output result obtained from the Transformer network with the user embedding features, input the result into a two-layer feedforward neural network, use GELU as the activation function to obtain the final probability distribution, and obtain the final predicted item through the probability distribution.

[0009] Furthermore, cluster the item embeddings interacted by the user over a relatively long period of time into several categories, and then generate user embedding features from the embeddings in each category.

[0010] Furthermore, adopt the Medoid method to find an item among all items in each cluster to represent this cluster, and this item satisfies that the sum of the squared distances from other members in the same cluster is the smallest;

[0011] embedding(C)←P m , where

[0012] Use the generated item embedding to represent this cluster and store it in the form of key-value pairs.

[0013] Furthermore, use GELU as the activation function to obtain the probability distribution:

[0014] P(v)=softmax(GELU(hW P +b P )E T +b O ) (5)

[0015] where W P is a learnable projection matrix, b P , b O are bias terms, E is the Embedding matrix of the commodity set V, and h is the output of the feedforward neural network.

[0016] Furthermore, the user features include user ID, gender, age, occupation, the average score of the user for the item, and the total number of items evaluated by the user.

[0017] Furthermore, the Transformer network adopts stacked Transformer layers, and each layer of Transformer includes a multi-head attention module and a feedforward neural network.

[0018] Furthermore, use the Cloze task to mask 15% of the items in the input sequence, and the loss function is:

[0019]

[0020] where S u ' is the masked version of the user behavior history S u ; is the randomly masked item, is the masked item, v m is the real item.

[0021] A deep interest evolution recommendation system, comprising a preprocessing module, an optimization training module and a search module;

[0022] The preprocessing module is configured to extract the user features and item features required by the model from the training dataset, group the extracted user features and item features by user, sort the grouped items by timestamp, and extract the user embedding features and item embedding features from the sorted items;

[0023] The prediction module is configured to generate position embedding features according to the extracted user features and item features, add the item embedding features and the position embedding features and input the result into the established Transformer network to obtain an output result; the output result obtained by the Transformer network is connected with the user embedding features and then input into a two-layer feed-forward neural network, and GELU is used as the activation function to obtain the final probability distribution, and the final predicted item is obtained through the probability distribution.

[0024] A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the deep interest evolution recommendation method when executing the computer program.

[0025] A computer-readable storage medium stores a computer program, and the computer program implements the steps of the deep interest evolution recommendation method when executed by a processor.

[0026] Compared with the prior art, the present invention has the following beneficial technical effects:

[0027] A deep interest evolution recommendation method of the present invention extracts user features and item features required by the model from the training dataset, groups the extracted user features and item features by user, sorts the grouped items by timestamp, extracts user embedding features and item embedding features from the sorted items, generates position embedding features according to the extracted user features and item features, adds the item embedding features and the position embedding features and inputs them into the established Transformer network to obtain an output result, connects the output result obtained by the Transformer network with the user embedding features and inputs them into a two-layer feedforward neural network, uses GELU as the activation function to obtain the final probability distribution, obtains the final predicted item through the probability distribution, and is a deep evolution recommendation method based on high-order feature fusion, which makes full use of user information and item information to improve the prediction ability of the model and the accuracy of prediction recommendation.

[0028] Furthermore, the present invention adopts the Medoid clustering method to divide the items interacted by the user within a relatively long time period into several categories, and then combines the time decay algorithm to obtain a feature embedding of the user as item classification information. By this method, user features are enriched. On the other hand, the generated feature embedding is stored in the form of key-value pairs, which is convenient for the system to quickly access and saves interaction time.

[0029] Furthermore, the present invention adopts an attention mechanism, which can model the user interaction sequence from both positive and negative time dimensions. In the experiment, the multi-head attention mechanism is used to learn the interaction information between the user and the item from different channels, so as to improve the performance ability of the model.

[0030] Furthermore, the present invention adopts a two-layer feedforward neural network to combine user information and item information. After the pre-model learns the relationship between items in the user interaction sequence, the user basic features and the learned user features are used as global user information, and the user features and the output of the Transformer model are used as the input of the feedforward neural network, so as to add user information to the whole model and make the model training no longer rely only on the item sequence. Description of the Drawings

[0031] Figure 1 is the hardware structure framework diagram of the application terminal in the embodiment of the present invention.

[0032] Figure 2 is the structural diagram of the Transformer network model in the embodiment of the present invention.

[0033] Figure 3 is the overall model structural diagram in the embodiment of the present invention.

[0034] Figure 4 is the schematic diagram of user feature cross in the embodiment of the present invention. Detailed implementation manners

[0035] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0037] As Figure 4 shown, a deep interest evolution recommendation method includes the following steps:

[0038] S1: Extract the user features and item features required by the model from the training dataset, group the extracted user features and item features by user, sort the grouped items by timestamp, and extract the user embedding features and item embedding features from the sorted items to provide input data for the model.

[0039] S2: Generate position embedding features according to the extracted user features and item features, add the item embedding features and the position embedding features, and input the result into the established Transformer network (Bert4Rec model) to obtain an output result;

[0040] S3: Connect the output result obtained by the Transformer network with the user embedding features, input the result into a two-layer feed-forward neural network, use GELU as the activation function to obtain the final probability distribution, and obtain the final predicted item through the probability distribution.

[0041] In S1, considering that the types of items that users may be interested in have a great impact on the final recommendation results, user embedding features are generated based on the item embeddings in the user's historical interaction sequence. The specific method is as follows: Take the item embeddings interacted by the user over a relatively long period of time and cluster them into several categories, and then generate the embedding features of the user from the embeddings in each category, that is, the user embedding features.

[0042] Specifically, obtain the historical sequence of interactions of the user in the past and divide it into several clusters; currently, the hierarchical clustering algorithm Ward is mainly used, and the measurement index for clustering is ESS (error sum of squares), and the formula is:

[0043]

[0044] The clustering process is as follows: First, initialize each point as a cluster. At this time, the ESS in each cluster is 0; then calculate the ESS of each cluster, and finally calculate the total ESS of all clusters; enumerate all binary clusters, calculate the total ESS value after merging all binary clusters, and select the two clusters with the smallest increase in the total ESS value to merge. Repeat the above steps until n is reduced to 1. It can be seen from the above steps that Ward clustering is very time-consuming. To calculate the ESS between any two clusters every time two clusters are merged, the time complexity of one calculation is O(n 2 ), considering that the number of items interacted by the recommended user may be large within a certain period of time, so this method is not very applicable.

[0045] This application adopts the Lance-Williams Algorithm calculation method. Let the initial five clusters be {A, B, C, D, E} respectively, and calculate the ESS between the five clusters. It is found through calculation that clusters A and B are the closest, so they are merged into cluster AB. Now there are 4 clusters {AB, C, D, E}, and the formula for calculating their ESS is:

[0046]

[0047] Where K represents the other clusters except the AB cluster, and n a , n b , n k represent the number of nodes in clusters A, B, and K. Initially, each node is a single cluster, so n = 1. Through this ESS calculation method, the amount of calculation in the clustering process is greatly reduced.

[0048] Calculate the medoid-based representation for each cluster; typical methods consider the clustering centroid, the time decay average model, or other more complex sequence models. However, there is a common problem with these methods: the embeddings obtained from them may be in different regions of the d-dimensional space, and when there are some outlier points assigned to the clusters, there will be a large within-cluster variance.

[0049] This application uses the Medoid method to find one of all items in each cluster to represent the cluster, and this item satisfies the minimum sum of squared distances from other members within the same cluster.

[0050] embedding(C)←P m ,where

[0051] Use the generated item embedding to represent the cluster and store it in a key-value pair manner for subsequent calculations of the model.

[0052] Calculate the importance score of each cluster for the user and calculate the user embedding feature. By introducing a time decay function to calculate the relative importance of different clusters to the user:

[0053]

[0054] where τ[i] is the interaction time between the user and the i-th item, C represents one of the clusters, and λ is a hyperparameter. The importance of the cluster is higher when the user interacts with the cluster more frequently or the activity time is closer. The user representation is the time decay average of the item embeddings.

[0055] Generate a sparse vector of other user features, then generate its dense vector through the embedding layer. Input all the dense vectors into a stacking layer, concatenate different embedding features and numerical features together to form a new feature vector containing all user features, and then input the feature vector into a fully connected layer for feature crossing.

[0056] Input the embedding vectors generated from the item features grouped by user into the Transformer network to model the dependencies between items. Connect its output with the generated user feature vector and then input it into a two-layer feedforward neural network, using GELU as the activation function to obtain the probability distribution:

[0057] P(v)=softmax(GELU(hW P +b P )E T +b O ) (5)

[0058] where W P is a learnable projection matrix, and b P , b O are bias terms, E is the Embedding matrix of the commodity set V, h is the output of the feedforward neural network. Here, shared commodity embeddings are used to alleviate the overfitting problem and reduce the size of the model. Finally, a multi-classifier is used to generate the final result.

[0059] In the BERT4Rec model, the size E of the item embedding features is the same as the hidden layer size H. From a modeling perspective, the item embedding features learn context-independent representations of words, while the hidden layer learns context-dependent representations. The hidden layer is more complex and requires more parameters, so H >> E is needed. However, in the actual model, the user interaction sequence V is usually large. If E = H at this time, when increasing the size of the hidden layer H, the dimension of the embedding matrix V×E will be very large. Here, the binding relationship between E and H is broken, and the embedding matrix is decomposed into two matrices with sizes V×E and E×H respectively, that is, the item is first projected into a low-dimensional embedding space E, and then projected into a high-dimensional hidden space H, reducing the dimension of the model's embedding matrix from O(V×H) to O(V×E + E×H). When H >> E, the number of parameters is significantly reduced. In implementation, the V×E and E×H matrices are randomly initialized. When calculating the item embedding features, the one-hot vector of the item is multiplied by the V×E-dimensional matrix, and then the obtained result is multiplied by the E×H-dimensional matrix.

[0060] The present invention provides a computing device, including at least one or more processors, a storage device, and an input unit. The computer storage device stores computer program code, and the processor executes the depth interest evolution recommendation method based on high-order feature fusion described above in the present invention by running the program code.

[0061] The method embodiment provided in this embodiment can be implemented in hardware, or can be implemented by a software module running on one or more processors, or can be implemented by any combination of hardware and software. Figure 1 is the hardware structure framework diagram of the application terminal of the depth interest evolution recommendation method based on high-order feature fusion in this embodiment. As Figure 1 shown, the computer may include at least one processor 102, a memory 103 for storing data, and an input unit 101. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only a schematic diagram, and it does not limit the structure of the above computer. For example, the computer may further include more or fewer components than Figure 1 shown, or have a different configuration from Figure 1 shown.

[0062] The memory 103 may include a high-speed RAM memory and may also include non-volatile storage devices, such as one or more disk storage devices, which can be used to store computer programs. The programs include, but are not limited to, software programs and modules of application software, such as the computer program corresponding to the deep interest evolution recommendation method based on high-order feature fusion in this embodiment. The processor 102 reads and runs the computer program stored in the memory 103, thereby implementing various application functions, such as implementing the above method. The processor 102 can also communicate with one or more input units 101 (such as a keyboard, etc.), and can also communicate with one or more devices that enable interaction between the user and the processor 102, or communicate with any device that enables the processor 102 to communicate with one or more other processors (such as a router, a modem, etc.). In addition, the processor 130 can also communicate with one or more networks through devices such as a network adapter. Examples of the above networks include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0063] As Figure 3 shown, a deep interest evolution recommendation system includes a preprocessing module, an optimization training module, and a search module;

[0064] The preprocessing module is used to extract the user features and item features required by the model from the training dataset, group the extracted user features and item features by user, sort the grouped items by timestamp, and extract the user embedding features and item embedding features from the sorted items;

[0065] The prediction module is used to generate position embedding features according to the extracted user features and item features, add the item embedding features and the position embedding features and input them into the established Transformer network to obtain an output result; the output result obtained by the Transformer network is connected with the user embedding features and then input into a two-layer feed-forward neural network, and GELU is used as the activation function to obtain the final probability distribution, and the final predicted item is obtained through the probability distribution.

[0066] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a terminal device, used to store programs and data. The computer-readable storage medium includes a built-in storage medium in the terminal device, which provides storage space and stores the operating system of the terminal. It may also include an extended storage medium supported by the terminal device. Moreover, one or more instructions suitable for being loaded and executed by a processor are stored in this storage space. These instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory or a non-volatile memory (Non-volatile memory), such as at least one disk memory. One or more instructions stored in the computer-readable storage medium can be loaded and executed by the processor to implement the corresponding steps in the above embodiments that can be used in the deep interest evolution recommendation method.

[0067] Preprocess the input data set. The specific processing flow is as follows:

[0068] Extract user features and item features, group the item features according to user features, and sort the grouped items according to the timestamp to generate the features of the user interaction item sequence input to the model. Additionally, randomly mask a part of the item features in the sequence through the Clozetask.

[0069] The required user features extracted include user ID, gender, age, occupation, the average score of the user for items, and the total number of items evaluated by the user. According to the user's interaction sequence with items, the embedding features of the user are obtained through the improved hierarchical clustering algorithm Ward.

[0070] Input the non-numerical features of the user into the Embedding layer to form its dense vector, concatenate different Embedding features and numerical features, and input them into the fully connected layer to fully cross-combine each dimension of the feature vector, enabling the model to capture more non-linear feature information.

[0071] Input the user's interaction sequence into the Transformer network to learn the relationship between each item in the user interaction sequence through a bidirectional model. As Figure 2 shown, the Transformer network adopts stacked Transformer layers, and each layer of Transformer includes a multi-head attention (Multi-Head Attention) module and a feed-forward neural network. The specific steps are as follows:

[0072] Input the item features into the embedding layer to generate their embedding features. The position of the item in the sequence, which is the time when the user interacts with the item, is important information. However, there is no iterative operation of the recurrent neural network in the Transformer network, so a position information, that is, position embedding features, must be generated for it. After summing the position embedding features and the item embedding features, a normalization operation is performed. In addition, the maximum sequence length N is set in the Transformer network. When the input sequence length exceeds N, the input sequence is truncated, [v1, v2,..., v t is truncated to the last N items [v t-n+1 ,..., v t .

[0073] Next, input the item embedding features and the position embedding features into the stacked Transformer layers. First, pass through the multi-head attention layer, linearly project H to the h subspace using different, learnable linear projections, and then apply the attention function h to generate the output result. In this way, the dependencies between item pairs are captured. Next, in order to let the model learn non-linear and different-dimensional interactions, the output of the attention sublayer is used as the input of the feed-forward neural network layer. Residual connections are added around each layer of the above two sublayers, and then layer normalization is performed. In addition, dropout is applied to the output of each sublayer. The output of each sublayer is LN(x + Dropout(sublayer(x))), where sublayer is the function implemented by the sublayer itself, and LN is the layer normalization function defined in the model. LN is used to normalize the input of all hidden units in the same layer to stabilize and accelerate the training of the network.

[0074] Connect the generated result with the user embedding features and then input it into a two-layer feed-forward neural network and use GELU as the activation function to obtain the final probability distribution, and obtain the final predicted item through the probability distribution.

[0075] Network Training

[0076] When processing the user's historical behavior sequence, use the Cloze task to mask 15% of the items in the input sequence, and let the model predict the masked items. The final loss function is:

[0077]

[0078] where S u ' is the masked version of the user behavior history S u , is the randomly masked item, is the masked item, v m is the real item.

[0079] When training the network, the number of Transformer layers L = 2, the number of heads h = 2, the dimension of each head d = 32, and the maximum sequence length N = 150. In this example, the stochastic gradient descent method with an Adam accelerator is used to optimize the objective function, where the momentum β1 = 0.9, β2 = 0.999, the weight decay (weight decay = 0.01), and the initial learning rate is 10e-4. The maximum number of iterations E = 100. In each iteration, this example jointly updates the segmentation network and the determination network. When the number of iterations is greater than the maximum number of iterations, the training stops and the trained model is saved.

Claims

1. A deep interest evolution recommendation method, characterized in that It includes the following steps: S1. Extract the user features and item features required by the model from the training dataset, group the extracted user features and item features by user, sort the grouped items by timestamp, and extract the user embedding features and item embedding features from the sorted items; S2. Generate position embedding features according to the extracted user features and item features, add the item embedding features and the position embedding features, and input the result into the established Transformer network to obtain the output result; S3: Connect the output result obtained by the Transformer network with the user embedding features, input the result into a two-layer feed-forward neural network, use GELU as the activation function to obtain the final probability distribution, and obtain the final predicted item through the probability distribution; Cluster the item embeddings interacted by the user in a relatively long time period into several categories, and then generate user embedding features from the embeddings in each category; Adopt the Lance-Williams Algorithm calculation method. Initialize it as 5 clusters, namely {A, B, C, D, E}, calculate the ESS between the five clusters. It is found that clusters A and B are the closest, so they are merged into cluster AB. Now there are 4 clusters {AB, C, D, E}, and the formula for calculating its ESS is: where K represents other clusters except the AB cluster, and n a , n b , n k represent the number of nodes in clusters A, B, and K. Initially, each node is a single cluster, so n = 1. Through this ESS calculation method, the computational complexity during the clustering process is greatly reduced; Adopt the Medoid method to find one of all the items in each cluster to represent this cluster, and this item satisfies that the sum of the squared distances from other members in the same cluster is the smallest; Use the generated item embedding to represent this cluster and store it in the form of key-value pairs for the subsequent calculation of the model; 2. The deep interest evolution recommendation method according to claim 1, wherein Use GELU as the activation function to obtain the probability distribution: P(v) = softmax(GELU(hW P + b P )E T + b O )(5) where W P is a learnable projection matrix, b P , b O are bias terms, E is the Embedding matrix of the commodity set V, and h is the output of the feed-forward neural network.

3. The deep interest evolution recommendation method according to claim 1, wherein The user features include user ID, gender, age, occupation, the average score of the user for the item, and the total number of items evaluated by the user.

4. A deep interest evolution recommendation method according to claim 1, characterized in that The Transformer network adopts stacked Transformer layers, and each layer of Transformer includes a multi-head attention module and a feed-forward neural network.

5. The deep interest evolution recommendation method according to claim 4, characterized in that Use the Cloze task to mask 15% of the items in the input sequence, and the loss function is: where S u ' is the masked version of the user behavior history S u , is the randomly masked item is the masked item, v m is the real item 6. A deep interest evolution recommendation system based on the deep interest evolution recommendation method according to claim 1, characterized in that It includes a preprocessing module, an optimization training module, and a search module; The preprocessing module is used to extract the user features and item features required by the model from the training dataset, group the extracted user features and item features by user, sort the grouped items by timestamp, and extract the user embedding features and item embedding features from the sorted items; The prediction module is used to generate position embedding features according to the extracted user features and item features, add the item embedding features and the position embedding features, and input the result into the established Transformer network to obtain the output result; Connect the output result obtained by the Transformer network with the user embedding features, input the result into a two-layer feed-forward neural network, use GELU as the activation function to obtain the final probability distribution, and obtain the final predicted item through the probability distribution.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images